Unable to compute metrics for model output/lego #481

yangqinhui0423 · 2023-11-18T10:36:55Z

Hello，everyone！
I trained it on the NeRF-synthetic dataset, and it worked, and it rendered. But when I run python metrics.py -m output/lego, I get an error saying Unable to compute metrics for model output/lego.

Has anyone encountered a similar problem and knew how to solve it?
PS：I was able to compute metrics on the Mip-NeRF360 dataset after colmap.

RetroFuturism7 · 2024-01-05T07:28:05Z

same issue.

curious-energy · 2024-01-24T08:48:10Z

I made a simple change, please see #632.

curious-energy · 2024-01-24T12:13:37Z

I made a simple change, please see #632.

and another reason is VRAM OOM when the progress bar is interrupted.

LakeofRain · 2024-01-25T07:16:38Z

I made a simple change, please see #632.

I have solved the problem of 'Unable to compute metrics for model ' by 'pip install torchmetrics', although don't know why, but it does work well. Now I can obtain the metrics.

leblond14u · 2024-03-25T14:21:47Z

Same problem here metrics.py does not work. Tried to install torchmetrics but:

No improvements with pip install
And the conda install of torchmetrics is messing with my conda environment

This is what I have after a full metrics evaluation :

All of the metrics are NaN ...

Sometimes it also crashes :

Traceback (most recent call last):
  File "/home/hleblond/Bureau/PhD/Depth_3DGS/metrics.py", line 150, in <module>
    evaluate(args.model_paths)
  File "/home/hleblond/Bureau/PhD/Depth_3DGS/metrics.py", line 116, in evaluate
    lpipss.append(lpips(renders[idx], gts[idx], net_type='vgg'))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/Bureau/PhD/Depth_3DGS/lpipsPyTorch/__init__.py", line 20, in lpips
    criterion = LPIPS(net_type, version).to(device)
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/Bureau/PhD/Depth_3DGS/lpipsPyTorch/modules/lpips.py", line 28, in __init__
    self.lin.load_state_dict(get_state_dict(net_type, version))
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/Bureau/PhD/Depth_3DGS/lpipsPyTorch/modules/utils.py", line 17, in get_state_dict
    old_state_dict = torch.hub.load_state_dict_from_url(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/anaconda3/envs/GS/lib/python3.11/site-packages/torch/hub.py", line 770, in load_state_dict_from_url
    return torch.load(cached_file, map_location=map_location, weights_only=weights_only)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/anaconda3/envs/GS/lib/python3.11/site-packages/torch/serialization.py", line 1028, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/anaconda3/envs/GS/lib/python3.11/site-packages/torch/serialization.py", line 1264, in _legacy_load
    typed_storage._untyped_storage._set_from_file(
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

shenqianxiang · 2024-06-26T14:13:34Z

我做了一个简单的更改，请参阅#632。

你好请问你是怎么解决的

shenqianxiang · 2024-06-26T15:06:46Z

同样的问题。

你解决了吗

LakeofRain · 2024-06-26T15:19:50Z

我是因为torchvision版本不对，算lpips时vgg网络没下载下来。发自我的荣耀手机-------- 原始邮件 --------发件人： shenqianxiang ***@***.***>日期： 2024年6月26日周三 23:07收件人： graphdeco-inria/gaussian-splatting ***@***.***>抄送： LakeofRain ***@***.***>, Comment ***@***.***>主题： Re: [graphdeco-inria/gaussian-splatting] Unable to compute metrics for model output/lego (Issue #481) 同样的问题。你解决了吗 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

biter0088 · 2024-09-05T17:16:36Z

I re-found a new environment on my computer with the torch package, ran pip install torchmetrics and re-executed python metrics.py -m “/home/xiu/Learning/gaussian-splatting/output/train” and was able to output the following results:

# Metric evaluation progress: 100%|█████████████████████████████████████████████████████████████████████████| 38/38 [00:39<00:00, 1.05 s/it]
# here
# SSIM : 0.8076466
# PSNR : 21.7869835
# LPIPS: 0.2143032

kellofcn · 2024-10-15T08:24:58Z

Same problem here metrics.py does not work. Tried to install torchmetrics but:

No improvements with pip install
And the conda install of torchmetrics is messing with my conda environment

This is what I have after a full metrics evaluation : All of the metrics are NaN ...

Sometimes it also crashes :

Traceback (most recent call last):
  File "/home/hleblond/Bureau/PhD/Depth_3DGS/metrics.py", line 150, in <module>
    evaluate(args.model_paths)
  File "/home/hleblond/Bureau/PhD/Depth_3DGS/metrics.py", line 116, in evaluate
    lpipss.append(lpips(renders[idx], gts[idx], net_type='vgg'))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/Bureau/PhD/Depth_3DGS/lpipsPyTorch/__init__.py", line 20, in lpips
    criterion = LPIPS(net_type, version).to(device)
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/Bureau/PhD/Depth_3DGS/lpipsPyTorch/modules/lpips.py", line 28, in __init__
    self.lin.load_state_dict(get_state_dict(net_type, version))
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/Bureau/PhD/Depth_3DGS/lpipsPyTorch/modules/utils.py", line 17, in get_state_dict
    old_state_dict = torch.hub.load_state_dict_from_url(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/anaconda3/envs/GS/lib/python3.11/site-packages/torch/hub.py", line 770, in load_state_dict_from_url
    return torch.load(cached_file, map_location=map_location, weights_only=weights_only)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/anaconda3/envs/GS/lib/python3.11/site-packages/torch/serialization.py", line 1028, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/anaconda3/envs/GS/lib/python3.11/site-packages/torch/serialization.py", line 1264, in _legacy_load
    typed_storage._untyped_storage._set_from_file(
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

This is what I have executed python metrics.py -m /home/lwk/gaussian-splatting/output/980df71c-a , metric evaluation progress is 0it , do u know why?
Scene: /home/lwk/gaussian-splatting/output/980df71c-a Method: ours_18000 Metric evaluation progress: 0it [00:00, ?it/s] SSIM : nan PSNR : nan LPIPS: nan

HanYan012 · 2024-11-23T18:04:18Z

Same problem here metrics.py does not work. Tried to install torchmetrics but:

No improvements with pip install
And the conda install of torchmetrics is messing with my conda environment

This is what I have after a full metrics evaluation : All of the metrics are NaN ...
Sometimes it also crashes :

Traceback (most recent call last):
  File "/home/hleblond/Bureau/PhD/Depth_3DGS/metrics.py", line 150, in <module>
    evaluate(args.model_paths)
  File "/home/hleblond/Bureau/PhD/Depth_3DGS/metrics.py", line 116, in evaluate
    lpipss.append(lpips(renders[idx], gts[idx], net_type='vgg'))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/Bureau/PhD/Depth_3DGS/lpipsPyTorch/__init__.py", line 20, in lpips
    criterion = LPIPS(net_type, version).to(device)
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/Bureau/PhD/Depth_3DGS/lpipsPyTorch/modules/lpips.py", line 28, in __init__
    self.lin.load_state_dict(get_state_dict(net_type, version))
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/Bureau/PhD/Depth_3DGS/lpipsPyTorch/modules/utils.py", line 17, in get_state_dict
    old_state_dict = torch.hub.load_state_dict_from_url(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/anaconda3/envs/GS/lib/python3.11/site-packages/torch/hub.py", line 770, in load_state_dict_from_url
    return torch.load(cached_file, map_location=map_location, weights_only=weights_only)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/anaconda3/envs/GS/lib/python3.11/site-packages/torch/serialization.py", line 1028, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hleblond/anaconda3/envs/GS/lib/python3.11/site-packages/torch/serialization.py", line 1264, in _legacy_load
    typed_storage._untyped_storage._set_from_file(
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

This is what I have executed python metrics.py -m /home/lwk/gaussian-splatting/output/980df71c-a , metric evaluation progress is 0it , do u know why? Scene: /home/lwk/gaussian-splatting/output/980df71c-a Method: ours_18000 Metric evaluation progress: 0it [00:00, ?it/s] SSIM : nan PSNR : nan LPIPS: nan

Same. Did you solve this?

tapowanliwuyun · 2024-12-24T11:45:25Z

Same. Did you solve this?

tapowanliwuyun · 2024-12-24T13:30:12Z

solved！please chekck /home/dkcs/.cache/torch/hub/checkpoints/vgg.pth ....

micki-37 · 2025-01-10T02:00:50Z

我是因为torchvision版本不对，算lpips时vgg网络没下载下来。发自我的荣耀手机-------- 原始邮件 --------发件人： shenqianxiang @.>日期： 2024年6月26日周三 23:07收件人： graphdeco-inria/gaussian-splatting @.>抄送： LakeofRain @.>, Comment @.>主题： Re: [graphdeco-inria/gaussian-splatting] Unable to compute metrics for model output/lego (Issue #481) 同样的问题。你解决了吗 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

我也是vgg网络没下完 重新跑metrics就显示算不了了 但并不会重新下vgg的权重 是得重新训练完重新下vgg吗

LakeofRain · 2025-01-10T05:12:30Z

我换了个torchvision的版本就解决了，没重新训练在 2025-01-10 10:01:12，"micki-37" ***@***.***> 写道：我是因为torchvision版本不对，算lpips时vgg网络没下载下来。发自我的荣耀手机-------- 原始邮件 --------发件人： shenqianxiang @.>日期： 2024年6月26日周三 23:07收件人： graphdeco-inria/gaussian-splatting @.>抄送： LakeofRain @.>, Comment @.>主题： Re: [graphdeco-inria/gaussian-splatting] Unable to compute metrics for model output/lego (Issue #481) 同样的问题。你解决了吗 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***> 我也是vgg网络没下完重新跑metrics就显示算不了了但并不会重新下vgg的权重是得重新训练完重新下vgg吗 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

king1111sadjfoisja · 2025-01-20T21:10:37Z

I re-found a new environment on my computer with the torch package, ran pip install torchmetrics and re-executed python metrics.py -m “/home/xiu/Learning/gaussian-splatting/output/train” and was able to output the following results:

Metric evaluation progress: 100%|█████████████████████████████████████████████████████████████████████████| 38/38 [00:39<00:00, 1.05 s/it]

here

SSIM : 0.8076466

PSNR : 21.7869835

LPIPS: 0.2143032

Hello, how did you create this environment? Did you follow the environment.yml file? Were there any modifications?

king1111sadjfoisja · 2025-01-20T21:11:13Z

我换了个torchvision的版本就解决了，没重新训

你好，换成什么版本了方便问一下吗？谢谢

lishuai-97 · 2025-03-23T08:59:10Z

Same issue! I installed the torchmetrics package with conda, but it not work for me.

mekeny1 · 2025-03-27T12:42:41Z

solved！please chekck /home/dkcs/.cache/torch/hub/checkpoints/vgg.pth ....

right! but i also noticed there are 2 pths totally to download(https://raw.githubusercontent.com/richzhang/PerceptualSimilarity/master/lpips/weights/v0.1/vgg.pth, https://download.pytorch.org/models/vgg16-397923af.pth), them should be downloaded by the script, but in my case, only vgg16-397923af.pth can be downloaded, but not vgg.pth. Finally, just put the two in the checkpoints

2229736580 · 2025-04-10T12:10:02Z

Here is my modification process that allowed me to successfully resolve the issue. The fix primarily involved updating metrics.py to skip hidden directories generated in the file system, though I have not yet investigated the root cause of these hidden

Switching the "except" clause
Initially, the script used a bare except: statement, which suppressed specific error messages. Replacing it with:
```
except Exception as e:
    print("Unable to compute metrics for model", scene_dir)
    print("Error:", e)
```
allowed the code to print the actual error details.
Inspecting the error message
After running the updated script, the error message indicated an “[Errno 21] Is a directory” issue, meaning the code attempted to open a directory (.ipynb_checkpoints) as if it were an image.
Confirming the hidden folder
Checking the target directory indeed revealed that.

Identifying the core function
The issue originated in the readImages function, which scanned all entries in the folder, unintentionally including hidden directories.
Filtering hidden folders
Within readImages, adding conditions to skip entries starting with “.” or that are directories prevented .ipynb_checkpoints from being processed:
```
for fname in os.listdir(renders_dir):
    if fname.startswith('.') or (renders_dir / fname).is_dir():
        continue
    ...
```
Verifying the fix
After testing, the script successfully avoided "Unable to compute metrics for model" errors by properly ignoring hidden directories (.ipynb_checkpoints) and produced the expected output.

Unable to compute metrics for model output/lego #481

Unable to compute metrics for model output/lego #481

Comments

yangqinhui0423 commented Nov 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

RetroFuturism7 commented Jan 5, 2024

Uh oh!

curious-energy commented Jan 24, 2024

Uh oh!

curious-energy commented Jan 24, 2024

Uh oh!

LakeofRain commented Jan 25, 2024

Uh oh!

leblond14u commented Mar 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shenqianxiang commented Jun 26, 2024

Uh oh!

shenqianxiang commented Jun 26, 2024

Uh oh!

LakeofRain commented Jun 26, 2024 via email

Uh oh!

biter0088 commented Sep 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kellofcn commented Oct 15, 2024

Uh oh!

HanYan012 commented Nov 23, 2024

Uh oh!

tapowanliwuyun commented Dec 24, 2024

Uh oh!

tapowanliwuyun commented Dec 24, 2024

Uh oh!

micki-37 commented Jan 10, 2025

Uh oh!

LakeofRain commented Jan 10, 2025 via email

Uh oh!

king1111sadjfoisja commented Jan 20, 2025

Metric evaluation progress: 100%|█████████████████████████████████████████████████████████████████████████| 38/38 [00:39<00:00, 1.05 s/it]

here

SSIM : 0.8076466

PSNR : 21.7869835

LPIPS: 0.2143032

Uh oh!

king1111sadjfoisja commented Jan 20, 2025

Uh oh!

lishuai-97 commented Mar 23, 2025

Uh oh!

mekeny1 commented Mar 27, 2025

Uh oh!

2229736580 commented Apr 10, 2025

Uh oh!

yangqinhui0423 commented Nov 18, 2023 •

edited

Loading

leblond14u commented Mar 25, 2024 •

edited

Loading

biter0088 commented Sep 5, 2024 •

edited

Loading