NAN appears when training

[JesseZZZZZ]

Hi @lahavlipson , have you tried training DPVO on other datasets beside TartanAir? I tried to train it on TUM (using _build_dataset from raw RGB and depth maps), but all values became NAN, do you know the issue? (t1[0] and t2[0] are all NANs). Thank you so much!

[JesseZZZZZ]

Hi, dear authors, it turns out that the nomalization step will cause NANs in poses

After I delete the two lines of codes like I did in the picture, the issue disappeared. Do you think that simply deleting the codes is available? Or are there any other ways to solve this issue? THANKS!

[markinruc]

@JesseZZZZZ Hello！ I met the same problem. Have you solved it?

[JesseZZZZZ]

@markinruc Well, for the NAN in the image it was because some values in my depth maps are 0, and the inverse depth calculation results in NAN (1/0). Another NAN in inference is that my model was so bad....After I trained it for some epochs the NAN disappeared

[JingruiYu]

Hello, have you successfully fine-tuned DPVO?
I have tried fine-tuning it, but the results are quite strange.
Would you be willing to give me some guidance?

[JesseZZZZZ]

I trained it on my own data, nan appears when I didn't normalize my gt data.maybe you can try normalizing everything to tartanair's distribution?

…

---- Replied Message ---- | From | ***@***.***> | | Date | 02/01/2025 10:04 | | To | ***@***.***> | | Cc | ***@***.***>***@***.***> | | Subject | Re: [princeton-vl/DPVO] NAN appears when training (Issue #54) | Hello, have you successfully fine-tuned DPVO? I have tried fine-tuning it, but the results are quite strange. Would you be willing to give me some guidance? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[JingruiYu]

I trained it on my own data, nan appears when I didn't normalize my gt data.maybe you can try normalizing everything to tartanair's distribution?
…

Wow, that's sound advice. I'd like to try it. Do you know what the distribution of tartan looks like?

[JesseZZZZZ]

tartan's range is about -10 to 10. I think the absolute value of gt should be less than 20 ...?

…

---- Replied Message ---- | From | ***@***.***> | | Date | 02/03/2025 10:02 | | To | ***@***.***> | | Cc | ***@***.***>***@***.***> | | Subject | Re: [princeton-vl/DPVO] NAN appears when training (Issue #54) | I trained it on my own data, nan appears when I didn't normalize my gt data.maybe you can try normalizing everything to tartanair's distribution? … Wow, that's sound advice. I'd like to try it. Do you know what the distribution of tartan looks like? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[JingruiYu]

tartan's range is about -10 to 10. I think the absolute value of gt should be less than 20 ...?
…

Thanks for your kindly repley.
I checked the Tartan dataset but didn’t quite understand what the range of -10 to 10 and the ground truth (GT) being less than 20 refer to. Are these values related to the variable stored in the .npy depth files? Or are they referring to a specific column in pose_left.txt, such as t_z (which is scaled in here)? But I checked both of them is not -10 to 10?

I have a few additional questions about preparing fine-tuning data, and I would greatly appreciate your help:

1. When preparing training data, we need to generate a pickle file. I followed a function mentioned in one of the author’s responses in another issue and only modified the paths without making any other code changes. I assume this is sufficient and no further modifications are needed—am I correct?

2. Does the resolution of my own dataset images need to match the resolution of TartanAir images? Or can they be mixed during training?

3. Another thing that has confused me is the format of pose_left.txt. Since I see that there’s a reordering operation when reading it in the code, I assume the format should be tz, tx, ty, qz, qx, qy, qw. However, this seems a bit odd—am I misunderstanding something?

4. Lastly, I’d like to ask how you prepared your own dataset when training on it. Is there anything different from what I’ve done? Specifically, here’s what I’ve prepared:

• image_left folder containing .jpg image files.
• depth_left folder containing .npy depth files.
• pose_left.txt file storing ground truth poses.
  Then, I used build_dataset funciton to construct a pickle file.