How to evaluate the model's bias and preference between the depth and RGB modalities?

[chushichong]

Hi
Very excited to see your great work.
I'm wondering have you evaluated the modalities preference?
Is the depth info playing more important role?
BR
Thanks

[danielS91]

We did such evaluations when working on the single-task predecessor ESANet.

These evaluations revealed that the models preference shifts (SE weighting) to depth if the input is dark or too bright. We also observed that the presence of "smaller objects" (see motivation for cluttered scenes) enforces depth preference. For real-world applications it is also important to train with "--raw-depth", otherwise the model will be less able to handle incomplete depth images.

[chushichong]

Thanks for your reply.
Do you think it'll works using random transform & shift / point cloud copy paste as the data augmentation methods in depth image which is usually used in point cloud based object detection?

[chushichong]

Also, could you upload the resnet18 encode pth file then I can reproduce the result mentioned in paper.

[danielS91]

Augmenting the depth in single RGB-D images is challenging as any transformation in 3D would also change the RGB part. We never found a promising augmentation pipeline for depth.

We still have some of the ResNet18 checkpoints – which result do you want to reproduce?