Regarding scaling of objects

[ArghyaChatterjee]

Hello,

I wanted to know if I change the object's scale during testing, from what it has seen during training, is it still able to identify the object and give the pose of the object correctly? You can take the example of a same-looking toy with a smaller scale fed during training and a larger scale seen during testing. Also scaling can happen in any one direction (x/y/z).

[mintar]

I wanted to know if I change the object's scale during testing, from what it has seen during training, is it still able to identify the object and give the pose of the object correctly?

Yes, that is possible. To be a bit more precise: I had once accidentally used the wrong scale for the mesh model that I used to generate synthetic training data. This was not a problem, because only the projected bounding box corners and projected centroid are used during training - the absolute scale of the mesh does not factor into it at this stage. The scale is only taken into account during inference (testing), and you have to specify it via the dimensions parameter in the config file.

I've only ever used 100% synthetic data for training and then real data for testing. What I haven't tried is using real data of a small scale toy model for training, and then real data of the full model for testing. I guess there will probably be some problems, because the toy model is not 100% accurate, the scene probably looks different from the real scene and so on, but it's worth trying. Just in case this is what you're asking.

Also scaling can happen in any one direction (x/y/z).

I don't understand the question. It will only work with uniform scaling (i.e., the x/y/z axes are all scaled by the same factor). If you give it a distorted model for training (where the axes are scaled differently), it won't work.

[ArghyaChatterjee]

Thanks for the reply. There are 2 questions here.

1. Scaling in any direction
   A pencil can be shorter only in z direction when it's' used for quite some time. A same looking same brand toothpaste can be short, medium and large.

Is dope going to fail in this case ?

2. Mesh modification:
   A hammer can be of different shapes.
   

Is dope going to identify all of them as hammer or it's going to identify only the one it is trained on ?

[mintar]

A pencil can be shorter only in z direction when it's' used for quite some time. A same looking same brand toothpaste can be short, medium and large. Is dope going to fail in this case ?

Yes.

A hammer can be of different shapes. Is dope going to identify all of them as hammer or it's going to identify only the one it is trained on?

Only the one it's trained on.

DOPE is an "instance-level" pose estimation algorithm. What you're looking for is called "category level" pose estimation. @TontonTremblay has written a paper about it, maybe he can help:

https://arxiv.org/abs/2109.06161

[ArghyaChatterjee]

Hi,

thanks again for the clarification. So do you think, for the 1st case also, it's better to use a category-level pose estimation approach (Center Pose) rather than an instance-level one (DOPE)?

[TontonTremblay]

Thank you @mintar for the great answer. Yeah when you are starting to mix different instances, you are looking at category pose estimation. You are also introducing an interesting problem, which is the scaling of the object, specially if you are using similar objects in the same frame, like your toothpastes. If you do not have a way to know the size of the objects you wont be able to get good poses. For example you could use a depth camera, which I have written code for centerpose to work with (might not be on the repo).

Doing multiple hammers is the same problem. Dope can be train to do "category level" but it does not have any mecanism to deal with scale, centerpose has one. Other methods rely on depth, which make it a little harder to use (you need a depth camera). We had plans to build a multiple camera system to deal with the scale issues, but this project is still on-going.