3D detections are far-off

[f1ren]

Problem

Getting 3D detections that are far from the object in the same frame_id.

Technical details

1. Running yolo_ros as follows:

ros2 launch yolo_bringup yolov11.launch.py use_3d:=True \
	input_image_topic:=/head_front_camera/rgb/image_raw \
	input_depth_topic:=/head_front_camera/depth/image_raw \
	input_depth_info_topic:=/head_front_camera/depth/camera_info

2. Topic /yolo/dbg_image looks good:

3. But, yolo/bdg_bb_markers (the yellow bounding-box) is far from the cup. It's actually inside the robot's body:

4. The "cup" detection.class_name on topic /yolo/detections_3d is at position:
   geometry_msgs.msg.Point(x=0.21692220892426856, y=0.02220525709998527, z=1.1176493412872952)

5. But, manually sampling the cup in RViz Publish Point, it seems to be at:

x=0.74, y=0.15, z=0.99

Both at frame_id == 'base_link'.

Question

Why are the coordinates so far off?

[f1ren]

Setting

depth_image_units_divisor:=1

seems much better:

Still having this y-axis offset (don't know why), and the x-axis stretch (perhaps some of the table's pixels were cropped along with the cup).

[mgonzs13]

Hi @f1ren, the depth_image_units_divisor depends on the camera, including the simulation camera. You can also try to change the maximum_detection_threshold.

[f1ren]

@mgonzs13 how will maximum_detection_threshold help in the case I've shared here?
Seems like maximum_detection_threshold is for z-axis, which is probably the vertical axis. The current problems (the image of bboxes about the cup and can) seems like horizontal stretch.

[mgonzs13]

The maximum_detection_threshold determines the maximum depth of an object and sometimes it can affect the 3D boxes. I have uploaded an experimental new version of the 3D node. Can you try it and tell if it improves your resutls?

[f1ren]

Pulled the latest code and executed as follows:

ros2 launch yolo_bringup yolov11.launch.py use_3d:=True \
	depth_image_units_divisor:=1 \
	input_image_topic:=/head_front_camera/rgb/image_raw \
	input_depth_topic:=/head_front_camera/depth/image_raw \
	input_depth_info_topic:=/head_front_camera/depth/camera_info 

Still getting this stretched, and a bit off, result:

Any idea how to fix the x-axis stretch and the x-y-axis offset?

[mgonzs13]

@f1ren can you share the simulation you are using?

[f1ren]

Sure. I'll wrap it up for you to try. It's basically PAL Developer Docker and a Gazebo sim world map. Fairly simple.

[f1ren]

My mistake: forgot to install the latest code after pulling.
The updated current result is this:

Regardless, @mgonzs13 , please let me know if you still want the simulation.

[mgonzs13]

Hi @f1ren, I have uploaded a new version of 3D detection. Testing it in the objects_on_table of PAL simulation.

[f1ren]

@mgonzs13 the latest committed code result is very similar to before in my setup:

[mgonzs13]

@f1ren, This is the result I got when testing in the Tiago simulation with a table. There are some errors due to the depth image errors. What is the orientation between your camera and your objects?

[f1ren]

The TIAGo in my setup is just in-front of the table, facing it almost directly.

The objects are a partial because the torso, and the camera, are a little higher compared to previous images.

This accuracy might be enough to finish the grasp. I will have to try to know.

Thanks

[github-actions]

This issue is stale because it has been open for 30 days with no activity.