Apply point cloud from realsense into OpenCV Surface matching algorithm

[anewworl]

So i have try to export the point cloud from the RealSense camera (I am using model D435i). I base on this link and try to replicate it by using point cloud from D435i https://docs.opencv.org/4.x/d9/d25/group__surface__matching.html and this is the reference i get from the opencv-contrib python here is the link to the document https://github.com/opencv/opencv_contrib/blob/4.x/modules/surface_matching/samples/ppf_load_match.py

import cv2 as cv

N = 2
modelname = "parasaurolophus_6700"
scenename = "rs1_normals"

detector = cv.ppf_match_3d_PPF3DDetector(0.025, 0.05)

print('Loading model...')
pc = cv.ppf_match_3d.loadPLYSimple("data/%s.ply" % modelname, 1)


print('Training...')
detector.trainModel(pc)

print('Loading scene...')
pcTest = cv.ppf_match_3d.loadPLYSimple("data/%s.ply" % scenename, 1)

print('Matching...')
results = detector.match(pcTest, 1.0/40.0, 0.05)

print('Performing ICP...')
icp = cv.ppf_match_3d_ICP(100)
_, results = icp.registerModelToScene(pc, pcTest, results[:N])

print("Poses: ")
for i, result in enumerate(results):
    #result.printPose()
    print("\n-- Pose to Model Index %d: NumVotes = %d, Residual = %f\n%s\n" % (result.modelIndex, result.numVotes, result.residual, result.pose))
    if i == 0:
        pct = cv.ppf_match_3d.transformPCPose(pc, result.pose)
        cv.ppf_match_3d.writePLY(pct, "%sPCTrans.ply" % modelname)

but i stuck at the step where i try to load the .ply data file export from the RealSense is not in correct format (https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/export_ply_example.py). Had anyone do this before can you have me with this problem pls.

[MartyG-RealSense]

Hi @anewworl I have not seen an example of a RealSense ply file being imported directly into an OpenCV script. My research indicates that imported ply data is typically converted into an OpenCV mat, like in the discussion at the link below.

https://stackoverflow.com/questions/29435273/pointcloud-data-to-mat-data

[anewworl]

Hi @MartyG-RealSense, After reading some article I had done some research and can finally do it but it come to another problem is that the point cloud from the realsense is kinda not meet my expectation, down here is some of the result and image in 2D (RGB, Depth), Point Cloud in Realsense:

Real world image

Left Image is file CAD, Right image is point cloud from realsense

Matching Reuslt

So i am try to do is detect pose(roll. pitch and yaw) of the free-object if you know any better solution just let me know.
Thansk for reading.

[MartyG-RealSense]

On the RealSense Viewer depth image, the background is likely missing because the Viewer has the threshold post-processing filter enabled by default, which restricts the maximum distance of the depth that can be displayed to 4 meters from the camera. Going to Stereo Module > Post-Processing, expanding open the list of post-processing filters and disabling the Threshold filter will enable depth up to 10 meters away to be included on the depth image.

The amount of detail on the depth image may also increase and the number of holes / gaps decrease if you maximize the value of the Laser Power setting to '360'. This setting can be found under Stereo Module > Controls.

Calculating roll, pitch and yaw with pyrealsense2 code by accessing the camera's IMU component is discussed at #4391