Point Cloud Generation

[tungyen]

Hi, thank you for this interesting work! I have question about generating point cloud from RGBD image. From my understanding, the point cloud generated by RGBD tend to be the surface of the object. In this case, the 3D box might not be accurate compared to the real location. Do you solve this problem?

[dagshub]

Join the discussion on DagsHub!

[tungyen]

One more question, do you test on MobileSAM before? I wonder if these two models has the same performace

[safouaneelg]

Hello @tungyen ,

Thank you for the interesting question.

Volumetric object estimation for accurate position estimation

Let's start with the first one.
What I do is estimating the mask using SAM => Translating it to the Depth Image => isolating the object in the depth image => reconstructing the object point cloud.

To answer you questions, I haven't studied the volumetric aspect of the objects to improve the 3D estimation.
But here's a suggestion:

Depending on the object shape and position of the camera with respect to the body, the surface reconstruction can give you an accurate estimation of object position. For example in rectangular and cubic form, the camera is able in a lot of cases to detect the Furthest points in the object. if you are dealing with round or complex shapes, I would suggest you to consider that the object is symmetrical and that you can duplicate the observed surface and add it to the reconstructed surface to estimate the volume of the object and result in a better estimation of the 3D bbox.

Here's a schematic :

SAM algorithms

I haven't personally tried MobileSAM, it might indeed worth trying. It's already implemented in ultralytics and easier to use compared to FastSAM. I encourage you to try it https://docs.ultralytics.com/models/mobile-sam/#testing-mobilesam-in-ultralytics.

Safouane,