ROS + DOPE demo

[rpapallas]

Hello,

I have been working on a ROS + DOPE for multi-object demo on my fork under the ros-example branch here. The demo file is here and the new config file for it here.

The demo simply subscribes to RGB and Depth topics of a RealSense camera and uses those frames as images for diff-dope, and subscribes to DOPE output topic for the pose. I recorded a ROS bag for this demo that I can share, so people can download the ROS bag and run the demo.

Are you interested in this demo? If so, I am happy to open a pull request and iterate over your feedback to merge it.

Although the demo is almost there, I still need to figure out certain things. I have a few comments / questions to make at this point:

1. Reading the code, it appears that the DiffDope class works on a per-object basis? If so, my workaround was to create a DiffDope object instance per object.
2. I had to modify slightly the data classes (Scene and Image) to escape the need to save the RGB and Depth frames to files. Those classes assumed the frames are stored in a file. My change should allow both file and in-memory representation.
3. It appears that the current implementation assumes a segmentation to be available. Do I remember correctly that in the paper you do not impose this modality, and diff-dope should still work with RGB+Depth? If so, then I think there is a bug in the code? Currently, in my demo, I generated a fixed segmentation for one of the objects.

These are the notes so far. I will continue working on the demo tomorrow and upload a ROS bag to make it easier for people to replicate the environment.

[TontonTremblay]

This is fantastic. I had a manual version of the earlier version of the code that did that. Here is an example.

But it sounds like you are making better progress than my hack that I had (which was very manual extensive).

To answer your comments:

1. you are correct, I think we could redesign to use subset of a batch per object. But this would be a somewhat involved re-coding, I think someone mentioned something about it (Rendering with different objects in batch #3).
2. That is a good catch, and oversight of mine for online versions of the code. Thank you.
3. You are correct the paper does not, I left it in this release as I wanted to focus on a working version, but you could simply write new losses without the mask multiplication. (or you could use the cuboid to get a mask, which would limit the loss to spread to the whole image).

I would for sure include the code if you make a PR and test it out on my end as well. Thank you so much for your contributions.

As a note, when I worked on this code base, one of my goal was to make the interface somewhat straight forward so someone like you could take the code and add functionalities to it, so any comments about how it is organized, what you like and did not like are welcome (you are allowed to not like anything btw).

[rpapallas]

Hi @TontonTremblay,

The ROS code is working; I implemented it using the actionlib model. I need to polish it a little and I will create a PR.

However, I can't seem to get diff-dope to work properly with a new scene. The refined pose I get is well off. I even tried to use your simple_scene.py example (unmodified) to ensure is not a glitch of my code, but even with this script the pose is well off. I modified these files to get your simple_scene.py to work with the new scene:

• diff-dope/data/example/scene to include my depth.png, rgb.png and seg.png.
• diff-dope/data/mesh/ to add bbq_sauce.ply and bbq_sauce.png (the object I am trying to refine its pose).
• configs/diffdope.yaml to update certain parameters; the camera intrinsic parameters, the image width/height, the object's initial position and rotation, the model path.

These files are attached to this issue too.

Running the simple_scene.py above does not produce a good result (it seems an orientation error).

Some notes:

• I realise that in the config file, under object3d/position you provide an initial guess of the position of the object in the camera in mm instead of meters? Initially I used meters and it took me a while to realise that the expected units are in mm.
• For object3d/position in the config file I am attaching I added DOPE's reported position (object in camera) converted to millimetres.
• Initially I tested the original 3D model of the BBQ sauce from the HOPE repo. However, after inspecting your provided example mesh in this repo, I realised that the object is scaled at a higher value. I therefore scaled the BBQ model to the same magnitude.

Here is DOPE's report of the BBQ sauce in the /dope/pose_BBQSauce topic:

---
header:
  seq: 172
  stamp:
    secs: 1705657887
    nsecs: 456136703
  frame_id: "camera_color_optical_frame"
pose:
  position:
    x: -0.09595088349679726
    y: 0.08746154327269216
    z: 0.5178489768791841
  orientation:
    x: 0.5365636268745633
    y: 0.18261931351690183
    z: 0.05214260540539413
    w: 0.8222109275285554
---

I used the K matrix from the /camera/color/camera_info topic to get the intrinsic parameters in the config file:

---
header:
  seq: 10432
  stamp:
    secs: 1705657884
    nsecs: 814515352
  frame_id: "camera_color_optical_frame"
height: 720
width: 1280
distortion_model: "plumb_bob"
D: [0.0, 0.0, 0.0, 0.0, 0.0]
K: [913.8265380859375, 0.0, 630.7417602539062, 0.0, 913.570068359375, 370.5628967285156, 0.0, 0.0, 1.0]
R: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
P: [913.8265380859375, 0.0, 630.7417602539062, 0.0, 0.0, 913.570068359375, 370.5628967285156, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi:
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False
---

DOPE's initial estimate, to me, seems very good:

I am not sure what I am missing here?

Thanks again for your time.

new_scene.zip

[TontonTremblay]

Thanks, there are some difference between opencv poses and opengl poses. You need to rotate them, and maybe some other things as well. I will check what I did, if you have a fork somewhere I could possibly test it this week.

[rpapallas]

I see. Please let me polish my ROS code and I will let you know once it's ready. Thank you.

[rpapallas]

Hi @TontonTremblay, here is the repo. Note the code is under the ros-example branch. I created a demo page under docs (which includes also a link to download a ROS bag), you may want to compile the docs and have a quick look there to see how to run it.

Hope the code structure isn't too complicated. Any feedback is welcome!

Thank you.

[TontonTremblay]

I just set up my machine back, I will check your demo next week.

[rpapallas]

Hi Jonathan, thank you. I am planning to have another look on Monday too. I feel it should be some misunderstanding of how the initial pose is expressed in the yaml file. If you do check out the code, please remember my ros branch in my fork includes changes to the DiffDope class to allow in-memory representation.

[rpapallas]

Hi Jonathan,

Sorry for the delay with this. I had a second look today, and I realised that the 3D model's coordinate frame was oriented differently from the one assumed by DOPE (which provides the initial estimate). As a result, I updated one of the models (BBQ Sauce) to match the frame assumed by DOPE. I didn't update the other models yet. However, this doesn't seem to have fixed the issue. Still, diff-dope initialisation (orientation-wise) is very off, despite good DOPE reading.

If you have the time at some point to let us know what you meant about the OpenGL representation, that would be great. Some details to get up and running:

• My fork is here: https://github.com/rpapallas/diff-dope and I implemented the ROS integration in the ros-example branch.
• If already cloned, please git pull as I just pushed some changes.
• Instructions on how to download an example ROS bag with DOPE readings and how to run the ROS demo can be found here.
• In this line is where I use the initial pose of the object from DOPE in diff-dope. Here is where we capture the pose from the ROS topic from DOPE.

[rpapallas]

Hi,

I think the problem is fixed. It was indeed an issue with the coordinate frame of the object, compared to what used by DOPE. The orientation is fixed now, there is a slight issue with scaling, the 3D model of the object seems a little smaller, but I am confident this could be fixed with appropriate tuning of the scaling parameter.

I will do some final checks, let me know if you want me to open a PR.

Rafael

[TontonTremblay]

Thanks for the update, I had plans to look into it today, but I am starting to feel sick and decided to not get my co-workers sick with me and stayed at home.

[rpapallas]

Sorry to hear that, Jonathan. I hope you feel better soon. Side note: I didn't push the corrected model yet, but will do tomorrow after I investigate the scaling issue.

[rpapallas]

For reference: NVlabs/Deep_Object_Pose#346

[rpapallas]

This is now completed from my side; see PR #7

Thanks,
Rafael

[TontonTremblay]

So the sickness went on for 3 weeks, and now I am starting to feel better and we have a family spring break! So I will look into this when I come back, feel free to ping me next week to take a look in case it gets lost in the todos.

[rpapallas]

That's great @TontonTremblay, glad that you feel better! Enjoy the break.