Created example with DOPE and DIff-DOPE integration through ROS actinolib. #7

rpapallas · 2024-04-05T21:33:16Z

Hello,

I created a demo that uses DOPE and DIff-DOPE through ROS and an actionlib server/client architecture. This PR brings the following:

Changes to Diff-DOPE core code to allow in-memory representation of the images (depth, rgb, segemntation mask) instead of the file representation used.
Some minor improvements to the diff-dope code too.
The config can now define more than one object, and the demo can refine the pose of more than one object (sequentially).
Implemented a class that allows automatic segmentation of an image from an in-memory representation image using the segment-anything library. It uses the DOPE initial position as a point to segment the image, and returns the largest mask around that point.
Camera intrinsic parameters and image dimensions are now derived from a camera info topic (defined in the config file).
The refine.py is the client which makes call to the server.py, the refine.py can send a request to refine a single object's pose or to refine all objects in the scene.
The refine node subscribes to topics for RGB and Depth, as well as the topics that publish the object pose from DOPE (can subscribe to the pose of more than one object). All defined in the config file.
Upon request, the server will segment the image on-the-fly (using the DOPE position) and using segment-anything, set up a Diff-DOPE Scene and Object3D based on the information derived from the ROS topics (rgb frame, depth frame, segmentation, DOPE pose), run the optimisation and return the refined pose (converted to OpenCV from OpenGL).
Added the ability to turn on / off saving of videos from the config file.
Added more 3D models (BBQ Sauce and Mustard) on top of Alphabet Soup. The provided ROS bag includes all three objects, so someone can use all three models to get a refined pose for all three.
A demo page is added to the docs, which includes details of how to run the demo and a link to download an example ROS bag. Please note that there is a link in there with a ROS bag that you can download to run the demo, this link will expire on the 5th of May. It would be great if you upload it to your own repository and update the link.

Please let me know if you have any questions or if you have any suggestions for improvement.

Thank you,
Rafael

The Scene dataclass used to rely on the path attributes to perform operations on the tensors; it seems unnecessary given that the tensors aren't None. This would allow one to set tensors directly without having to provide paths to files (say someone using ROS topics and reads images from a topic message).

…strucitons of how to run the demos.

…o install SAM.

rpapallas · 2024-04-05T22:07:37Z

examples/diffdope_ros/scripts/server.py

+        transform[0, 3] /= 10
+        transform[1, 3] /= 10
+        transform[2, 3] /= 10


One small issue that is currently hard-coded is that I observed the output of diff-dope was off by a fixed error. Here, I divide the output by 10 to correct it. I understand this is not ideal and ugly, but I am not sure where this error comes from. This is strange because it seems that Diff-DOPE expects the initial position estimate in millimetres, yet the output seems more close to metres? The 3D model is scaled to match the scale of your provided example by 10, but I don't understand how this would affect the output.

Yeah I had some issues in there, and these are somehow are to pin point, thank you for this, maybe we should make this a hyper params, as the 3d model will have an impact on the representation. I have a vague recollection that I needed this, but did it just to match my older code.

rpapallas · 2024-04-05T22:09:18Z

docs/demos.rst

+sensor). It also uses a camera info topic to retrieve camera intrinsic
+parameters as well as the image dimensions.
+
+You can also download a ROS bag from `here <https://leeds365-my.sharepoint.com/:u:/g/personal/scsrp_leeds_ac_uk/Ec-TbyOr1QVIt6NQQP7E4pABkEUmaEGByVjLHugY7Als_A?e=JES96n>`_


Please consider replacing this link with yours, as this link will expire on 5th of May.

will do, I am in vacation atm, but I will test this out when I am back.

rpapallas · 2024-04-05T22:10:52Z

examples/diffdope_ros/scripts/server.py

+        ddope.make_animation(output_file_path=video_path)
+        rospy.loginfo(f"Video saved at {video_path}")
+
+    def __convert_opengl_pose_to_opencv(self, transform):


Diff-DOPE output seems to be in the OpenGL coordinate system, so this aims to apply a rotation of 180 degrees around the x-axis to bring this to the OpenCV coordinate system used by DOPE.

diff-dope is in opengl, nvdiffrast is in opengl. So this is good.

rpapallas · 2024-04-05T22:12:15Z

examples/diffdope_ros/scripts/server.py

+        position = np.array([dope_position.x, dope_position.y, dope_position.z])
+
+        # Convert to mm
+        position *= 1_000


From your provided example, it seems that Diff-DOPE expects the input position in millimetres, here I convert DOPE's reading from metres to millimetres if my assumption is correct.

rpapallas · 2024-04-22T13:16:14Z

@TontonTremblay a gentle reminder about this PR.

TontonTremblay · 2024-04-24T18:04:26Z

Ok I am looking at it today, could you add instructions on how to run the code with dope to the readme, or make an other readme in a folder ros or something like that. I am not sure where to start to run an example atm.

TontonTremblay · 2024-04-24T18:05:34Z

Ok I see this docs/demo. Could you add a reference to it in the readme?

TontonTremblay · 2024-04-24T18:12:16Z

Maybe diffdope_ros should be its own folder instead of example? What do you think?

TontonTremblay · 2024-04-24T19:00:28Z

Maybe in the doc adding a step where you would run DOPE model?

TontonTremblay · 2024-04-24T19:01:24Z

(john) kinova@kinova:~/catkin_ws_with_conda$ roslaunch diffdope_ros server.py 
RLException: multiple files named [server.py] in package [diffdope_ros]:
- /home/kinova/catkin_ws_with_conda/devel/lib/diffdope_ros/server.py
- /home/kinova/catkin_ws_with_conda/src/diffdope_ros/scripts/server.py
Please specify full path instead
The traceback for the exception was written to the log file

TontonTremblay · 2024-04-24T19:04:25Z

#7 (comment)
This one is not working for me. diffdope ros launch is not working for me. I did catkin_make and source, but I dont get the launch.

rpapallas · 2024-04-24T19:15:02Z


(john) kinova@kinova:~/catkin_ws_with_conda$ roslaunch diffdope_ros server.py 

RLException: multiple files named [server.py] in package [diffdope_ros]:

- /home/kinova/catkin_ws_with_conda/devel/lib/diffdope_ros/server.py

- /home/kinova/catkin_ws_with_conda/src/diffdope_ros/scripts/server.py

Please specify full path instead

The traceback for the exception was written to the log file

Hi, it should be server.launch not .py sorry on mobile to double check but I think it's .launch not .py

TontonTremblay · 2024-04-24T19:18:48Z

I was able to run the code all the way to refining bbq_sauce. I would say that the doc is great, but I think we would need a little bit more step by step work. But I dont know where the refined pose gets published.

TontonTremblay · 2024-04-24T19:19:36Z

Could the code say something about waiting for image topic or other topics it cannot find?

rpapallas · 2024-04-24T19:22:46Z

The refine.py will subscribe to rgb, depth and dope pose topics. It will then make an actionlib request to refine the pose given an rgb and depth frame and the DOPE pose. The server responds back with the refined pose. There is no topic in this example for the refined pose.

TontonTremblay · 2024-04-24T20:34:17Z

Ok I think we would need so ways to republish the pose no? Also if something is not loading it would be nice that it said so.

rpapallas · 2024-04-27T21:50:31Z

@TontonTremblay pushed some more commits addressing some of your comments. I will add checks in refine.py to make sure certain things are being published before trying to refine a pose, I agree with this. I will work on a demo that publishes the refined pose to a topic.

…ontinuous tracking.

…opic.

rpapallas · 2024-04-30T13:41:10Z

Hi @TontonTremblay,

I think I have addressed all the comments:

Added demos page to main README (a3198fc).
Moved diffdope_ros to the root directory (48bd007).
Added step in docs for running DOPE (4eeeceb).
Added code to check if RGB, Depth and pose topics are being published before starting diff-dope refinement (6a8f526).
Implemented a separate demo called refine_continuous.launch that will constantly refine the pose of an object or multiple objects and publish the refined poses to a topic with name /diffdope_OBJECTNAME (6c335fe).
Updated the demos page to include how to run the new continuous demo (4b7d46f).

Please let me know what you think.

TontonTremblay · 2024-04-30T22:00:50Z

server.py -> server.launch
I cannot do ctrl-c to kill process on refine.launch -- I had to do ps aux | grep refine.py
I dont think the piping of the object_name:=bbq_sauce work, have you tried to run it without dope topics of mustard and the other one. I get waiting for these topics even though I passed a single name.
Is there a sort of feedback showing the refinement happening? like would be nice to have a count down?

process[diffdope_refiner-1]: started with pid [2085063]
[INFO] [1714514232.567412]: Waiting for all data to become available ...
[INFO] [1714514232.569854]: All data available. Starting refinement.
[INFO] [1714514232.572003]: Refining bbq_sauce only...

this is what I have at the moment, no success or failure message and it does not seem to be running.

Thank you for continuous :P I like it.

I feel like we are getting close. Probably just some clarifications we can add to the doc and I think we are up and running.

rpapallas · 2024-06-03T12:14:39Z

Sorry a combination of work and travelling put that on the side.

The server shows the progress of the refinement. If you don't want to get certain objects tracked (i.e. the pose topic isn't available) you need to remove it from the config file.

Strange, my version works with object_name:=bbq_sauce and it will spit out a pose after the refinement.

TontonTremblay · 2024-06-03T18:28:58Z

Yeah no worries! Work and life happen. I really appreciate all the effort you put in tbh and I am quite impressed.

rpapallas · 2024-06-03T18:43:34Z

Thanks, Jonathan!

I encountered the issue with killing too just today. I will fix that. I will also try to update docs to be more helpful.

Is there anything else you want to see to this before it gets merged?

Rafael

rpapallas and others added 30 commits January 18, 2024 12:56

Created a config file for demo with multi-object with ROS and DOPE

3a6d6f2

Updated model paths and added rgb/depth topics.

053f252

Corrected typo

55e1d0a

Minor changes to yaml file

0a465bf

Added some HOPE object models as examples.

b361465

Added .mp4 in gitignore

deb545f

Implemented first version of ROS+DOPE demo with multi-object tracking.

e7ccf35

Ignore _build directory in docs

e77f4eb

Created a new docs page for demos

66d318c

Added URL to the ROS bag.

eeea44d

Added readme to demos to help people realise that the docs include in…

c36572b

…strucitons of how to run the demos.

Converted the demo into a catkin package

e6fe8f3

Updated msgs

6d08c96

Implemented first version of actionlib server

0b335f3

Implemented a class to support segmentation based on segment-anything

40fa711

Reformmating based on pre-commit rules

4406f5a

Added some segmentation test images

3eb186c

Renamed segmentation class

195f74d

Added script to build and zip website for deployment

239bc07

Updated meshes of HOPE objects.

a699d7f

Implemented actionlib demo

e932625

Updated the mesh for bbq sauce

aadfe1f

Updated meshes for other HOPE objects.

ddc0dd7

Camera intrinsic params dynamically retrieved from ROS topic.

3dbd21f

General refactoring

fdffe0e

Updated docs to include instructions of how to run the demo and how t…

b7bdf99

…o install SAM.

Updated the roslaunch commands for refine client

a9cd963

Updated model's frame orientation to match DOPE

463c8cb

Update demos.rst

56a8248

rpapallas commented Apr 5, 2024

View reviewed changes

rpapallas added 3 commits April 27, 2024 21:15

Added ROS example in README.

a3198fc

Moved diffdope_ros to root dir.

48bd007

Updated docs to include step to run DOPE.

4eeeceb

rpapallas added 5 commits April 30, 2024 10:23

Made rgb and depth callbacks atomic and not one per object.

8aaa647

Added check to ensure all data available before refining.

6a8f526

Added limit of saved videos per run to avoid saving too many during c…

13a41fd

…ontinuous tracking.

Implemented continuous tracking and publishing of refined pose to a t…

6c335fe

…opic.

Updated docs to include details about continuous tracking launch file

4b7d46f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Created example with DOPE and DIff-DOPE integration through ROS actinolib. #7

Created example with DOPE and DIff-DOPE integration through ROS actinolib. #7

rpapallas commented Apr 5, 2024

rpapallas Apr 5, 2024

TontonTremblay Apr 7, 2024

rpapallas Apr 5, 2024

TontonTremblay Apr 7, 2024

rpapallas Apr 5, 2024

TontonTremblay Apr 7, 2024

rpapallas Apr 5, 2024

TontonTremblay Apr 7, 2024

rpapallas commented Apr 22, 2024

TontonTremblay commented Apr 24, 2024

TontonTremblay commented Apr 24, 2024

TontonTremblay commented Apr 24, 2024

TontonTremblay commented Apr 24, 2024

TontonTremblay commented Apr 24, 2024

TontonTremblay commented Apr 24, 2024

rpapallas commented Apr 24, 2024

TontonTremblay commented Apr 24, 2024

TontonTremblay commented Apr 24, 2024

rpapallas commented Apr 24, 2024

TontonTremblay commented Apr 24, 2024

rpapallas commented Apr 27, 2024

rpapallas commented Apr 30, 2024

TontonTremblay commented Apr 30, 2024

rpapallas commented Jun 3, 2024

TontonTremblay commented Jun 3, 2024

rpapallas commented Jun 3, 2024

Created example with DOPE and DIff-DOPE integration through ROS actinolib. #7

Are you sure you want to change the base?

Created example with DOPE and DIff-DOPE integration through ROS actinolib. #7

Conversation

rpapallas commented Apr 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rpapallas commented Apr 22, 2024

TontonTremblay commented Apr 24, 2024

TontonTremblay commented Apr 24, 2024

TontonTremblay commented Apr 24, 2024

TontonTremblay commented Apr 24, 2024

TontonTremblay commented Apr 24, 2024

TontonTremblay commented Apr 24, 2024

rpapallas commented Apr 24, 2024

TontonTremblay commented Apr 24, 2024

TontonTremblay commented Apr 24, 2024

rpapallas commented Apr 24, 2024

TontonTremblay commented Apr 24, 2024

rpapallas commented Apr 27, 2024

rpapallas commented Apr 30, 2024

TontonTremblay commented Apr 30, 2024

rpapallas commented Jun 3, 2024

TontonTremblay commented Jun 3, 2024

rpapallas commented Jun 3, 2024