ROS2 Foxy Communication across multiple local users [10599]

[cplasberg]

When starting nodes as multiple users on a single machine (e.g. some as root others as normal users) communication only succeeds if there is no network connection.

As soon as the machine is connected to a network, topics from other local users can not be seen. Without a network connection topics are visible. Other machines can see topics of all users (beside different local ones of course).

Issue came up here.

[IkerLuengo]

We were able to reproduce the issue regarding the communication between multiple users. However, we didn't find any difference with the network connection up or down.

Since Foxy, FastDDS includes a shared memory transport that is active by default, meaning that if two processes are on the same host, they will choose this transport over the other alternatives. However, shared memory segments are limited by user permissions, just like files, and this is intended for security reasons: you don't want a user having access to other user's data. So right now FastDDS does not support shared memory transport on multi-user environments. We are looking for alternatives to detect multi user scenarios and disable shared memory or somehow support the scenario.

Meanwhile, if UDP transport is enough for your use case, you can disable shared memory transport. This can be done in ROS2 Foxy through XML configuration. Create an XML file disabling the builtin transports and adding a custom UDP transport:

<?xml version="1.0" encoding="UTF-8" ?>
    <profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles" >
        <transport_descriptors>
            <transport_descriptor>
                <transport_id>CustomUdpTransport</transport_id>
                <type>UDPv4</type>
            </transport_descriptor>
        </transport_descriptors>

        <participant profile_name="participant_profile" is_default_profile="true">
            <rtps>
                <userTransports>
                    <transport_id>CustomUdpTransport</transport_id>
                </userTransports>

                <useBuiltinTransports>false</useBuiltinTransports>
            </rtps>
        </participant>
    </profiles>

Set the environment variable FASTRTPS_DEFAULT_PROFILES_FILE to the path where the XML was created and relaunch your applications. They should now be using UDP transport and communicate normally.

[tiany44]

@IkerLuengo I also run into the same issue. Only I am using a super client config with the discovery server

<?xml version="1.0" encoding="UTF-8" ?>
<dds>
    <profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
    
	<transport_descriptors>
	    <transport_descriptor>
	        <transport_id>UdpTransport</transport_id>
	        <type>UDPv4</type>
	    </transport_descriptor>
	</transport_descriptors>
    
        <participant profile_name="super_client_profile" is_default_profile="true">
            <userTransports>
		<transport_id>UdpTransport</transport_id>
            </userTransports>
            <useBuiltinTransports>false</useBuiltinTransports>
            <rtps>
            	<userTransports>
		    <transport_id>UdpTransport</transport_id>
                </userTransports>
                <useBuiltinTransports>false</useBuiltinTransports>
                   <builtin>
		        <discovery_config>
			   <discoveryProtocol>SUPER_CLIENT</discoveryProtocol>
			   <discoveryServersList>
		               <RemoteServer prefix="44.53.00.5f.45.50.52.4f.53.49.4d.41">
				    <metatrafficUnicastLocatorList>
				        <locator>
			                    <udpv4>
                                               <address>10.x.x.x</address>
                                                   <port>11811</port>
                                               </udpv4>
                                        </locator>
				     </metatrafficUnicastLocatorList>
			        </RemoteServer>
			   </discoveryServersList>
                      </discovery_config>	
            	</builtin>
            </rtps>
        </participant>
    </profiles>
</dds>

This configuration did not work. Any suggestions

[EduPonz]

Hi @tiany44 ,

First of all, are you also using different users? Else, please consider opening a separate issue.

Looking at the remote Server GUID, I'm assuming you are instantiating the server with the CLI and id 0, i.e. fastdds discovery -i 0, is that correct? Where is your Discovery Server listening specifically? If you're not setting a specific address within the server command, then it will listen on all available interfaces (on the server's machine). Your XML however does not point to a specific IP address but to a range (10.x.x.x) instead, which is something that Fast DDS does not support. You'd need to set either a specific IP address, or a host name that the SUPER_CLIENT's OS can resolve either because it's on its list of known hosts, or because your DNS returns a reachable IP for that name. I'm also not sure why you have the user_transports tag twice; you'd only need the one within rtps.

[tiany44]

@EduPonz

First of all, are you also using different users? Else, please consider opening a separate issue.

Yes, I am using different users. My discovery server works well. I just cannot see nodes/topic started by sudo

Looking at the remote Server GUID, I'm assuming you are instantiating the server with the CLI and id 0, i.e. fastdds discovery -i 0, is that correct?

Correct

Where is your Discovery Server listening specifically? If you're not setting a specific address within the server command, then it will listen on all available interfaces (on the server's machine).

Inside a VPN hence the 10.x.x.x
I just omitted the full ip. Again, this is accessible to me and ros2 once I join this VPN
Yes, the fast DDS server does listen on all available interfaces on a machine inside the VPN. The machine's IP address is 10.bla.bla.bla

I'm also not sure why you have the user_transports tag twice; you'd only need the one within rtps

This is a mistake from trying which configuration would not result in an error. Within rtps, the server configuration also needs to be inside <builtin></builtin>. This kinda throws me off given I need it to use UDP and not builtin shared memory. Am I understanding this wrong?

My main issue is that I do not know a way off configuring the super client to not use shared memory and use UDP instead (locally). I believe this should solve the issue where separate users on the same machine and even different machines do not see each other. Correct?

While I understand why shared memory is the main issue for users on the same machine, I do not understand why I cannot see the nodes/topics started by sudo on a separate machine given that I am using a discovery server.

[tiany44]

@EduPonz I also started the ros2 daemon as sudo. Still could not see nodes and topics started as sudo through the discovery server

[EduPonz]

Hi @tiany44 ,

When you say you cannot see the topics/nodes, do you mean using ros2 topic list and ros2 node list? Could you please provide the steps you're taking to launch the nodes and the Daemon?

[tiany44]

Hello @EduPonz

With the discovery server running, I connect my laptop to the vpn and run ros2 run demo_nodes_cpp listener.
ros2 topic list at this point shows the chatter topic.
I then ssh into the robot that is also connected to the vpn and run ros2 run demo_nodes_cpp talker and as expected I get the message exchange between the robot and my laptop.
I do this everytime I resume working on my project just as a sanity check.

The node on the robot has to do stuff with input and output devices and launching the node without sudo will not work as expected.
The next step is to run the node as sudo through the robot ssh terminal. I run

sudo su
source ros2_foxy/ros2-linux/setup/bash
source Documents/Nodes/my_node/install/setup.bash
ros2 daemon start # sometimes this is successful meaning the daemon was not automatically started
ros2 run my_node my_node

With my_node running. executing ros2 topic list on another ssh terminal connected to the robot does not show the nodes/topics as expected. Switching the ssh session to sudo does show the topics and nodes.

From my laptop, the ros2 topic list does not show topics created by my_node with or without sudo.

This behaviour can be tested by running the demo_nodes_cpp listener and talker as sudo from different machines connecting to the same discovery server

[tiany44]

@EduPonz were you able to reproduce this behaviour? Also, how soon can we expect version 2.6.0 and this PR merged?

[tiany44]

@MiguelCompany any chance we can have your fixes in the next release?

[MiguelCompany]

@tiany44 I don't think it will make it to the next release, but it should be possible to workaround the issue with the XML files provided along the comments.

I think there may be another unrelated issue here. Maybe the communication from the ros2 daemon and the ros2 topic list is not possible between users with different permissions. What I mean is that perhaps you need to start another superclient daemon on the second ssh terminal.

[tiany44]

Both of my clients use the same super client configuration. Without sudo, ros2 topic list shows topic and nodes started from the other client. I also run sudo su && ros2 daemon start on both super clients to see if a sudo initiated daemon would fix the issue. It did not

[JLBuenoLopez]

eProsima/Fast-DDS-docs#497 introduced a note stating this caveat related to SHM communications. I am labelling this as an enhancement and moving to the corresponding forum discussion according to Fast DDS CONTRIBUTING guidelines.