Hello everyone, during the payload test our team encountered a lot of communication issues that made basically impossible to execute the whole localization and navigation pipeline.
We noticed that ROS messages failed to arrive to the nodes, especially when many nodes are active.
I’ll list what we tried and what we could try next time, but even with the fixes we didn’t manage to having reliable messages when all nodes are running.
What we tried during the test:
Having the ROS subscriber use best effort QoS (instead of reliable):
i don’t think this will improve a lot communication and it could cause the opposite effect. If a message is very large it is divided into packets. With reliable QoS every packet is guaranteed to arrive, even if with a delay. With Best effort instead if only one of this packet is lost, the whole message is discarded. Some more info in this discussion
Changing network settings:
As described in this post on the forum. These commands can’t be executed on the VM so they were applied by the organizers during the test. This actually improved a lot the performances but there were still issues when more nodes were activated. (later I suggest other settings)
Change DDS settings:
We tried tuning the CycloneDDS XML as described here: DDS settings for ROS 2 and Autoware - Autoware Documentation
This also seems to have improved communication during the test. In previous tests and projects we found these settings to improve a lot the internal communication.
Other things that we do not tested but could be useful:
different network settings:
Rather than the settings mentioned above, in previous projects I used these settings (disclaimer: i didn’t test these settings on the VM, maybe they can’t be used because of VM limitations):
# Increase the maximum receive buffer size for network packets
sudo sysctl -w net.core.rmem_max=2147483647 # 2 GiB, default is 208 KiB
# IP fragmentation settings
sudo sysctl -w net.ipv4.ipfrag_time=3 # in seconds, default is 30 s
sudo sysctl -w net.ipv4.ipfrag_high_thresh=134217728 # 128 MiB, default is 256 KiB
sudo sysctl -w net.ipv4.udp_mem="102400 873800 16777216"
sudo sysctl -w net.core.netdev_max_backlog="30000"
sudo sysctl -w net.core.rmem_max="20971520"
sudo sysctl -w net.core.wmem_max="20971520"
sudo sysctl -w net.core.rmem_default="20971520"
sudo sysctl -w net.core.wmem_default="20971520"
Point Cloud compression:
Organizer suggested that point cloud compression could be used, (some examples of compression algorithms integrated with ROS: GitHub - ros-perception/point_cloud_transport: Point Cloud Compression for ROS).
The problem with this is that if open source ROS packages are used it is possible that they do not handle compressed point clouds, so the cloud has to be decompressed before reaching the node. This would probably still improve communication if the bottleneck is from the rover to the VM and not inside the VM.
Composable nodes: Composition — ROS 2 Documentation: Jazzy documentation
Using composable nodes messages could be exchanged without using topics but internally, more info on the documentation.
Not all packages supports this but it could be added for custom nodes and enabled for Nav2.
Did anyone else encountered similar issues?
Did you manage to solve them?
In our case in the moment we enabled Nav2 the messages stopped arriving to the localization nodes. And to have a working localization I had to disable foxglove but I don’t have any feedback then, except for the Zoom Webcam.
Hi
We also faced similar issues in the Payload test, where when we ran all the nodes for our nav2 stack, the topics from the VM did not get populated. For example, we had written our script that decoded the ffmpeg image messages, and it was working in the test runs. However, for some reason, it was not working in the Payload test. We faced a similar issue while running our package for mapping.
Another issue was that we could not get Ros2Router to work, so we could not even attempt the anomaly detection task. Did any other teams face or solve this issue?
Enabling composable nodes for the Nav2 nodes did not solve the issue, the messages still stop arriving when I start Nav2.
A big issue we found is that it seems that the Lidar stops working at random moments.
For example: using ros2 topic Hz on the lidar topic returns 20 Hz, so the lidar is working correctly. Then I run the slam node and it doesn’t start because the lidar messages are not arriving.
Ros2 topic Hz now returns nothing so the messages are not arriving and the lidar topic doesn’t appear even in ros2 topic list.
After some time it randomly reappear and start working again.
Due to this issue is really difficult to understand when a problem is related to dds communication issues or when is the problem above.
We also faced similar issues while running the panther on autonomous mode.If any other team faced these issues then can also comment or if you got it running please let us know any fixes
Limitations of Compute(Nodes/Docker) on VM
When we ran ROS2Router and the decompressor Docker containers, we were able to get the camera feeds to the remote site and on the VM, but for some reason, the pointcloud data stopped getting populated, and hence, we were not able to run our Auto Nav Stack.
The solution provided by the organizers was to stop the Ros2Router and to minimize the number of docker containers running. We were able to get the point cloud data but at the expense of the camera topic at the remote site.
When we ran our auto_nav stack, we faced these errors
[WARN] [1754998033.523027220] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/battery/battery_status' with type 'sensor_msgs::msg::dds_::BatteryState_' from USER_DATA '(null)'.
[WARN] [1754998033.523101981] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/battery/charging_status' with type 'husarion_ugv_msgs::msg::dds_::ChargingStatus_' from USER_DATA '(null)'.
[WARN] [1754998033.523123092] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/camera_back/ffmpeg' with type 'ffmpeg_image_transport_msgs::msg::dds_::FFMPEGPacket_' from USER_DATA '(null)'.
[WARN] [1754998033.523131259] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/camera_front/ffmpeg' with type 'ffmpeg_image_transport_msgs::msg::dds_::FFMPEGPacket_' from USER_DATA '(null)'.
[WARN] [1754998033.523140403] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/camera_left/ffmpeg' with type 'ffmpeg_image_transport_msgs::msg::dds_::FFMPEGPacket_' from USER_DATA '(null)'.
[WARN] [1754998033.523147722] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/camera_right/camera_info' with type 'sensor_msgs::msg::dds_::CameraInfo_' from USER_DATA '(null)'.
[WARN] [1754998033.523156767] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/camera_right/ffmpeg' with type 'ffmpeg_image_transport_msgs::msg::dds_::FFMPEGPacket_' from USER_DATA '(null)'.
[WARN] [1754998033.523163541] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/cmd_vel' with type 'geometry_msgs::msg::dds_::TwistStamped_' from USER_DATA '(null)'.
[WARN] [1754998033.523167032] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/controller_manager/activity' with type 'controller_manager_msgs::msg::dds_::ControllerManagerActivity_' from USER_DATA '(null)'.
[WARN] [1754998033.523170612] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/controller_manager/introspection_data/full' with type 'pal_statistics_msgs::msg::dds_::Statistics_' from USER_DATA '(null)'.
[WARN] [1754998033.523364002] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/controller_manager/introspection_data/names' with type 'pal_statistics_msgs::msg::dds_::StatisticsNames_' from USER_DATA '(null)'.
[WARN] [1754998033.523393886] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/controller_manager/introspection_data/values' with type 'pal_statistics_msgs::msg::dds_::StatisticsValues_' from USER_DATA '(null)'.
[WARN] [1754998033.523411701] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/cx/lslidar_driver_node/transition_event' with type 'lifecycle_msgs::msg::dds_::TransitionEvent_' from USER_DATA '(null)'.
[WARN] [1754998033.523425534] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/cx/lslidar_point_cloud' with type 'sensor_msgs::msg::dds_::PointCloud2_' from USER_DATA '(null)'.
[WARN] [1754998033.523441251] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/diagnostics' with type 'diagnostic_msgs::msg::dds_::DiagnosticArray_' from USER_DATA '(null)'.
[WARN] [1754998033.523572112] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/dynamic_joint_states' with type 'control_msgs::msg::dds_::DynamicJointState_' from USER_DATA '(null)'.
[WARN] [1754998033.523590643] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/hardware/e_stop' with type 'std_msgs::msg::dds_::Bool_' from USER_DATA '(null)'.
[WARN] [1754998033.523603361] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/hardware/io_state' with type 'husarion_ugv_msgs::msg::dds_::IOState_' from USER_DATA '(null)'.
[WARN] [1754998033.523624001] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/hardware/robot_driver_state' with type 'husarion_ugv_msgs::msg::dds_::RobotDriverState_' from USER_DATA '(null)'.
[WARN] [1754998033.523639350] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/imu/data' with type 'sensor_msgs::msg::dds_::Imu_' from USER_DATA '(null)'.
[WARN] [1754998033.523752041] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/joint_states' with type 'sensor_msgs::msg::dds_::JointState_' from USER_DATA '(null)'.
[WARN] [1754998033.523770750] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/joy/set_feedback' with type 'sensor_msgs::msg::dds_::JoyFeedbackArray_' from USER_DATA '(null)'.
[WARN] [1754998033.523783736] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/joy' with type 'sensor_msgs::msg::dds_::Joy_' from USER_DATA '(null)'.
[WARN] [1754998033.523796188] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/lights/channel_1_frame' with type 'sensor_msgs::msg::dds_::Image_' from USER_DATA '(null)'.
[WARN] [1754998033.523808129] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/lights/channel_2_frame' with type 'sensor_msgs::msg::dds_::Image_' from USER_DATA '(null)'.
[WARN] [1754998033.523935467] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/localization/set_pose' with type 'geometry_msgs::msg::dds_::PoseWithCovarianceStamped_' from USER_DATA '(null)'.
[WARN] [1754998033.523954920] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/odometry/wheels' with type 'nav_msgs::msg::dds_::Odometry_' from USER_DATA '(null)'.
[WARN] [1754998033.523967558] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/robot_description' with type 'std_msgs::msg::dds_::String_' from USER_DATA '(null)'.
[WARN] [1754998033.523977550] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/system_status' with type 'husarion_ugv_msgs::msg::dds_::SystemStatus_' from USER_DATA '(null)'.
[WARN] [1754998033.523985664] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/parameter_events' with type 'rcl_interfaces::msg::dds_::ParameterEvent_' from USER_DATA '(null)'.
[WARN] [1754998033.524064678] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/rosout' with type 'rcl_interfaces::msg::dds_::Log_' from USER_DATA '(null)'.
[WARN] [1754998033.524079702] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/tf_static' with type 'tf2_msgs::msg::dds_::TFMessage_' from USER_DATA '(null)'.
[WARN] [1754998033.524102139] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/tf' with type 'tf2_msgs::msg::dds_::TFMessage_' from USER_DATA '(null)'.
[WARN] [1754998033.524117319] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/camera_front/camera_info' with type 'sensor_msgs::msg::dds_::CameraInfo_' from USER_DATA '(null)'.
[WARN] [1754998033.524129024] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/camera_back/camera_info' with type 'sensor_msgs::msg::dds_::CameraInfo_' from USER_DATA '(null)'.
[WARN] [1754998033.524246552] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/odometry/filtered' with type 'nav_msgs::msg::dds_::Odometry_' from USER_DATA '(null)'.
[WARN] [1754998033.524269900] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/camera_main/ffmpeg' with type 'ffmpeg_image_transport_msgs::msg::dds_::FFMPEGPacket_' from USER_DATA '(null)'.
[WARN] [1754998033.524285186] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/battery/battery_status' with type 'sensor_msgs::msg::dds_::BatteryState_' from USER_DATA '(null)'.
[WARN] [1754998033.524298752] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/battery/charging_status' with type 'husarion_ugv_msgs::msg::dds_::ChargingStatus_' from USER_DATA '(null)'.
[WARN] [1754998033.524310761] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/camera_back/camera_info' with type 'sensor_msgs::msg::dds_::CameraInfo_' from USER_DATA '(null)'.
[WARN] [1754998033.524322945] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/camera_back/ffmpeg' with type 'ffmpeg_image_transport_msgs::msg::dds_::FFMPEGPacket_' from USER_DATA '(null)'.
[WARN] [1754998033.524334432] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/camera_front/camera_info' with type 'sensor_msgs::msg::dds_::CameraInfo_' from USER_DATA '(null)'.
[WARN] [1754998033.524478499] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/camera_left/camera_info' with type 'sensor_msgs::msg::dds_::CameraInfo_' from USER_DATA '(null)'.
[WARN] [1754998033.524498588] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/camera_left/ffmpeg' with type 'ffmpeg_image_transport_msgs::msg::dds_::FFMPEGPacket_' from USER_DATA '(null)'.
[WARN] [1754998033.524512686] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/camera_right/camera_info' with type 'sensor_msgs::msg::dds_::CameraInfo_' from USER_DATA '(null)'.
[WARN] [1754998033.524522144] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/camera_right/ffmpeg' with type 'ffmpeg_image_transport_msgs::msg::dds_::FFMPEGPacket_' from USER_DATA '(null)'.
[WARN] [1754998033.524542433] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/cmd_vel' with type 'geometry_msgs::msg::dds_::TwistStamped_' from USER_DATA '(null)'.
[WARN] [1754998033.524624801] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/controller_manager/activity' with type 'controller_manager_msgs::msg::dds_::ControllerManagerActivity_' from USER_DATA '(null)'.
[WARN] [1754998033.524642365] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/controller_manager/introspection_data/full' with type 'pal_statistics_msgs::msg::dds_::Statistics_' from USER_DATA '(null)'.
[WARN] [1754998033.524654845] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/controller_manager/introspection_data/names' with type 'pal_statistics_msgs::msg::dds_::StatisticsNames_' from USER_DATA '(null)'.
[WARN] [1754998033.524667154] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/controller_manager/introspection_data/values' with type 'pal_statistics_msgs::msg::dds_::StatisticsValues_' from USER_DATA '(null)'.
[WARN] [1754998033.524679471] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/cx/lslidar_driver_node/transition_event' with type 'lifecycle_msgs::msg::dds_::TransitionEvent_' from USER_DATA '(null)'.
[WARN] [1754998033.524827582] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/cx/lslidar_point_cloud' with type 'sensor_msgs::msg::dds_::PointCloud2_' from USER_DATA '(null)'.
[WARN] [1754998033.524855190] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/cx/scan' with type 'sensor_msgs::msg::dds_::LaserScan_' from USER_DATA '(null)'.
[WARN] [1754998033.524868977] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/diagnostics' with type 'diagnostic_msgs::msg::dds_::DiagnosticArray_' from USER_DATA '(null)'.
[WARN] [1754998033.524879075] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/dynamic_joint_states' with type 'control_msgs::msg::dds_::DynamicJointState_' from USER_DATA '(null)'.
[WARN] [1754998033.524892235] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/hardware/e_stop' with type 'std_msgs::msg::dds_::Bool_' from USER_DATA '(null)'.
[WARN] [1754998033.524991655] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/hardware/io_state' with type 'husarion_ugv_msgs::msg::dds_::IOState_' from USER_DATA '(null)'.
[WARN] [1754998033.525008340] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/hardware/robot_driver_state' with type 'husarion_ugv_msgs::msg::dds_::RobotDriverState_' from USER_DATA '(null)'.
[WARN] [1754998033.525023125] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/imu/data' with type 'sensor_msgs::msg::dds_::Imu_' from USER_DATA '(null)'.
[WARN] [1754998033.525037061] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/joint_states' with type 'sensor_msgs::msg::dds_::JointState_' from USER_DATA '(null)'.
[WARN] [1754998033.525056886] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/joy/set_feedback' with type 'sensor_msgs::msg::dds_::JoyFeedbackArray_' from USER_DATA '(null)'.
[WARN] [1754998033.525161546] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/joy' with type 'sensor_msgs::msg::dds_::Joy_' from USER_DATA '(null)'.
[WARN] [1754998033.525177983] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/lights/channel_1_frame' with type 'sensor_msgs::msg::dds_::Image_' from USER_DATA '(null)'.
[WARN] [1754998033.525204268] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/lights/channel_2_frame' with type 'sensor_msgs::msg::dds_::Image_' from USER_DATA '(null)'.
[WARN] [1754998033.525217451] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/localization/set_pose' with type 'geometry_msgs::msg::dds_::PoseWithCovarianceStamped_' from USER_DATA '(null)'.
[WARN] [1754998033.525230075] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/odometry/filtered' with type 'nav_msgs::msg::dds_::Odometry_' from USER_DATA '(null)'.
[WARN] [1754998033.525328852] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/odometry/wheels' with type 'nav_msgs::msg::dds_::Odometry_' from USER_DATA '(null)'.
[WARN] [1754998033.525346036] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/robot_description' with type 'std_msgs::msg::dds_::String_' from USER_DATA '(null)'.
[WARN] [1754998033.525358549] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/panther/system_status' with type 'husarion_ugv_msgs::msg::dds_::SystemStatus_' from USER_DATA '(null)'.
[WARN] [1754998033.525370849] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/rosout' with type 'rcl_interfaces::msg::dds_::Log_' from USER_DATA '(null)'.
[WARN] [1754998033.525447632] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/parameter_events' with type 'rcl_interfaces::msg::dds_::ParameterEvent_' from USER_DATA '(null)'.
[WARN] [1754998033.525533247] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/tf' with type 'tf2_msgs::msg::dds_::TFMessage_' from USER_DATA '(null)'.
[WARN] [1754998033.525542496] [rmw_cyclonedds_cpp]: Failed to parse type hash for topic 'rt/tf_static' with type 'tf2_msgs::msg::dds_::TFMessage_' from USER_DATA '(null)'.
Even after stopping the ros2router, we were able to get the pointcloud data as topics , but while running our nav2 stack, we were still getting this error in the pointcloud data. We had to switch to the scan topic, and that seemed to be working, and we were able to generate the map but time ran out when we tried to run our auto nav part and we were not able to detect the error
Thanks for sharing your issues! In our case we didn’t use ros2router so the problems were happening with only nodes in the VM.
I never got that cyclonedds error, maybe is a version mismatch between messages between VM, your local machine and ros2router. You could try running sudo apt update on both (and save the docker image for ros2 router to avoid doing it every time)
Hi
We checked, and there was no version mismatch between the Docker containers on our VM and our local machines. It would be great if the organizers could help with this issue or if other teams could share their auto-nav experience.