NVIDIA-accelerated packages for depth segmentation.
Learn how to use this package by watching our on-demand webinar: Using ML Models in ROS 2 to Robustly Estimate Distance to Obstacles
Isaac ROS Depth Segmentation provides NVIDIA NVIDIA-accelerated packages for
depth segmentation. The isaac_ros_bi3d package uses the
optimized Bi3D DNN
model
to perform stereo-depth estimation via binary classification, which is
used for depth segmentation. Depth segmentation can be used to
determine whether an obstacle is within a proximity field and to avoid
collisions with obstacles during navigation.
Bi3D is used in a graph of nodes to provide depth segmentation from a time-synchronized input left and right stereo image pair. Images to Bi3D need to be rectified and resized to the appropriate input resolution. The aspect ratio of the image needs to be maintained; hence, a crop and resize may be required to maintain the input aspect ratio. The graph for DNN encode, to DNN inference, to DNN decode is part of the Bi3D node. Inference is performed using TensorRT, as the Bi3D DNN model is designed to use optimizations supported by TensorRT.
Compared to other stereo disparity functions, depth segmentation provides a prediction of whether an obstacle is within a proximity field, as opposed to continuous depth, while simultaneously predicting freespace from the ground plane, which other functions typically do not provide. Also unlike other stereo disparity functions in Isaac ROS, depth segmentation runs on NVIDIA DLA (deep learning accelerator), which is separate and independent from the GPU. For more information on disparity, refer to this page.
Note
This DNN is optimized for and evaluated with RGB global shutter camera images, and accuracy may vary on monochrome images.
This package is powered by NVIDIA Isaac Transport for ROS (NITROS), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.
| Sample Graph |
Input Size |
AGX Orin |
Orin NX |
x86_64 w/ RTX 4090 |
|---|---|---|---|---|
| Depth Segmentation Node |
576p |
45.8 fps 79 ms @ 30Hz |
28.2 fps 99 ms @ 30Hz |
105 fps 25 ms @ 30Hz |
Please visit the Isaac ROS Documentation to learn how to use this repository.
Update 2024-12-10: Update to be compatible with JetPack 6.1