Merge pull request #36 from NVIDIA-ISAAC-ROS/release-dp-2

Isaac ROS 0.20.0 (DP2)
NVIDIA-ISAAC-ROS · Oct 19, 2022 · 17759d5 · 17759d5
2 parents 2c66866 + f650c3b
commit 17759d5
Show file tree

Hide file tree

Showing 86 changed files with 1,696 additions and 666 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -0,0 +1,14 @@
+# Isaac ROS Contribution Rules
+
+Any contribution that you make to this repository will
+be under the Apache 2 License, as dictated by that
+[license](http://www.apache.org/licenses/LICENSE-2.0.html):
+
+> **5. Submission of Contributions.** Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
+
+Contributors must sign-off each commit by adding a `Signed-off-by: ...`
+line to commit messages to certify that they have the right to submit
+the code they are contributing to the project according to the
+[Developer Certificate of Origin (DCO)](https://developercertificate.org/).
+
+[//]: # (202201002)
diff --git a/LICENSE b/LICENSE
diff --git a/README.md b/README.md
diff --git a/docs/technical-details.md b/docs/technical-details.md
@@ -1,4 +1,5 @@
 # Nvblox Technical Details
+
 Nvblox builds the reconstructed map in the form of a Truncated Signed Distance Function (TSDF) stored in a 3D voxel grid. This approach is similar to 3D occupancy grid mapping approaches in which occupancy probabilities are stored at each voxel. However, TSDF-based approaches like nvblox store the (signed) distance to the closest surface at each voxel. The surface of the environment can then be extracted as the zero-level set of this voxelized function. Typically, TSDF-based reconstructions provide higher quality surface reconstructions.
 
-In addition to their use in reconstruction, distance fields are also useful for path planning, because they provide an immediate means of checking whether potential future robot positions are in collision. The dual utility of distance functions for both reconstruction and planning motivates their use in nvblox (a reconstruction library for path planning).
+In addition to their use in reconstruction, distance fields are also useful for path planning, because they provide an immediate means of checking whether potential future robot positions are in collision. The dual utility of distance functions for both reconstruction and planning motivates their use in nvblox (a reconstruction library for path planning).
diff --git a/docs/troubleshooting-nvblox-vslam-realsense.md b/docs/troubleshooting-nvblox-vslam-realsense.md
@@ -0,0 +1,193 @@
+# Troubleshoot for RealSense-based Reconstruction Example
+
+This page contains troubleshooting for the nvblox-VSLAM-realsense example found [here](tutorial-nvblox-vslam-realsense.md). The example combines nvblox-based reconstruction with VSLAM. Both of these processes, nvblox and VSLAM, are driven by the realsense depth camera.
+
+The setup for the example is shown below.
+
+<div align="center"><img src="../resources/nvblox_realsense_example_diagram.png"/></div>
+
+Images (2 x IR images and 1 x Depth image) are published at 60Hz from the realsense ROS2 node. At the time of publishing we are using the `ros2-beta` branch of [realsense-ros](https://github.com/IntelRealSense/realsense-ros/tree/ros2-beta). This is the branch used by the ROS2 apt packages (viewable on the [ros package index](https://index.ros.org/p/realsense2_camera/github-IntelRealSense-realsense-ros/#humble)).
+
+> **Why do we include the realsense splitter?**: The depth output from the realsense is significantly improved when using its inbuilt projector which increases the amount of texture in the scene. However, VSLAM performance is disrupted by the projector. We, therefore, use the realsense's toggling feature such that the projector is switched on and off in interleaved frames (on, off, on, off, on, off,...etc.). The splitter splits this interleaved image stream into two channels, one with the projector on and one with the projector off. IR images with the projector off are passed to VSLAM, while depth images with the projector on are passed to nvblox. This splitting occurs based on the metadata associated with each frame, published by the realsense, which contains a field which states if the projector was on at the time of frame capture.
+
+## Symptoms
+
+No reconstruction appears in RVIZ, and the terminal reports:
+
+```bash
+[nvblox_node-3] Warning: Invalid frame ID "odom" passed to canTransform argument target_frame - frame does not exist
+[nvblox_node-3] at line 93 in /opt/ros/humble/src/geometry2/tf2/src/buffer_core.cpp
+[nvblox_node-3] Warning: Invalid frame ID "odom" passed to canTransform argument target_frame - frame does not exist
+[nvblox_node-3] at line 93 in /opt/ros/humble/src/geometry2/tf2/src/buffer_core.cpp
+```
+
+**Explanation:** This message is repeating because nvblox can't get the camera pose in the odom frame. It can't get the pose because VSLAM is not publishing it. VSLAM is not publishing anything because (likely) it is not receiving input data.
+
+So the issue here is that VSLAM is not getting data. This is caused either by one of:
+1) Images not coming off the camera into Linux.
+2) Images not being published by the ROS node.
+3) Images not making it through the splitter (usually because the metadata does not contain the projector state).
+
+I have seen all three of these issues. All three of them have the same symptom (no pose for nvblox), but all three have different solutions.
+
+## Issue
+
+From what I can tell, the underlying issue is that the realsense does not support recent versions of the Linux kernel (see [here](https://github.com/IntelRealSense/librealsense/blob/master/doc/distribution_linux.md) for the list of supported kernel versions). There are some related issues [here](https://github.com/IntelRealSense/librealsense/issues/10439) and [here](https://github.com/IntelRealSense/librealsense/issues/5229).
+
+To determine what to do we have to do a few checks.
+
+### Check #1 - Images and Metadata are coming into Linux
+
+Check that images are making it off the camera. Start the `realsense-viewer`.
+
+```bash
+realsense-viewer
+```
+Then:
+
+1) Before turning on the "Stereo Module" click the drop down arrow **(red box below)** and turn on Infrared camera. 
+2) Turn on the stereo module
+3) Click the further drop down labels "controls" and click on the option "Emitter On Off". **(purple box below)**
+4) In the upper right change from 3D view to 2D. **(green box below)**
+5) You should see something like the image below.
+6) On the greyscale image click the left-most icon in the upper-right-hand corner which looks like three horizontal bars with two tiny check marks. This displays the meta data. **(orange box below)**
+
+Do you see two images like below?
+
+<div align="center"><img src="../resources/realsense_troubleshooting_viewer_ir.png"/></div>
+
+Do you see the metadata labelled "Frame Laser Pose Mode" **(green box above)** which should be flicking between 0 and 1 with the emitter. If no, preceded to the section below for [Solution 1 - Install DKMS](#solution-1---install-dkms)
+
+
+### Check #2 - Check Images are Being Published in ROS
+
+You're here because your realsense-viewer is operating as expected, i.e. is showing IR images and emitter-state metadata, but the nvblox, VSLAM, realsense example still isn't generating a reconstruction.
+
+Start the example:
+
+```bash
+ros2 launch nvblox_examples_bringup nvblox_vslam_realsense.launch.py
+```
+
+In rviz check that the following images can be viewed.
+
+- Topics before splitter
+  - `/camera/color/image_raw`
+  - `/camera/depth/image_rect_raw`
+  - `/camera/infra1/image_rect_raw`
+  - `/camera/infra2/image_rect_raw`
+- Topics after splitter
+  - `/camera/realsense_splitter_node/output/depth`
+  - `/camera/realsense_splitter_node/output/infra_1`
+  - `/camera/realsense_splitter_node/output/infra_2`
+
+The topics can be viewed by changing the topic field of the image topic **(red box below)**
+
+<div align="center"><img src="../resources/realsense_troubleshooting_rviz.png"/></div>
+
+If the topics before the splitter are not all viewable, preceded to [Solution 2](#solution-2---build-without-cuda) - build realsense-ros without CUDA.
+
+If only the topics after the splitter are not available and [Check 1](#check-1---images-and-metadata-are-coming-into-linux) passed. I'm out of ideas and you should probably get in contact.
+
+
+
+## Solutions
+
+The solutions I have found boil down to:
+
+- Install the realsense DKMS, or
+- Build without CUDA, or
+- Both of the above.
+
+Try the solution the above checks directed you to. Repeat the checks 1 and 2. If you still have no luck, attempt the other solution.
+
+### Solution 1 - Install DKMS
+
+You're here because either you get no images off the camera, or you don't get the correct metadata field.
+
+On a supported kernel version, the apt package patches the kernel when you install `librealsense-dkms` such that the realsense operates as expected. 
+
+If everything is as expected on a supported kernel and you run
+
+```bash
+helen@holeynikova-dt:~/workspaces/isaac_ros-dev/src/isaac_ros_common (dev)$ uname -r
+5.8.0-43-generic
+
+helen@holeynikova-dt:~/workspaces/isaac_ros-dev/src/isaac_ros_common (dev)$ dkms status
+librealsense2-dkms, 1.3.18, 5.8.0-43-generic, x86_64: installed
+librealsense2-dkms, 1.3.18, 5.8.0-67-generic, x86_64: installed
+nvidia, 510.85.02, 5.13.0-40-generic, x86_64: installed
+nvidia, 510.85.02, 5.8.0-43-generic, x86_64: installed
+
+helen@holeynikova-dt:/lib/udev/rules.d$ modinfo uvcvideo | grep "version:"
+version: 1.1.2.realsense-1.3.18
+srcversion: 964E8D3335D17053B8EEDD2
+```
+
+The important thing to see here is that the word "realsense" appears:
+- after the second command, it appears **next to your version of the kernel**, and
+- after the third command, it appears at all.
+
+Likely, if you're here, you have a recent kernel and the words realsense do not appear, or appear in the second command next to an old kernel version. Something like:
+
+```bash
+shubham@shubham-linuxbox:~$ uname -r
+5.15.0-48-generic
+shubham@shubham-linuxbox:~$ dkms status
+nvidia, 510.85.02, 5.15.0-48-generic, x86_64: installed
+shubham@shubham-linuxbox:~$ modinfo uvcvideo | grep "version:"
+version:        1.1.1
+srcversion:     18D809600E7D1E107042647
+```
+
+**The solution** is to install the dkms for recent versions of the kernel. We found these here:
+
+https://github.com/mengyui/librealsense2-dkms/releases/tag/initial-support-for-kernel-5.15
+
+Download the .deb file named `librealsense2-dkms-dkms_1.3.14_amd64.deb`
+In the directory where you downloaded the file run: `sudo apt install ./librealsense2-dkms-dkms_1.3.14_amd64.deb`
+Now the commands above, on my machine, return:
+
+```bash
+alex@shallowinsight:~$ uname -r
+5.15.0-48-generic
+
+alex@shallowinsight:~$ dkms status
+librealsense2-dkms, 1.3.14, 5.15.0-48-generic, x86_64: installed
+librealsense2-dkms, 1.3.14, 5.15.0-50-generic, x86_64: installed
+nvidia, 510.85.02, 5.15.0-48-generic, x86_64: installed
+nvidia, 510.85.02, 5.15.0-50-generic, x86_64: installed
+nvidia, 510.85.02, 5.4.0-126-generic, x86_64: installed
+nvidia, 510.85.02, 5.4.0-128-generic, x86_64: installed
+
+alex@shallowinsight:~$ modinfo uvcvideo | grep "version:"
+version:        1.1.2.realsense-1.3.14
+srcversion:     26234508927E0F6886C9A48
+```
+**Go back and repeat check 1. You should have images + metadata.**
+
+
+### Solution 2 - Build without CUDA
+
+You're here because check 1 passed; you have images on your machine + metadata, but you get no images in ROS. The solution here is to build librealsense without CUDA.
+
+Manually build librealsense within the container without CUDA. Run the following commands within the isaac_ros_common docker container:
+
+```bash
+/opt/realsense/build-librealsense.sh -n
+```
+
+now rebuild the ROS2 workspace
+
+```bash
+rm -rf build/ install/ log/
+colcon build --symlink-install --continue-on-error
+source install/setup.bash
+```
+
+**Go back and repeat check 2. You should have images in ROS before and after the splitter.**
+
+# Things still aren't working
+Well this isn't going well. One thing to try is to build with `-DFORCE_RSUSB_BACKEND=true` as reported in this [issue](https://github.com/IntelRealSense/librealsense/issues/10439).
+
+Otherwise, file an issue, and describe the steps you've taken here.
diff --git a/docs/tutorial-isaac-sim.md b/docs/tutorial-isaac-sim.md
@@ -1,4 +1,12 @@
-# Tutorial with Isaac Sim
+# Tutorial for NVBLOX with Isaac Sim
+
+<div align="center"><img src="../resources/isaac_sim_nvblox_nav2.gif"/></div>
+
+## Overview
+
+This tutorial walks you through generating a 3D reconstruction with nvblox using image and LiDAR data from Isaac Sim. The reconstruction is displayed as a 3D mesh in RVIZ. Nvblox also converts the reconstruction into a 2D costmap, which is fed into the Nav2 stack for path planning.
+
+## Tutorial Walkthrough
 
 1. Complete the [Quickstart section](../README.md#quickstart) in the main README.
 2. Launch the Docker container using the `run_dev.sh` script:
@@ -24,14 +32,12 @@
       omni_python ~/workspaces/isaac_ros-dev/src/isaac_ros_nvblox/nvblox_isaac_sim/omniverse_scripts/carter_warehouse.py
     ```
 
-> **Note:** If you are using a different version of Isaac Sim, replace `isaac_sim-2022.1.1-release.1` with your version found at `'~/.local/share/ov/pkg/`
+    > **Note:** If you are using a different version of Isaac Sim, replace `isaac_sim-2022.1.1-release.1` with your version found at `'~/.local/share/ov/pkg/`
 
 6. Launch the pre-composed pipeline launchfile:
 
     ```bash
     ros2 launch nvblox_nav2 carter_sim.launch.py
     ```
 
-7. Click on the `2D Goal Pose` button. You should see the mesh, costmap, and the robot moving towards the goal location as shown below:
-
-<div align="center"><img src="../resources/isaac_sim_nvblox_nav2.gif"/></div>
+7. Click the **2D Goal Pose** button. You should see the mesh and costmap, as well as the robot moving towards the goal location, as shown at the top of this page.
diff --git a/docs/tutorial-nvblox-vslam-realsense.md b/docs/tutorial-nvblox-vslam-realsense.md
@@ -1,59 +1,38 @@
-# Tutorial For Realsense-based Reconstruction
+# Tutorial For RealSense-based Reconstruction
 
-In this tutorial we'll demonstrate how one can perform depth-camera based reconstruction using a [realsense](https://www.intel.com/content/www/us/en/architecture-and-technology/realsense-overview.html) camera, NVIDIA [VSLAM](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_visual_slam), and [nvblox](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nvblox).
+This tutorial demonstrates how to perform depth-camera based reconstruction using a [Realsense](https://www.intel.com/content/www/us/en/architecture-and-technology/realsense-overview.html) camera, NVIDIA [VSLAM](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_visual_slam), and [nvblox](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nvblox).
 
-> Note: This tutorial has been tested with a realsense D455 or D435 connected to an x86 computer with an NVIDIA graphics card, as well as Jetson Xavier AGX.
+> Note: This tutorial has been tested with a Realsense D455/D435 camera connected to an x86 computer with an NVIDIA graphics card, as well as a Jetson Xavier AGX.
 
 ## Installing the Dependencies  
 
-Complete steps 1, 2, and 3 described in the [quickstart guide](../README.md#quickstart) to clone `isaac_ros_common` and `isaac_ros_nvblox` if you haven't done it already.
+1. Complete steps 1, 2, and 3 described in the [quickstart guide](../README.md#quickstart) to clone `isaac_ros_common` and
+   `isaac_ros_nvblox` and to build [librealsense](https://github.com/IntelRealSense/librealsense) as part of the Docker container.
 
-Then:
+2. Complete the [RealSense setup tutorial](https://github.com/NVIDIA-ISAAC-ROS/.github/blob/main/profile/realsense-setup.md).
 
-1. Download the code for the realsense ROS2 interface: 
-   ```bash
-   cd ~/workspaces/isaac_ros-dev/src
-   ```
-   ```bash
-    git clone https://github.com/IntelRealSense/realsense-ros.git -b ros2
-   ```
-
-> Note: Follow this [workaround](https://gitlab-master.nvidia.com/isaac_ros/isaac_ros_common/-/blob/release-dp1.1/docs/troubleshooting.md#realsense-driver-doesnt-work-with-ros2-humble) to build realsense2_camera package on ROS2 Humble.
+3. Download the code for [Isaac ROS Visual SLAM](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_visual_slam.git).
 
-2. Download the code for [Isaac ROS Visual SLAM](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_visual_slam.git).
    ```bash
     git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_visual_slam.git
    ```
 
-3. Plug in your realsense camera (it needs to be plugged in before launching the container).
-
 4. Launch the Docker container using the `run_dev.sh` script:
+
     ```bash
     cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \
       ./scripts/run_dev.sh
     ```
 
-5. At this point we can check that the realsense camera is connected by running `realsense-viewer`:
-
-   ```bash
-   realsense-viewer
-   ```
-   and turning on the "Stereo Module" in the GUI. You should see something like:
-
-<div align="center"><img src="../resources/realsense_viewer.png" width=800px/></div>
-
-6. If the `realsense-viewer` GUI prompts you to update the camera firmware, do so.
-
-> Note: In our testing it was important that the camera had a recent version of the firmware installed.
-
-7. Inside the container, install package-specific dependencies via `rosdep`:
+5. Inside the container, install package-specific dependencies via `rosdep`:
 
     ```bash
     cd /workspaces/isaac_ros-dev/ && \
         rosdep install -i -r --from-paths src --rosdistro humble -y --skip-keys "libopencv-dev libopencv-contrib-dev libopencv-imgproc-dev python-opencv python3-opencv nvblox"
     ```
-
-8. Build and source the workspace:
+
+6. Build and source the workspace:
+
     ```bash
     cd /workspaces/isaac_ros-dev && \
       colcon build --symlink-install && \
@@ -62,18 +41,22 @@ Then:
 
 ## Running the Example
 
-To run the example, source the workspace 
+To run the example, source the workspace:
 
 ```bash
 source /workspaces/isaac_ros-dev/install/setup.bash
 ```
 
-And run the launchfile which launches the example
+Then run the launch file, which launches the example:
 
 ```bash
 ros2 launch nvblox_examples_bringup nvblox_vslam_realsense.launch.py
 ```
 
-Here is a few seconds of the result from running the example from my desk:
+Here is a few seconds of the result from running the example:
 
 <div align="center"><img src="../resources/realsense_example.gif"/></div>
+
+## Troubleshooting
+
+See our troubleshooting page [here](troubleshooting-nvblox-vslam-realsense.md).
diff --git a/isaac_ros_nvblox/CMakeLists.txt b/isaac_ros_nvblox/CMakeLists.txt
@@ -1,10 +1,19 @@
-# Copyright (c) 2022, NVIDIA CORPORATION.  All rights reserved.
+# SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
+# Copyright (c) 2021-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 #
-# NVIDIA CORPORATION and its licensors retain all intellectual property
-# and proprietary rights in and to this software, related documentation
-# and any modifications thereto.  Any use, reproduction, disclosure or
-# distribution of this software and related documentation without an express
-# license agreement from NVIDIA CORPORATION is strictly prohibited.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# SPDX-License-Identifier: Apache-2.0
 
 cmake_minimum_required(VERSION 3.5)
 project(isaac_ros_nvblox)