add training and update readme

README.md

@@ -8,6 +8,110 @@ Official Implementation for [Agile But Safe: Learning Collision-Free High-Speed
   <img src="images/Youtube-Cover[2M].png" width="80%"/>
 </p>
 
+This codebase is under [CC BY-NC 4.0 license](https://creativecommons.org/licenses/by-nc/4.0/deed.en), with inherited license in [Legged Gym](training/legged_gym) and [rsl rl](training/rsl_rl) from *ETH Zurich, Nikita Rudin* and *NVIDIA CORPORATION & AFFILIATES*. You may not use the material for commercial purposes, e.g., to make demos to advertise your commercial products.
+
+## Training in Simulation
+
+### Pipeline to install and train ABS
+
+**Note**: Before running our code, it's highly recommended to first play with [RSL's Legged Gym version](https://github.com/leggedrobotics/legged_gym) to get a basic understanding of the Isaac-LeggedGym-RslRL framework.
+   <!-- <br/><br/> -->
+
+1. create environment, install torch
+
+   ```text
+   conda create -n xxx python=3.8  # or use virtual environment/docker
+   
+   pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116  
+   # used version during this work: torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2
+   # for older cuda ver:
+   pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
+   ```
+
+
+
+2. install isaac gym preview 4 release https://developer.nvidia.com/isaac-gym
+
+   unzip files to a folder, then install with pip:
+
+   `cd isaacgym/python && pip install -e .`
+
+   check it is correctly installed by playing: 
+
+   ```cmd
+   cd examples && python 1080_balls_of_solitude.py
+   ```
+
+
+
+3. clone this codebase and install our `rsl_rl` in the training folder
+
+   ```cmd
+   pip install -e rsl_rl
+   ```
+
+
+
+4. install our `legged_gym`
+
+   ```cmd
+   pip install -e legged_gym
+   ```
+
+   Ensure you have installed the following packages:
+    + pip install numpy==1.20 (must < 1.24)
+    + pip install tensorboard
+    + pip install setuptools==59.5.0
+
+5. try training.
+
+   can use "--headless" to disable gui, press "v" to pause/resume gui play.
+
+   for go1, in `legged_gym/legged_gym`,
+    ```text
+   # agile policy
+   python scripts/train.py --task=go1_pos_rough --max_iterations=4000 
+   
+   # agile policy, lagrangian ver
+   python scripts/train.py --task=go1_pos_rough_ppo_lagrangian --max_iterations=4000
+   
+   # recovery policy
+   python scripts/train.py --task=go1_rec_rough --max_iterations=1000
+   ```
+
+
+6. play the trained policy
+
+   ```cmd
+   python scripts/play.py --task=go1_pos_rough
+   python scripts/play.py --task=go1_rec_rough
+   ```
+
+7. Use the testbed, and train/test Reach-Avoid network:
+
+   ```text
+   # try testbed
+   python scripts/testbed.py --task=go1_pos_rough [--load_run=xxx] --num_envs=1
+
+   # train RA (be patient it will take time to converge) 
+   # make sure you have at least exported one policy by play.py so the exported folder exists
+   python scripts/testbed.py --task=go1_pos_rough --num_envs=1000 --headless --trainRA
+   
+   # test RA (only when you have trained one RA)
+   python scripts/testbed.py --task=go1_pos_rough --num_envs=1 --testRA
+   
+   # evaluate
+   python scripts/testbed.py --task=go1_pos_rough --num_envs=1000 --headless [--load_run=xxx] [--testRA]
+   ```
+
+8. Sample Dataset for ray-prediction network training
+   ```cmd
+   python scripts/camrec.py --task=go1_pos_rough --num_envs=3
+   ```
+   + Tips 1: You can edit the `shift` value in Line 93 and the `log_root` in Line 87 to collect different dataset files in parallel (so you can merge them by simply moving the files), and manually change the obstacles in `env_cfg.asset.object_files` in Line 63.
+   + Tips 2: After collecting the data, there's a template code in [`train_depth_resnet.py`](training/legged_gym/legged_gym/scripts/train_depth_resnet.py) to train the ray-prediction network, but using what you like for training CV models is highly encouraged!
+   + Tips 3: You may change camera configs of resolution, position, FOV, and depth range in the [config file](training/legged_gym/legged_gym/envs/go1/go1_pos_config.py) Line 151.
+
 
 ## Hardware Deployment 
 ### System overview
@@ -59,8 +163,40 @@ Official Implementation for [Agile But Safe: Learning Collision-Free High-Speed
 
 
 
-## TODO:
-
-- [ ] Upload Sim Training Code
-- [x] Upload Deployment Code
+## Troubleshooting:
+### Contact
++ Deployment and Ray-Prediction: Tairan He, [email protected]
++ Policy Learning in Sim: Chong Zhang, [email protected]
++ PPO-Lagrangian Implementation: Wenli Xiao, [email protected]
+
+### Issues
+You can create an issue if you meet any bugs except:
++ If you cannot run the [vanilla RSL's Legged Gym](https://github.com/leggedrobotics/legged_gym), it is expected that you first go to the vanilla Legged Gym repo for help.
++ There can be CUDA-related errors when there are too many parallel environments on certain PC+GPU+driver combination: we cannot solve this, you can try to reduce num_envs
++ Our codebase is only for our system showcased in the 
+
+## Credit
+If our work does help you, please consider citing us and the following works:
+```bibtex
+@inproceedings{he2024agile,
+  author    = {He, Tairan and Zhang, Chong and Xiao, Wenli and He, Guanqi and Liu, Changliu and Shi, Guanya},
+  title     = {Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion},
+  booktitle = {arXiv},
+  year      = {2024},
+}
+```
+We used codes in [Legged Gym](training/legged_gym) and [rsl rl](training/rsl_rl), based on the paper:
+  + Rudin, Nikita, et al. "Learning to walk in minutes using massively parallel deep reinforcement learning." CORL 2022.
+
+Previsou works that heavily inspired the policy training designs:
+ + Rudin, Nikita, et al. "Advanced skills by learning locomotion and local navigation end-to-end." 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022.
+ + Zhang, Chong, et al. "Learning Agile Locomotion on Risky Terrains." arXiv preprint arXiv:2311.10484 (2023).
+ + Zhang, Chong, et al. "Resilient Legged Local Navigation: Learning to Traverse with Compromised Perception End-to-End." ICRA 2024.
+
+Previsou works that heavily inspired the RA value design:
++ Hsu, Kai-Chieh, et al. "Safety and liveness guarantees through reach-avoid reinforcement learning." RSS 2021.
+
+Previsou works that heavily inspired the perception design:
++ Hoeller, David, et al. "Anymal parkour: Learning agile navigation for quadrupedal robots." Science Robotics 9.88 (2024): eadi7566.
++ Acero, F., K. Yuan, and Z. Li. "Learning Perceptual Locomotion on Uneven Terrains using Sparse Visual Observations." IEEE Robotics and Automation Letters 7.4 (2022): 8611-8618.
 

training/.gitignore

@@ -0,0 +1,3 @@
+logs/
+sync_mac_to_server.sh
+sync_server_to_mac.sh

training/legged_gym/.gitattributes

@@ -0,0 +1,4 @@
+*.dae filter=lfs diff=lfs merge=lfs -text
+*.obj filter=lfs diff=lfs merge=lfs -text
+*.obj text !filter !merge !diff
+*.dae text !filter !merge !diff

training/legged_gym/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,30 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: bug
+assignees: nikitardn
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Execute `./path/script arg1 arg2`
+1. Then ... 
+1. See error: ...
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**System (please complete the following information):**
+ - Commit: [e.g. 8f3b9ca]
+ - OS: [e.g. Ubuntu 20.04]
+ - GPU: [e.g. RTX 2060 Super]
+ - CUDA: [e.g. 11.4]
+ - GPU Driver: [e.g. 470.82.01]
+
+**Additional context**
+Add any other context about the problem here.

training/legged_gym/.gitignore

@@ -0,0 +1,62 @@
+# These are some examples of commonly ignored file patterns.
+# You should customize this list as applicable to your project.
+# Learn more about .gitignore:
+#     https://www.atlassian.com/git/tutorials/saving-changes/gitignore
+
+# Node artifact files
+node_modules/
+dist/
+
+# Compiled Java class files
+*.class
+
+# Compiled Python bytecode
+*.py[cod]
+
+# Log files
+*.log
+
+# Package files
+*.jar
+
+# Maven
+target/
+dist/
+
+# JetBrains IDE
+.idea/
+
+# Unit test reports
+TEST*.xml
+
+# Generated by MacOS
+.DS_Store
+
+# Generated by Windows
+Thumbs.db
+
+# Applications
+*.app
+*.exe
+*.war
+
+# Large media files
+*.mp4
+*.tiff
+*.avi
+*.flv
+*.mov
+*.wmv
+
+# VS Code
+.vscode
+# logs
+logs
+runs
+
+# other
+*.egg-info
+__pycache__
+
+depth_data
+depth_logs

training/legged_gym/LICENSE

@@ -0,0 +1,31 @@
+Copyright (c) 2021, ETH Zurich, Nikita Rudin
+Copyright (c) 2021, NVIDIA CORPORATION & AFFILIATES
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice,
+   this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its contributors
+   may be used to endorse or promote products derived from this software without
+   specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+See licenses/assets for license information for assets included in this repository.
+See licenses/dependencies for license information of dependencies of this package.

training/legged_gym/README.md

@@ -0,0 +1,93 @@
+# Isaac Gym Environments for Legged Robots #
+This repository provides the environment used to train ANYmal (and other robots) to walk on rough terrain using NVIDIA's Isaac Gym.
+It includes all components needed for sim-to-real transfer: actuator network, friction & mass randomization, noisy observations and random pushes during training.  
+**Maintainer**: Nikita Rudin  
+**Affiliation**: Robotic Systems Lab, ETH Zurich  
+**Contact**: [email protected]  
+
+### Useful Links ###
+Project website: https://leggedrobotics.github.io/legged_gym/
+Paper: https://arxiv.org/abs/2109.11978
+
+### Installation ###
+1. Create a new python virtual env with python 3.6, 3.7 or 3.8 (3.8 recommended)
+2. Install pytorch 1.10 with cuda-11.3:
+    - `pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html`
+3. Install Isaac Gym
+   - Download and install Isaac Gym Preview 3 (Preview 2 will not work!) from https://developer.nvidia.com/isaac-gym
+   - `cd isaacgym/python && pip install -e .`
+   - Try running an example `cd examples && python 1080_balls_of_solitude.py`
+   - For troubleshooting check docs `isaacgym/docs/index.html`)
+4. Install rsl_rl (PPO implementation)
+   - Clone https://github.com/leggedrobotics/rsl_rl
+   -  `cd rsl_rl && pip install -e .` 
+5. Install legged_gym
+    - Clone this repository
+   - `cd legged_gym && pip install -e .`
+
+### CODE STRUCTURE ###
+1. Each environment is defined by an env file (`legged_robot.py`) and a config file (`legged_robot_config.py`). The config file contains two classes: one conatianing all the environment parameters (`LeggedRobotCfg`) and one for the training parameters (`LeggedRobotCfgPPo`).  
+2. Both env and config classes use inheritance.  
+3. Each non-zero reward scale specified in `cfg` will add a function with a corresponding name to the list of elements which will be summed to get the total reward.  
+4. Tasks must be registered using `task_registry.register(name, EnvClass, EnvConfig, TrainConfig)`. This is done in `envs/__init__.py`, but can also be done from outside of this repository.  
+
+### Usage ###
+1. Train:  
+  ```python issacgym_anymal/scripts/train.py --task=anymal_c_flat```
+    -  To run on CPU add following arguments: `--sim_device=cpu`, `--rl_device=cpu` (sim on CPU and rl on GPU is possible).
+    -  To run headless (no rendering) add `--headless`.
+    - **Important**: To improve performance, once the training starts press `v` to stop the rendering. You can then enable it later to check the progress.
+    - The trained policy is saved in `issacgym_anymal/logs/<experiment_name>/<date_time>_<run_name>/model_<iteration>.pt`. Where `<experiment_name>` and `<run_name>` are defined in the train config.
+    -  The following command line arguments override the values set in the config files:
+     - --task TASK: Task name.
+     - --resume:   Resume training from a checkpoint
+     - --experiment_name EXPERIMENT_NAME: Name of the experiment to run or load.
+     - --run_name RUN_NAME:  Name of the run.
+     - --load_run LOAD_RUN:   Name of the run to load when resume=True. If -1: will load the last run.
+     - --checkpoint CHECKPOINT:  Saved model checkpoint number. If -1: will load the last checkpoint.
+     - --num_envs NUM_ENVS:  Number of environments to create.
+     - --seed SEED:  Random seed.
+     - --max_iterations MAX_ITERATIONS:  Maximum number of training iterations.
+2. Play a trained policy:  
+```python issacgym_anymal/scripts/play.py --task=anymal_c_flat```
+    - By default the loaded policy is the last model of the last run of the experiment folder.
+    - Other runs/model iteration can be selected by setting `load_run` and `checkpoint` in the train config.
+
+### Adding a new environment ###
+The base environment `legged_robot` implements a rough terrain locomotion task. The corresponding cfg does not specify a robot asset (URDF/ MJCF) and no reward scales. 
+
+1. Add a new folder to `envs/` with `'<your_env>_config.py`, which inherit from an existing environment cfgs  
+2. If adding a new robot:
+    - Add the corresponding assets to `resourses/`.
+    - In `cfg` set the asset path, define body names, default_joint_positions and PD gains. Specify the desired `train_cfg` and the name of the environment (python class).
+    - In `train_cfg` set `experiment_name` and `run_name`
+3. (If needed) implement your environment in <your_env>.py, inherit from an existing environment, overwrite the desired functions and/or add your reward functions.
+4. Register your env in `isaacgym_anymal/envs/__init__.py`.
+5. Modify/Tune other parameters in your `cfg`, `cfg_train` as needed. To remove a reward set its scale to zero. Do not modify parameters of other envs!
+
+
+### Troubleshooting ###
+1. If you get the following error: `ImportError: libpython3.8m.so.1.0: cannot open shared object file: No such file or directory`, do: `sudo apt install libpython3.8`
+
+### Known Issues ###
+1. The contact forces reported by `net_contact_force_tensor` are unreliable when simulating on GPU with a triangle mesh terrain. A workaround is to use force sensors, but the force are propagated through the sensors of consecutive bodies resulting in an undesireable behaviour. However, for a legged robot it is possible to add sensors to the feet/end effector only and get the expected results. When using the force sensors make sure to exclude gravity from trhe reported forces with `sensor_options.enable_forward_dynamics_forces`. Example:
+```
+    sensor_pose = gymapi.Transform()
+    for name in feet_names:
+        sensor_options = gymapi.ForceSensorProperties()
+        sensor_options.enable_forward_dynamics_forces = False # for example gravity
+        sensor_options.enable_constraint_solver_forces = True # for example contacts
+        sensor_options.use_world_frame = True # report forces in world frame (easier to get vertical components)
+        index = self.gym.find_asset_rigid_body_index(robot_asset, name)
+        self.gym.create_asset_force_sensor(robot_asset, index, sensor_pose, sensor_options)
+    (...)
+
+    sensor_tensor = self.gym.acquire_force_sensor_tensor(self.sim)
+    self.gym.refresh_force_sensor_tensor(self.sim)
+    force_sensor_readings = gymtorch.wrap_tensor(sensor_tensor)
+    self.sensor_forces = force_sensor_readings.view(self.num_envs, 4, 6)[..., :3]
+    (...)
+
+    self.gym.refresh_force_sensor_tensor(self.sim)
+    contact = self.sensor_forces[:, :, 2] > 1.
+```