RSL RL

Fast and simple implementation of RL algorithms, designed to run fully on GPU. This code is an evolution of rl-pytorch provided with NVIDIA's Isaac GYM.

Environment repositories using the framework:

• Isaac Lab (built on top of NVIDIA Isaac Sim): https://github.com/isaac-sim/IsaacLab
• Legged-Gym (built on top of NVIDIA Isaac Gym): https://leggedrobotics.github.io/legged_gym/

The main branch supports PPO with additional features from our research. These include:

• Random Network Distillation (RND) - Encourages exploration by adding a curiosity driven intrinsic reward.
• Symmetry-based Augmentation - Makes the learned behaviors more symmetrical.

We welcome contributions from the community. Please check our contribution guidelines for more information.

Maintainer: Mayank Mittal and Clemens Schwarke
Affiliation: Robotic Systems Lab, ETH Zurich & NVIDIA
Contact: [email protected]

Note: The algorithms branch supports additional algorithms (SAC, DDPG, DSAC, and more). However, it isn't currently actively maintained.

Setup

The package can be installed via PyPI with:

pip install rsl-rl-lib

or by cloning this repository and installing it with:

git clone https://github.com/leggedrobotics/rsl_rl
cd rsl_rl
pip install -e .

The package supports the following logging frameworks which can be configured through logger:

• Tensorboard: https://www.tensorflow.org/tensorboard/
• Weights & Biases: https://wandb.ai/site
• Neptune: https://docs.neptune.ai/

For a demo configuration of PPO, please check the dummy_config.yaml file.

Contribution Guidelines

For documentation, we adopt the Google Style Guide for docstrings. Please make sure that your code is well-documented and follows the guidelines.

We use the following tools for maintaining code quality:

• pre-commit: Runs a list of formatters and linters over the codebase.
• black: The uncompromising code formatter.
• flake8: A wrapper around PyFlakes, pycodestyle, and McCabe complexity checker.

Please check here for instructions to set these up. To run over the entire repository, please execute the following command in the terminal:

# for installation (only once)
pre-commit install
# for running
pre-commit run --all-files

Citing

We are working on writing a white paper for this library. Until then, please cite the following work if you use this library for your research:

@InProceedings{rudin2022learning,
  title = 	 {Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning},
  author =       {Rudin, Nikita and Hoeller, David and Reist, Philipp and Hutter, Marco},
  booktitle = 	 {Proceedings of the 5th Conference on Robot Learning},
  pages = 	 {91--100},
  year = 	 {2022},
  volume = 	 {164},
  series = 	 {Proceedings of Machine Learning Research},
  publisher =    {PMLR},
  url = 	 {https://proceedings.mlr.press/v164/rudin22a.html},
}

If you use the library with curiosity-driven exploration (random network distillation), please cite:

@InProceedings{schwarke2023curiosity,
  title = 	 {Curiosity-Driven Learning of Joint Locomotion and Manipulation Tasks},
  author =       {Schwarke, Clemens and Klemm, Victor and Boon, Matthijs van der and Bjelonic, Marko and Hutter, Marco},
  booktitle = 	 {Proceedings of The 7th Conference on Robot Learning},
  pages = 	 {2594--2610},
  year = 	 {2023},
  volume = 	 {229},
  series = 	 {Proceedings of Machine Learning Research},
  publisher =    {PMLR},
  url = 	 {https://proceedings.mlr.press/v229/schwarke23a.html},
}

If you use the library with symmetry augmentation, please cite:

@InProceedings{mittal2024symmetry,
  author={Mittal, Mayank and Rudin, Nikita and Klemm, Victor and Allshire, Arthur and Hutter, Marco},
  booktitle={2024 IEEE International Conference on Robotics and Automation (ICRA)},
  title={Symmetry Considerations for Learning Task Symmetric Robot Policies},
  year={2024},
  pages={7433-7439},
  doi={10.1109/ICRA57147.2024.10611493}
}