This work is an extension of Coach-RL. The idea is to study how can we improve the algorithm Direct Future Prediction by adding depth information and time information.
You can have a look to the report that I have made here.
In short, adding information helped the agent to learn a better behavior in a small amount of training steps (~ 200 000 steps). Indeed, the agent with only 3 frames as input (on the top left) is lost and does not know where to go. On the contrary, the other agents understand that they have to search for medkits.
 |
 |
| Input: 3 frames (like in the original paper) | Input: 3 frames + depth from the game engine |
 |
 |
| Input: 3 frames + self predicted depth (by pix2pix) | Input: 10 frames |
You can quickly get access to the colab notebook I used to train pix2pix here:
Coach is a python reinforcement learning framework containing implementation of many state-of-the-art algorithms.
It exposes a set of easy-to-use APIs for experimenting with new RL algorithms, and allows simple integration of new environments to solve. Basic RL components (algorithms, environments, neural network architectures, exploration policies, ...) are well decoupled, so that extending and reusing existing components is fairly painless.
Training an agent to solve an environment is as easy as running:
coach -p CartPole_DQN -r
- Release 0.8.0 (initial release)
- Release 0.9.0
- Release 0.10.0
- Release 0.11.0
- Release 0.12.0
- Release 1.0.0 (current release)
- Benchmarks
- Installation
- Getting Started
- Supported Environments
- Supported Algorithms
- Citation
- Contact
- Disclaimer
One of the main challenges when building a research project, or a solution based on a published algorithm, is getting a concrete and reliable baseline that reproduces the algorithm's results, as reported by its authors. To address this problem, we are releasing a set of benchmarks that shows Coach reliably reproduces many state of the art algorithm results.
Note: Coach has only been tested on Ubuntu 16.04 LTS, and with Python 3.5.
For some information on installing on Ubuntu 17.10 with Python 3.6.3, please refer to the following issue: IntelLabs#54
In order to install coach, there are a few prerequisites required. This will setup all the basics needed to get the user going with running Coach on top of OpenAI Gym environments:
# General
sudo -E apt-get install python3-pip cmake zlib1g-dev python3-tk python-opencv -y
# Boost libraries
sudo -E apt-get install libboost-all-dev -y
# Scipy requirements
sudo -E apt-get install libblas-dev liblapack-dev libatlas-base-dev gfortran -y
# PyGame
sudo -E apt-get install libsdl-dev libsdl-image1.2-dev libsdl-mixer1.2-dev libsdl-ttf2.0-dev
libsmpeg-dev libportmidi-dev libavformat-dev libswscale-dev -y
# Dashboard
sudo -E apt-get install dpkg-dev build-essential python3.5-dev libjpeg-dev libtiff-dev libsdl1.2-dev libnotify-dev
freeglut3 freeglut3-dev libsm-dev libgtk2.0-dev libgtk-3-dev libwebkitgtk-dev libgtk-3-dev libwebkitgtk-3.0-dev
libgstreamer-plugins-base1.0-dev -y
# Gym
sudo -E apt-get install libav-tools libsdl2-dev swig cmake -y
We recommend installing coach in a virtualenv:
sudo -E pip3 install virtualenv
virtualenv -p python3 coach_env
. coach_env/bin/activate
Finally, install coach using pip:
pip3 install rl_coach
Or alternatively, for a development environment, install coach from the cloned repository:
cd coach
pip3 install -e .
If a GPU is present, Coach's pip package will install tensorflow-gpu, by default. If a GPU is not present, an Intel-Optimized TensorFlow, will be installed.
In addition to OpenAI Gym, several other environments were tested and are supported. Please follow the instructions in the Supported Environments section below in order to install more environments.
To allow reproducing results in Coach, we defined a mechanism called preset.
There are several available presets under the presets directory.
To list all the available presets use the -l flag.
To run a preset, use:
coach -r -p <preset_name>For example:
-
CartPole environment using Policy Gradients (PG):
coach -r -p CartPole_PG
-
Basic level of Doom using Dueling network and Double DQN (DDQN) algorithm:
coach -r -p Doom_Basic_Dueling_DDQN
Some presets apply to a group of environment levels, like the entire Atari or Mujoco suites for example.
To use these presets, the requeseted level should be defined using the -lvl flag.
For example:
-
Pong using the Neural Episodic Control (NEC) algorithm:
coach -r -p Atari_NEC -lvl pong
There are several types of agents that can benefit from running them in a distributed fashion with multiple workers in parallel. Each worker interacts with its own copy of the environment but updates a shared network, which improves the data collection speed and the stability of the learning process.
To specify the number of workers to run, use the -n flag.
For example:
-
Breakout using Asynchronous Advantage Actor-Critic (A3C) with 8 workers:
coach -r -p Atari_A3C -lvl breakout -n 8
It is easy to create new presets for different levels or environments by following the same pattern as in presets.py
More usage examples can be found here.
Training an agent to solve an environment can be tricky, at times.
In order to debug the training process, Coach outputs several signals, per trained algorithm, in order to track algorithmic performance.
While Coach trains an agent, a csv file containing the relevant training signals will be saved to the 'experiments' directory. Coach's dashboard can then be used to dynamically visualize the training signals, and track algorithmic behavior.
To use it, run:
dashboard
As of release 0.11.0, Coach supports horizontal scaling for training RL agents on multiple nodes. In release 0.11.0 this was tested on the ClippedPPO and DQN agents. For usage instructions please refer to the documentation here.
Training and evaluating an agent from a dataset of experience, where no simulator is available, is supported in Coach. There are example presets and a tutorial.
-
OpenAI Gym:
Installed by default by Coach's installer (see note on MuJoCo version below).
-
ViZDoom:
Follow the instructions described in the ViZDoom repository -
https://github.com/mwydmuch/ViZDoom
Additionally, Coach assumes that the environment variable VIZDOOM_ROOT points to the ViZDoom installation directory.
-
Roboschool:
Follow the instructions described in the roboschool repository -
-
GymExtensions:
Follow the instructions described in the GymExtensions repository -
https://github.com/Breakend/gym-extensions
Additionally, add the installation directory to the PYTHONPATH environment variable.
-
PyBullet:
Follow the instructions described in the Quick Start Guide (basically just - 'pip install pybullet')
-
CARLA:
Download release 0.8.4 from the CARLA repository -
https://github.com/carla-simulator/carla/releases
Install the python client and dependencies from the release tarball:
pip3 install -r PythonClient/requirements.txt pip3 install PythonClientCreate a new CARLA_ROOT environment variable pointing to CARLA's installation directory.
A simple CARLA settings file (
CarlaSettings.ini) is supplied with Coach, and is located in theenvironmentsdirectory. -
Starcraft:
Follow the instructions described in the PySC2 repository -
-
DeepMind Control Suite:
Follow the instructions described in the DeepMind Control Suite repository -
-
Note: To use Robosuite-based environments, please install Coach from the latest cloned repository. It is not yet available as part of the
rl_coachpackage on PyPI.Follow the instructions described in the robosuite documentation (see note on MuJoCo version below).
OpenAI Gym supports MuJoCo only up to version 1.5 (and corresponding mujoco-py version 1.50.x.x). The Robosuite simulation framework, however, requires MuJoCo version 2.0 (and corresponding mujoco-py version 2.0.2.9, as of robosuite version 1.2). Therefore, if you wish to run both Gym-based MuJoCo environments and Robosuite environments, it's recommended to have a separate virtual environment for each.
Please note that all Gym-Based MuJoCo presets in Coach (rl_coach/presets/Mujoco_*.py) have been validated only with MuJoCo 1.5 (including the reported benchmark results).
- Deep Q Network (DQN) (code)
- Double Deep Q Network (DDQN) (code)
- Dueling Q Network
- Mixed Monte Carlo (MMC) (code)
- Persistent Advantage Learning (PAL) (code)
- Categorical Deep Q Network (C51) (code)
- Quantile Regression Deep Q Network (QR-DQN) (code)
- N-Step Q Learning | Multi Worker Single Node (code)
- Neural Episodic Control (NEC) (code)
- Normalized Advantage Functions (NAF) | Multi Worker Single Node (code)
- Rainbow (code)
- Policy Gradients (PG) | Multi Worker Single Node (code)
- Asynchronous Advantage Actor-Critic (A3C) | Multi Worker Single Node (code)
- Deep Deterministic Policy Gradients (DDPG) | Multi Worker Single Node (code)
- Proximal Policy Optimization (PPO) (code)
- Clipped Proximal Policy Optimization (CPPO) | Multi Worker Single Node (code)
- Generalized Advantage Estimation (GAE) (code)
- Sample Efficient Actor-Critic with Experience Replay (ACER) | Multi Worker Single Node (code)
- Soft Actor-Critic (SAC) (code)
- Twin Delayed Deep Deterministic Policy Gradient (TD3) (code)
- Direct Future Prediction (DFP) | Multi Worker Single Node (code)
- Behavioral Cloning (BC) (code)
- Conditional Imitation Learning (code)
- E-Greedy (code)
- Boltzmann (code)
- Ornstein–Uhlenbeck process (code)
- Normal Noise (code)
- Truncated Normal Noise (code)
- Bootstrapped Deep Q Network (code)
- UCB Exploration via Q-Ensembles (UCB) (code)
- Noisy Networks for Exploration (code)
If you used Coach for your work, please use the following citation:
@misc{caspi_itai_2017_1134899,
author = {Caspi, Itai and
Leibovich, Gal and
Novik, Gal and
Endrawis, Shadi},
title = {Reinforcement Learning Coach},
month = dec,
year = 2017,
doi = {10.5281/zenodo.1134899},
url = {https://doi.org/10.5281/zenodo.1134899}
}
We'd be happy to get any questions or contributions through GitHub issues and PRs.
Please make sure to take a look here before filing an issue or proposing a PR.
The Coach development team can also be contacted over email
Coach is released as a reference code for research purposes. It is not an official Intel product, and the level of quality and support may not be as expected from an official product. Additional algorithms and environments are planned to be added to the framework. Feedback and contributions from the open source and RL research communities are more than welcome.
Connect to the virtual machine and copy the .ssh folder to ubuntu user. Connect from local host by ssh with ubuntu user. Clone repository this at /home/theovincent/MVA/ObjectRecognition/LearningToAct/ Open the devcontainer from vs-code. Create a virtual environment and install coach has an editable package. export VIZDOOM_ROOT=/home/developer/LearningToAct/ViZDoom
# Build a VM with compute engine having debian bullseye (same as Dockerfile), http(s) connections and remove vTPM security but allow your ssh public key.
# Set .ssh/config locally so that we launch at IP with theo.vincent
ssh DirectFuturePredictionTrain
# Download docker
sudo apt-get update
sudo apt-get install docker.io htop -y
# Allow non-root user to use it
sudo chmod 666 /var/run/docker.sock
sudo mkdir -p /home/theovincent/MVA/ObjectRecognition/LearningToAct/
sudo chown -R theo.vincent:theo.vincent ../theovincent/
cd /home/theovincent/MVA/ObjectRecognition/LearningToAct/
git clone [email protected]:theovincent/coach.git
cd coach
git checkout # Lastest branch
cd ..
git clone [email protected]:mwydmuch/ViZDoom.git
# Open the devcontainer from vs-code to build the dev-container in the folder coach
python -m venv env_container
source env_container/bin/activate
pip install --upgrade pip
pip install -e .
echo "export VIZDOOM_ROOT=/home/developer/LearningToAct/ViZDoom" >> ~/.bashrc
cd ..
# For GPU usage only
curl https://raw.githubusercontent.com/GoogleCloudPlatform/compute-gpu-installation/main/linux/install_gpu_driver.py --output install_gpu_driver.py
# Remove linux-headers-{kernel_version} line 243 from the installed packages
sudo python3 install_gpu_driver.py
sudo apt install cuda
```pip install -e .