tips_and_tricks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Useful tips and tricks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook you have a few useful tricks, that can impact your performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:\n",
      "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
      "For more information, please see:\n",
      "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
      "  * https://github.com/tensorflow/addons\n",
      "  * https://github.com/tensorflow/io (for I/O related ops)\n",
      "If you depend on functionality not listed there, please file an issue.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Create virtual display to render on remote machine\n",
    "from pyvirtualdisplay import Display\n",
    "display = Display(visible=0, size=(1, 1))\n",
    "display.start()\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "from IPython import display\n",
    "import gym\n",
    "import numpy as np\n",
    "from pathlib import Path\n",
    "\n",
    "from stable_baselines.common.vec_env import DummyVecEnv, SubprocVecEnv\n",
    "from stable_baselines.common.evaluation import evaluate_policy\n",
    "from stable_baselines.common import set_global_seeds, make_vec_env\n",
    "\n",
    "from utils import evaluate_model_vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stable_baselines.common.policies import MlpPolicy # our policy (neural network that will perform actions)\n",
    "from stable_baselines import PPO2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensorboard\n",
    "We also can use tensorboard when we train stable-baselines, then we will be able to explore more details about out training process. \\\n",
    "To use tensorboard you need:\n",
    "1. Pass tensorboard_log argument to you model which will indicate path where you want to have logs written;\n",
    "2. Run tensorboard: \n",
    "    - at your VM **you have active tensorboard** at port 6006 that is looking for logs in two folder `your_work` and `tensorboard`. So you just need to open the same ip adress with 6006 port to get to the tensorboard.\n",
    "\n",
    "Here we set tensorboard logs directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensorboard_dir = \"/root/tensorboard\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.5/dist-packages/gym/logger.py:30: UserWarning: \u001b[33mWARN: Box bound precision lowered by casting to float32\u001b[0m\n",
      "  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('CartPole-v0')\n",
    "env = DummyVecEnv([lambda: env]) # Should be used in 1 core case"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Q9EpDF-ntg81"
   },
   "source": [
    "Train our model. Note that we use **magic command ```%%time```** to track time for this cell. \\\n",
    "Directory to the tensorboard is just passed as a `tensorboard_dir` argument to our model.  \\"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "PQvrSiYNn3Vb",
    "outputId": "03bb8e0e-aa44-48f2-a448-f9579a6aa204"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/stable_baselines/common/tf_util.py:58: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/stable_baselines/common/tf_util.py:67: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/stable_baselines/common/policies.py:115: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/stable_baselines/common/input.py:25: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/stable_baselines/common/policies.py:560: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.flatten instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow_core/python/layers/core.py:332: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `layer.__call__` method instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/stable_baselines/a2c/utils.py:156: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/stable_baselines/common/distributions.py:326: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/stable_baselines/common/distributions.py:327: The name tf.log is deprecated. Please use tf.math.log instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/stable_baselines/ppo2/ppo2.py:194: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/stable_baselines/ppo2/ppo2.py:202: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/stable_baselines/ppo2/ppo2.py:210: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/stable_baselines/ppo2/ppo2.py:244: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/stable_baselines/ppo2/ppo2.py:246: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/stable_baselines/common/base_class.py:1082: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/stable_baselines/a2c/utils.py:581: The name tf.Summary is deprecated. Please use tf.compat.v1.Summary instead.\n",
      "\n",
      "-------------------------------------\n",
      "| approxkl           | 0.0068953326 |\n",
      "| clipfrac           | 0.109375     |\n",
      "| explained_variance | -0.0591      |\n",
      "| fps                | 264          |\n",
      "| n_updates          | 1            |\n",
      "| policy_entropy     | 0.6864212    |\n",
      "| policy_loss        | -0.018464664 |\n",
      "| serial_timesteps   | 128          |\n",
      "| time_elapsed       | 4.05e-06     |\n",
      "| total_timesteps    | 128          |\n",
      "| value_loss         | 34.298294    |\n",
      "-------------------------------------\n",
      "--------------------------------------\n",
      "| approxkl           | 0.0032370728  |\n",
      "| clipfrac           | 0.041015625   |\n",
      "| explained_variance | 0.0894        |\n",
      "| fps                | 882           |\n",
      "| n_updates          | 100           |\n",
      "| policy_entropy     | 0.53069425    |\n",
      "| policy_loss        | -0.0012431142 |\n",
      "| serial_timesteps   | 12800         |\n",
      "| time_elapsed       | 14.9          |\n",
      "| total_timesteps    | 12800         |\n",
      "| value_loss         | 7.6743298     |\n",
      "--------------------------------------\n",
      "CPU times: user 48.6 s, sys: 9.02 s, total: 57.6 s\n",
      "Wall time: 30.1 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<stable_baselines.ppo2.ppo2.PPO2 at 0x7f4b7b6984a8>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "n_steps = 128\n",
    "model = PPO2(\n",
    "    MlpPolicy, \n",
    "    env, \n",
    "    verbose=1, \n",
    "    n_steps=n_steps, \n",
    "    learning_rate=0.003, \n",
    "    ent_coef=0.0,\n",
    "    tensorboard_log=tensorboard_dir # path to store tensorboard logs\n",
    ")\n",
    "model.learn(total_timesteps=25000, log_interval=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parallelized environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use baselines documentation as a cookbook and run our traing in parallel.  \n",
    "https://stable-baselines.readthedocs.io/en/master/guide/examples.html#multiprocessing-unleashing-the-power-of-vectorized-environments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do quick performance comparision. \\\n",
    "First, we will run single-core training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.5/dist-packages/gym/logger.py:30: UserWarning: \u001b[33mWARN: Box bound precision lowered by casting to float32\u001b[0m\n",
      "  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('CartPole-v0')\n",
    "env = DummyVecEnv([lambda: env])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------\n",
      "| approxkl           | 0.0012090533   |\n",
      "| clipfrac           | 0.001953125    |\n",
      "| explained_variance | -0.00806       |\n",
      "| fps                | 310            |\n",
      "| n_updates          | 1              |\n",
      "| policy_entropy     | 0.6919473      |\n",
      "| policy_loss        | -0.00010843319 |\n",
      "| serial_timesteps   | 128            |\n",
      "| time_elapsed       | 3.34e-06       |\n",
      "| total_timesteps    | 128            |\n",
      "| value_loss         | 38.902607      |\n",
      "---------------------------------------\n",
      "CPU times: user 18.8 s, sys: 3.77 s, total: 22.6 s\n",
      "Wall time: 11.7 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<stable_baselines.ppo2.ppo2.PPO2 at 0x7f4bfb84e2e8>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "n_steps = 128\n",
    "model = PPO2(\n",
    "    MlpPolicy, \n",
    "    env, \n",
    "    verbose=1, \n",
    "    n_steps=n_steps, \n",
    "    learning_rate=0.003, \n",
    "    ent_coef=0.0,\n",
    ")\n",
    "model.learn(total_timesteps=10000, log_interval=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's run parallelized training. \\\n",
    "The only thing we need is to create vectorized environment, each of them on separate threads."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.5/dist-packages/gym/logger.py:30: UserWarning: \u001b[33mWARN: Box bound precision lowered by casting to float32\u001b[0m\n",
      "  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n"
     ]
    }
   ],
   "source": [
    "env_name = \"CartPole-v0\"\n",
    "num_cpu = 4\n",
    "vec_env = make_vec_env(env_name, n_envs=num_cpu, seed=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------\n",
      "| approxkl           | 0.0006608284  |\n",
      "| clipfrac           | 0.0           |\n",
      "| ep_len_mean        | 16.5          |\n",
      "| ep_reward_mean     | 16.5          |\n",
      "| explained_variance | -0.00216      |\n",
      "| fps                | 381           |\n",
      "| n_updates          | 1             |\n",
      "| policy_entropy     | 0.6925019     |\n",
      "| policy_loss        | -0.0010020696 |\n",
      "| serial_timesteps   | 32            |\n",
      "| time_elapsed       | 3.1e-06       |\n",
      "| total_timesteps    | 128           |\n",
      "| value_loss         | 24.10811      |\n",
      "--------------------------------------\n",
      "CPU times: user 11.6 s, sys: 2.81 s, total: 14.4 s\n",
      "Wall time: 6.28 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<stable_baselines.ppo2.ppo2.PPO2 at 0x7f4b7b748d68>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "model = PPO2(\n",
    "    MlpPolicy, \n",
    "    vec_env, \n",
    "    verbose=1, \n",
    "    n_steps=n_steps // num_cpu, # Scaling number of steps\n",
    "    learning_rate=0.003, \n",
    "    ent_coef=0.0,\n",
    ")\n",
    "model.learn(total_timesteps=10000, log_interval=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, parallel execution of 4 environments speeds up training by 2 times."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Faster evaluation for parallelized environments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In `utils.py` in this folder (which you can also find on our [github](https://github.com/nextgrid/notebooks)) you can find `evaluate_model_vec` function.  \\\n",
    "This function could be very usefull if you use parallelized environemnts because it gives you parallelized evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do simple comparision - run both evaluation functions on the same model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.5/dist-packages/gym/logger.py:30: UserWarning: \u001b[33mWARN: Box bound precision lowered by casting to float32\u001b[0m\n",
      "  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n"
     ]
    }
   ],
   "source": [
    "env_name = \"LunarLander-v2\"\n",
    "num_cpu = 4\n",
    "train_env = make_vec_env(env_name, n_envs=num_cpu, seed=0)\n",
    "\n",
    "model = PPO2(\n",
    "    MlpPolicy, \n",
    "    train_env, \n",
    "    verbose=1, \n",
    "    n_steps=n_steps // num_cpu, # Scaling number of steps\n",
    "    learning_rate=0.003, \n",
    "    ent_coef=0.0,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run our function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.5/dist-packages/gym/logger.py:30: UserWarning: \u001b[33mWARN: Box bound precision lowered by casting to float32\u001b[0m\n",
      "  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean reward: -482.1 Num episodes: 100\n",
      "CPU times: user 8.09 s, sys: 328 ms, total: 8.42 s\n",
      "Wall time: 7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "vec_env = make_vec_env(env_name, n_envs=num_cpu, seed=0)\n",
    "rewards = evaluate_model_vec(model, vec_env, num_episodes=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And evaluation from stable-baselines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean reward is -505.01254\n",
      "CPU times: user 18.6 s, sys: 1.7 s, total: 20.3 s\n",
      "Wall time: 14.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "eval_env = DummyVecEnv([lambda: gym.make(env_name)])\n",
    "new_evaluation = evaluate_policy(model, eval_env, n_eval_episodes=100, deterministic=True)\n",
    "print(\"Mean reward is\", new_evaluation[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see we have x2 speed-up due to parallelization of our evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom policy networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Policy is neural network that performs actions and it's possible to configure it architecture with stable-baselines tools:  \n",
    "https://stable-baselines.readthedocs.io/en/master/guide/custom_policy.html\n",
    "\n",
    "Here we will use baselines FeedForwardPolicy network to construct our custom feed forward net.  \n",
    "We define `CustomPolicy` that will consist of two networks:\n",
    "- Policy with 2 layers of 64 neurons each;\n",
    "- Value function with 3 layers: 1 and 2 have 64 neurons and the last has 32;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stable_baselines.common.policies import FeedForwardPolicy, register_policy\n",
    "\n",
    "# Custom MLP policy of three layers of size 128 each\n",
    "class CustomPolicy(FeedForwardPolicy):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super(CustomPolicy, self).__init__(*args, **kwargs,\n",
    "                                           net_arch=[dict(pi=[64, 64],          # Policy network layers size\n",
    "                                                          vf=[64, 64, 32])],    # Value function layers size\n",
    "                                           feature_extraction=\"mlp\") # The feature extraction type (\"cnn\" or \"mlp\") (could be also custom network)\n",
    "\n",
    "# Register the policy, it will check that the name is not already taken\n",
    "register_policy('CustomPolicy', CustomPolicy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "env_name = \"CartPole-v0\"\n",
    "num_cpu = 4\n",
    "vec_env = make_vec_env(env_name, n_envs=num_cpu, seed=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------------\n",
      "| approxkl           | 0.004477359  |\n",
      "| clipfrac           | 0.048828125  |\n",
      "| ep_len_mean        | 22.2         |\n",
      "| ep_reward_mean     | 22.2         |\n",
      "| explained_variance | -0.014       |\n",
      "| fps                | 342          |\n",
      "| n_updates          | 1            |\n",
      "| policy_entropy     | 0.6891707    |\n",
      "| policy_loss        | -0.004464022 |\n",
      "| serial_timesteps   | 32           |\n",
      "| time_elapsed       | 3.58e-06     |\n",
      "| total_timesteps    | 128          |\n",
      "| value_loss         | 33.687138    |\n",
      "-------------------------------------\n",
      "--------------------------------------\n",
      "| approxkl           | 0.002528545   |\n",
      "| clipfrac           | 0.009765625   |\n",
      "| ep_len_mean        | 120           |\n",
      "| ep_reward_mean     | 120           |\n",
      "| explained_variance | 0.000405      |\n",
      "| fps                | 1901          |\n",
      "| n_updates          | 100           |\n",
      "| policy_entropy     | 0.6042605     |\n",
      "| policy_loss        | -0.0017906395 |\n",
      "| serial_timesteps   | 3200          |\n",
      "| time_elapsed       | 7.34          |\n",
      "| total_timesteps    | 12800         |\n",
      "| value_loss         | 95.36445      |\n",
      "--------------------------------------\n",
      "CPU times: user 28.4 s, sys: 6.03 s, total: 34.5 s\n",
      "Wall time: 15 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<stable_baselines.ppo2.ppo2.PPO2 at 0x7f4b70d267b8>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "model = PPO2(\n",
    "    CustomPolicy, # Instead of MlpPolicy now we have our custom policy\n",
    "    vec_env, \n",
    "    verbose=1, \n",
    "    n_steps=n_steps // num_cpu, # Scaling number of steps\n",
    "    learning_rate=0.003, \n",
    "    ent_coef=0.0,\n",
    ")\n",
    "model.learn(total_timesteps=25000, log_interval=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.5/dist-packages/gym/logger.py:30: UserWarning: \u001b[33mWARN: Box bound precision lowered by casting to float32\u001b[0m\n",
      "  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean reward: 200.0 Num episodes: 100\n",
      "CPU times: user 5.69 s, sys: 692 ms, total: 6.38 s\n",
      "Wall time: 4.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "vec_env = make_vec_env(env_name, n_envs=num_cpu, seed=0)\n",
    "rewards = evaluate_model_vec(model, vec_env, num_episodes=100)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}