gym_cartpole.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "gym_cartpole",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [
        "odNaDE1zyrL2"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ymiftah/machine_learning/blob/master/gym_cartpole.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "68uPXO-ZLYR1",
        "colab_type": "text"
      },
      "source": [
        "# The CartPole\n",
        "\n",
        "The cartpole stabilization is a classic problem in control systems. It also serves as a good topical introduction to reinforcement learning.\n",
        "\n",
        "This notebook contains some of my experimentations with openai/gym"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "odNaDE1zyrL2",
        "colab_type": "text"
      },
      "source": [
        "## Install dependancies to visualise OpenAI Gym results on colab\n",
        "\n",
        "Credits to [Paul Steven Conyngham, Star-AI](https://star-ai.github.io/Rendering-OpenAi-Gym-in-Colaboratory/) for the code in this section.\n",
        "\n",
        "We recommend running this notebook on Google Colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8-AxnvAVyzQQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Rendering Dependancies\n",
        "!pip install -qq gym pyvirtualdisplay\n",
        "!apt-get install -y -qq xvfb python-opengl ffmpeg\n",
        "\n",
        "!apt-get update -qq\n",
        "!apt-get install -qq cmake\n",
        "!pip install --upgrade -qq setuptools\n",
        "!pip install -qq ez_setup\n",
        "\n",
        "## Imports and helper function\n",
        "# Install TensorFlow\n",
        "try:\n",
        "  # %tensorflow_version only exists in Colab.\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass\n",
        "\n",
        "import tensorflow as tf\n",
        "\n",
        "import gym\n",
        "from gym import logger as gymlogger\n",
        "from gym.wrappers import Monitor\n",
        "gymlogger.set_level(40) #error only\n",
        "import numpy as np\n",
        "import random\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "import math\n",
        "import glob\n",
        "import io\n",
        "import base64\n",
        "from IPython.display import HTML\n",
        "\n",
        "from IPython import display as ipythondisplay\n",
        "\n",
        "from pyvirtualdisplay import Display\n",
        "display = Display(visible=0, size=(1400, 900))\n",
        "display.start()\n",
        "\n",
        "\"\"\"\n",
        "Utility functions to enable video recording of gym environment and displaying it\n",
        "To enable video, just do \"env = wrap_env(env)\"\"\n",
        "\"\"\"\n",
        "\n",
        "def show_video(folder):\n",
        "    mp4list = glob.glob(folder+'/*.mp4')\n",
        "    if len(mp4list) > 0:\n",
        "        mp4 = mp4list[0]\n",
        "        video = io.open(mp4, 'r+b').read()\n",
        "        encoded = base64.b64encode(video)\n",
        "        ipythondisplay.display(HTML(data='''<video alt=\"test\" autoplay \n",
        "                    loop controls style=\"height: 400px;\">\n",
        "                    <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n",
        "                 </video>'''.format(encoded.decode('ascii'))))\n",
        "    else: \n",
        "        print(\"Could not find video\")\n",
        "    \n",
        "\n",
        "def wrap_env(env, folder='./video'):\n",
        "    env = Monitor(env, folder, force=True)\n",
        "    return env\n",
        "\n",
        "def play(env, agent):\n",
        "    observation = env.reset()\n",
        "    reward = 0\n",
        "    counter = 0\n",
        "    done = False\n",
        "\n",
        "    while True:\n",
        "        counter += 1\n",
        "        env.render()\n",
        "        \n",
        "        #your agent goes here\n",
        "        action = agent.act(observation, reward, done) \n",
        "            \n",
        "        observation, reward, done, info = env.step(action)\n",
        "\n",
        "        if done:\n",
        "            print('Total steps taken : ', counter)\n",
        "            print(info)\n",
        "            break;\n",
        "                \n",
        "    env.close()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W3BGbWOu179M",
        "colab_type": "text"
      },
      "source": [
        "## Cart Pole\n",
        "\n",
        "The cartpole problem is a classical problem of control theory.\n",
        "The goal is to find the right controls for the cart such that the inverted pendulum stays in a stable upright position.\n",
        "\n",
        "<center> <img src=https://upload.wikimedia.org/wikipedia/commons/thumb/0/00/Cart-pendulum.svg/300px-Cart-pendulum.svg.png> </center>\n",
        "\n",
        "We remind below the equations ruling the dynamics of the cartpole system (frictionless), and which are implemented on OpenAI CartPole environment.\n",
        "\n",
        "$$ \\ddot{x} = \\frac{F+m_p l\\ [\\sin(\\theta)\\ \\dot{\\theta}^2- \\cos(\\theta) \\ddot{\\theta}]}{M} $$\n",
        "\n",
        "$$\n",
        "\\ddot{\\theta}=\n",
        "\\frac{g\\sin(\\theta) - \\frac{F}{M}\\cos(\\theta) - \\frac{m_p}{M}l \\cos(\\theta)\\sin(\\theta)\\dot{\\theta}^2}{l [\\frac{4}{3} - \\frac{m_p}{M} \\ \\cos^2\\theta]}\n",
        "$$\n",
        "\n",
        "We stress that contrary to the above picture, the angle in the gym environment is positive when the ball is on the right.\n",
        "\n",
        "Let us see how fares a random agent"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6CE1dj519YBZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class RandomAgent(object):\n",
        "    def act(self, observation, reward, done):\n",
        "        return np.random.randint(0,2) # [0,2) is [0,1]\n",
        "\n",
        "env = wrap_env(gym.make(\"CartPole-v1\"), folder='./video')\n",
        "env.theta_threshold_radians = 1\n",
        "agent = RandomAgent()\n",
        "\n",
        "play(env, agent)\n",
        "\n",
        "show_video(folder='./video')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_HBCoEiP-F7R",
        "colab_type": "text"
      },
      "source": [
        "As expected the pole falls quite quickly"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T8mMfi_cqzTr",
        "colab_type": "text"
      },
      "source": [
        "## Classic control\n",
        "\n",
        "Let us check how good old control theory performs on this task.\n",
        "\n",
        "The equations of dynamics of the cartpole are known. We can linearize these equations around the unstable equilibrium position at $\\theta = 0$, and $x=0$.\n",
        "\n",
        "We define $\\mathbf{X}$ as\n",
        "\n",
        "$$\n",
        "\\mathbf{X}\n",
        "=\n",
        "\\begin{bmatrix} x \\\\ \\dot{x} \\\\  \n",
        "\\theta \\\\ \\dot{\\theta} \\end{bmatrix}\n",
        "$$.\n",
        "\n",
        "We have the following linearized system:\n",
        "\n",
        "$$\n",
        "\\mathbf{\\dot{X}}\n",
        "=\n",
        "\\begin{bmatrix}\\dot{x} \\\\ \\ddot{x} \\\\  \n",
        "\\dot{\\theta} \\\\ \\ddot{\\theta}\n",
        "\\end{bmatrix}\n",
        "=\n",
        "\\begin{bmatrix}\n",
        "0 & 1 & 0 & 0 \\\\\n",
        "0 & 0 & - \\frac{m_p}{\\frac{4}{3}M - m_p}g & 0 \\\\  \n",
        "0 & 0 & 0 & 1 \\\\\n",
        "0 & 0 & \\frac{M}{\\frac{4}{3}M - m_p}\\frac{g}{l} & 0 \n",
        "\\end{bmatrix}\n",
        "\\begin{bmatrix} x \\\\ \\dot{x} \\\\  \n",
        "\\theta \\\\ \\dot{\\theta} \\end{bmatrix}\n",
        "+\n",
        "\\begin{bmatrix}\n",
        "0 \\\\ \\frac{m_p}{M} \\frac{F}{\\frac{4}{3}M - m_p} + \\frac{F}{M}\\\\  \n",
        "0 \\\\ -\\frac{1}{\\frac{4}{3}M - m_p}\\frac{F}{l}\n",
        "\\end{bmatrix}u\n",
        "$$\n",
        "\n",
        "Introducing the matrices $A$ and $B$, and $u$ the direction of the force (control variable), we have the following form:\n",
        "\n",
        "$$\n",
        "\\mathbf{\\dot{X}} = A \\mathbf{X} + bu\n",
        "$$\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xq5Em4Yd28tn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "env = gym.make(\"CartPole-v1\")\n",
        "m_p = env.masspole\n",
        "M = env.total_mass\n",
        "bla = (1/(4./3 * M - m_p))\n",
        "F = env.force_mag\n",
        "l = env.length\n",
        "g = env.gravity\n",
        "\n",
        "A = np.array([[0, 1, 0, 0],\n",
        "              [0, 0, -m_p*bla*g, 0],\n",
        "              [0, 0, 0, 1],\n",
        "              [0, 0, M*g/l*bla, 0]])\n",
        "B = np.array([0, F*bla*m_p/M + F/M, 0, - F/l*bla])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4yeE1DUWMZ-S",
        "colab_type": "text"
      },
      "source": [
        "## Linear Feedback\n",
        "\n",
        "We will find a linear feedback control $R$ such that $u = - \\mathbf{r^TX}$. then\n",
        "\n",
        "$$\n",
        "\\mathbf{\\dot{X}} = \\mathbf{(A-br^T)X}\n",
        "$$\n",
        "\n",
        "Can we find the vector r such that the eigen values of the matrix $\\mathbf{(A-br^T)}$ all have a negative real part ?\n",
        "\n",
        "We will use the scipy \"place_poles\" function to select the gain vector $\\mathbf{r}$ such that the poles are around -1.\n",
        "\" Around \" -1 because the rank of B is 1, so the multiplicity of the poles must be no greater than 1 for the place_poles algorithm to function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w3J2XjBeREbk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from scipy.signal import place_poles\n",
        "\n",
        "r = place_poles(A,B.reshape(4,1),[-1.1,-1.02,-1.05,-1.01]).gain_matrix.flatten()\n",
        "print(r)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ExvU4kyn5YVA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "env = wrap_env(gym.make(\"CartPole-v1\"), folder='./video')\n",
        "\n",
        "class ClassicControlAgent(object):\n",
        "    def __init__(self, gain):\n",
        "        self.gain = gain\n",
        "    def act(self, observation, reward, done):\n",
        "        x, dx, theta, dtheta = observation\n",
        "\n",
        "        u = - np.dot(self.gain, np.array(observation))\n",
        "        sign = u > 0\n",
        "        return int(sign)\n",
        "\n",
        "agent = ClassicControlAgent(r)\n",
        "\n",
        "play(env, agent)\n",
        "\n",
        "show_video(folder='./video')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8zgzgXL5-yVJ",
        "colab_type": "text"
      },
      "source": [
        "This looks much better : the pole is held perfectly upright, with a limitation that the pole may still go off bounds (this could be corrected with an improved controller).\n",
        "\n",
        "Of course in this case our agent knows exactly the dynamics of the problem, in a way we have hardcoded the agent with the right control logic.\n",
        "\n",
        "What if the dynamics are unknown or too complex to approach by a classical method ?\n",
        "\n",
        "We will try to have an agent learn the right control logic through experience, instead of telling the agent how."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LPV_GA_wdlML",
        "colab_type": "text"
      },
      "source": [
        "## Cross Entropy Learning\n",
        "\n",
        "Our agent will now learn which action to take through learning from experience.\n",
        "\n",
        "First we will apply the cross entropy method (implementation from [Max Lapan](https://github.com/PacktPublishing/Deep-Reinforcement-Learning-Hands-On) )\n",
        "\n",
        "The method is as follows:\n",
        "- First the agent is a neural network which input is the current observation, and output the probability of taking action 0 or 1.\n",
        "- The agent plays BATCH_SIZE of games, and keeps the best performing plays to train the network.\n",
        "\n",
        "The idea is that the agent will progressively learn the replicate the best plays."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UzWW--wDf0XE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "\n",
        "from collections import namedtuple\n",
        "\n",
        "HIDDEN_SIZE = 128\n",
        "BATCH_SIZE = 32\n",
        "PERCENTILE = 70\n",
        "\n",
        "# We define a two-layer neural network of dense layers\n",
        "class Net(nn.Module):\n",
        "    def __init__(self, obs_size, hidden_size, n_actions):\n",
        "        super(Net, self).__init__()\n",
        "        self.net = nn.Sequential(\n",
        "            nn.Linear(obs_size, hidden_size),\n",
        "            nn.ELU(),\n",
        "            nn.Linear(hidden_size, hidden_size//2),\n",
        "            nn.ELU(),\n",
        "            nn.Linear(hidden_size//2, n_actions)\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.net(x)\n",
        "\n",
        "\n",
        "class NetAgent(object):\n",
        "    def __init__(self, net):\n",
        "        self.nn = net\n",
        "    \n",
        "    def act(self, obs, reward, done):\n",
        "        assert isinstance(observation, np.ndarray) and observation.ndim == 1, 'unsupported observation type for now.'\n",
        "        \n",
        "        sm = nn.Softmax(dim=1)\n",
        "        obs_v = torch.FloatTensor([obs])\n",
        "        act_probs_v = sm(self.nn(obs_v))\n",
        "        act_probs = act_probs_v.data.numpy().flatten()\n",
        "        # We choose the action according to the model output distribution\n",
        "        action = np.random.choice(len(act_probs), p=act_probs)\n",
        "        return action\n",
        "\n",
        "Episode = namedtuple('Episode', field_names=['reward', 'steps'])\n",
        "EpisodeStep = namedtuple('EpisodeStep', field_names=['observation', 'action'])\n",
        "\n",
        "# Helper function to create a batch of \n",
        "def iterate_batches(env, net, batch_size):\n",
        "    batch = []\n",
        "    episode_reward = 0.0\n",
        "    episode_steps = []\n",
        "    obs = env.reset()\n",
        "    sm = nn.Softmax(dim=1)\n",
        "    while True:\n",
        "        # Get the action from the observation\n",
        "        obs_v = torch.FloatTensor([obs])\n",
        "        act_probs_v = sm(net(obs_v))\n",
        "        act_probs = act_probs_v.data.numpy()[0]\n",
        "        action = np.random.choice(len(act_probs), p=act_probs)\n",
        "\n",
        "        # Take action and record\n",
        "        next_obs, reward, is_done, _ = env.step(action)\n",
        "        episode_reward += reward\n",
        "        episode_steps.append(EpisodeStep(observation=obs, action=action))\n",
        "        if is_done:\n",
        "            # Add the full episode and total reward\n",
        "            batch.append(Episode(reward=episode_reward, steps=episode_steps))\n",
        "\n",
        "            #reset the episode records and start a new game\n",
        "            episode_reward = 0.0\n",
        "            episode_steps = []\n",
        "            next_obs = env.reset()\n",
        "\n",
        "            # We played batch_size number of games, yield the result\n",
        "            if len(batch) == batch_size:\n",
        "                yield batch\n",
        "                batch = []\n",
        "        \n",
        "        # Not done, we keep on playing\n",
        "        obs = next_obs\n",
        "\n",
        "# Helper function to filter from the batch of episode those which\n",
        "# total reward is in the Xth percentile\n",
        "def filter_batch(batch, percentile):\n",
        "    rewards = list(map(lambda s: s.reward, batch))\n",
        "    reward_bound = np.percentile(rewards, percentile)\n",
        "    reward_mean = float(np.mean(rewards))\n",
        "\n",
        "    train_obs = []\n",
        "    train_act = []\n",
        "    for example in batch:\n",
        "        if example.reward < reward_bound:\n",
        "            continue\n",
        "        train_obs.extend(map(lambda step: step.observation, example.steps))\n",
        "        train_act.extend(map(lambda step: step.action, example.steps))\n",
        "\n",
        "    # make then torch tensors\n",
        "    train_obs_v = torch.FloatTensor(train_obs)\n",
        "    train_act_v = torch.LongTensor(train_act)\n",
        "    return train_obs_v, train_act_v, reward_bound, reward_mean"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Rhi9ErxpqXxX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "env = gym.make(\"CartPole-v1\")\n",
        "env._max_episode_steps = 500\n",
        "# env = gym.wrappers.Monitor(env, directory=\"mon\", force=True)\n",
        "obs_size = env.observation_space.shape[0]\n",
        "n_actions = env.action_space.n\n",
        "\n",
        "# Define our network, objective and optimizer\n",
        "net = Net(obs_size, HIDDEN_SIZE, n_actions)\n",
        "objective = nn.CrossEntropyLoss()\n",
        "optimizer = optim.Adam(params=net.parameters(), lr=0.01)\n",
        "\n",
        "def CrossEntropyTraining(env, net):\n",
        "    # Logs of the mean total reward of a batch to monitor learning progress\n",
        "    log_mean = [0]*10\n",
        "\n",
        "    for iter_no, batch in enumerate(iterate_batches(env, net, BATCH_SIZE)):\n",
        "        obs_v, acts_v, reward_b, reward_m = filter_batch(batch, PERCENTILE)\n",
        "\n",
        "        optimizer.zero_grad()\n",
        "        action_scores_v = net(obs_v)\n",
        "        loss_v = objective(action_scores_v, acts_v)\n",
        "        loss_v.backward()\n",
        "        optimizer.step()\n",
        "        print(\"\\r %d: loss=%.3f, reward_mean=%.1f, reward_bound=%.1f\" % (\n",
        "            iter_no, loss_v.item(), reward_m, reward_b), end=\"\", flush=True)\n",
        "        log_mean.append(reward_m)\n",
        "\n",
        "        if iter_no > 100 or sum(log_mean[-10:])/5 >= 490:\n",
        "            print(\"\\n Solved!\")\n",
        "            break\n",
        "    return log_mean\n",
        "\n",
        "log_mean = CrossEntropyTraining(env, net)\n",
        "# writer.close()\n",
        "plt.plot(log_mean)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3UJyDU6otSzY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "env = wrap_env(gym.make(\"CartPole-v1\"), folder='./video')\n",
        "env._max_episode_steps = 500\n",
        "observation = env.reset()\n",
        "reward = 0\n",
        "counter = 0\n",
        "done = False\n",
        "\n",
        "while True:\n",
        "    counter += 1\n",
        "    env.render()\n",
        "    #your agent goes here\n",
        "    sm = nn.Softmax(dim=1)\n",
        "    obs_v = torch.FloatTensor([observation])\n",
        "    act_probs_v = sm(net(obs_v))\n",
        "    act_probs = act_probs_v.data.numpy()[0]\n",
        "    action = np.random.choice(len(act_probs), p=act_probs)\n",
        "        \n",
        "    observation, reward, done, info = env.step(action)\n",
        "\n",
        "    if done:\n",
        "        print('Total steps taken : ', counter)\n",
        "        break;\n",
        "            \n",
        "env.close()\n",
        "\n",
        "show_video(folder='./video')\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l84Ct6-yWfsx",
        "colab_type": "text"
      },
      "source": [
        "Compared to the feedback controller method we see that our pole is much less stable with this simple algorithm, but it does learn to keep the pole up !\n",
        "\n",
        "We will try to have the agent keep the pole at the central position. For this we\n",
        "will slightly modify our rewards to add a penalty for deviating from the central position."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "84AgbGKFmoUw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class MyRewardWrapper(gym.RewardWrapper):\n",
        "    def __init__(self, env):\n",
        "        super(MyRewardWrapper, self).__init__(env)\n",
        "\n",
        "    def reset(self, **kwargs):\n",
        "        return self.env.reset(**kwargs)\n",
        "\n",
        "    def step(self, action):\n",
        "        observation, reward, done, info = self.env.step(action)\n",
        "        reward += np.exp(-10 * np.abs(np.linalg.norm(observation)))\n",
        "        return observation, reward, done, info\n",
        "\n",
        "env = MyRewardWrapper(gym.make(\"CartPole-v1\"))\n",
        "env._max_episode_steps = 1000\n",
        "env.reset()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6W9VFxglXW7x",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Define our network, objective and optimizer\n",
        "net = Net(obs_size, HIDDEN_SIZE, n_actions)\n",
        "objective = nn.CrossEntropyLoss()\n",
        "optimizer = optim.Adam(params=net.parameters(), lr=0.01)\n",
        "\n",
        "log_mean = CrossEntropyTraining(env, net)\n",
        "# writer.close()\n",
        "plt.plot(log_mean)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d5h4c_PhYIkZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "env = wrap_env(gym.make(\"CartPole-v1\"), folder='./video')\n",
        "env._max_episode_steps = 500\n",
        "observation = env.reset()\n",
        "reward = 0\n",
        "counter = 0\n",
        "done = False\n",
        "\n",
        "while True:\n",
        "    counter += 1\n",
        "    env.render()\n",
        "    #your agent goes here\n",
        "    sm = nn.Softmax(dim=1)\n",
        "    obs_v = torch.FloatTensor([observation])\n",
        "    act_probs_v = sm(net(obs_v))\n",
        "    act_probs = act_probs_v.data.numpy()[0]\n",
        "    action = np.random.choice(len(act_probs), p=act_probs)\n",
        "        \n",
        "    observation, reward, done, info = env.step(action)\n",
        "\n",
        "    if done:\n",
        "        print('Total steps taken : ', counter)\n",
        "        break;\n",
        "            \n",
        "env.close()\n",
        "\n",
        "show_video(folder='./video')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uh3H1cpO6oJL",
        "colab_type": "text"
      },
      "source": [
        "Pretty, neat, it does seem to bring the pole at the $0$ position as much as possible"
      ]
    }
  ]
}