Notebook/GTAV-Self-driving-car.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Grand Theft Auto V Driving learning with Deep Learning (CNN+RNN)\n",
    "---\n",
    "Self driving car in Grand Theft Auto V with Deep Learning.\n",
    "https://github.com/eritzyg/GTAV-Self-driving-car\n",
    "\n",
    "### Authors: Iker García and Eritz Yerga\n",
    "\n",
    "See [this README](https://github.com/eritzyg/GTAV-Self-driving-car/blob/master/Notebook/README.md) for info about the notebook and required libraries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Index:\n",
    "\n",
    "1. <a href=\"#1.-Generate-dataset\">Generate dataset</a>\n",
    "    * <a href=\"#Frame-capture-functions\">Frame capture functions</a>\n",
    "    * <a href=\"#Image-preprocessing-functions\">Image preprocess functions</a>\n",
    "    * <a href=\"#Game-control-and-input-reading-functions\">Game control and input reading functions</a>\n",
    "    * <b><a href=\"#Generate-dataset\">Generate dataset</a></b>\n",
    "2. <a href=\"#2.-Dataset-processing-utilities\">Dataset processing utilities</a>\n",
    "3. <a href=\"#3.-Define-the-model\">Define our model</a>\n",
    "4. <a href=\"#4.-Train\">Train</a>\n",
    "5. <a href=\"#5.-Run-our-model-in-the-game\">Run our model in the game</a>\n",
    "\n",
    "<a href=\"\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "First we import the libraries we are going to need to run this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from PIL import ImageGrab\n",
    "import cv2\n",
    "import time\n",
    "from sys import stdout\n",
    "from IPython.display import clear_output\n",
    "import os\n",
    "from grabber import Grabber\n",
    "import threading\n",
    "import matplotlib.pyplot as plt\n",
    "from collections import Counter\n",
    "from random import shuffle\n",
    "import glob\n",
    "import tensorflow as tf\n",
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import TimeDistributed, LSTM, Flatten, Dense, InputLayer, MaxPooling2D, Dropout, Activation, Embedding, GRU, ConvLSTM2D\n",
    "from keras.layers.convolutional import Convolution2D\n",
    "from keras import optimizers\n",
    "from keras.models import load_model\n",
    "from keras import initializers\n",
    "import h5py\n",
    "import log\n",
    "from heapq import nlargest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Generate dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will generate the dataset for training the model later on, it is important that we set the game on the first person view and take into account certain conditions for the dataset. Check the \"Generation of the dataset\" section in the [documentation](https://github.com/eritzyg/GTAV-Self-driving-car#documentation) for more information on this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Frame capture functions\n",
    "These functions capture the game's frames in 1600x900 Windowed mode.\n",
    "\n",
    "Screen record is the method to get one frame and img_thread is the thread we will later use to constantly capture the game's output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "global grb\n",
    "grb = Grabber(bbox=(1,26,1601,926))\n",
    "def screen_record(method = 'grabber'):\n",
    "    if method == 'ImameGrab':\n",
    "        printscreen =  ImageGrab.grab(bbox=(1,26,1601,926))\n",
    "        generalIMG = np.array(printscreen)\n",
    "    \n",
    "    elif method == 'grabber':\n",
    "        global grb\n",
    "        printscreen = None\n",
    "        printscreen = grb.grab(printscreen)\n",
    "        generalIMG = np.array(printscreen)\n",
    "    \n",
    "    return generalIMG          "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "global front_buffer\n",
    "global back_buffer\n",
    "front_buffer = np.zeros((1600, 900), dtype=np.int8)\n",
    "back_buffer = np.zeros((1600, 900), dtype=np.int8)\n",
    "\n",
    "global fps\n",
    "fps = 0\n",
    "\n",
    "def img_thread():\n",
    "    global front_buffer\n",
    "    global back_buffer\n",
    "    global fps\n",
    "    \n",
    "    last_time = time.time()\n",
    "    while True:\n",
    "        front_buffer = screen_record()\n",
    "        # Swap buffers\n",
    "        front_buffer, back_buffer = back_buffer, front_buffer\n",
    "        fps = int(1.0/(time.time()-last_time))\n",
    "        last_time = time.time()\n",
    "    return\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Image preprocessing functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will define a function that will apply the preprocessing we want to the images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_image(image):\n",
    "    proccessed_image = cv2.resize(image,(480,270))\n",
    "    \n",
    "    return proccessed_image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Game control and input reading functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These functions will read the inputs and generate a array for later use when generating the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from game_control import PressKey, ReleaseKey\n",
    "from getkeys import key_check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def keys_to_output(keys):\n",
    "    '''\n",
    "    Convert keys to a ...multi-hot... array\n",
    "\n",
    "    [A,W,D] boolean values.\n",
    "    '''\n",
    "    output = [0,0,0,0]\n",
    "    \n",
    "    if 'A' in keys:\n",
    "        output[0] = 1\n",
    "    if 'D' in keys:\n",
    "        output[1] = 1\n",
    "    if 'W' in keys:\n",
    "        output[2] = 1\n",
    "    if 'S' in keys:\n",
    "        output[3] = 1\n",
    "    \n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This sequencer thread will capture the sequences of 5 frames with a separation of 1/capturerate ms each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "global seq\n",
    "global num\n",
    "num = 0\n",
    "seq = []\n",
    "\n",
    "global key_out\n",
    "key_out = [0, 0, 0, 0]\n",
    "\n",
    "def image_sequencer_thread():\n",
    "    global back_buffer\n",
    "    global seq\n",
    "    global key_out\n",
    "    global num\n",
    "    \n",
    "    # Frames per second capture rate\n",
    "    capturerate = 10.0\n",
    "    while True:\n",
    "        last_time = time.time()\n",
    "        if len(seq) == 5:\n",
    "            del seq[0]\n",
    "\n",
    "        seq.append(preprocess_image(np.copy(back_buffer)))\n",
    "        num = num + 1\n",
    "        keys = key_check()\n",
    "        key_out = keys_to_output(keys)\n",
    "        waittime = (1.0/capturerate)-(time.time()-last_time)\n",
    "        if waittime>0.0:\n",
    "            time.sleep(waittime)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function will be useful to check which class corresponds the input we captured to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def counter_keys(key):\n",
    "        if np.array_equal(key , [0,0,0,0]):\n",
    "            return 0\n",
    "        elif np.array_equal(key , [1,0,0,0]):\n",
    "            return 1\n",
    "        elif np.array_equal(key , [0,1,0,0]):\n",
    "            return 2\n",
    "        elif np.array_equal(key , [0,0,1,0]):\n",
    "            return 3\n",
    "        elif np.array_equal(key , [0,0,0,1]):\n",
    "            return 4\n",
    "        elif np.array_equal(key , [1,0,1,0]):\n",
    "            return 5\n",
    "        elif np.array_equal(key , [1,0,0,1]):\n",
    "            return 6\n",
    "        elif np.array_equal(key , [0,1,1,0]):\n",
    "            return 7\n",
    "        elif np.array_equal(key , [0,1,0,1]):\n",
    "            return 8\n",
    "        else:\n",
    "            return -1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the function that the data saving threads will run to save the dataset to compressed files (change the path if needed):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_data(data,number):\n",
    "    file_name = 'E:\\\\GTAV_AI\\\\training_data'+str(number)+'.npz'\n",
    "    np.savez_compressed(file_name,data)\n",
    "    del data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will run this function to generate the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run():\n",
    "    global fps\n",
    "    global front_buffer\n",
    "    global back_buffer\n",
    "    global seq\n",
    "    global key_out\n",
    "    global num\n",
    "    training_data = []\n",
    "    threads = list()\n",
    "    th_img = threading.Thread(target=img_thread)\n",
    "    th_seq = threading.Thread(target=image_sequencer_thread)\n",
    "    threads.append(th_img)\n",
    "    threads.append(th_seq)\n",
    "    th_img.start()\n",
    "    time.sleep(1)\n",
    "    th_seq.start()\n",
    "    l = 0\n",
    "    fn = 0\n",
    "    time.sleep(4)\n",
    "    last_num = 0\n",
    "    \n",
    "    number_of_keys = [0,0,0,0,0,0,0,0,0]\n",
    "    \n",
    "    while True:\n",
    "        img_seq = seq.copy()\n",
    "        output = key_out.copy()\n",
    "        \n",
    "        while len(img_seq) != 5 or last_num==num:\n",
    "            del img_seq, output\n",
    "            img_seq = seq.copy()\n",
    "            output = key_out.copy()\n",
    "        last_num = num\n",
    "        \n",
    "        clear_output(wait=True)\n",
    "        stdout.write('Recording at {} FPS \\n'.format(fps))\n",
    "        stdout.write('Images in sequence {} \\n'.format(len(img_seq)))\n",
    "        stdout.write('Training data len {} secuences \\n'.format(l))\n",
    "        stdout.write('Number of archives {}\\n'.format(fn))\n",
    "        stdout.write('Keys pressed: ' + str(output) + ' \\n')\n",
    "        stdout.write('Keys samples in this file: ' + 'none:' + str(number_of_keys[0]) + ' A:' + str(number_of_keys[1])+ ' D:' + str(number_of_keys[2]) + ' W:' + str(number_of_keys[3])+ ' S:' + str(number_of_keys[4]) + ' AW:'  + str(number_of_keys[5]) + ' AS:' + str(number_of_keys[6]) + ' WD:' + str(number_of_keys[7]) + ' SD:' + str(number_of_keys[8]) + ' \\n')\n",
    "        stdout.flush()\n",
    "        \n",
    "        key  = counter_keys(output)\n",
    "        \n",
    "        if key != -1:\n",
    "            larg = nlargest(9,number_of_keys)\n",
    "            prop = (9. - float(larg.index(number_of_keys[key])))/10\n",
    "            if(number_of_keys[key]  > np.mean(number_of_keys) * 1.25):\n",
    "                prop = prop + 0.05\n",
    "            if (np.random.rand() > prop):\n",
    "                number_of_keys[key] += 1\n",
    "                l = l+1\n",
    "                training_data.append([img_seq[0],img_seq[1],img_seq[2],img_seq[3],img_seq[4], output])\n",
    "            \n",
    "        if cv2.waitKey(1) & 0xFF == ord('q'):\n",
    "            cv2.destroyAllWindows()\n",
    "            break\n",
    "\n",
    "        if len(training_data) % 500 == 0:\n",
    "            print(len(training_data))\n",
    "            threading.Thread(target=save_data, args=(training_data.copy(), fn,)).start()\n",
    "            fn = fn + 1\n",
    "            del training_data\n",
    "            training_data = []\n",
    "            \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will run the capture and generation of the dataset, remember that the game's window must be in 1600x900 resolution and located in the top left part of the screen ((0,0) coordinates).\n",
    "\n",
    "Once you captured all the data you need you can interrupt the kernel to stop the run function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time.sleep(5)\n",
    "run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Dataset processing utilities\n",
    "Some of these functions are not used, but we include them for debug purposes. Others are used later on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can plot some data to see if it captured correctly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plotImgs = False\n",
    "if plotImgs:\n",
    "    with np.load('E:\\\\GTAV_AI\\\\filename.npz') as data:\n",
    "        training_data = data['arr_0']\n",
    "    i= 502\n",
    "    plt.imshow(cv2.cvtColor(training_data[i][0], cv2.COLOR_BGR2RGB))\n",
    "    plt.show()\n",
    "    plt.imshow(cv2.cvtColor(training_data[i][1], cv2.COLOR_BGR2RGB))\n",
    "    plt.show()\n",
    "    plt.imshow(cv2.cvtColor(training_data[i][2], cv2.COLOR_BGR2RGB))\n",
    "    plt.show()\n",
    "    plt.imshow(cv2.cvtColor(training_data[i][3], cv2.COLOR_BGR2RGB))\n",
    "    plt.show()\n",
    "    plt.imshow(cv2.cvtColor(training_data[i][4], cv2.COLOR_BGR2RGB))\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function is used to count the number of instances per class in a set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def number_instances_per_class(data):\n",
    "    \n",
    "    nonekey = []\n",
    "    A = []\n",
    "    D = []\n",
    "    W = []\n",
    "    S = []\n",
    "    \n",
    "    AD = []\n",
    "    AW = []\n",
    "    AS = []\n",
    "    DW =[]\n",
    "    DS = []\n",
    "    WS =[]\n",
    "    \n",
    "    ADW = []\n",
    "    AWS =[]\n",
    "    ADS = []\n",
    "    DWS = []\n",
    "    \n",
    "    ASWS = []\n",
    "    \n",
    "    np.random.shuffle(data)\n",
    "    \n",
    "    for d in data:\n",
    "        if np.array_equal(d[5] , [0,0,0,0]):\n",
    "            nonekey.append(d)\n",
    "        elif np.array_equal(d[5] , [1,0,0,0]):\n",
    "            A.append(d)\n",
    "        elif np.array_equal(d[5] , [0,1,0,0]):\n",
    "            D.append(d)\n",
    "        elif np.array_equal(d[5] , [0,0,1,0]):\n",
    "            W.append(d)\n",
    "        elif np.array_equal(d[5] , [0,0,0,1]):\n",
    "            S.append(d)\n",
    "        elif np.array_equal(d[5] , [1,1,0,0]):\n",
    "            AD.append(d)\n",
    "        elif np.array_equal(d[5] , [1,0,1,0]):\n",
    "            AW.append(d)\n",
    "        elif np.array_equal(d[5] , [1,0,0,1]):\n",
    "            AS.append(d)\n",
    "        elif np.array_equal(d[5] , [0,1,1,0]):\n",
    "            DW.append(d)\n",
    "        elif np.array_equal(d[5] , [0,1,0,1]):\n",
    "            DS.append(d)\n",
    "        elif np.array_equal(d[5] , [0,0,1,1]):\n",
    "            WS.append(d)\n",
    "        elif np.array_equal(d[5] , [1,1,1,0]):\n",
    "            ADW.append(d)\n",
    "        elif np.array_equal(d[5] , [1,1,0,1]):\n",
    "            AWS.append(d)\n",
    "        elif np.array_equal(d[5] , [1,1,0,1]):\n",
    "            ADS.append(d)\n",
    "        elif np.array_equal(d[5] , [0,1,1,1]):\n",
    "            DWS.append(d)\n",
    "        elif np.array_equal(d[5] , [1,1,1,1]):\n",
    "            ASWS.append(d)\n",
    "    return [nonekey,A,D,W,S,AW,AS,DW,DS]\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function will balance the number of instances of the classes in a set by deleting extra instances after shuffling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def balance_data(data_in_clases):\n",
    "    balanced_data = []\n",
    "    data_in_clases.sort(key=len)\n",
    "    max_len = len(data_in_clases[0])\n",
    "        \n",
    "    for data in data_in_clases:\n",
    "        if len(data) > max_len:\n",
    "            data=data[:max_len]\n",
    "        for d in data:\n",
    "            balanced_data.append(d)\n",
    "        \n",
    "    np.random.shuffle(balanced_data)\n",
    "    \n",
    "    return balanced_data\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For debug purposes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "debug = False\n",
    "if debug:\n",
    "    with np.load('E:\\\\GTAV_AI\\\\training_data'+str(20)+'.npz') as data:\n",
    "         training_data = data['arr_0']\n",
    "    number = number_instances_per_class(training_data)\n",
    "    print('none: ' + str(len(number[0])))\n",
    "    print('A: ' + str(len(number[1])))\n",
    "    print('D ' + str(len(number[2])))\n",
    "    print('W ' + str(len(number[3])))\n",
    "    print('S ' + str(len(number[4])))\n",
    "    print('AW ' + str(len(number[5])))\n",
    "    print('AS ' + str(len(number[6])))\n",
    "    print('DW ' + str(len(number[7])))\n",
    "    print('DS ' + str(len(number[8])))\n",
    "\n",
    "    balanced_data = balance_data(number)\n",
    "    number = number_instances_per_class(balanced_data)\n",
    "    print('none: ' + str(len(number[0])))\n",
    "    print('A: ' + str(len(number[1])))\n",
    "    print('D ' + str(len(number[2])))\n",
    "    print('W ' + str(len(number[3])))\n",
    "    print('S ' + str(len(number[4])))\n",
    "    print('AW ' + str(len(number[5])))\n",
    "    print('AS ' + str(len(number[6])))\n",
    "    print('DW ' + str(len(number[7])))\n",
    "    print('DS ' + str(len(number[8])))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Define the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will define the model to use in our training, please select the model you want to use in the cell bellow (the models are described in the \"Definition of the DNN\" of the [documentation](https://github.com/eritzyg/GTAV-Self-driving-car#documentation)):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select the model to use:\n",
    "# 'CNN+MPL' : Convolutional neural network with multi layer perceptron.\n",
    "# 'CNN+RNN' : Convolutional neural network with recurrent neural network.\n",
    "selected_model = 'CNN+RNN'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we define the corresponding model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if selected_model == 'CNN+RNN':\n",
    "    model = Sequential()\n",
    "\n",
    "    model.add(InputLayer(input_shape=(5, 270, 480, 3)))\n",
    "\n",
    "    model.add(TimeDistributed(Convolution2D(32, (4,4), data_format='channels_last')))\n",
    "    model.add(TimeDistributed(Activation('relu')))\n",
    "    print(model.output_shape)\n",
    "\n",
    "    model.add(TimeDistributed(Convolution2D(32, (4,4), data_format='channels_last')))\n",
    "    model.add(TimeDistributed(Activation('relu')))\n",
    "    print(model.output_shape)\n",
    "\n",
    "    model.add(TimeDistributed(MaxPooling2D(pool_size=(5, 5), data_format='channels_last')))\n",
    "    model.add(TimeDistributed(Dropout(0.25)))\n",
    "    print(model.output_shape)\n",
    "\n",
    "    model.add(TimeDistributed(Convolution2D(16, (3,3), data_format='channels_last')))\n",
    "    model.add(TimeDistributed(Activation('relu')))\n",
    "    print(model.output_shape)\n",
    "\n",
    "\n",
    "    model.add(TimeDistributed(MaxPooling2D(pool_size=(5, 5), data_format='channels_last')))\n",
    "    model.add(TimeDistributed(Dropout(0.25)))\n",
    "    print(model.output_shape)\n",
    "\n",
    "    model.add(TimeDistributed(Flatten()))\n",
    "    print(model.output_shape)\n",
    "\n",
    "    model.add(GRU(256, kernel_initializer=initializers.RandomNormal(stddev=0.001))) #128\n",
    "    model.add(Dropout(0.25))\n",
    "    print(model.output_shape)\n",
    "\n",
    "    model.add(Dense(100))\n",
    "    print(model.output_shape)\n",
    "\n",
    "    model.add(Dense(80))\n",
    "    print(model.output_shape)\n",
    "\n",
    "    model.add(Dense(40))\n",
    "    print(model.output_shape)\n",
    "\n",
    "    model.add(Dense(9, activation='sigmoid'))\n",
    "    print(model.output_shape)\n",
    "\n",
    "    opt = optimizers.rmsprop(lr=0.001)\n",
    "    model.compile(loss='mean_squared_error', optimizer=opt, metrics=['accuracy']) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if selected_model == 'CNN+MLP':\n",
    "    model = Sequential()\n",
    "\n",
    "    model.add(InputLayer(input_shape=(5, 270, 480, 3)))\n",
    "\n",
    "    model.add(TimeDistributed(Convolution2D(16, (4,8), data_format='channels_last')))\n",
    "    model.add(TimeDistributed(Activation('relu')))\n",
    "    print(model.output_shape)\n",
    "\n",
    "    model.add(TimeDistributed(Convolution2D(16, (4,4), data_format='channels_last')))\n",
    "    model.add(TimeDistributed(Activation('relu')))\n",
    "    print(model.output_shape)\n",
    "\n",
    "    model.add(TimeDistributed(MaxPooling2D(pool_size=(5, 5), data_format='channels_last')))\n",
    "    model.add(TimeDistributed(Dropout(0.25)))\n",
    "    print(model.output_shape)\n",
    "\n",
    "    model.add(TimeDistributed(Convolution2D(12, (3,3), data_format='channels_last')))\n",
    "    model.add(TimeDistributed(Activation('relu')))\n",
    "    print(model.output_shape)\n",
    "\n",
    "    model.add(TimeDistributed(MaxPooling2D(pool_size=(5, 5), data_format='channels_last')))\n",
    "    model.add(TimeDistributed(Dropout(0.25)))\n",
    "    print(model.output_shape)\n",
    "\n",
    "    model.add(Flatten())\n",
    "    print(model.output_shape)\n",
    "\n",
    "    model.add(Dense(300))\n",
    "    print(model.output_shape)\n",
    "    model.add(Dense(100))\n",
    "    print(model.output_shape)\n",
    "    print(model.output_shape)\n",
    "    model.add(Dense(9, activation='sigmoid'))\n",
    "    print(model.output_shape)\n",
    "\n",
    "    opt = optimizers.rmsprop(lr=0.001)\n",
    "    model.compile(loss='mean_squared_error', optimizer=opt, metrics=['accuracy']) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will define some functions to reshape the data according to the input of our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reshape_custom_X(data, verbose = 1):\n",
    "    reshaped = np.zeros((data.shape[0], 5, 270, 480, 3), dtype=np.float32)\n",
    "    for i in range(0, data.shape[0]):\n",
    "        for j in range(0, 5):\n",
    "            if (verbose == 1):\n",
    "                clear_output(wait=True)\n",
    "                stdout.write('Reshaped image: ' + str(i))\n",
    "                stdout.flush()\n",
    "            reshaped[i][j] = data[i][j]/255.\n",
    "            \n",
    "    return reshaped\n",
    "\n",
    "def reshape_custom_y(data):\n",
    "    reshaped = np.zeros((data.shape[0], 9), dtype=np.float32)\n",
    "    for i in range(0, data.shape[0]):\n",
    "            if np.array_equal(data[i][0] , [0,0,0,0]):\n",
    "                reshaped[i][0] = 1.\n",
    "            elif np.array_equal(data[i][0] , [1,0,0,0]):\n",
    "                reshaped[i][1] = 1.\n",
    "            elif np.array_equal(data[i][0] , [0,1,0,0]):\n",
    "                reshaped[i][2] = 1.\n",
    "            elif np.array_equal(data[i][0] , [0,0,1,0]):\n",
    "                reshaped[i][3] = 1.\n",
    "            elif np.array_equal(data[i][0] , [0,0,0,1]):\n",
    "                reshaped[i][4] = 1.\n",
    "            elif np.array_equal(data[i][0] , [1,0,1,0]):\n",
    "                reshaped[i][5] = 1.\n",
    "            elif np.array_equal(data[i][0] , [1,0,0,1]):\n",
    "                reshaped[i][6] = 1.\n",
    "            elif np.array_equal(data[i][0] , [0,1,1,0]):\n",
    "                reshaped[i][7] = 1.\n",
    "            elif np.array_equal(data[i][0] , [0,1,0,1]):\n",
    "                reshaped[i][8] = 1.\n",
    "    return reshaped"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These functions will help for picking the number of batches and the start and end indexes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_num_batches(length, BATCH_SIZE):\n",
    "    if (int(length/BATCH_SIZE)*BATCH_SIZE == length):\n",
    "        return int(length/BATCH_SIZE)\n",
    "    else:\n",
    "        return int(length/BATCH_SIZE)+1\n",
    "\n",
    "def get_start_end(iteration, BATCH_SIZE, max_length):\n",
    "    start = iteration*BATCH_SIZE\n",
    "    if (start > max_length):\n",
    "        print(\"ERROR: Check iterations made! Must be wrong\")\n",
    "        return -1, -1\n",
    "    end = (iteration+1)*BATCH_SIZE\n",
    "    if (end > max_length):\n",
    "        end = max_length\n",
    "    return start, end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we run the training process, the accuracy can be seen after each epoch (all the inputs will be saved to a \"log.txt\" file, so don't worry if you miss out some output). Also remember to change the path to the same path where you saved the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "log.openlog()\n",
    "\n",
    "# Define the sizes and epoches\n",
    "BATCH_SIZE = 10\n",
    "TEST_BATCH_SIZE = 10\n",
    "n_epochs = 6\n",
    "\n",
    "# This path will be the path used to load the dataset files!\n",
    "files = glob.glob(\"E:\\\\GTAV_AI\\\\merged\\\\*.npz\")\n",
    "\n",
    "\n",
    "actual_file = 0\n",
    "acc_for_files = []\n",
    "for fil in files:\n",
    "    actual_file = actual_file + 1\n",
    "    log.output(\"\\n Loading input: \"+fil)\n",
    "    with np.load(fil) as data:\n",
    "         training_data = data['arr_0']\n",
    "\n",
    "    #print(\"\\n Balancing data...\")\n",
    "    #number = number_instances_per_class(training_data)\n",
    "    #training_data = np.array(balance_data(number))\n",
    "\n",
    "    log.output(\"\\n Reshaping data...\")\n",
    "    np.random.shuffle(training_data)\n",
    "    train = training_data[0:int(len(training_data)*0.90)]\n",
    "    test = training_data[int(len(training_data)*0.90):len(training_data)]\n",
    "    del training_data\n",
    "    X_train = reshape_custom_X(train[:, 0:5])\n",
    "    y_train = reshape_custom_y(train[:, 5:6])\n",
    "    X_test = reshape_custom_X(test[:, 0:5])\n",
    "    y_test= reshape_custom_y(test[:, 5:6])\n",
    "    del train, test\n",
    "    log.output(\"\\n Training...\")\n",
    "    data_length = len(X_train)\n",
    "    n_batch = get_num_batches(data_length, BATCH_SIZE)\n",
    "    for epoch in range(n_epochs):\n",
    "        for iteration in range(n_batch):\n",
    "            start, end = get_start_end(iteration, BATCH_SIZE, data_length)\n",
    "            model.fit(x=X_train[start:end], y=y_train[start:end], epochs=1, verbose=0)\n",
    "            clear_output(wait=True)\n",
    "            log.output('\\n => File : ' + str(actual_file) + ' of ' + str(len(files)))\n",
    "            log.output('\\n ==> EPOCH : ' + str(epoch+1) + ' of ' + str(n_epochs))\n",
    "            log.output('\\n ===> Iteration: ' + str(iteration+1) + ' of ' + str(n_batch))\n",
    "            #score = model.evaluate(X_train[start:end], y_train[start:end], verbose=0)\n",
    "            #log.output(\"\\n Train batch accuracy: %.2f%%\" % (score[1]*100))\n",
    "            stdout.flush()\n",
    "        prec = np.zeros((9))\n",
    "        for i in range(len(y_test)):\n",
    "            p = model.predict_classes(X_test[i:i+1])\n",
    "            prec[p] = prec[p] + 1\n",
    "        log.output(\"\\n ==> Predictions:\"+str(prec))\n",
    "        stdout.flush()   \n",
    "    total_acc = 0.0\n",
    "    num = 0\n",
    "    data_length = len(X_test)\n",
    "    n_batch = get_num_batches(data_length, TEST_BATCH_SIZE)\n",
    "    for iteration in range(n_batch):\n",
    "        start, end = get_start_end(iteration, TEST_BATCH_SIZE, data_length)\n",
    "        score = model.evaluate(X_test[start:end], y_test[start:end], verbose=0)\n",
    "        log.output(\"\\n => Batch %s: %.2f%%\" % (model.metrics_names[1], score[1]*100))\n",
    "        total_acc = total_acc + score[1]\n",
    "        num = num + 1\n",
    "    total_acc = total_acc/float(num)\n",
    "    log.output(\"\\n ==> Total acc for file %s: %.2f%%\" % (fil, total_acc*100))\n",
    "    acc_for_files.append(total_acc*100)\n",
    "\n",
    "\n",
    "    prec = np.zeros((9))\n",
    "    for i in range(len(y_test)):\n",
    "        p = model.predict_classes(X_test[i:i+1])\n",
    "        prec[p] = prec[p] + 1\n",
    "    log.output(\"\\n ==> Predictions:\"+str(prec))\n",
    "    time.sleep(5)\n",
    "\n",
    "for acc in range(len(acc_for_files)):\n",
    "    log.output(\"\\n ==> Total acc after file %d: %.2f%%\" % (acc, acc_for_files[acc]))\n",
    "    \n",
    "    \n",
    "log.closelog()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we save the learned network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import h5py as h5py\n",
    "model.save('E:\\\\GTAV_AI\\\\'+selected_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Run our model in the game"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it is time to test our model in the game, we will reduce the amount of VRAM tensorflow can use so that the game has some VRAM in spare. We will also import the library needed to send inputs to the game:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let the game have some VRAM (needed or the game will crash)\n",
    "from keras.backend.tensorflow_backend import set_session\n",
    "config = tf.ConfigProto()\n",
    "config.gpu_options.per_process_gpu_memory_fraction = 0.4\n",
    "set_session(tf.Session(config=config))\n",
    "\n",
    "# For controlling the game\n",
    "from inputsHandler import select_key\n",
    "from tkinter import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def key_press(key):\n",
    "    if key == 1:\n",
    "        return'A'\n",
    "    if key == 2:\n",
    "        return'D'\n",
    "    if key == 3:\n",
    "        return'W'\n",
    "    if key == 4:\n",
    "        return'S'\n",
    "    if key == 5:\n",
    "        return'AW'\n",
    "    if key == 6:\n",
    "        return'AS'\n",
    "    if key == 7:\n",
    "        return'DW'\n",
    "    if key == 8:\n",
    "        return'DS'\n",
    "    return 'none'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This fuction calculates the Mean Squared Error between 2 images. Is used to detect if the car is stuck somewhere. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mse(imageA, imageB):\n",
    "\n",
    "    err = np.sum((imageA.astype(\"float\") - imageB.astype(\"float\")) ** 2)\n",
    "    err /= float(imageA.shape[0] * imageA.shape[1])\n",
    "    return err"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define the run function for our network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Run Configurations\n",
    "show_current_control = False #It will show a windows with a message indicating if the car is currently be controlled by\n",
    "                            #Network  or by a Human\n",
    "    \n",
    "show_whatAIsees = False #It will show the 5 images that the netowrk uses the predict the output \n",
    "\n",
    "enable_evasion = False #If the program detects that the car is not moving (for example because it is stuck facing a wall and\n",
    "                        #the network is not able to return to the road) It will make the car move backwards for a second."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_IA():\n",
    "    global fps\n",
    "    global front_buffer\n",
    "    global back_buffer\n",
    "    global seq\n",
    "    global key_out\n",
    "    global num\n",
    "    \n",
    "    model = load_model('E:\\\\GTAV_AI\\\\'+selected_model)\n",
    "    \n",
    "    training_data = []\n",
    "    threads = list()\n",
    "    th_img = threading.Thread(target=img_thread)\n",
    "    th_seq = threading.Thread(target=image_sequencer_thread)\n",
    "    threads.append(th_img)\n",
    "    threads.append(th_seq)\n",
    "    th_img.start()\n",
    "    time.sleep(1)\n",
    "    th_seq.start()\n",
    "    time.sleep(4)\n",
    "    \n",
    "    last_num = 0\n",
    "    \n",
    "    last_time = time.time()\n",
    "    \n",
    "    if show_current_control:\n",
    "        root = Tk()\n",
    "        var = StringVar()\n",
    "        var.set('IA CONDUCIENDO')\n",
    "        l = Label(root, textvariable = var, fg='green', font=(\"Courier\", 44))\n",
    "        l.pack()\n",
    "\n",
    "    \n",
    "    while True:\n",
    "       \n",
    "        img_seq = seq.copy()\n",
    "        while len(img_seq) != 5 or last_num==num:\n",
    "            del img_seq\n",
    "            img_seq = seq.copy()\n",
    "        last_num = num\n",
    "        array=[]\n",
    "        array.append([img_seq[0],img_seq[1],img_seq[2],img_seq[3],img_seq[4]])\n",
    "        NNinput = np.array(array)\n",
    "        \n",
    "        x = reshape_custom_X(NNinput[:,0:5],0)\n",
    "        p = model.predict_classes(x)\n",
    "        \n",
    "        if not 'J' in key_check():\n",
    "            select_key(p[0])\n",
    "            if show_current_control:\n",
    "                var.set('IA CONDUCIENDO')\n",
    "                l.config(fg='green')\n",
    "                root.update()\n",
    "        else:\n",
    "            if show_current_control:\n",
    "                var.set('CONTROL MANUAL')\n",
    "                l.config(fg='red')\n",
    "                root.update()\n",
    "\n",
    "        #This is used to detect if the car is stuck somewhere (for example facing a wall) and the network does not know what to do. It will move the car\n",
    "        #backward for a second.\n",
    "        \n",
    "        if enable_evasion:\n",
    "            score = mse(img_seq[0],img_seq[4])\n",
    "            if score < 1000:\n",
    "                if show_current_control:\n",
    "                    var.set('MANIOBRA DE EVASIÓN')\n",
    "                    l.config(fg='blue')\n",
    "                    root.update()\n",
    "                select_key(4)\n",
    "                time.sleep(1)\n",
    "                if np.random.rand()>0.5:\n",
    "                    select_key(6)\n",
    "                else:\n",
    "                    select_key(8)\n",
    "                time.sleep(0.2)\n",
    "                if show_current_control:\n",
    "                    var.set('IA CONDUCIENDO')\n",
    "                    l.config(fg='green')\n",
    "                    root.update()\n",
    "\n",
    "        time_act = time.time()\n",
    "        clear_output(wait=True)\n",
    "        stdout.write('Recording at {} FPS \\n'.format(fps))\n",
    "        stdout.write('Images in sequence {} \\n'.format(len(img_seq)))\n",
    "        stdout.write('Keys pressed: ' + key_press(p[0]) + '\\n')\n",
    "        stdout.write('Actions per second: ' + str(1/(time_act-last_time)) + '\\n')\n",
    "        if enable_evasion:\n",
    "            stdout.write('Diference from img 1 to img 5: ' + str(score))\n",
    "        stdout.flush()\n",
    "        last_time = time.time()\n",
    "        \n",
    "        if show_whatAIsees:\n",
    "            cv2.imshow('window1',np.array(img_seq[0])) \n",
    "            cv2.imshow('window2',np.array(img_seq[1]))\n",
    "            cv2.imshow('window3',np.array(img_seq[2]))\n",
    "            cv2.imshow('window4',np.array(img_seq[3]))\n",
    "            cv2.imshow('window5',np.array(img_seq[4]))\n",
    "        \n",
    "        if cv2.waitKey(1) & 0xFF == ord('q'):\n",
    "            cv2.destroyAllWindows()\n",
    "            break\n",
    "     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we run the network in the game:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_IA()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}