Deep_Learning_Project-Pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align='center' style=\"margin-bottom: 0px\"> An end to end implementation of a Machine Learning pipeline </h1>\n",
    "<h4 align='center' style=\"margin-top: 0px\"> SPANDAN MADAN</h4>\n",
    "<h4 align='center' style=\"margin-top: 0px\"> Visual Computing Group, Harvard University</h4>\n",
    "<h4 align='center' style=\"margin-top: 0px\"> Computer Science and Artificial Intelligence Laboratory, MIT</h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 align='center' style=\"margin-top: 0px\"><a href='https://github.com/Spandan-Madan/DeepLearningProject'>Link to Github Repo</a></h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 1. Introduction\n",
    "\n",
    "### Background\n",
    "In the fall of 2016, I was a Teaching Fellow (Harvard's version of TA) for the graduate class on \"Advanced Topics in Data Science (CS209/109)\" at Harvard University. I was in-charge of designing the class project given to the students, and this tutorial has been built on top of the project I designed for the class.\n",
    "\n",
    "### Why write yet another Tutorial on Machine Learning and Deep Learning?\n",
    "As a researcher on Computer Vision, I come across new blogs and tutorials on ML (Machine Learning) every day. However, most of them are just focussing on introducing the syntax and the terminology relevant to the field. For example - a 15 minute tutorial on Tensorflow using MNIST dataset, or a 10 minute intro to Deep Learning in Keras on Imagenet. \n",
    "\n",
    "While people are able to copy paste and run the code in these tutorials and feel that working in ML is really not that hard, it doesn't help them at all in using ML for their own purposes. For example, they never introduce you to how you can run the same algorithm on your own dataset. Or, how do you get the dataset if you want to solve a problem. Or, which algorithms do you use - Conventional ML, or Deep Learning? How do you evaluate your models performance? How do you write your own model, as opposed to choosing a ready made architecture? All these form fundamental steps in any Machine Learning pipeline, and it is these steps that take most of our time as ML practitioners. \n",
    "\n",
    "This tutorial breaks down the whole pipeline, and leads the reader through it step by step in an hope to empower you to actually use ML, and not just feel that it was not too hard. Needless to say, this will take much longer than 15-30 minutes. I believe a weekend would be a good enough estimate.\n",
    "\n",
    "### About the Author\n",
    "\n",
    "I am <a href=\"http://spandanmadan.com/\">Spandan Madan</a>, a graduate student at Harvard University working on Computer Vision. My research work is supervised collaboratively by Professor Hanspeter Pfister at Harvard, and Professor Aude Oliva at MIT. My current research focusses on using Computer Vision and Natural Language Techniques in tandem to build systems capable of reasoning using text and visual elements simultaneusly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 2. Project Outline : Multi-Modal Genre Classification for Movies "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wow, that title sounds like a handful, right? Let's break it down step by step.\n",
    "\n",
    "### Q.1. what do we mean by Classification?\n",
    "\n",
    "In machine learning, the task of classification means to use the available data to learn a <i>function</i> which can assign a category to a data point. For example, assign a genre to a movie, like \"Romantic Comedy\", \"Action\", \"Thriller\". Another example could be automatically assigning a category to news articles, like \"Sports\" and \"Politics\". \n",
    "\n",
    "### More Formally \n",
    "\n",
    "#### Given:\n",
    "- A data point $x_i$ \n",
    "- A set of categories $y_1,y_2...y_n$ that $x_i$ can belong to. <br>\n",
    "\n",
    "#### Task : \n",
    "Predict the correct category $y_k$ for a new data point $x_k$ not present in the given dataset.\n",
    "\n",
    "#### Problem : \n",
    "We don't know how the $x$ and $y$ are related mathematically.\n",
    "\n",
    "#### Assumption : \n",
    "We assume there exists a function $f$ relating $x$ and $y$ i.e. $f(x_i)=y_i$\n",
    "\n",
    "#### Approach : \n",
    "Since $f$ is not known, we learn a function $g$, which approximates $f$. \n",
    "\n",
    "#### Important consideration : \n",
    "- If $f(x_i)=g(x_i)=y_i$ for all $x_i$, then the two functions $f$ and $g$ are exactly equal. Needless to say, this won't realistically ever happen, and we'll only be able to approximate the true function $f$ using $g$. This means, sometimes the prediction $g(x_i)$ will not be correct. And essentially, our whole goal is to find a $g$ which makes a really low number of such errors. That's basically all that we're trying to do. \n",
    "\n",
    "- For the sake of completeness, I should mention that this is a specific kind of learning problem which we call \"Supervised Learning\". Also, the idea that $g$ approximates $f$ well for data not present in our dataset is called \"Generalization\". It is absolutely paramount that our model generalizes, or else all our claims will only be true about data we already have and our predictions will not be correct. \n",
    "\n",
    "- We will look into generalization a little bit more a little ahead in the tutorial. \n",
    "\n",
    "- Finally, There are several other kinds, but supervised learning is the most popular and well studied kind."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q.2. What's Multi-Modal Classification then?\n",
    "\n",
    "In the machine learning community, the term Multi-Modal is used to refer to multiple <i>kinds</i> of data. For example, consider a YouTube video. It can be thought to contain 3 different modalities -\n",
    "\n",
    "- The video frames (visual modality)\n",
    "- The audio clip of what's being spoken (audio modality)\n",
    "- Some videos also come with the transcription of the words spoken in the form of subtitles (textual modality)\n",
    "\n",
    "Consider, that I'm interested in classifying a song on YouTube as pop or rock. You can use any of the above 3 modalities to predict the genre - The video, the song itself, or the lyrics. But, needless to say, you can predict it much better if you could use all three simultaneously. This is what we mean by multi-modal classification. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# For this project, we will be using visual and textual data to classify movie genres."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project Outline\n",
    "\n",
    "- **Scraping a dataset** : The first step is to build a rich data set. We will collect textual and visual data for each movie.\n",
    "- **Data pre-processing**\n",
    "- **Non-deep Machine Learning models : Probabilistic and Max-Margin Classifiers.**\n",
    "- **Intuitive theory behind Deep Learning**\n",
    "- **Deep Models for Visual Data**\n",
    "- **Deep Models for Text**\n",
    "- **Potential Extensions**\n",
    "- **Food for Thought**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 3. Building your very own DataSet.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For any machine learning algorithm to work, it is imperative that we collect data which is \"representative\". Now, let's take a moment to discuss what the word representative mean.\n",
    "\n",
    "### What data is good data? OR What do you mean by data being \"representative\"?\n",
    "Let's look at this from first principles. Mathematically, the premise of machine learning (to be precise, the strand of machine learning we'll be working with here) is that given input variable X, and an output variable y, **IF** there is a function such that g(X)=y, then if g is unknown, we can \"learn\" a function f which approximates g. At the very heart, its not at all different from what you may have earlier studied as \"curve fitting\". For example, if you're trying to predict someone's movie preferences then X can be information about the person's gender, age, nationality and so on, while y can be the genre they most like to listen to!\n",
    "\n",
    "Let's do a thought experiment. Consider the same example - I'm trying to predict people's movie preferences. I walk into a classroom today, and collect information about some students and their movie preferences. Now, I use that data to build a model. How well do you think I can predict my father's movie preferences? The answer is - probably not very well. Why? Intuitively, there was probably no one in the classroom who was my father's age. My model can tell me that as people go from age 18 to 30, they have a higher preference for documentaries over superhero movies. But does this trend continue at 55? Probably, they may start liking family dramas more. Probably they don't. In a nutshell, we cannot say with certainty, as our data tells us nothing about it. So, if the task was to make predictions about ANYONE's movie preferences, then the data collected from just undergraduates is NOT representative.\n",
    "\n",
    "Now, let's see why this makes sense Mathematically. Look at the graph below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"files/contour.png\">\n",
    "<center>Fig.1: Plot of a function we are trying to approximate(<a href=\"http://www.jzy3d.org/js/slider/images/ContourPlotsDemo.png\">source</a>)</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we consider that the variable plotted on the vertical axis is $y$, and the values of the 2 variables on the horizontal axes make the input vector $X$, then, our hope is that we are able to find a function $f$ which can approximate the function plotted here. If all the data I collect is such that $x_1$ belongs to (80,100) and $x_2$ belongs to (80,100), the learned function will only be able to learn the \"yellow-green dipping bellow\" part of the function.  Our function will never be able to predict the behavior in the \"red\" regions of the true function. So, in order to be able to learn a good function, we need data sampled from a diverse set of values of $x_1$ and x2. That would be representative data to learn this contour."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore, we want to collect data which is representative of all possible movies that we want to make predictions about. Or else (which is often the case), we need to be aware of the limitations of the model we have trained, and the predictions we can make with confidence. The easiest way to do this is to only make predictions about the domain of data we collected the training data from. For example, in our case, let us start by assuming that our model will predict genres for only English movies. Now, the task is to collect data about a diverse collection of movies.\n",
    "\n",
    "So how do we get this data then? Neither google, nor any university has released such a dataset. We want to collect visual and textual data about these movies. The simple answer is to scrape it from the internet to build our own dataset. For the purpose of this project, we will use movie posters as our visual data, and movie plots as textual data. Using these, we will build a model that can predict movie genres! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# We will be scraping data from 2 different movie sources - IMDB and TMDB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>IMDB:http://www.imdb.com/</h3>\n",
    "\n",
    "For those unaware, IMDB is the primary source of information about movies on the internet. It is immensely rich with posters, reviews, synopsis, ratings and many other information on every movie. We will use this as our primary data source. \n",
    "\n",
    "<h3>TMDB:https://www.themoviedb.org/</h3>\n",
    "\n",
    "TMDB, or The Movie DataBase, is an open source version of IMDB, with a free to use API that can be used to collect information. You do need an API key, but it can be obtained for free by just making a request after making a free account."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Note - \n",
    "IMDB gives some information for free through the API, but doesn't release other information about movies. Here, we will keep it legal and only use information given to us for free and legally. However, scraping does reside on the moral fence, so to say. People often scrape data which isn't exactly publicly available for use from websites. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "import torchvision\n",
    "import urllib2\n",
    "import requests\n",
    "import json\n",
    "import imdb\n",
    "import time\n",
    "import itertools\n",
    "import wget\n",
    "import os\n",
    "import tmdbsimple as tmdb\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "import pickle\n",
    "import sys\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Here is a broad outline of technical steps to be done for data collection\n",
    "\n",
    "\n",
    "* Sign up for TMDB (themoviedatabase.org), and set up API to scrape movie posters for above movies.\n",
    "* Set up and work with TMDb to get movie information from their database\n",
    "* Do the same for IMDb\n",
    "* Compare the entries of IMDb and TMDb for a movie\n",
    "* Get a listing and information of a few movies\n",
    "* Think and ponder over the potential challenges that may come our way, and think about interesting questions we can answer given the API's we have in our hands.\n",
    "* Get data from the TMDb\n",
    "\n",
    "Let's go over each one of these one by one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Signing up for TMDB and getting set up for getting movie metadata.\n",
    "\n",
    "* Step 1. Head over to [tmdb.org](https://www.themoviedb.org/?language=en) and create a new account there by signing up.\n",
    "* Step 2. Click on your account icon on the top right, then from drop down menu select \"Settings\".\n",
    "* Step 3. On the settings page, you will see the option \"API\" on the left pane. Click on that.\n",
    "* Step 4. Apply for a new developer key. Fill out the form as required. The fields \"Application Name\" and \"Application URL\" are not important. Fill anything there.\n",
    "* Step 5. It should generate a new API key for you and you should also receive a mail.\n",
    "\n",
    "Now that you have the API key for TMDB, you can query using TMDB. Remember, it allows only 40 queries per 10 seconds.\n",
    "\n",
    "An easy way to respect this is to just have a call to <i>time.sleep(1)</i> after each iteration. This is also being very nice to the server.\n",
    "\n",
    "If you want to try and maximize your throughput you can embed every TMDB request in a nested try except block. If the first try fails, the second try first uses python's sleep function to give it a little rest, and then try again to make a request. Something like this -\n",
    "\t\n",
    "~~~~\n",
    "try:\n",
    "    search.movie(query=movie) #An API request\n",
    "except:\n",
    "    try:\n",
    "        time.sleep(10) #sleep for a bit, to give API requests a rest.\n",
    "        search.movie(query=<i>movie_name</i>) #Make second API request\n",
    "    except:\n",
    "        print \"Failed second attempt too, check if there's any error in request\"\n",
    "~~~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using TMDB using the obtained API Key to get movie information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have made these functions which make things easy. Basically, I'm making use of a library called tmdbsimple which makes TMDB using even easier. This library was installed at the time of setup.\n",
    "\n",
    "However, if you  want to avoid the library, it is also easy enough to load the API output directly into a dictionary like this without using tmdbsimple:\n",
    "\n",
    "~~~\n",
    "url = 'https://api.themoviedb.org/3/movie/1581?api_key=' + api_key\n",
    "data = urllib2.urlopen(url).read()\n",
    "\n",
    "# create dictionary from JSON \n",
    "dataDict = json.loads(data)\n",
    "~~~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Folder already exists\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "# set here the path where you want the scraped folders to be saved!\n",
    "poster_folder='posters_final/'\n",
    "if poster_folder.split('/')[0] in os.listdir('./'):\n",
    "    print('Folder already exists')\n",
    "else:\n",
    "    os.mkdir('./'+poster_folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'posters_final/'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "poster_folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For the purpose of this example, i will be working with the 1999 Sci-Fi movie - \"The Matrix\"!\n",
    "import tmdbsimple as tmdb\n",
    "api_key = '' #Enter your own API key here to run the code below. \n",
    "# Generate your own API key as explained above :)\n",
    "\n",
    "\n",
    "tmdb.API_KEY = api_key #This sets the API key setting for the tmdb object\n",
    "search = tmdb.Search() #this instantiates a tmdb \"search\" object which allows your to search for the movie\n",
    "import os.path\n",
    "# These functions take in a string movie name i.e. like \"The Matrix\" or \"Interstellar\"\n",
    "# What they return is pretty much clear in the name - Poster, ID , Info or genre of the Movie!\n",
    "def grab_poster_tmdb(movie):\n",
    "    response = search.movie(query=movie)\n",
    "    id=response['results'][0]['id']\n",
    "    movie = tmdb.Movies(id)\n",
    "    posterp=movie.info()['poster_path']\n",
    "    title=movie.info()['original_title']\n",
    "    if os.path.isfile(poster_folder+title+'.jpg '):\n",
    "        return\n",
    "    url='image.tmdb.org/t/p/original'+posterp\n",
    "    title='_'.join(title.split(' '))\n",
    "    strcmd='wget -O '+poster_folder+title+'.jpg '+url\n",
    "    os.system(strcmd)\n",
    "\n",
    "def get_movie_id_tmdb(movie):\n",
    "    response = search.movie(query=movie)\n",
    "    movie_id=response['results'][0]['id']\n",
    "    return movie_id\n",
    "\n",
    "def get_movie_info_tmdb(movie):\n",
    "    response = search.movie(query=movie)\n",
    "    id=response['results'][0]['id']\n",
    "    movie = tmdb.Movies(id)\n",
    "    info=movie.info()\n",
    "    return info\n",
    "\n",
    "def get_movie_genres_tmdb(movie):\n",
    "    response = search.movie(query=movie)\n",
    "    id=response['results'][0]['id']\n",
    "    movie = tmdb.Movies(id)\n",
    "    genres=movie.info()['genres']\n",
    "    return genres"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the above functions have been made to make it easy to get genres, posters and ID, all the information that can be accessed can be seen by calling the function get_movie_info() as shown below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{u'id': 28, u'name': u'Action'}, {u'id': 878, u'name': u'Science Fiction'}]\n"
     ]
    }
   ],
   "source": [
    "print get_movie_genres_tmdb(\"The Matrix\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All the Movie information from TMDB gets stored in a dictionary with the following keys for easy access -\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[u'poster_path',\n",
       " u'production_countries',\n",
       " u'revenue',\n",
       " u'overview',\n",
       " u'video',\n",
       " u'id',\n",
       " u'genres',\n",
       " u'title',\n",
       " u'tagline',\n",
       " u'vote_count',\n",
       " u'homepage',\n",
       " u'belongs_to_collection',\n",
       " u'original_language',\n",
       " u'status',\n",
       " u'spoken_languages',\n",
       " u'imdb_id',\n",
       " u'adult',\n",
       " u'backdrop_path',\n",
       " u'production_companies',\n",
       " u'release_date',\n",
       " u'popularity',\n",
       " u'original_title',\n",
       " u'budget',\n",
       " u'vote_average',\n",
       " u'runtime']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "info=get_movie_info_tmdb(\"The Matrix\")\n",
    "print \"All the Movie information from TMDB gets stored in a dictionary with the following keys for easy access -\"\n",
    "info.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, to get the tagline of the movie we can use the above dictionary key - "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Welcome to the Real World.\n"
     ]
    }
   ],
   "source": [
    "info=get_movie_info_tmdb(\"The Matrix\")\n",
    "print info['tagline']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting movie information from IMDB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know how to get information from TMDB, here's how we can get information about the same movie from IMDB. This makes it possible for us to combine more information, and get a richer dataset. I urge you to try and see what dataset you can make, and go above and beyond the basic things I've done in this tutorial. Due to the differences between the two datasets, you will have to do some cleaning, however both of these datasets are extremely clean and it will be minimal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All the information we can get about this movie from IMDB-\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[u'production managers',\n",
       " u'rating',\n",
       " u'casting directors',\n",
       " u'distributors',\n",
       " u'music department',\n",
       " u'runtimes',\n",
       " u'special effects',\n",
       " u'year',\n",
       " u'color info',\n",
       " u'composers',\n",
       " u'costume designers',\n",
       " u'votes',\n",
       " u'top 250 rank',\n",
       " u'title',\n",
       " u'writer',\n",
       " u'editors',\n",
       " u'visual effects',\n",
       " u'cinematographers',\n",
       " u'writers',\n",
       " u'camera department',\n",
       " u'directors',\n",
       " u'certificates',\n",
       " u'countries',\n",
       " u'country codes',\n",
       " u'language codes',\n",
       " u'production designers',\n",
       " u'director',\n",
       " u'casting department',\n",
       " u'special effects companies',\n",
       " u'assistant directors',\n",
       " u'sound mix',\n",
       " u'location management',\n",
       " u'genres',\n",
       " u'miscellaneous',\n",
       " u'production companies',\n",
       " u'producers',\n",
       " u'set decorators',\n",
       " u'original air date',\n",
       " u'costume departmen',\n",
       " u'akas',\n",
       " u'aspect ratio',\n",
       " u'sound department',\n",
       " u'stunts',\n",
       " u'kind',\n",
       " u'make up department',\n",
       " u'other companies',\n",
       " u'art department',\n",
       " u'languages',\n",
       " u'transportation department',\n",
       " u'plot outline',\n",
       " u'plot',\n",
       " u'cast',\n",
       " u'synopsis',\n",
       " u'art directors',\n",
       " u'animation department',\n",
       " u'cover url',\n",
       " u'editorial department',\n",
       " u'canonical title',\n",
       " u'long imdb title',\n",
       " u'long imdb canonical title',\n",
       " u'smart canonical title',\n",
       " u'smart long imdb canonical title',\n",
       " u'full-size cover url']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "import imdb\n",
    "# Create the IMDB object that will be used to access the IMDb's database.\n",
    "imbd_object = imdb.IMDb() # by default access the web.\n",
    "\n",
    "# Search for a movie (get a list of Movie objects).\n",
    "results = imbd_object.search_movie('The Matrix')\n",
    "#print(len(results))\n",
    "# As this returns a list of all movies containing the word \"The Matrix\", we pick the first element\n",
    "movie = results[0]\n",
    "\n",
    "imbd_object.update(movie)\n",
    "\n",
    "print \"All the information we can get about this movie from IMDB-\"\n",
    "movie.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The genres associated with the movie are -  [u'Action', u'Sci-Fi']\n"
     ]
    }
   ],
   "source": [
    "print \"The genres associated with the movie are - \",movie['genres']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A small comparison of IMDB and TMDB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have both systems running, let's do a very short comparison for the same movie?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The genres for The Matrix pulled from IMDB are - [u'Action', u'Sci-Fi']\n",
      "The genres for The Matrix pulled from TMDB are - [{u'id': 28, u'name': u'Action'}, {u'id': 878, u'name': u'Science Fiction'}]\n"
     ]
    }
   ],
   "source": [
    "print \"The genres for The Matrix pulled from IMDB are -\",movie['genres']\n",
    "print \"The genres for The Matrix pulled from TMDB are -\",get_movie_genres_tmdb(\"The Matrix\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, both the systems are correct, but the way they package information is different. TMDB calls it \"Science Fiction\" and has an ID for every genre. While IMDB calls it \"Sci-Fi\". Thus, it is important to keep track of these things when making use of both the datasets simultaneously."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know how to scrape information for one movie, let's take a bigger step towards scraping multiple movies?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with multiple movies : Obtaining Top 20 movies from TMDB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first instantiate an object that inherits from class Movies from TMDB. Then We use the **popular()** class method (i.e. function) to get top movies. To get more than one page of results, the optional page argument lets us see movies from any specified page number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20\n"
     ]
    }
   ],
   "source": [
    "all_movies=tmdb.Movies()\n",
    "top_movies=all_movies.popular()\n",
    "\n",
    "# This is a dictionary, and to access results we use the key 'results' which returns info on 20 movies\n",
    "print(len(top_movies['results']))\n",
    "top20_movs=top_movies['results']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at one of these movies. It's the same format as above, as we had information on the movie \"The Matrix\", as you can see below. It's a dictionary which can be queried  for specific information on that movie"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here is all the information you can get on this movie - \n",
      "{u'poster_path': u'/2uNW4WbgBXL25BAbXGLnLqX71Sw.jpg', u'title': u'Venom', u'overview': u'When Eddie Brock acquires the powers of a symbiote, he will have to release his alter-ego \"Venom\" to save his life.', u'release_date': u'2018-10-03', u'popularity': 371.566, u'original_title': u'Venom', u'backdrop_path': u'/VuukZLgaCrho2Ar8Scl9HtV3yD.jpg', u'vote_count': 1576, u'video': False, u'adult': False, u'vote_average': 6.6, u'genre_ids': [878, 28, 80, 28, 27], u'id': 335983, u'original_language': u'en'}\n",
      "\n",
      "\n",
      "The title of the first movie is -  Venom\n"
     ]
    }
   ],
   "source": [
    "first_movie=top20_movs[0]\n",
    "print \"Here is all the information you can get on this movie - \"\n",
    "print first_movie\n",
    "print \"\\n\\nThe title of the first movie is - \", first_movie['title']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print out top 5 movie's titles! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Venom\n",
      "Halloween\n",
      "A Star Is Born\n",
      "The Seven Deadly Sins: Prisoners of the Sky\n",
      "Incredibles 2\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(top20_movs)):\n",
    "    mov=top20_movs[i]\n",
    "    title=mov['title']\n",
    "    print title\n",
    "    if i==4:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Yes, I know. I'm a little upset too seeing The Nun above Avengers: Infinity War! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Moving on, we can get their genres the same way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[878, 28, 80, 28, 27]\n",
      "[27]\n",
      "[18, 10402, 10749]\n",
      "[28, 12, 14, 16]\n",
      "[28, 12, 16, 10751]\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(top20_movs)):\n",
    "    mov=top20_movs[i]\n",
    "    genres=mov['genre_ids']\n",
    "    print genres\n",
    "    if i==4:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, TMDB doesn't want to make your job as easy as you thought. Why these random numbers? Want to see their genre names? Well, there's the Genre() class for it. Let's get this done!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tmdb genre object!\n",
    "genres=tmdb.Genres()\n",
    "# the list() method of the Genres() class returns a listing of all genres in the form of a dictionary.\n",
    "list_of_genres=genres.list()['genres']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's convert this list into a nice dictionary to look up genre names from genre IDs!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "Genre_ID_to_name={}\n",
    "for i in range(len(list_of_genres)):\n",
    "    genre_id=list_of_genres[i]['id']\n",
    "    genre_name=list_of_genres[i]['name']\n",
    "    Genre_ID_to_name[genre_id]=genre_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's re-print the genres of top 20 movies? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Venom [u'Science Fiction', u'Action', u'Crime', u'Action', u'Horror']\n",
      "Halloween [u'Horror']\n",
      "A Star Is Born [u'Drama', u'Music', u'Romance']\n",
      "The Seven Deadly Sins: Prisoners of the Sky [u'Action', u'Adventure', u'Fantasy', u'Animation']\n",
      "Incredibles 2 [u'Action', u'Adventure', u'Animation', u'Family']\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(top20_movs)):\n",
    "    mov=top20_movs[i]\n",
    "    title=mov['title']\n",
    "    genre_ids=mov['genre_ids']\n",
    "    genre_names=[]\n",
    "    for id in genre_ids:\n",
    "        genre_name=Genre_ID_to_name[id]\n",
    "        genre_names.append(genre_name)\n",
    "    print title,genre_names\n",
    "    if i==4:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 4 - Building a dataset to work with : Let's take a look at the top 1000 movies from the database"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Making use of the same api as before, we will just pull results from the top 50 pages. As mentioned earlier, the \"page\" attribute of the command top_movies=all_movies.popular() can be used for this purpose."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please note: Some of the code below will store the data into python \"pickle\" files so that it can be ready directly from memory, as opposed to being downloaded every time. Once done, you should comment out any code which generated an object that was pickled and is no longer needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_movies=tmdb.Movies()\n",
    "top_movies=all_movies.popular()\n",
    "\n",
    "# This is a dictionary, and to access results we use the key 'results' which returns info on 20 movies\n",
    "len(top_movies['results'])\n",
    "top20_movs=top_movies['results']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pulling movie list, Please wait...\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "# Comment out this cell once the data is saved into pickle file.\n",
    "all_movies=tmdb.Movies()\n",
    "top1000_movies=[]\n",
    "print('Pulling movie list, Please wait...')\n",
    "for i in range(1,51):\n",
    "    if i%15==0:\n",
    "        time.sleep(7)\n",
    "    movies_on_this_page=all_movies.popular(page=i)['results']\n",
    "    top1000_movies.extend(movies_on_this_page)\n",
    "len(top1000_movies)\n",
    "f3=open('movie_list.pckl','wb')\n",
    "pickle.dump(top1000_movies,f3)\n",
    "f3.close()\n",
    "print('Done!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "f3=open('movie_list.pckl','rb')\n",
    "top1000_movies=pickle.load(f3)\n",
    "f3.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pairwise analysis of Movie Genres"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As our dataset is multi label, simply looking at the distribution of genres is not sufficient. It might be beneficial to see which genres co-occur, as it might shed some light on inherent biases in our dataset. For example, it would make sense if romance and comedy occur together more often than documentary and comedy. Such inherent biases tell us that the underlying population we are sampling from itself is skewed and not balanced. We may then take steps to account for such problems. Even if we don't take such steps, it is important to be aware that we are making the assumption that an unbalanced dataset is not hurting our performance and if need be, we can come back to address this assumption. Good old scientific method, eh?\n",
    "\n",
    "So for the top 1000 movies let's do some pairwise analysis for genre distributions. Our main purpose is to see which genres occur together in the same movie. So, we first define a function which takes a list and makes all possible pairs from it. Then, we pull the list of genres for a movie and run this function on the list of genres to get all pairs of genres which occur together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function just generates all possible pairs of movies\n",
    "def list2pairs(l):\n",
    "    # itertools.combinations(l,2) makes all pairs of length 2 from list l.\n",
    "    pairs = list(itertools.combinations(l, 2))\n",
    "    # then the one item pairs, as duplicate pairs aren't accounted for by itertools\n",
    "    for i in l:\n",
    "        pairs.append([i,i])\n",
    "    return pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned, now we will pull genres for each movie, and use above function to count occurrences of when two genres occurred together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get all genre lists pairs from all movies\n",
    "allPairs = []\n",
    "for movie in top1000_movies:\n",
    "    allPairs.extend(list2pairs(movie['genre_ids']))\n",
    "    \n",
    "nr_ids = np.unique(allPairs)\n",
    "visGrid = np.zeros((len(nr_ids), len(nr_ids)))\n",
    "for p in allPairs:\n",
    "    visGrid[np.argwhere(nr_ids==p[0]), np.argwhere(nr_ids==p[1])]+=1\n",
    "    if p[1] != p[0]:\n",
    "        visGrid[np.argwhere(nr_ids==p[1]), np.argwhere(nr_ids==p[0])]+=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the structure we just made. It is a 19X19 structure, as shown below. Also, see that we had 19 Genres. Needless to say, this structure counts the number of simultaneous occurrences of genres in same movie."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(19, 19)\n",
      "19\n"
     ]
    }
   ],
   "source": [
    "print visGrid.shape\n",
    "print len(Genre_ID_to_name.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x11e9d1690>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAE9CAYAAACiDN36AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXecVcX5/9+fpShKUexiwa5YIIhEsQRrjH5ji0aNRjH5\nfokpliQmMeWXYGISjSZG4zfxi8Ru1Gg0lhgVUWxgQRFQ7GLDgogIBBB29/n9MXPhcLnl3LN39967\nPG9e57WnzDMz5+xynjMzz3xGZobjOI7j1IKmWlfAcRzHWXVxJ+Q4juPUDHdCjuM4Ts1wJ+Q4juPU\nDHdCjuM4Ts1wJ+Q4juPUDHdCjuM4Ts1wJ+Q4juPUDHdCjuM4Ts3oWusKrAosuvfSTLIUv/3GxEzl\nTWr9OJPdjMWzM9m9tWBWJruxfXbNZHdh96WZ7ADufO/pzLZZ2H297TPZTZk7I5PdwqWfZrI7deO9\nMtndMndqJruPFs7LZLdWj56Z7I5ae5dMdle//0QmO4Dm1pZsdktmKnOhkaWzX0/9zum27pZtLq8t\neEvIcRzHqRk1d0KSjpBkkgp+Mkq6StLRVS5zuKRh1czTcRynbmhZmn6rMTV3QsDxwKPxZ0cxHKjI\nCUnyrkvHcRqD1tb0W42pqROS1BPYC/g6cFw8J0mXSnpJ0v3A+vH8wZJuTtgOl3RX3D9I0kRJz0i6\nOeaLpDcknRPPT5O0vaT+wKnAdyU9K2nv/NaWpAWJMh6RdAcwPZ47UdKT0fb/JHVp9wflOI5TAWat\nqbdaU+uW0OHAPWb2MvCRpF2BI4HtgAHASSxvsdwPfFbSmvH4WOBGSesCPwMOMLPBwCTge4kyZsfz\nfwHOMrM3gMuAi8xskJk9UqaOg4EzzGxbSTvEcvc0s0FAC3BCG+7fcRyn+nhLKDXHAzfG/Rvj8T7A\nDWbWYmbvAg8AmFkzcA/wxdg1dihwO7A7wWE9JulZ4GRg80QZt8afTwP9M9TxSTPLhSrtD+wKPBXL\n2h/YspCRpJGSJkma9Ne7H8tQrOM4TkasNf1WY2o2ziGpL7AfsLMkA7oABtxWwuxG4DvAHGCSmc2X\nJGCsmRUbU8rFrLZQ/H6biQ5ZUhPQPXHtP8lqA1eb2Y9L1BEAMxsNjIbsIdqO4ziZyBgeXgtq2RI6\nGrjWzDY3s/5mtikwA/gIOFZSF0kbAfsmbB4idI/9D8tbUI8De0raGkDSmpK2LVP2fKBX4vgNQgsH\n4DCgWxG7ccDRknLjVH0lbV4kreM4Tm1oaU6/1ZhaOqHjWbnV8w9gI+AVQiDANcCyGZtm1gLcBXwh\n/sTMPgRGADdImhrTl5sheCdwZC4wAbgc+JykKcAerNj6WYaZTSeMP90Xyxob6+s4jlM3NFJgQs26\n48xs3wLnLklh9x1Cl1zy3APAbgXS9k/sTyKEZhMDIfKnUO+e2P9RTDceGJ+X503ATeXq6TiOUzPq\nIOAgLT73pQPIKr9zWHPBBllZbl6STX6n2bL1I/fo2r18ogKc221RJrtZS+ZnsoMw6JiFrLomLy+Y\nmclucfOSjCVm498LXslkt2DJ4irXpDTdm7K9sma0ZPubaa2DlkImGqje7oQcx3E6Gw0UmOBOyHEc\np7NRBwEHaan1PKGKkdQSAwpyW/+M+ZwpaY3q1s5xHKcO8HlC7cqiqFbQVs4ErgMWViEvx3Gc+qGB\nAhMariVUCEn9o8bbM3EbFs8PlzRe0i2SXpR0fdSmOx3YGHhQ0oMx7V+iwsHzks5J5H2epOmSpkq6\nUFIvSTMkdYvXeyePHcdxao1ZS+qt1jRiS6hHlMwBmGFmRwKzgAPNbLGkbYAbgCExzWeAHYF3gccI\num+XSPoesK+Z5ULJfmpmc6Ig6ThJuwAzCVp225uZSVorqjSMJ8gG/ZMgvHqrma2giS5pJDAS4NC+\nQxnca+v2eBaO4zgrUwfdbGlpxJbQoig8Oig6IAgKB5dLmgbcTNCSy/Gkmb1jYVbWsxTXj/uypGeA\nyQSnNQD4BFgM/FXSUSzvuhsDnBL3TwGuzM/MzEab2RAzG+IOyHGcDqWBBEwbsSVUiO8CHwADCY41\nOXkhud5xQf04SVsAZwG7mdnHkq4CVjezZklDCUKlRxMmye5nZo/FLsDhQBcze64d7slxHCcbdbBY\nXVoasSVUiD7Ae7G181WCGGo5kvpxvQlSPZ9I2oAgC5Rb76iPmd1NcHQDE/bXAH+jQCvIcRynpnh0\nXIfzZ+Afkk4iLPeQRmpgNHCPpHfNbF9Jk4EXgbcJY0cQnNTtklYnTJpPrlN0PXAuYfzJcRynfqiD\nbra0NJwTMrOeBc69wopacAW136LuXG7/T8CfEscjihQ5tMj5vYBbzGxuupo7juN0EHXQwklLwzmh\nekDSnwhddoekST+p9eNM5WTVgDt19XIrWRRm1JzHM9nNX5JNA+7Jua9mshu6VscHemTVnPtoUXad\nu47k7fmzMtk1KVuPflNTNrs5ixdkshvUZ+1Mdg9Ygy4F5i2hzo2ZnVbrOjiO4xSlgZxQXQQmSDpC\nkkkqtw4QksZIGlAuXYp8+kv6SuJ4iKSyS0k4juPUO9ayNPVWa+rCCREWuHs0/iyJmf13XFyurfQH\nljkhM5tkZqdXIV/HcZza0kDRcTV3QjEMei/g6wT1gaJyO/HaeElD4v4CSRdEqZ37JQ2N11+XdFhM\nU1DSBzgP2DuKoH43lnlXtOkr6Z9RqufxqJ6ApFGSrkiU4U7LcZz6o4Emq9bcCQGHA/fE1U4/krRr\nPP8ZgsjoAGBLYM8CtmsCD5jZjoR5P+cCBxKkdn4Z0+QkfQYDxwK5LrezgUei8sJFefmeA0w2s12A\nnxDmBOXYHvg8IWruF64Z5zhO3VHFlpCk1SU9KWlKUltT0haSnpD0qqSbJHWP51eLx6/G6/1L5V8P\nTuh44Ma4fyPLu+TSyO0sIcwLApgGPBQ13KYl0peS9CnGXsC1sGzp8HUk9Y7X/mVmn0bNuVnABoUy\nkDQyCqJOemvBWymKdBzHqRLVbQl9SlCKGQgMAg6WtDtwPnCRmW0NfEzozSL+/DievyimK0pNnZCk\nvsB+wBhJbwA/AL5MmBhaVm4HWGq2LIayNWcTHVcufVLSZwiQbS3q5aSp1wracZv13KyNRTqO41RA\nS3P6rQwWyMXGd4ubEd7dt8TzVwNHxP3D4zHx+v654ZRC1LoldDRwrZltbmb9zWxTYAawdxXLKCbp\nk5TtyecR4AQI41PAbDObV8U6OY7jtB8VtISSvTZxG5mfnaQucfWCWcBY4DVgrpnlvNg7QL+434+g\nPEO8/gmwTrGq1nqe0PGs3FT7B/BNwk1Wg2KSPlOBFklTgKsI6tk5RgFXSJpKUM4+uUp1cRzHaX8q\niHozs9EEGbNSaVqAQZLWAm4jjI1XhZo6ITPbt8C5S1gePJA7l5TbGZ7Y75nYH5Vn0zP+LCbps5TQ\nnEwyPl6bw/KmZTLP/DJ2WvmuHMdxakw7Rb2Z2dy4EOgewFqSusbWziaE9deIPzcF3pHUldAb9VGx\nPGvdElolmLE4m/xOc8ZVD7PK78x6475Mdltte3g2uzU2zGT3cUs2mSCAjXv2zWQ3a+EnmezO3iBb\nz/ITrXMy2Y39YGomuy37bJTJrkdTtiHWl+fNLJ+oAJ9bN9s89b/Nm5bJLuvfC8B7C7L9DqtCFef/\nSFqPMP4+V1IPQgTy+cCDhCGVGwm9RbdHkzvi8cR4/YHE2P1KuBNyHMfpbFS3JbQRcHVcdboJ+LuZ\n3SVpOnCjpHMJwxl/jen/Clwr6VVgDnH+ZzHcCTmO43Q2UkS9pcXMphLmbeaff50CqwyY2WLgmLT5\ndxonJKmFMD+oG9BMmGB6UYyKcxzHWXWoAyWEtHQaJwQsMrNBAJLWJ6x62hv4RTJRYiDNcRync9JA\nS1DUep5Qu2Bms4CRwHcUGCHpDkkPAOMk9ZQ0LmrJTZN0OCzTmXtR0lWSXo6adQdIekzSK5KGxnRD\nJU2UNFnSBEnb1fB2HcdxVqSBtOM6U0toBczs9TiQtn48NRjYxczmxLDBI81snqR1gccl3RHTbU3o\nz/wa8BRBaXsv4DCCjtwRhGXA9zazZkkHAL8BvpQsP074GgmwYc/NWavH+jiO43QIdeBc0tJpnVAB\nxsb5PxBkgX4jaR+C3E8/lmvAzTCzaQCSngfGmZlF7bn+MU0fQrTINgT5ipVETJMTwHZYf2jjtI0d\nx2l8GmgovNM6IUlbErTdcusW/ydx+QRgPWBXM1sadetWj9eS2nCtieOkHt2vgAfN7MioEDu+ytV3\nHMfJTku2OYa1oFM6oTi56jLg0tiKyU/SB5gVHdC+wOYVFtGH5bODR7Slro7jOFXHu+NqQo8osJcL\n0b4W+EORtNcDd8YutkmEMZ5K+B2hO+5nwL8y1tdxHKd9cCfU8ZhZlxLXriKIlOaOZxO0jwqxUyLd\niMT+G7lrZjYR2DZh87PKa+w4jtNO+JiQk+StBbPKJypAj67ZdLnmL8mmrZZVA+77PQdmsjtv3jOZ\n7LJq6gF8vGhB+URVZLJl05wbP3t6lWtSmoXNi7PZkc3u0+almezue39KJrum4svZtBu1jEay1saJ\nhXIn5DiO09moomxPe9OpJqtKWpB3PELSpbWqj+M4Tk1otfRbjfGWUIJ8SZ+0Ej8uBeQ4Tl3hgQn1\nR5zPcwWwLvAhcIqZvSXpKmAxQSX2MUnzgK2ALYG3JJ0C/AUYQoi6+56ZPShpBHAU0JOwZPjnOvJ+\nHMdxiuJOqGbkwrRz9CUssATwJ+BqM7ta0tcIq7fmVk/dBBhmZi2SRgEDgL3MbJGk7wNmZjtL2h64\nT1IuMm6ZFFA735fjOE56GkjAtLM5oWVK2hDGhAgtGAgh2UfF/WsJc31y3BzXUM9xh5nlQsz2Ijgw\nzOxFSW+yPDx7bDEHlNSO696tL1279sp8U47jOBXhLaGG4z9ljtPaLSOpHbfmGv0b57PEcZzGp4Fk\nezpVdFwZJrB8mdkTgEdS2j0S0xO74TYDXqp67RzHcaqFR8fVJacBV0r6ATEwIaXdn4G/RImfZmCE\nmX1aQI/OcRynLjDvjqsNZtYz7/gqolyPmb0J7FfAZkTe8ai848UUcFj5UkCO4zh1Qx20cNLSqZxQ\nvTK2z66Z7M7tlk1+58m5r2ay22qNDTPZZZXfeePxv2SyG77/qEx2AM98mu3ZWMZoo+Oa+2Sym9N3\n60x2Ez+sVIs3cEZG6aU/zJ+cyS4rXZuKSkSWpNdqPTLZNbdhbCWrfFZVcO04x3Ecp2Y0N05ggjsh\nx3GczkYDdcd1yug4SUdIsji5tFS6EZI2ThyPkTSg/WvoOI7Tjlhr+q3GdEonBBwPPBp/lmIEsMwJ\nmdl/m1nHaug7juNUmwYK0e50TkhST4LKwddZPi8IST+SNE3SFEnnSTqaoKZwvaRnJfWQNF7SkJj+\n+Jj+OUnnJ/JZIOnXMZ/HJW3QwbfoOI5TEmttTb3Vmk7nhIDDgXvM7GXgI0m7SvpCPP9ZMxsI/M7M\nbiEs7X2CmQ1KyPQQu+jOJ4R0DwJ2k5TTmVsTeDzm8zDwP4UqIWmkpEmSJt2+cEY73arjOE4BvCVU\nU44Hboz7N8bjA4ArzWwhQArB0d2A8Wb2YVyi4Xpgn3htCXBX3H8a6F8oAzMbbWZDzGzI4WtskfVe\nHMdxKqelJf1WYzqVE5LUl9B6GSPpDeAHwJerXMxSWz5ppAWPMHQcp96oYktI0qaSHpQ0XdLzks7I\nu/79GAi2bjyWpEskvSppqqTBpfLvVE4IOBq41sw2N7P+ZrYpMAP4BDhF0hqwzFkBzAcKyVs/CXxO\n0rqSuhBaUw+1f/Udx3HajrVa6i0FzcD3zWwAsDvw7VwUsaRNgYOAtxLpvwBsE7eRhPXYitLZnNDx\nwG155/4BbERYV2hSXG/orHjtKuCyXGBCzsDM3gPOBh4EpgBPm9nt7Vx3x3Gc6lDFlpCZvWdmz8T9\n+cALQL94+SLgh0Ayo8OBayzwOLCWpI2K5a+sciROeo7a/LBMD/mtJR9nKm/9btnWLvq4JZvMyKsL\n3s1kt3XPjcsnKsD938hmB9Dn3PGZbbOwRrfVMtn16No9k91Hi+Znslt3jd6Z7D5evCCTXbembL3Y\n2/bpVz5RAXZdreg7sCxXvjshk11WieOlS2a2WR15/ncOSf3O6XXp3anLiytUPwzsBOwL7GdmZ8Th\njyFmNlvSXcB5ZvZotBkH/MjMJhXK08czHMdxipDVAdWcCqLekgtwRkbH9dDy0/Uk9CydSeii+wmh\nK65NuBNyHMfpZFhL+vk/yQU4iyGpG8EBXW9mt0raGdgCmBKXtdkEeEbSUGAmsGnCfJN4riANNSYk\naUNJN0p6TdLTku6OC811RNlv5KI/HMdx6prqRscJ+Cvwgpn9AcDMppnZ+jEArD/wDjDYzN4njL+f\nFKPkdgc+iePsBWmYllB8ELcBV5vZcfHcQGAD4OVa1s1xHKeuqO4k1D2BrwLTYmAXwE/M7O4i6e8G\nDgFeBRZSZgHRhnFChEGwpWZ2We6EmU2J3vYCQligAeea2U2ShgPnAHOBnYG/A9OAM4AewBFm9pqk\n9YDLCMt2A5xpZo9JWge4gRAFMpE4zijpl8AcM/tjPP41MMvMLm7f23ccx0lHytDrdHmFAIOSwQux\nNZTbN+DbafNvpO64nQgKBfkcRZDWGUhQRrggEQ44EDgV2IHgybc1s6HAGMJy3wAXAxeZ2W7Al+I1\ngF8Aj5rZjoQWWM5JXQGcBCCpiaBPd12V7tFxHKftNJBsTyO1hIqxF3CDmbUAH0h6iCC7Mw94KtcX\nKek14L5oM43QsoLguAbEwTWA3jEKZB+Cg8PM/iXp47j/hqSPJH2G0BU42cw+yq9UMuJkUN9d2KLn\n5lW+bcdxnMJYc+2dS1oayQk9T1BEqIRPE/utieNWlt97E7C7mS1OGiacUiHGEJaB2JDQMlqJZMRJ\n1nlCjuM4maiDFk5aGqk77gFgtdjCAEDSLoQxn2MldYnjO/sQZHfSch/Lu+aQNCjuPgx8JZ77ArB2\nwuY24GBCi+veym/FcRynHWmtYKsxDdMSMjOTdCTwR0k/AhYDbxAmTvUkyOsY8EMze7/cqqoJTgf+\nV9JUwvN4mDCOdA5wg6TngQkktJHMbImkB4G5sRvQcRynbqhmYEJ70zBOCMDM3qWwKvYP4pZMOx4Y\nnzgeXuiamc0Gji1Q1kcUmQ0cAxJ2B46ppP6O4zgdQh20cNLi2nEVEtVj7wJuM7Pvp7Hp1r1fpofc\n0b+ZjXv2LZ+oAO8uKLc8U2G6NGXrDW5pw2qQ/Xqtk8lu5vyVYk9S0a1Ltu+8ltZsDexW//9cVbJq\n/wEsXPpp+UQFaK6CdtycIz+X+g+h720Ptbm8ttBQLaF6wMymA1vWuh6O4zjFsOZa1yA97oQcx3E6\nGw3UHddI0XEFkbQg73iEpEvj/qmSTiphO1zSsPauo+M4Tkdirem3WtOpW0JJiZ8iDAcWEKLfUiGp\nq1kjNXYdx1nlqAPnkpaGbwmVQtIoSWfF/dPjGulToxJ3f0Io9nfjyqp7S+ov6YGYZpykzaLtVZIu\nk/QE8DtJr8Q5SUhqimupr1ej23Qcx1kBbwl1LD0Syq4AfQlS4vmcDWxhZp9KWsvM5kq6DFhgZhcC\nSLqToNJ9taSvAZcAR0T7TYBhZtYi6RPgBOCPBNmfKWb2YbKwpGxPU5c+NDWtWbUbdhzHKUU9OJe0\ndIaW0CIzG5TbgJ8XSTcVuF7SiYRVAQuxB/C3uH8tQZcux82JianLREyBrwFX5mdkZqPNbIiZDXEH\n5DhOR2ItSr3Vms7ghNJyKPC/wGDgKUmVtgL/k9sxs7cJYqn7AUOBf1etlo7jOG2kkbrjVgknFBUO\nNjWzB4EfAX0IUj/zgV6JpBMISzNA6G57pES2YwhLONzs0j2O49QT1qrUW61ZJZwQ0AW4TtI0YDJw\niZnNBe4EjswFJhCETE+JOnJfJSyAV4w7CI5spa44x3GcWtJILSGX7cmIpCGExfD2Lpc2q2xPVrIW\n1rWpSya75owSM02ll8soSi2kabLKtyzKKN3i/ytXXaoh2zNzj/1S/wn1m/iAy/Y0GpLOBr5J6LJz\nHMepK1qba9/NlpZVpTuuqpjZeWa2eVx73XEcp64wS7/VmoZwQpIuknRm4vheSWMSx7+X9L0K8zxT\n0hrVrKfjOE494IEJ1ecxYBgsi3RbF9gxcX0YFUjvRM4EKnJCkrINmjiO43Qg7oSqzwTCRFIIzuc5\nYL6ktSWtBuwAPCPpB5KeirI75wBIWlPSvyRNkfScpGMlnQ5sDDwYV0hF0kGSJkp6RtLNknrG829I\nOl/SM8AxksbH4yclvRyj6hzHceqGRuqOa4jABDN7V1Jz1HIbBkwE+hEc0yfANIIY6TaEyaMC7pC0\nD7Ae8K6ZHQogqY+ZfRK77/Y1s9mS1gV+BhxgZv+Jy4d/D/hlrMJHZjY42p8KdDWzoZIOAX5BkO5x\nHMepC+qhhZOWhnBCkQkEBzQM+APBCQ0jOKHHCEtxH0SYBwRhDs82hAmnv5d0PnCXmRWagLo7MAB4\nTCFsuDvB0eW4KS/9rfHn00D/QpV17TjHcWpFax3I8aSlkZxQblxoZ0J33NvA94F5hAmjnwN+a2b/\nl28oaTBwCHCupHFm9sv8JMBYMzu+SNn/yTvOTf5oocgzNLPRwGjo+HlCjuOs2rRa4zihRhkTgtAS\n+i9gjpm1mNkcYC1Cl9wE4F7ga4mxnH6S1pe0MbDQzK4DLiBox8GKkj2PA3tK2jrarilp2466Mcdx\nnGpiptRbrWmkltA0QlTc3/LO9TSz2cB9knYAJsYutQXAicDWwAWSWoGlhEmmEFop90h618z2lTQC\nuCEGOkAYI3q5ne/JcRyn6jTSmJDL9nQALttTGJftKY7/r1x1qYZszwvbHJL6T2iHV+4uWZ6kKwi9\nULPMbKd4bhBwGbA6YWmcb5nZkwotgIsJwx8LgRFm9kyp/BupJdSw7L7e9pnsXl4wM5PdR4vmZ7I7\ne4Ns0eaT7ZNMdsc198lk941PJpZPVISlGR1mVmcy75bvZrI76bRSAu7Fue29SZnsDt5wUCY7ke19\n+e/3J5dPVIDVunbLZPdp89JMdt26ZH9FNrcUW7as/alyS+gq4FLgmsS53wHnmNm/Y5Tw7wgRyl8g\nBIRtA3wW+Ev8WRR3Qo7jOJ2MltbqDfeb2cOS+uefBnrH/T7Au3H/cOAaC11sj0taS9JGZvZesfwb\nKTBhJSStE5dheFbS+5Jmxv25kqanzONUSSfF/askHR33x0elbMdxnIaiAyarnkkYa38buBD4cTzf\njxC5nOOdeK4oDe2EzOyjxLLelxGWVhgEDALKrpQhqauZXWZm15RLmyIvl/RxHKcuaDWl3iSNlDQp\nsY1MUcQ3ge+a2abAd4G/Zq1rZ+6O6yLpcsLcopnA4Wa2SNJ44FlgL0I0XC9ggZldWCwjSQcB5wCr\nAa8Bp5jZAklvECayHkjoE72xHe/HcRwnFZWEXifnNFbAySxf9PNmwkrTEN61mybSbRLPFaWhW0Jl\n2Ab4XzPbEZgLfClxrbuZDTGz35fLJE/SZzAwiSDpk+MjMxtsZu6AHMepCzqgO+5dgkAAwH7AK3H/\nDuAkBXYHPik1HgSduyU0w8yejfv58jr5MjylqFTSB1hRtmfrPtux4Zolu0Udx3GqRjUDEyTdQIh8\nW1fSOwS9zP8BLpbUFVhMfNcBdxPCs18lhGifUi7/zuyEkjG1LUCPxHG+DE8pKpX0AVZs4u7db3+f\n9uE4TodRTdmeEu++XQukNeDbleTfmbvjqoVL+jiO01BYBVut6cwtoapgZh+6pI/jOI1EIwmYdhon\nZGajEvtvADslji9M7A8vYTeiUDozewDYrUCZ/dtUacdxnHagHoRJ09JpnFA9M2XujEx2i5uXVLkm\npXmidU4mu/GzU80LXok5fbfOZNeja/dMdgCLFy/IZJe12yKr/M4+rb3KJyrAbZms4L4PpmSy26hn\n34wlZiOr/E7WV/LSGkrvtIWykyTrCHdCjuM4nYwWbwk5juM4taI1c9uv4+m00XGSNpR0o6TXJD0t\n6e5CUW2SJtSifo7jOO2FodRbremULaG4psVtwNVmdlw8NxDYgBjVFnXjms1sWO1q6jiOU30aaUyo\ns7aE9gWWmtlluRNmNoWgJ/eIpDuA6QCSFsSfwyU9JOl2Sa9LOk/SCZKelDRN0lYx3XqS/iHpqbjt\nWYP7cxzHKYq3hGrPTgSpnkIMBnYys0IhawOBHYA5wOvAGDMbKukM4DSCfPnFBLXuRyVtBtwbbVYg\nKduzWvd16N61d34Sx3GcdqGRYvo6qxMqxZNFHBDAUzmxPUmvAffF89MIrSuAA4ABWr40dW9JPc1s\nhdjfpGxP7zW3rIeJyY7jrCLUQwsnLZ3VCT0PHF3kWinduKTeXGviuJXlz6oJ2N3MFrepho7jOO1E\ndVf3bl8665jQA8BqycWZJO0C7F2FvO8jdM3l8h1UhTwdx3GqRitKvdWaTumEopLrkcABMUT7eeC3\nwPtVyP50YIikqXEJ8VOrkKfjOE7VcAHTOsDM3gW+XODS5Xnpesaf44HxifPDE/vLrpnZbODY6tbW\ncRynejRSiLasDUvrOeno2r2fP2SnQ1i7R89Mdh8vyqap51Sf5iUz29xHdvNGJ6R+5xzz3vU17ZPr\ntC0hx3GcVZVGagm12QlJaiGEMHcjhKdfQ5hHU/fPIQYVbGxmd9e6Lo7jONWikaLjqtESWmRmgwAk\nrQ/8DehNWIe83hkEDCGsi56KnNxP+1XJcRynbdRD1FtaqhodZ2azCCoB31FgdUlXRtmbyZL2BZDU\nRdKFkp6LUWanxfNvSFo37g+RND7uj5J0dZTceVPSUZJ+F/O9R1K3mG7XKL3ztKR7JW0Uz4+XdH6U\n4HlZ0t6SugO/BI6V9KykYyUNlTQx1nWCpO2i/QhJd0h6ABgn6RpJR+TuW9L1kg6v5rN0HMfJyiod\nHWdmr0vqAqwPnBhO2c6Stgfui0rWpwD9gUFm1iwpzcpYWxFUCwYAE4EvmdkPJd0GHCrpX8CfgMPj\nktzHAr9j0vEXAAAgAElEQVQGvhbtu0YJnkOAX5jZAZJ+Dgwxs+8ASOoN7B3rdADwG+BL0X4wsIuZ\nzZH0OeC7wD8l9QGGASdnf2qO4zjVY1XrjivFXgTHgJm9KOlNYFuC9M1luW4tM0uzpOe/zWyppGlA\nF+CeeH4awaFtR9CMGxsldboA7yXsb40/n47pC9EHuFrSNoSPhG6Ja2Nz9TSzhyT9WdJ6BCf1j/wu\nuqR2nLr0oalpzRS36DiO03Zaal2BCqi6E5K0JeEZzMpg3szyLsLV8659CmBmrZKW2vLY8pykjoDn\nzWyPInnnJHhaKH7fvwIeNLMjJfUnMW+IleV+riG09I4jtOxWIKkd5yHajuN0JI3UEqrqmFBsGVwG\nXBqdxCPACfHatsBmwEvAWOAbkrrGa7nuuDeAXeP+l6iMl4D1JO0R8+wmaccyNvOBXonjPsDMuD+i\njO1VBFVtzGx6hXV1HMdpN1or2GpNNZxQjziw/zxwP0Fb7Zx47c9AU+xCuwkYYWafAmOAt4CpkqYA\nX4npzwEuljSJCluUZraEIFp6fszzWcJYTSkeJChiPxvHkH4H/FbSZMq0Es3sA+AF4MpK6uk4jtPe\nNJITcsWEjEhagzAeNdjMPimV1rvjnI7CFRMan2ooJly26Ymp3zmnvn2dKyY0GjFy7q+ESbklHRDA\nqRvvlamcfy94JZPd2/OzDMfBln02ymS3sDnbqhZn9ByYye6CecXWKyzP7IXzMttm4eANs4ms3/fB\nlEx2WZ3JtM2z/S72mfVaJrus9ezWJdsrq1tTl0x2i5uXZLIDaK3hB34jTWR0J5QBM7sf2LzW9XAc\nxylEI3W9uBNyHMfpZHSq6DhJP5X0fFQ2eFbSZ0ukHSLpkupWsTyS+ktaFOuX27pLOkzS2SXs1pL0\nrcTxxpJu6ZhaO47jtA/VDEyQdIWkWZKeS5y7QNKL0S/cJmmtxLUfS3pV0kuSPl8u/5JOKIY7/xdh\n8H0XwiTTt4ulN7NJZnZ6ivtqD14zs0GJbYmZ3WFm55WwWQtY5oTM7F0zK7YsuOM4TkNQ5ei4q4CD\n886NBXaKfuFl4McAkgYQ5k7uGG3+HBV0ilKuJbQRMDuGVWNms+NicUjaLeqrTYmabL0kDZd0V7y+\nZvSgT0YttsPj+RGSbo2ab69I+l2uMEkHS3om5jmuVD5piGVdGvc3iB57StyGAecBW8WW0wWxRfVc\nTF9M965o/R3HceqBamrHmdnDwJy8c/clVGIeBzaJ+4cDN5rZp2Y2A3gVGFoq/3JjQvcBP5f0MmEO\n0E1RsqY7Yd7PsWb2VNRcW5Rn+1PgATP7WmyqPSnp/nhtEPAZgorBS5L+BCwmrHq6j5nNSExgLZiP\nmeUrGGwl6dm4/5iZfTvv+iXAQ1ENoQvQEzib4M1zKuD9E+m/TWHdu4L1N7MVWohJ2Z7hfXdlx15b\nFXq+juM4Vae5gjGh5LsqMjoqvqTlawR/ANCP4JRyvBPPFaXchMwFknYF9iaIh94Ux1ieBt4zs6di\nunkA0gp3fhBwmKSz4vHqBMUEgHG50GZJ0wmRZmsDD0fvmdSTK5bPC3nVfS3nTIqwH3BSzLsF+ETS\n2iXSF9O9K1b/FZxQUrbnO/2PbaRgFcdxGpxKXjjJd1WlSPopISL8+iz2kCI6Lr6wxwPjo/LByQQn\nVLZ+BKXrl1Y4GQIbPk2cKqXlVjSfGlNJ/R3HcTqU1g4I0pY0ghAzsH9Cy3MmsGki2SYsl0IrSLnA\nhO0UFKVzDALeJOi0bSRpt5iul6IOXIJ7gdMUm0eSPlPyjkITbh9JW8T0ue64SvMpxjjgmzGPLgpL\nMORrxyUppnvnOI5T17S3bI+kg4EfAoeZ2cLEpTuA4yStFt/l2wBPlsqrXGBCT8LSBtMlTSWs5TMq\n6rQdC/xJQadtLCurXv+KsBTCVAVduV+VKsjMPiT0S94a88z1MVaUTwnOAPaNrbmngQFm9hHwmMLi\nehfkpS+me+c4jlPXVDMwQdINhDXctpP0jqSvA5cSPuDHxsCuywDM7Hng78B0wnI73469acXzd+24\n9ifrmJDL9hTGZXuKk1UqxmV7ClML2Z5qaMeN2vyE1IWPevN6147r7Nwyd2omuwVLsr3cm5RNHL1H\nU/dMdgvJVs8/zJ+cye7jxY0jtimy/f/eqGeaxYZXZub8jzLZZXUmX1k7W+/4/y56JJNdS2u25dqy\n2tVS/60tNKtx6u1OyHEcp5PROC6oyovadQSSTNJ1ieOukj7MTZKtMK9Bkg6pbg0dx3FqSyOtJ9Rw\nToiwzPZOknrE4wMpEwJYgkFARU6oQBSg4zhOXdGKpd5qTSM6IYC7gUPj/vHADQCSmqKUznqJ41cl\nrSfpmBgFN0XSw1H14ZfAsTG649hiEkFRqucOSQ8A4yRdI+mIXGUkXV+JnJDjOE57Us3ouPamUZ3Q\njYRY9NWBXYAnAMysFbiOOL+HILg6JYZ//xz4vJkNJMS2L4nnboqCpzexXCJoKEEh4gJJa8a8BgNH\nm9nnCAvajQCI842GAf9q53t2HMdJRTOWeqs1DemEzGwq0J/QCro77/IVRHkegqbRlXH/MeAqSf8D\nFIvXPAg4O2rQjWdFqaGxOSkhM3sI2Ca2uI4H/pEQ8wOCHpOkSZImLVwyN9N9Oo7jZKGRWkKNPL5x\nB3AhMBxYJ3fSzN6W9IGk/QjqrSfE86dGyaBDgaejJl4+paSG8gVTrwFOJMiWn5KfUVKPacO1dqiH\n37XjOKsI9RBwkJaGbAlFrgDOMbNpBa6NIXTL3ZybrStpKzN7wsx+DnxI0DfKl+2pRCLoKuBMADOb\n3sZ7cRzHqRpWwb9a07BOyMzeMbNiq7jeQZAcujJx7oK4NtBzwARgCvAgMCAXmEAFEkFm9gFByfvK\nYmkcx3FqQSOFaDdcd5yZ9SxwbjxhDCfHQEJAwouJNEcVyG4OsFveuW8UyP8qQstnGZLWIIjz3ZCq\n4o7jOB1EPYRep6XhnFA54npH32R5hFx7lHEAIULuoty6QqX4qIP1ypqasjVwX56XbbrVp81LM9ll\nZfWu2eSFAFpas2uBZeHf72eTJuposmq5ZZXfWadHMfH60sxbkr92ZjqyasctXNqYmsUt7oRqh5md\nR1i2uz3LuJ+wkJ3jOE7dUQ/dbGnpdE7IcRxnVaceAg7S0rCBCWmops5ctB8jaUD1aug4jlN9PDCh\nflimM2dmi2ibzhxm9t9Vq5njOE474S2h+qKgzhyApFGSzkocPyepf9SQ+1fUmXsuhm8jabykIXH/\nYEnPxDTjOvB+HMdxStJILaFVwQkV1Jkrw8HAu2Y20Mx2IixTu4wo13M5QV1hIHBMfgZJ2Z7W1nyx\nBcdxnPajxSz1Vms6vRMqozNXjGnAgZLOl7R3gTDs3YGHzWxGLGNOgXJHm9kQMxvS1LRm/mXHcZx2\nw5dyqD9yOnP5E0ubWfEZrA5gZi8TVLOnAedK+nlHVNJxHKcauGxP/VFMZ+4NgrNB0mBgi7i/MbDQ\nzK4DLsilSfA4sI+kXPq+7Vd1x3GcymikMaHOHh0HBJ05oJDO3D+Ak6JO3BPAy/H8zgStuVZgKUGB\nIZnfh5JGArdKagJmESLvHMdxak49dLOlRVYHA1OdnfX6bJfpIXdvyvaNMGdxNgmWfdfbMZPdfe9P\nyWTXNaOUyoC1NyufqAhTP5qR2TYLq3Xtlsmuo6WQunXJ9rfW0tqSya5Lxt/9z9bfK5Pd7+c8mckO\nYEFGqaDWjO/W5iUzlckwwVGbH5a68FvfvKPN5bWFVaIl5DiOk4WsDqjWNFLjwp2Q4zhOJ6ORuuPq\n0glJaiFEpnUFZgBfNTNfI9txHCcF9RBwkJZ6jY5bZGaD4kTROcC3a10hx3GcRsFDtKvLRKAfgAIX\nRCmdaQk5neGSHpJ0u6TXJZ0n6QRJT8Z0W8V0X5T0hKTJku6XtEE8P0rSFVGW53VJp+cKl3SSpKlR\nnufaeG49Sf+Q9FTc9uzwp+I4jlOERpqsWpfdcTkkdQH2JywgB3AUMIiwcuq6wFOSHo7XBgI7EFpO\nrwNjzGyopDOA04AzgUeB3c3MJP038EPg+9F+e2BfoBfwkqS/ANsCPwOGmdnsxHygiwkL2j0qaTPg\n3li24zhOzam2HI+ktYAxwE6AAV8DXgJuIijSvAF82cw+rjTvenVCPSQ9S2gBvQCMjef3Am4wsxbg\nA0kPEZbnngc8ZWbvAUh6Dbgv2kwjOBeATYCbJG0EdCeMN+X4l5l9CnwqaRawAbAfcLOZzYYV5HkO\nAAZIyyIbe0vqaWbLYqPjPKKRAD1XX5/Vu6/V1mfiOI6TinboZrsYuMfMjpbUHVgD+AkwzszOiyta\nnw38qNKM67U7bpGZDSKsXirSjQkl1+FtTRy3stzZ/gm41Mx2Br5BlOkpYN9CaQfdRGhRDYpbv6QD\nghW149wBOY7TkVSzO05SH2AfYo+UmS2JgWKHA1fHZFcDR2Spa706IQDMbCFwOvB9SV2BR4BjJXWJ\nStb7AJXMQuvD8vWETk6R/gHgGEnrwAryPPcRuviI5wdVUAfHcZx2xcxSb0nF/7iNzMtuC+BD4Mo4\nnj5G0prABrneJ+B9Qu9RxdS1EwIws8nAVIIK9m1xfwrBQfzQzN6vILtRwM2SngZmpyj7eeDXwEOS\npgB/iJdOB4bEgIXpwKkV1MFxHKddqaQllOy1idvovOy6EvQz/2JmnyEsFnp2MoGF2bGZ+gDrckzI\nzHrmHX8xcfiDuCWvjwfGJ46HF7pmZrcDtxcob1Te8U6J/atZ3uTMnZsNHJviVhzHcTqcFqvqTKF3\ngHfMLLcW2y0EJ/SBpI3M7L04zj4rS+Z16YQ6G0etvUsmuxkt8zPZDeqzdia7v83LFxlPR5OySU/1\nWq1HJrtdV9sokx3AVDpWOy6rBlxWMa+sw9HdMmq5ZdWOy1peVg2429ccmMlu308nZrKD7L/DalDN\nsAQze1/S25K2M7OXCBHL0+N2MnBe/LnSB34a3Ak5juN0Mtph/s9pwPUxMu514BTCcM7fJX0deBP4\ncpaM3Qk5juN0MqrthMzsWWBIgUv7tzXvug9MSIukFknPJrb+VcjzVEknxf2rJB3d1jwdx3Ham0qi\n42pNZ2oJ5eYWVQ0zu6ya+TmO43QE9SDHk5ZO0xIqhKT+kh6R9EzchsXzabXmRkk6Ky/P/ST9M3F8\noKTbOvbOHMdxitNqram3WtOZnFCPRFdczinMAg40s8GEkOrkEt8DCfN7dgC+CmxrZkMJ+kinUZwH\nge3jZFkIA3RX5CdKTgB7Yf7rbboxx3GcSmgkAdPO5IRyyz8MMrMj47luwOWSpgE3AwMS6Z8ys/ei\nXly+1lz/YoXESVnXAidGUb89gH8XSLdsAtgOvbZs6705juOkxseE6ofvAh8QWj1NwOLEtTRac8W4\nErgz5nezmTVXpbaO4zhVoB5aOGnp7E6oD2Gmb6ukk4FsM+TyMLN3Jb1LWObhgGrk6TiOUy3qYbG6\ntHSm7rhC/Bk4Oeq+bU/QPKoW1wNvm9kLVczTcRynzbSapd5qjeqhT7ARkXQpMNnM/lou7eqrb5bp\nIWeNXMn6h7Vxz77lExXg/f9UvI4VAGt2W718ogLMX7Iokx3AGt1Wy2S3cOmn5RMVoFuXbJ0NS1s6\ntoc3q/RSR7/EOrqem/fOJAwNwJvzPshk17xkZpsVf3ZYf2jqG35h1pO1VBjq9N1x7UJU4f4Py1dl\ndRzHqRsaqTvOnVAGzGzXWtfBcRynGPXQzZaWzj4mVBJJF0k6M3F8r6QxiePfS/pebWrnOI6TDavg\nX61ZpZ0Q8BiQU1FoAtYFdkxcHwZMKJeJAqv6s3Qcp05opMCEVf3FOYEw2RSC83kOmC9pbUmrEdQU\npksaF2V/pkk6HJZJAr0k6Zpot2ktbsBxHCefVmtJvdWaVXpMKM73aZa0GaHVMxHoR3BMnxDUExYC\nR5rZPEnrAo9LuiNmsQ1wspk9XoPqO47jFMQnqzYWEwgOaBjwB4ITGkZwQo8RFkj8jaR9CGoK/YBc\n3OabxRyQpJHASICuXdemS5eehZI5juNUnUaaeuNOaPm40M6EbrW3CaHX8wjyPCcA6wG7mtlSSW8A\nuQkuRSe/mtloYDRknyfkOI6ThUZqCa3qY0IQWkL/BcwxsxYzmwPkhEknEKR/ZkUHtC+wee2q6jiO\nUx4XMG0sphGi4v6Wd66nmc2WdD1wZ1TingS8WIM6Oo7jpKYeot7Ssso7ITNrAXrnnRuR2J/N8gi6\nfHZqv5o5juNkox4Wq0vLKu+EOoLm1tqHQabhvQVzMtll/ebKqgHXFqGrrBpwWWnuYA24rDTKl3PW\nemb9m8mq/wbZdQOrQSONCbkTchzH6WTUw1hPWuoyMEHSOomlut+XNDPum6TP56U9U9JfCuRhkq5L\nHHeV9KGkuzLW6VRJJ2WxdRzH6UgaSTGhLltCZvYRMAhA0ihggZldGOfeHAfcm0h+HPDDAtn8B9hJ\nUg8zWwQcCMxsQ50uy2rrOI7TkXhLqP24BThUUncI0jnAxsAjRdLfDRwa948HbshdkNRX0j8lTZX0\nuKRdJDVJekPSWol0r0jaQNIoSWfFc1tJukfS05IekbR91e/UcRwnI61Y6q3WNJQTinN4ngS+EE8d\nB/zdirv9G4HjJK0O7AI8kbh2DmFRul2AnwDXmFkrcDtwJICkzxJUEfJHJ0cDp8UlHc4irODqOI5T\nF7S0tqbeak1DOaHIDQTnQ/x5Q7GEZjYV6E9oBd2dd3kv4NqY7gFgHUm9gZuAYxP535Q0ktSToLBw\ns6Rngf8DNsovW9JISZMkTWptreaq4o7jOKVppKUc6nJMqAy3AxdJGgysYWZPl0l/B3AhMBxYJ0X+\nE4GtJa0HHAGcm3e9CZhrZoNKZZKU7enavV/tf9OO46wy1EPAQVoariVkZguAB4ErKNEKSnAFcI6Z\nTcs7/whBFw5Jw4HZZjYvdu3dRhAzfSEGSSTLnwfMkHRMtJWkgW24JcdxnKpSTdkeSQfHZWtelXR2\ntevacE4ocgMwkBROyMzeMbNLClwaBewqaSpwHnBy4tpNwInkdcUlOAH4uqQpwPPA4emr7jiO075U\nqztOUhfgfwnj8AOA4yUNqGZd1UihfI1Ko3THZZ1V3tE31xbFhEapa0P8wTQQtfg9ZFVMWLTozbb8\niYeyK3jnLF0ys2h5kvYARpnZ5+PxjwHM7LdtrWOORm0JOY7jOEWwCrYy9CMsb5PjnXiuipWtoO/Q\nt+pvwMiOtKtFmW7X2HaNVNfObtceG2HxzUmJbWTi2tHAmMTxV4FLq1m+t4Rqz8gOtqtFmW7X2Ha1\nKNPtOggzG21mQxLb6MTlmcCmieNNaIPyTCHcCTmO4zjFeArYRtIWUanmOMK0l6rRiPOEHMdxnA7A\nzJolfYeg19kFuMLMnq9mGe6Eas/o8kmqaleLMt2use1qUabb1QlmdjcrK85UDQ/RdhzHcWqGjwk5\njuM4NcOdkOM4jlMzfEzIcToASU3A0Wb291rXJS2S+gGbk3hPmNnDtatRbZEkYBMze7tsYic13hLq\nYKLg6YmSfh6PN5M0NKVtlzaU20XSxrG8zSRtljWvlGU92F75FyjvNElrZ7RNo6zeZiysVVVoBeCK\nkLRGhem/GB1gpeWcDzwG/Az4QdzOas8y20qGZ3OrpEPT1tXCAHqmAXpJ20oaJ+m5eLyLpJ9lyauz\n4U6o4/kzsAdhjSOA+QSBwDS8IumCSgUEJZ0GfACMBf4Vt7tS2m4j6RZJ0yW9nttK2ZhZC9AqqU+F\n9VxP0k8kjZZ0RW5LYboB8JSkv0fF30q0tx6XdLOkQ9LaSdpT0lhJL8fnMaPcM4ncL+ksSZvGlX37\nSuqbssxhkqYDL8bjgZLSLKZ4LOHv5ncVrgB8BLCdmR1iZl+M22EpbTOVGZ9pclXjtSXdm8Iu67P5\nM/CVWNfzJG2XwuYZSbulSJfP5cCPgaWwbK2z40parCrUWjJiVduAZ+LPyYlzU1La9gL+B5gAPE6Y\ndd07hd2rwDoZ6/sosD8wldA1Mwr4ZQq724G3gL8Cl+S2MjYTgPOBLwNfym0p6yng84TVdF8FfgNs\nldLuQIIie85u2zI2LxJUhdcnrFG1TprnC8wosL2e8v6eIMxcT/7dPJfStjfwjfg3MzH+3fQqY/Nv\noGcb/s6zlDk5zblqPpuYtg9wKkEjbQJwCtCtxO++GXgt/p+YBkxNUcZT+fcDPJv1+XamzceEOp6l\nsVstvAHD4nmp1tg1s/mEL6rLJX0O+Bthgb9bgF+Z2atFTN8GPslY3x5mNk6SzOxNYJSkp4Gfl7G7\nNW6VsIaZ/ShLJc3MJL0PvE94SawN3CJprJkV7Qaz8DYYC4yVtC9wHfCtuEzH2WY2sYDZJ2b27wx1\n3KJSmzz7t/Maay0p7ebFv5EewJmE5et/IOkSM/tTEbOFwLOSxgGfJvI6vR3LbJW0mZm9BSBpc1IK\nWWd9NrE79kSCJtpk4HrCqssnExbCzOfzafItwGxJW7H8//3RwHsZ8+pUuBPqeC4hLJq3vqRfEwQC\nU/UNR+d1KOFLrT/we8J/mr0JfdXbFjF9HRgv6V+s+EL5Q4piP4195q/EmdMzgZ7ljMzs6ijzkavT\nS2a2tIzZXZIOsTA5LjWSzgBOAmYDY4AfmNnSXL0pMRaT9xL6ADiNIEsyCLgZKOQ4HpR0AcHJJp/n\nM2XquQbwPWAzMxspaRtCl1eartG3JQ0DTFI34AzghXJGkg4HRgBbA9cAQ81sVqzLdKCYQ7iDjPIs\nbSjzp8Cjkh4itFD3Jp3GWtZncxuwHXAt8EUzyzmFmyRNKmQTP8SQtD6weoq65fg2YYLq9pJmElrB\nJ1Zg32nxyao1IPaT70/4jzbOzMr+h4l2rxNWlf2rmU3Iu3ZJsa9USb8odN7MzklR5m6E/9BrAb8i\ndF38zsweL2M3HLgaeINwn5sCJ1uJ6CpJ84E1gSXEvvNQTetdpqxzCHIibxa4tkOp5yvpZcJL6Eoz\neyfv2o/M7PwCNoWCLszM9itTz5uAp4GTzGyn+FKeYGWWio+26wIXAwcQnud9wBmWt/JvAburCM9m\npecuaX8zG1fCttKPiGqUuS6wezx83Mxmpygv67PZ18wqCqCRdBjh429jYBahi/oFM9sxpf2aQFPs\n1XBwJ9ShxJbM82ZWyQBx0r6nheXNs5bfE5Ytkd6uxC67r5jZS/F4W+AGM9u1imWUHNQ3szll7LsQ\nHOr3Kygzc6i1pElmNkTSZDP7TDw3xczaZXn4eH/3m9m+GWyHU+FHRNYyJW1vZi9KGlzoerkWZqVI\nOqrUdTMr2o0cu2n3I9zjZ2IX7olm9vUi6U80s+skfa9IWWl6Izo13h3XgZhZi8Ja7cv6vSvk55LO\nBRYB9wC7AN81s+tKGUnaifC13zcezyZ8jZcVIpQ0hNBNkj9fZJcypt1yDiimfzl2lZQr7zBgn3g4\nvkxX1dOEPnYBmwEfx/21CEERJcdg4u9jWLk65dm0SvohkGW+zxJJPVg+LrAVie68UkjagtBV2J8V\nfw9FI9bi/bVK6mNmlY4J/h44KP8jAij5EZGxzO8Rut1+XyhLwku/KJKuJrR85sbjtYHfm9nXiph8\nsUR2RumxzKVm9pGkJklNZvagpD+WSL9m/NmrRJpVGndCHc/awPOSngT+kztZ6mWS4CAz+6GkIwlf\nqEcBDxMG00sxGvherushfuVeDqR5AV9PmCMyjZQBFJFJksYk6nYCYcGsokg6D9gtlglwhqQ9zezH\nhdLnBvolXQ7clhtLkvQFQohxGp6VdAdh/Cf5+yj1Irpf0lnATXk2JVtewC8IHw+bSroe2JMwdpKG\nfxIiDe+kst/DAmCapLF5dS0XYJDpIyJLmWY2Mv6suMUW2SXngGI+H0v6TLHEZnZKxnIA5sYehUeA\n6yXNInGPBcr6v7j7ZzP7sA3ldlq8O66DUYhqWwkzeyiF7fNmtmN8ud9iZvek6c4plCZtN5CkR81s\nr3LpCtitRhiMzdk+QviPWPTLX9JUYJCFiZ25rp3J5VpdkqaZ2c7lzhWxvbLAaSvxFY2kGUVstkxR\n3jqEMQ+Rcswj2j1hZp9NkzbP7uRC583s6jJ2VxCcXfIjokup51KFMo8B7jGz+QoTOQcToj4nl7Gb\nAgw3s4/jcV/goWK//yxdZJLOJIRvTydEDjYRnkkf4PoU408vEz4cbwJuzdXV8ZZQh5PG2ZTgTkkv\nErrjvqkQ3r04hd3rkv4foUsOQlROmsmVAL+ITi8/VLdUv3lu3ZETgEr7vNcCci2KtJNd340vreQL\n8900hlm+iq3CUOsCYx25KKzNYtdsmjGPi2OAyX1UEJFX7sVfgm8SPiJyrZdHCJM7y2IhMrIHIQrw\npbIGy/l/ZnazpL0IQQYXAJcB5Zzv74GJkm4mOPejgV+XSJ+li2wT4I/A9oRegccITunOFC1gzGxb\nBWWU44CfKkyuvbFcV/qqgLeEOpgYAZZ76N2BbsB/ykWAJez7EuaptMToqt5m9n4Zm7WBc1ixVTIq\nzdeYpOsI//GeZ3k3UMmWQrR7FNjPzJaUKyNhczxwHiECUISxobPN7KYydn0JXV37EJ7tw4QJtWVf\nDpI2IYQM7xlPPUIYX3inhE1FodZFoulymJWJqot5/JYQRv4aK/4eyo2XbAP8FhhAIqQ4TastK5K+\nCFwIdDezLSQNIvw+SnY55wI24r1OM7O/JYM4ytjuCOS68x4ws+ltvY8i5XQHhhC6sveI21wzS61i\nohDN9wfgBDPLLMXVWXAnVEMkCTgc2N3Mzk5psxMrv1CuaZ8agqSXzCyNnEm+3TXADoS5JslxgZIt\nI0kbEcaFAJ4s52DzbNc0s6L980VsxhIm/SZbiSeY2YElbDKHWmdF0qvAgEqcerR7lOCgLyIMyJ9C\nCBEuONlY0t/N7MuSplFgomiKgJRcZOR+hMCSXBTgc2a2Uxm7uwjz0A4kdMUtIvwNpOk27kKQb0oG\nbY89xa4AABXwSURBVJQM/skS7KEgRbUH4aNlD0LLfVq5FrWk3oQJu8cBWxHmCv7dzJ4uZbcq4E6o\nDqjga+8XhFncAwiTU78APGpmRxdJ/0czO1PSnRR+oZQNhohjJhdU+mWpCuYmqY0hujHCbQxBZmYz\nSQOBb5jZt1LU89l851HoXN71ikKt1YaQ4EQe/wRGmtmscmnz7J42s12TY2S5c0XSb2Rm7ymoFRSq\n60pzsQrk8biZ7Z73fKamGNtbAziY8FJ/JX6Q7Gxm95WxO43gaD8gKCUoVLVseVMIwR4rBN0U6jKX\nNBrYkaD1+ARBjujxtGM7cRzxnwTHU0iFY5XFx4Q6mLwXUhOhaZ9mXAdCX/dAwmD9KZI2oHRkXO7r\n/sKKK7qc3QkRZDMIYxFl/4PHr9JeZpZKdZk2hugSvvI/T5zhb2ZTJO1T2mQZH0k6kRB+DEFYtuQg\nM5WHWrclJDjHWsCLkp5ixTGhch8SFSle2HLVgG9ZnoSSgrJ2Glml5yV9BegSuwNPJ4yflGNdYgSl\nlqu8v5jC7gxCd2i531s+i83skpRpNwNWIyhwzATeAeaWtFiRLc3MJPVUG+f7dTa8JdTB5EVjNRMi\nZi5P84Ur6UkzGxq7O/YlfJW9YGUmv0o6w8wuLneuiG2mL2JJE81sj3L559msbmaLy50rYPeEmX02\nbcskz3ZzwpjQHgSHMAE4zUqsGSPpIMLcqQGEQIE9gVOswtn3laCMUZVaWfGiN2GC7hNl7J4xs8F5\n58q2ZmK6NQjP56B46l5ClFvJOVGJLkARupu3ICg1lFQjiGNuB5pZc7m65dl9BdiGlMEesft8R8J4\n0DBgJ0IQzUQzK9jyT9gm5+oJ+JAw+fe5SurcGfGWUMczxsweS56QtCdBAqQckxSk7i8njEksICgU\nl+NkgqxJkhEFzq2EZdfKyjL/ZgJhLKDcuXwyaYdFNslvTcTfR1EnZGb3xQ+BXKj1GVYi1FpVmDVf\nztmUoL+ZPUX4Wzkl1ucYQpdSobp+E/gWsJVCyHyOXqRrzQAcamY/JTiiXL7HEP4WimIrh9kPjnUp\nR1ZtxJ0JwR77kQj2oEjL28IX+3OS5hIEgT8B/gsYSugOLEWhuXqjSTdXr1PjLaEOpsgX5krnUuTT\nnxAZN7VEmuMJ66XsRYj6ytELaDWz/VOUk0krSxXMv5G0IdCP0LX4FcKLHcJX+2UpWnqZtMOibcW/\nD0nj8p9doXOJa98ws/+rZJwsYfuome2lFaMqYXm3aDldvYruLw68r02IqEsGy8y3FNGGWcosk1fZ\n+V5Znmu0Sx3sIel0lreAlhIccm6bZv+/vTMPlqyu7vjnyzAMWxGDA7JD0GSAqCBCwrBYEhIWgYih\n2LRIUhjACkvYK4YYBlSiggEdAmFRCbIFMuDKjOxIQKrYxgIXCGGkikgFCKsMKbaTP86v593X033v\n795+vbx+51P16r2+3b/+/fp19z33d5bvSbVtJeMb1+qNO7ETGhCS5uMf4PXarojXAbLSNIsnOjP7\nVfuxDtyL16TMZXK85VW8F0oOn8ev+CdpZVUNqsoWamMvfGe2SVpnywi9AvxdxlzP47VB2TR5PySt\nDqwJzJWnvReN5cYl67s4xcleMbPz6qzTUqGwmdWSfZGrRnwM2FhSMe6xDu4G7jbfy8DLkr4GvGBJ\naFPSOpL+sMyN13TOwvji+7AKvgOurPeqMjYlPIq7KXO8EFvgO7kTC3GzOvRSqzfWhBEaHKvhAeFV\nmVwk9wqecNCVHk5+TwFP4fGOptTVymqtObv+xryg8l8lHWhmi+ouUA1SbWn2fhyN98bZCHeHFo3l\nBWVrNK/rOgxPoqiFmgnf/hoP8v9pWmuLV4ETM8ZfxGQ36G86HJvqOYvvw1t4B+DKz4O8aPs0PF5T\nLF2oSmjJTvYws46u1BocgdfqtdzRd6djM55wxw0YSZvnpLm2jfkbJk5+/83kk9+lZlZ6ApS0E24Q\ntsZPvrPILJCVdCuuw/aP+I7qWWBHMyv1ZatZ/c3ZeNC8KER5spmV9ltSjVTbDmNXvB8pi2xtM3ul\nYsxx1r0xW9m48/Di5HbNuUrFBEnfxRMmagnfSpptmS0Y2sZ1Sl3PTUxYMWd6Dzctcxv3iqSb8f/p\nKXiH1L8AnmvP7uswrrGEVjB1hBEaMHI14lNY+ao9p2q+6cnvAbxI7no8JfzP8RbWHYVB28auhRcN\n1tXKalJ/s1K9VE4sQQ111dLYq/ET19vA/fju8mtmdk7JmE4aZ1+oMiZq2Icojf0x8CGglvBtSrJY\nwIQKeiuWVKqYIOkG4E589wOeILC7mVUKw0q6E98NrYrviJ7Fi3k77oZSAktXMl5jqxZqhZGUdL+Z\n7Vg2bhD0+tpmAuGOGzzX43pYl5HZgriFmS1MWWBbMNmAVSommNkTkmaZ2dvAtyQ9DJQaoeQG+oG5\nuvE7eH+ZXJrU38ySNMdSKq+8FmdOxlyNdNUS25i3ov4UsBgPxj+I65Z1o5PG2UVUaJxZs74+78OV\nAD7Xdtdu5LWH/gbuCnuQep+3z+BdgP8eT4i4jbwupwC/lf6nfwVcYWZntGXatTMfz0a8Bs/aU8lj\nO9Ha6T0jaV/cLVjaawp68xDUoNfXNvaEERo8b5nZRdUPWxlJ38YlP5YycUIxvIVyGcvlmldLJX0F\nP3mtUjWf9daP5gj8C34eE/U3VckKVwG3pcw64ckKOYavVqptG7Plad0HABeYtwWvcg+0/vf7ApeY\n2Q/lfZ5KkSuLH8jKFxFnlQw7H/ismT3S9lwvAGfjRqaMl81scdXa2jGvWzu07rjEqnK1g4MppGmX\nsAEu1dPK5vwh3gCxst9V4gspq+9k/DO3DnkxqAvo4CHInDOXXl/b2BPuuAEjaQHunriRyVftOWKb\nv8Cv3Gu9afKCzGfxeMSJuEvtQjN7ImPsd3E3UN1+NI2QtDe+uzA85rWBmR1TMaaRrloaezyuAvBT\n3KhsBlxpZruVjGmkcSZpCV5bMmlXYmadlCJaY7q6lTLTl7+EX+HfQF5B5mlm9hVJC+ks9VT5vid3\n5edwSam/lrQlLv10YMbYOfgJ+xzgzKp4Zy9oQn6p6MbLktBqON/AXtt0InZCg6fVa+XUwjEDclSN\nH8WvrGqliBYSIV7HM3TqcAN5sjIAdDt5FdZSdRL7nzT+IGAZGdlR1Eu1bV/P13G3U4unUhp6GQfj\nGmfnmtlL6ar/1Iox4IWxe9dc4rtK7lsjY3zLRbhD4VjZLrFV5FvagLCC281sRWGqmT2J7wC7kk7Q\n++In6S3w9+TGnMkaZkdCQw9BXXp5bTOBMEIDxmr2omljLvBzeVfWbP0wSfvh9T7twelK37d5b5j1\n0t85nSGLJ68zqa4kbyVrHJZ+nscznVQjhlJbV00VKgaU9EEys+Xyjpq74lpib6XfVdwr6QPtrrUK\nHpB0pJldWjyY4i2VCswN4lCL07imfYgA7pO0FPgWsLhq5y5XXH8/Lsp7ptWXsmnadfZw3Ogci3sI\nNqXCWNZlCl7b2BPuuAGjmr1o2sY21Q97Am8F/kiuK0+ScANyLP5FFX6yXVgRwyg+R646+Dt43cSn\nWy5CSU9WZXAVxtf+v6g3FYMz8J3FPPNmZRsB15vZLl0e/yh+clwV1yp7knwx2PfgV81vMGF0dsAD\n6Z+w6l5S78FjRxuZ2T6StgHmm1nHWFIxG1HSQjM7ruz5uzyHcJfqEXhbjuuAy83s8S6Pf4cJV28T\nVYha2ZHyRoK1Ut2b0utrmwmEERowGk4vmjuAPaxCWqRtzEl4q4ijzGxZOrYlngW2xDIq/3PSq9Pj\nDsADxLsAS4BrcY297F1jOtkW+xDVds3VmGspHid7yDJaFUh6Eej6/lpee4Td8Stq8MLV2zPXuhjf\nkZxuZttKWhVXYe/W+rooAttIaqfDuq/Eu5n+FG9SOKWtDFRfiLRoaBflxKqC/hHuuMHzXjM7RF49\n33LtZKVt9pBSehpwk6S7yBd4PBxXJn6+8PgnU9r1zTSo/O+GmX0H+I68JunjeGHu+pIuAm606n4y\nB+PB3jvxK8yFkk41s38vGVMq4V8Ru3rDzKyVRZfWXcayHENTsZ478I6zdZlrZtdJ+mx6nrcklaVq\n93xVKundeHHy4XiM7zi8zcZ2eCZaLy7pTtTNjix+3/rWYTbII4zQ4Knbi6ZI05TSL+KyK6vjxiuH\n2dZBGdrMnkspzR3RZKHNNSW11Acq3Q/mXVGvBq6WV9ofhGeulRohPA14x9buJ8WwbgW6GiEmx1Oy\nYlcFrpN0MfAuSUfibqdLSx6/fknsKUtFuwdeS0ah9XnbCc/Q68ZW8poeMVlJO6tRXOInuFLGATZZ\npukBSf9S+xVUcxDeryc3O9K6/B0MgTBCg2cB7nLaVNJVuAvqL3MHW4OiUzweUNpauQNlX+iu91lN\noc2S53kRl7q/JOPhq7S53/6XiiynYuBd0gk5gXhJJ+D1Tufj/ZxeAeYB/2Bmt5QMnYXr1A2jUPEk\nfBfyXkn3AOtRrlW49RTMOa9b7NHMvjwFz99O3ezIbdPFkYA16lwoBVNPGKEBYzV70bTRNKX0Jkl7\nVrm12ti28OUs0mo4NkoskfQjJtQZDsGzkXLJvRreBDdAW+E6dffgRqkqS+2Z3GSOqcbMHkqJG/Pw\n9+4xK9GS68VtqIJETScPc0bKdFNqZUeaWZZq/VQg6Z+Bq62th1gwQSQmDBhJ38ddTt9L7qc6YzfH\nfeyrUaPoNLnI1sK/oG8yJld8SpI2ZnaPvG36rumul3B9u//KfJ5aAfh0IbAD3gpifvp5ycy26fL4\nvhVAViGXXtqXlWtoptwFKOk5SiRqqrI4e5h3ZIVI5eLDhwIb4lmC15jZw8Nd1WgRRmjApC/MIfiJ\n4X48E+wHVtLCepAppdMJuXJBJ0mbDwBnm9n+JWMnxa6A5a27qDDQcomY+bgrdT5+Jf6IdemhJGld\ny2wIN9VIugn4P1ZWGG/ag6dsrllMSNR8kAFK1AwyO7IJ6QLy0PSzBm6or+mWtj6TCCM0JNIX9o+A\nI4G9K056jVJKJW1lZr+Ut0leiW4prNMF9Shp02C+S/CeNa/iV/r3Afel+NVIUpY6njF2Dbye7bEG\nYwcpv9OeHbkbUJodOUwkfQj4JvDBQboGR5WICQ2B9OXeH98RbU+1SGfTlNKTcOXjTtpkuQKfo0yv\nkjZ12QxX9f5PXDvuadz1N8osbhAPRNL+wLm46/d3JG0HnFUV19FwJGqaZEcOlFSftQ++E9oDN5gL\nhrikkSGM0ICRdB3wB3iG3AXAXRlFpI1SSs3sqPS7dguBaUJPkjZ1MbO9U03X7+PxoJOB98sVrX9i\nZnXSvAfFfcCN8oZ9deKBC/DP6Z34gKVyjbauaHgSNbWzIweFpJZ78mN4L6hr8QLwWvHgcSbccQNG\n0l7ArSnFOnfM27j0h/Ar/Oz4ReE5GvUhGmXUo6RNj3NvgseEdgb2A95tZmU7s6EgaRleAJwt2ZTG\n3WdmO7UpKJS69jQkiRpJ5+AxqGJ25CNmdlo/5qtD+u4eDSwaZbftMAkjNCBS9lZXzCxbqbrB3B37\nEFWoAkwbmkraNJjneNzo7IzvKu4t/DySsaMdOPKOrB+tuzZJ38Ab2f0tLup5PF7A/JmpX2XvtGVH\n3m1mI6FSPczMyOlCGKEBIW/UBrA+fhJrnSh3x7Xj9uvj3I36EAWTkfRPpNogM6vVTmNYSLocjyMu\nJl+yqSW0ezqwZzr0I7yFedcszlEhuR4PM7OrRmAtT1OuyN5PtYxpQcSEBkQrfVfSLbhBeCbd3hC4\nvM/TN+pDFEzGzLpK74wwy9LPauRLNmFmy3EjlNMZdShIWgc4BtgYV4W4Jd0+BRdLHboRwtUypkRF\nZFyJndCAkfQLM9u6cHsV4OdmtlUf57wDF48s9iEyM/t4v+YMpjfpYukgM3sp3f5t4Foz22u4K5tA\n3vX3RVyrbg/cy9BSIVk6zLW1qFsIPROJndDgua2DxEyt9NkGLCj83aqjOLTPcwYjQLoA6dSmuyo9\nf27LAKXHvyhp/aleX49s2aoFk3QZvtPfbMRchsPQC5xWhBEaMGZ2rKRPAB9Jh+7FXWX9nPOuVCD3\nSSbaZvdDzTgYPU4p/L06nmTwVsa4d4pKHanif9TcJis08MzsbUlPj5gBAt+hBSWEERoOv8KTE1oG\nYVE/JlHvbbODaY6ZtddL3SNvD1/F6cB/yHtQtXbPR031+nqkKLJbVMQeGW3EYck1TSciJjQguhiE\nU8xs8z7O2VPb7GD6I2ndws1VgA8DXzezeRlj5+Jq7+DyRLlq70GQTeyEBscvcYOwX8EgnNjnOf8M\nj/3cIanVNjt81DOLB3E3mnA33DLg05lj5wAv4OeJbSRhZj/uyyqDGUvshAaEpANwg7ALLtlzLXCZ\nmU11q+NOc7faZh+G68VdQUbb7GDmIunLeNLMzyi0zK7SjguCuoQRGjDDNgiaaJt9iJlF0HTMkXQM\n3lupmGp9mJldWDHuMVzlObf1fBA0IozQEAmDEPQbSUvNbLu2Y5VSMpIW43VCv+nrAoMZT8SEhkgS\nNLwk/QRBP5glSS3JptTHKkc5YTneRv42Jsv9jIXeYDA6hBEKgvFmCfBvki5Ot49Ox6r4XvoJgr4S\n7rggGGOSLNTRTBRN3oInxFS2Eumls2oQ5BJGKAjGHEmrAfPwVO3HzOzNiiGTOquaWXZn1SCoy0h0\nHwyCoD9I+ijejvwC4ELgcUkfKR3kLMA7q74E3lmVeq3lgyCLiAkFwXjzVWDPlkstKXdcgysnlPGm\nmb3s3cxXMHJN+4LpT+yEgmC8mV2M6ZjZ48DsjHE/k/RJPLvudyUtxMV2g2BKiZhQEIwxkr6J72Cu\nTIc+BcwysyMqxhU7qwrvrPr5EVSpDqY5YYSCYIyRNAfvNrprOnQ3cGEoIQSjQhihIBhzJK0HYGbP\nZTz2fDM7QdL36dwML7LjgiklEhOCYAyRZxScARxLiv1KehtYaGZnlQz9dvp9bn9XGARO7ISCYAyR\ndBKwD3CUmS1Lx7YELgKWmNl5FePXAl43s3fS7VnAHDNb3t+VBzONMEJBMIZIehj4k/ZGdMk1d3OG\ngOl9wB+3BEwlrZ3G7dyvNQczk0jRDoLxZHanTqgpLpSTor16UUE7/b3mFK4vCIAwQkEwrrzR8L4W\nr0navnVD0oeB13teVRC0Ee64IBhDUhLCa53uwnc5pbshSTvi3X9/ncZsgPe9enCq1xrMbMIIBUHQ\nEUmzceFTyBQ+DYK6hDsuCIIVSNpR0gYAyehsD3wR+KqkdYe6uGAsCSMUBEGRi0kxo6S2/SXgCuBl\nogNw0AeiWDUIgiKzzOyF9PchwCVmtghYJGnpENcVjCmxEwqCoMgsSa2L0z2A2wv3xUVrMOXEhyoI\ngiLXAHdJeh5Pyb4bQNL7cJdcEEwpkR0XBMEkJO0EbIgrJLyWjv0esLaZPTTUxQVjRxihIAiCYGhE\nTCgIgiAYGmGEgiAIgqERRigIgiAYGmGEgiAIgqERRigIgiAYGv8PyUdpOiVAZz4AAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11cfbea50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "annot_lookup = []\n",
    "for i in xrange(len(nr_ids)):\n",
    "    annot_lookup.append(Genre_ID_to_name[nr_ids[i]])\n",
    "\n",
    "sns.heatmap(visGrid, xticklabels=annot_lookup, yticklabels=annot_lookup)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above image shows how often the genres occur together, as a heatmap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Important thing to notice in the above plot is the diagonal. The diagonal corresponds to self-pairs, i.e. number of times a genre, say Drama occurred with Drama. Which is basically just a count of the total times that genre occurred! \n",
    "\n",
    "As we can see there are a lot of dramas in the data set, it is also a very unspecific label. There are nearly no documentaries or TV Movies. Horror is a very distinct label, and romance is also not too widely spread. \n",
    "\n",
    "To account for this unbalanced data, there are multiple things we can try to explore what interesting relationships can be found."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Delving Deeper into co-occurrence of genres"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we want to do now is to look for nice groups of genres that co-occur, and see if it makes sense to us logically? Intuitively speaking, wouldn't it be fun if we saw nice boxes on the above plot - boxes of high intensity i.e. genres that occur together and don't occur much with other genres. In some ways, that would isolate the co-occurrence of some genres, and heighten the co-occurrence of others.\n",
    "\n",
    "While the data may not show that directly, we can play with the numbers to see if that's possible. The technique used for that is called biclustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import SpectralCoclustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAFJCAYAAAAyv5ItAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXm8XdP5/9+fezPLJMQUJIogRUIiNYuhqmiVUlQR1ear\n39ZQ9e3op6i2VFst2ipqjlBzqqoIiZkkZJCYJUQMkZDIJMO9z++PtU6yc3LOPefue+49Q553XvuV\nPaxn7bX32Xc/e631rM+SmeE4juM45aCu3AVwHMdx1l3cCTmO4zhlw52Q4ziOUzbcCTmO4zhlw52Q\n4ziOUzbcCTmO4zhlw51QKyGps6R/SVog6Y5WOsdwSU82cfw/kk4uIp+Zkg4qbemaj6QTJD1U7nKs\nC0i6QdJFbXzOfpJMUrs2POf5km5p4vg0ScOKyMckbVPSwjmAO6EWI2mspE8kdcw6dDSwMbCBmR1T\nyGG0Bmb2ZTO7sS3OVeiPvRjMbKSZHVyqMjnpKMezWi7M7PNmNrYtzlUOx18NuBNqAZL6AfsABnw1\n63Bf4DUzW1mic7XZ12M5KOf1lfrcufKr9d/PaX1q9hkyM19SLsB5wFPAH4H7E/svAJYDK4BFwPeB\nz4CGuD0/pusI/B54B/gQuAroHI8NA94FfgJ8ANyc4/zD4/mvBBYArwAHJo6PBb6T2P4u8DKwEJgO\n7Br3zwQOius3ABclbIYB7ya2fwLMjnm8ChwIHJJ1vZNj2h7AP4D3o81FQH1W2S8D5sVjw4EnE+cy\n4DTgdWA+8BdA8Vg98AdgLjAD+EFM3y4e/2nyN8lx72bGa5kCLAPaAZsBdwEfxTzPSKQfCjwTy/F+\nvOcdssr6/VjWGU3s+zMwC/gUmAjsk8jjfOCfwE3x/k4DhiSO7wq8GI/dAdye9VsdDkyKZXwa2Dlx\nbBfghWh7O3Bb0jaRbgdyP6s9Yrk+At4GzgXq8tzbocCEeI0fAn+M+/vFe3Iy4ZmfC/wiYdcR+BPw\nXlz+BHSMx8YBX4/re8V8DovbBwKT8pTlfODOeM0L4z0YmPUcZJ79euDnwJsx7URgi8RvuU2ev6vh\nxOcWEOGZnhOvfyqwIzCC8PexPN7Xf8X0TT1zmbLfEvP6Tr57W81L2QtQzQvwBvC/wOD4gG2cOHY+\ncEtie9WDmth3GTAa6AV0A/4F/DYeGwasBC6Jf5ydc5x/eEzzQ6A9cCzBGfWKx1f9sQDHEBzBbvEP\nZRugbzyW/EO8gTxOCNiO8ALdLG73A7bOdb1x3z3A34H1gI2A54H/ySr76QQH0Dn7HhH+8O8HegJb\nxj/UQ+Kx0wiOdHNgfeAREk6oiN9uJuGFvUU8dx3hpXMe0AH4HPAW8KWYfjCweyxrP4IzPyurrA/H\n37JzE/u+BWwQ8/kR4QOjU+IefgYcSngh/hZ4Nh7rQHj5nxl/66MIL7SL4vFdCC++L0Tbk+M1dkzY\nZp6TownP61pOqIln9SbgPsJz2g94DTg1j/0zwIlxvSuwe+J5MeCaeM8HEj4AdojHLwSejc9Kb4Ij\n/VXi2BVxPeMoLkkc+3Oespwfr/XoeO3nEF727XM8+/9HcBrbEf5GBhKa0zO/ZTFO6EuE56hnzGMH\nYNM8f1uFnrlM2b8W03bOd2+refHmuJRI2pvQ5PZPM5tI+KP4ZjPsRfg6+qGZfWxmC4HfAMclkjUC\nvzSzZWa2NE9Wc4A/mdkKM7udUDs5LEe67wC/M7PxFnjDzN4utryRBsJLbYCk9mY208zezHN9GxNe\npmeZ2WIzm0Nwusnre8/MrjCzlU1c38VmNt/M3gEeAwbF/d8gvHjeNbNPgIubeS0Al5vZrHju3YDe\nZnahmS03s7cIL8vjAMxsopk9G8s6k+Bc98vK77fxt1yab5+Z3WJm82I+fyDcz+0S6Z80swfMrAG4\nmfAihNUO8PL4W99NcOoZRgB/N7PnzKzBQl/gsmi3O+EFnHlO7gTGF3uTJNXH+/AzM1sYr/8PwIl5\nTFYA20ja0MwWmdmzWccvMLOlZjYZmJy4xhOAC81sjpl9RGhRyJxjHKvv974EB53Z3i8ez8dEM7vT\nzFYQWi06Ee5JNt8BzjWzV+PfyGQzm9dEvrlYQXDU2xNq7S+b2ft50jb5zEWeMbN7zawxPkOF7m3V\n4U4oPScDD5nZ3Lh9a9xXLL2BLsBESfMlzQcejPszfGRmnxXIZ7aZJVVo3yZU8bPZguAoU2NmbwBn\nEb7Q5ki6TVKuc0Fw0O2B9xPX93fCV26GWUWc9oPE+hLC1x+Ea0zaF5NXNkmbvsBmmbLG8v6cEFyC\npP6S7pf0gaRPCR8MGzaRX859ks6R9HKMmpxPaOZK5pN9vZ1iX8BmrP1bZ5f/R1nl3yLa5bJtzgfI\nhoTfMmnzNtAnT/pTgf7AK5LGSzo863hTv2n2OTLP1zNA//hxM4hQM9tC0oaEJqrHmyj/qvtkZo2E\nZu7W+ht5lNBU+xfC38jVkrrnSd7kM5dd9kihe1t1uBNKgaTOhC/x/eJL6QNCU8dASQPzmGXLlc8F\nlgKfN7OecelhZl2bsMlFn1iryrAloT09m1nA1kXkt5jgHDNskjxoZreaWaYWaITmwlxlnUX4Et8w\ncX3dzezzyeyKKE8+3ic0xWXYIkUe2S/0GYmy9jSzbmZ2aDz+N0Kf27Zm1p3wslAT+a21T9I+wI8J\nz876ZtaT0HyanU8u3mft3zp5zbOAX2eVv4uZjcpju2UT58r1rK4g/OZJ+9k5jc1eN7PjCR8clwB3\nSlqvqYuLvJfjHO/FPJcQmq7OBF4ys+WE5rqzgTcTH4O5WHWfJNURnpvW/Bu53MwGAwMIDuP/Mody\nnK+pZ24tmxbc24rFnVA6vkZomhpA+CobRGj7fQI4KY/Nh8DmkjrAqi+ya4DLJG0EIKmPpC81sywb\nAWdIai/pmFiOB3KkuxY4R9JgBbaR1DdHuknAoZJ6SdqEUPMhlm87SQfEcPTPCE60MXF9/eIfObEJ\n4iHgD5K6S6qTtLWk7CastPwTODPes56EIINVxJDxsc3I73lgoaSfxDFe9ZJ2lLRbPN6N0Bm8SNL2\nwPdSlLkboR/sI6CdpPOAfF/J2TxDeOZ+IKmdpCMINYAM1wCnSfpC/H3Xk3SYpG7RdiWrn5Ojsmyz\nyX5WGwj3+9eSusXn5mxCh/laSPqWpN7xGZ8fdzfmSpvFKOBcSb1jDee8rHOMIwSgZJrexmZt52Ow\npKNijfIswsdRrmasa4FfSdo23sOdJW2QI90k4ChJXRTGDp2aOSBpt/gbtCc4q89Y82/kc4l8Cj1z\na9GCe1uxuBNKx8nA9Wb2jpl9kFkI1fATlDuU8lFCtNMHkjJfbT8hBDc8G5t4HmHN/oFieA7YlvC1\n+mvg6Fzt2GZ2Rzx+KyHy515Ch3k2NxPa6WcSnMjtiWMdCX0vcwlNKhsBP4vHMgNy50l6Ia6fROhw\nnQ58Qoj02bSZ15ePa2L5phAixh4gvGgb4vEtCNF3RRFftIcTPihmEK7xWkJzGYQO7W8S7t01rHlf\niuW/hCbX1whNTZ9RZDNi/PI/ivDCm08IcLif8ELFzCYQoh+vJNzrNwgd5knb4cDHhACWu5s4Xa5n\n9XTCS/Ut4EnCc3RdHvtDgGmSFhGiAY9ros8vyUWEyK8phACBF+K+DOMIjvzxPNv5uI9wzZ8Q+piO\niv1D2fyR4GwfInxw/IMQDJDNZYSgkA+BG4GRiWPdCc/HJ4TfeB5waTz2D0J/6nxJ9xbxzOUi7b2t\nWDLhro5T1Uj6MnCVmfWN25MI4erN7ViuGiQ9R7jm68tdFsdJi9eEnKokNl8cGpum+gC/JISEA2Bm\ng2rNAUnaT9Im8ZpPBnYm1Kwcp2pxJ+RUKyKE8H5CaI57mdCHUMtsR2gqnU8YY3R0E+G/jlMVeHOc\n4ziOUza8JuQ4juOUjdoUxKswlo7+farq5jd+MDbV+dJ+WRy3sqmgnPyMbDe/cKIcdE6px3jP+xNS\n2QF0bNc+ld2ylbmCqQpTzACgUvKLTYelsrtkTtuKZrevq09lt2TFslR2G3Tulspu3tKFqewA9ui9\nfSq7J2aPafFjs2LuW0W/c9pv+Lm2fkzXwGtCjuM4TtnwmpDjOE6t0ZCu5l4OqromJGkDSZPi8oGk\n2XF9vqTpReZxmqST4voNko6O62MlDWnN8juO47QKjY3FL2WmqmtCcRzIIAgyLcAiM/u9wmRz9xey\nl9TOzK4qRVkk1ccR0I7jOGUlqPpUB1XthApQL+kaYE+C0OIRZrY06olNAvYGRkVtrUVm9vt8GUk6\nmDAmpSNBZfcUM1skaSZBvuWLwO8IE4U5juOUlwqo4RRLVTfHFWBb4C9RtXk+8PXEsQ5mNiTO59Ik\nUUjxXMLEV7sStK3OTiSZZ2a7mtltWXYjJE2QNOEf/636KT8cx6kmrLH4pczUck1ohplNiusTCbM6\nZmiO+OTuBLXsp6ISfgeCKnGTeZnZ1cDVkD5E23EcJxWN1dMzUMtOKDmgoIE11XAXNyMfAQ/HOTxy\n0Zy8HMdxWp+GleUuQdHUcnNcqXgW2CvOG0Kcp6V/mcvkOI6TF7PGopdyU8s1oZJgZh9JGk4IYugY\nd59LmBPGcRyn8qiiwAQXMG0DvrLl4alu8ufquhZOlIN7Pp2Wyq4h5VfR8sZ0Vf8u7ToWTpSD2QvT\nz9BQp3QKJY1V8neyU69+qeymfjyzpOUoRH1dukYYpRRCapdSJmh5CwZ99kopFfTB/JdbLKOz7LUn\ni35gO/bfu6yyPV4TchzHqTU8MMFxHMcpGx6Y0HpIMkm3JLbbSfpIUkGFhBx5DZJ0aGlL6DiOU2aq\naJxQ1TkhQkj0jpIyIddfJCgipGEQ0CwnJKWcf8BxHKetqCLtuGp0QgAPAIfF9eOBUQCS6iS9Lql3\nYvsNSb0lHSPpJUmTJT0uqQNwIXBsFD09NoZfXyfpeUkvSjoi5jNc0mhJjwJjJN0k6WuZwkgamUnr\nOI5Tbswail7KTbU6oduA4yR1AnYGngOwEPR+C3BCTHcQMNnMPgLOA75kZgOBr5rZ8rjvdjMbZGa3\nA78AHjWzocD+wKWS1ot57QocbWb7Af8AhgNI6kHQp/t3soBJ2Z63F73TKjfBcRwnJ94c17qY2RSC\nDM/xhFpRkuuAk+L6t4Hr4/pTwA2Svgvki9c8GPippEnAWKATsGU89rCZfRzPPw7YNta4jgfuMrM1\negLN7OqoTzekb9ctcRzHaTOqqDmumvs3RgO/B4YBG2R2mtksSR9KOgAYSqwVmdlpkr5AaMabKGlw\njjwFfN3MXl1jZ7DLlue5CfgWcBxwSkmuyHEcpxT4pHZtwnXABWY2NcexawnNcndk5viRtLWZPWdm\n5wEfAVsAC4HkiLL/AqcrKpVK2qWJ898AnAVgZkVNoOc4jtMmeHNc62Nm75rZ5XkOjwa6sropDkL/\nzlRJLwFPA5OBx4ABmcAE4FdAe2CKpGlxO9/5PwRezjqH4zhO+fHmuNbDzNbSsjGzsYQ+nAwDCQEJ\nryTSHJUju4+B3bL2/U+O/G8g1HxWIakLYc6iUUUV3HEcp62ogBpOsVSdEyqEpJ8C32N1hFxrnOMg\nQoTcZWa2oFD6tNXNtBpwP+o6MJXdxZ++kMquc32HVHZzlhS8dSWnWjTg0jJv+adter60WnzdO3ZJ\nZffJ0kWp7Lp26JTKriXacQuXL01t22IqoIZTLDXnhMzsYuDiVj7HI0Df1jyH4zhOatwJOY7jOOXC\nPDquPEhalLU9XNKV5SqP4zhOWaii6DivCSWQ1C456DR7u1g7x3GcsuLNcZWHpH6EsUUbEsYJnWJm\n70i6AfgM2AV4StKnwNbA54B3JJ0C/A0YAqwEzjazx+Jsq0cRQsHrgf3a8nocx3HyUsIaTpRHexzo\nSPAZd5rZLyVtRZBQ2wCYCJxoZsvjDNQ3AYOBecCxZjYzX/411RwHdI5jfiZF6Z0LE8euAG40s52B\nkUByjNHmwJ5mdnbcHgAcZGbHA98HzMx2Ikj03Bh/FFhTT24NktpxM107znGctqS044SWAQdE3c1B\nwCGSdgcuIUQIbwN8Apwa058KfBL3XxbT5aXWnNDSKEY6yMwGEQRKM+wB3BrXbwb2ThxbpawQGW1m\nmfjKvQnqC8RxR28D/eOxVXpy2SS14/q5dpzjOG1Jw8rilwJYINPf3j4uBhwA3Bn33whkZhY4Im4T\njx+YUaHJRa05obRk68Jlbxdr5ziOU36aURNKttrEZUR2dpLqY+vSHOBh4E1gfqIv/F2gT1zvA8wC\niMcXkND3zGZdckJPE8RGIQxkfaJIuydieiT1J6hqv9qkheM4TjlpRnRcstUmLlevlZ1ZQ2xd2pwg\nDL19qYq6Ljmh04FTJE0BTgTOLNLur0CdpKnA7cBwM1vWSmV0HMdpOa2kHWdm8wmam3sAPRMzTW/O\n6hmuZxMEojMzUfcgBCjkpKai47J15ZKab2b2NqENM9tmeNb2+Vnbn5FjqoZcenL5OG5lj2KSrcWE\nlBEuaeV3Zj77t1R2Vx58TSq7ezu/n8rumY9eKZwoD+3q8k0l1TQrG9t2Bsq0cjhbd9kkld2HS+an\nsuvSrmMqu226bpbK7nWbXThRDob23CaV3fgFb6ayA7ivy46pbVtMaaPjegMrzGy+pM7AFwnBBo8B\nRxMi5E4G7osmo+P2M/H4o2b59bJqygk5juM4lHqc0KaEqOB6QuvZP83sfknTgdskXQS8SNDTJP5/\ns6Q3CCLRx+XKNIM7IcdxnFqjiKi3YokzWa81t5qZvUXoH8re/xlwTLH512SfkKSvSTJJTXaeRVmf\nzRLb10oa0PoldBzHaUWqaD6hmnRChEGlT8b/m2I4sMoJmdl3fJZUx3GqHrPilzJTc05IUlfCANNT\nSbRFSvpJnFl1sqSLJR1NkOIZGRUWOksaK2lITH98ZiZWSZck8lkk6dcxn2clbdzGl+g4jtM0XhMq\nK0cAD5rZa8A8SYMlfTnu/0KUnvidmd0JTABOiAoLq2agik10lxCi6QYBu0nKjAZeD3g25vM48N1c\nhUgOABuz5I1WulTHcZwcuBMqK8cTQgaJ/x8PHARcb2ZLAPJJ7STYDRhrZh/FEb8jgX3jseXA/XF9\nItAvVwbJAWAHdkkXHuo4jpMKn8qhPEjqRai97CTJCOrWBtxRwtOsSMS8N1Bj99BxnBqgoW3HtbWE\nWqsJHQ3cbGZ9zayfmW0BzCBoF50iqQusclYAC4FuOfJ5HthP0oYxNv54YFzrF99xHKcEeHNc2Tge\nuCdr312EwVajgQlRhO+ceOwG4KpMYELGwMzeB35KGBE8GZhoZvfhOI5TDVSRE6qppiQz2z/HvuS8\nQRdnHbuL4KQyDEscGwWMypFf18T6nayWMnccx6kMKqCvp1hqyglVKiPbpdPlWt6YbtRz5/oOqezS\nasAd2rFQnEdufvZO24uRNzGtSUXRmHL8xvvL0z1rDSm/iBev+CyV3YzFH6Sy+3TZklR2ry5Jd75l\nK1eksgO4tWO61+ueqc+4Gmss//ifYnEn5DiOU2uUULantXEn5DiOU2tUUU2oYGCCpF9ImiZpSuzA\n/0ITaYdIujzf8dZCUj9JS2P5MksHSV+V9NMm7HpK+t/E9maSvI/HcZzqplYCEyTtARwO7GpmyyRt\nCOTtcDCzCQQVgnLwZpz5L8nouOSjJ/C/hInrMLP3CGHejuM41UsFOJdiKVQT2hSYm5lJ1Mzmxhc1\nknaT9HTUUHteUjdJwyTdH4+vJ+m6eOxFSUfE/cMl3S3pQUmvS/pd5mSSDpH0QsxzTFP5FEM815Vx\nfWNJ98S8J0vakxAtt3WsOV0aa1QvxfSdJF0f9eNelLR/ofI7juNUBFUkYFqoT+gh4DxJrwGPALeb\n2ThJHQhTXR9rZuMldQeWZtn+gjCj3rcl9QSel/RIPDaIMD/FMuBVSVcAnwHXAPua2YzEgNKc+ZjZ\n4qzzbR3HAAE8ZWbfzzp+OTDOzI6MA1C7EsYC7ZipQUnql0j/fcDMbKc4JcRDkvrnK7+ZzUqeTNII\nYATATuvvyJZdt8x1fx3HcUpPFdWEmnRCZrZI0mBgH2B/4PbYxzIReN/Mxsd0n8Ja4a8HA1+VlBkY\n2gnIvInHmNmCaDMd6AusDzxuZjNinh8XyOflrOLmao5LcgBwUsy7AVggaf0m0u8NXBHTvyLpbSDj\nhHKVfw0nZGZXA1cDHL7lYeX/3HAcZ92himR7CkbHxRf2WGCspKmEucMnFpG3gK+b2RqDQWJgw7LE\nrkL6aznzKTPNKb/jOE7bUivRcZK2k7RtYtcg4G3gVWBTSbvFdN0kZb+I/wucrlg9krTW9LBZPAvs\nK2mrmD7THNfcfPIxBvhezKNeUg/ya8cBPAGcENP3J9S+KskROo7j5MQaG4teyk2hwISuwI2Spkua\nAgwAzjez5cCxwBWSJgMPE5rJkvwKaA9MkTQtbufFzD4i9KHcHfO8PU0+TXAmsH+szU0EBpjZPOAp\nhYnrLs1K/1egLqa/HRieCdBwHMepaBqt+KXMyCogOqLWOabvEalu8vOLZqQ635wlC1LZDemVbt6j\n5+amqyD+YaNhqex++OFjqezWBb648c6p7MbMmZrKLq280Aad8zVANM28pQtT2dWllGtqyfuxc/uO\nqew+XfxWi7WlFl/0raILvt65t5RVy8r7MhzHcWqNlTUUmOA4juNUGRXQzFYstTaf0CokbSLpNklv\nSpoo6YHEOJ9kuqfLUT7HcZxWw6f3Li8xku4e4EYzOy7uGwhsDLwWt9uZ2UozK4VyuuM4TuXgNaGy\nsz+wwsyuyuwws8lAvaQnJI0GpgNIWhT/HyZpnKT7JL0l6WJJJ0S5oKmSto7peku6S9L4uOxVhutz\nHMfJSy2FaFcrO5J/QO2uwJlmtlbTHDAQOA3YATgR6G9mQ4FrgdNjmj8Dl5nZbsDX47G1kDRC0gRJ\nE95aNDP1hTiO4zSbKgrRrsnmuAI8n5EGysF4M3sfQNKbBO08gKmE2hXAQcCAhERRd0ldzWxRMqOk\nbE/aEG3HcZxU1JJsT5UyjfxTMmQLnyZJDkZtTGw3svpe1QG7m1m6eY0dx3Famwqo4RRLrTbHPQp0\njErWAEjamSDE2lIeYnXTHJKaEk11HMdpc6zRil4KIWkLSY9F5Zxpks7MOv4jSaYw3xwKXC7pDYXJ\nUHdtKv+adEIWhjkfCRwUQ7SnAb8FPihB9mcAQ+LNnU7oQ3Icx6kcStsntBL4kZkNAHYHvi9pAAQH\nRZjp4J1E+i8D28ZlBPC3pjJ32Z42oF2HPn6TS8j6nbumtv1k6aLCiRynjKxcPrvFMjoLf3Bo0e+c\nblc+0KzzSboPuNLMHpZ0J0HP8z5giJnNlfR3YKyZjYrpXwWGZfrbs6nJmpDjOM46TTNqQslI3riM\nyJdtnPhzF+A5hVmuZ8fhL0n6sOb8au/GfTmp1cAEx3GcdRZrKH78TzKStykkdQXuAs4iNNH9nNAU\n1yLcCTmO49QaJY6Ok9Se4IBGmtndknYCtgImx+EqmwMvSBoKzAa2SJhvHvflpMXNcZIaJE2KUROT\nY6REVTTzSRok6dByl8NxHKeklDAwIcqg/QN42cz+CGBmU81sIzPrZ2b9CE1uu5rZB8Bo4KQYJbc7\nsCBffxCUpia01MwGxcJuBNwKdAd+WYK8W5tBwBDggWINMppzrVckx3GcllFM6HUz2IugIDNV0qS4\n7+dmlu+9+QBwKPAGsAQ4panMWxwdJ2mRmXVNbH8OGA9sCHQkhOcNIbQhnm1mj0mqBy4BDiEMBL3G\nzK6QNJPVERZDgN+b2TBJ5xOqfp8jTLP9Q0Ko4JcJ1byvmNkKSYOBPxJmhJ1LmA31fUljgecIqgc9\ngVPj9htA55jHb4EZBFmeTsBS4BQze1XScOComG89YYrzu83s3njNI4F/mtl9ue6RR8eVFo+Oc2qZ\nUkTHLTj5wKLfOT1uHFNbk9qZ2VvRyWwEfCvssp0kbQ88FKdTOAXoBwwys5WSehWR9dYEJzIAeAb4\nupn9WNI9wGGS/g1cARxhZh9JOhb4NfDtaN/OzIbG5rdfmtlBks4jOL0fAEjqDuwTy3QQ8BuCPhwE\nzbmdzexjSfsRHOG9knoAewInJwsbI0xGAKi+B3V16zXrPjqO46TFVlbPd29rBybsTXAMmNkrkt4G\n+hP0167KNGuZ2cdF5PWfWNuZSqiNPBj3TyU4tO0IwqUPx46yeiDZDnl3/H9iTJ+LHsCNkrYFDGif\nOPZwppxmNk7SXyX1Jjipu7Kb6JIRJ14TchynTaki2Z6SO6HYHNcAzElhvpLVwRKdso4tAzCzRkkr\nbHU7YkbXTcA0M9sjT94ZHbgG8l/3r4DHzOzIGA8/NnEsW3PuJkJN7zgKtHk6juO0KeWfoaFoShrF\nFmsGVxFG0xrwBHBCPNaf0J/zKvAw8D+S2sVjmea4mcDguP51mserQG9Je8Q820v6fAGbhUC3xHYP\nVocSDi9gewMhXh4zm97MsjqO47QapdSOa21K4YQ6Z0K0gUcIAp8XxGN/BepiE9rthECBZYQ5eN4B\npkiaDHwzpr8A+LOkCYQaS9GY2XKCcvYlMc9JhL6apniMMC3DpNiH9Dvgt5JepEAt0cw+BF4Grm9O\nOR3HcVqdxmYsZca141IiqQuhP2pXM1vQVNr1uvRLdZNXNKSLBG9M+Zu2q6tPZZeYW6lZpL2+lnDB\npsNS2V0058lUdm19jQdvMjCV3UMfZCuvFEddyt9+x179Utm9PH9W4UQ56NUpXUTl0pXLU9kB1Nel\n+8b/aMGrLY5W+/jI/Yp+CfS6Z1xZo+OqYlBppREj514GrijkgBzHcdoaW1n8Um5cticFZvYI0Lfc\n5XAcx8lJBTSzFUtFOiFJDYSmrnaEAaQnmtn88pbKcRynOrAqckKV2hy31MwGmdmOwMfA98tdIMdx\nnKqhigITKtUJJXmGOBdFFMS7VNJLkqbGiDYkDZM0TtJ9kt6SdLGkEyQ9H9NtHdN9RdJzkl6U9Iik\njeP+8yVdJ2lstD8jc3JJJ8VZVCdLujnu6y3pLknj47JXm98Vx3GcPFhj8Uu5qcjmuAxR/udAgoIr\nBP22QcAMZIb4AAAgAElEQVRAgjbdeEmPx2MDgR0INae3gGujTM+ZwOmEMT1PArubmUn6DvBj4EfR\nfnuCLFA34FVJfyOoO5wL7Bn17DLjmf4MXGZmT0raEvhvPHey7Ktkezq070W7dsnhSI7jOK1HJTiX\nYqlUJ9Q5qrX2IUShPRz37w2MMrMG4ENJ44DdgE+B8Rm5cElvEsYrQehb2j+ubw7cLmlToAOhvynD\nv+MYpmWS5gAbAwcAd5jZXFhDXuggwviijG13SV3NbJU6ZlK2J22ItuM4ThqsoaxR182iUpvjMtND\n9CXI8RTTJ7Qssd6Y2M7I+kDQsbvSzHYC/oc1pYGS9k1J+0C4b7vHfqtBZtYn6YAcx3HKSTU1x1Wq\nEwLAzJYAZwA/ihI/TwDHSqqPEkH7As83I8ukLM/JTSWMPAocI2kDWENe6CFCEx9x/6BmlMFxHKdV\nsUYVvZSbinZCAGb2IjAFOB64J65PJjiIH8eZ/IrlfOAOSRMJ8w0VOvc0wnQQ46IU0B/joTOAITFg\nYTpwWjPK4DiO06pUU03IZXvaAJ/KoXJoX5+uG7R7h86p7OYtXZjKLi0d27UvnCgHy1auKHFJmqZT\nuw6p7FY0ppSyakz3tu3cvmMqO4ClK5YVTpSDFSWY1G72HgcU/c7p88yjtTWpneM4jlNeGleWv5mt\nWNwJOY7j1BjV1MBV8X1CLUGSSbolsd1O0keS7k+Z37WSBpSuhI7jOKWnmgITar0mtBjYUVJnM1sK\nfJHV0XHNxsy+U7KSOY7jtBKV4FyKpaZrQpEHgMPi+vHAqMyBKNdzTmL7JUn9JK0n6d9RquelhDzQ\nWElD4vohkl6Iaca04fU4juM0iVnxS7mp9ZoQwG3AebEJbmfgOmCfAjaHAO+Z2WEAknokD8YxStcA\n+5rZjMT4IcdxnLLjNaEKwsymAP0ItaAHijSbCnxR0iWS9skxcd3uwONmNiOe4+PsDCSNkDRB0oTG\nxsXpL8BxHKeZNDao6KXc1LwTiowGfk+iKS6ykjXvQScAM3sN2JXgjC6SdF5zT2hmV5vZEDMbUle3\nXrpSO47jpKDRVPRSbtYVJ3QdcIGZTc3aP5PgbJC0K7BVXN8MWGJmtwCXZtIkeBbYV1ImvTfHOY5T\nMZip6KXcrAt9QpjZu8DlOQ7dBZwkaRrwHPBa3L8TcKmkRmAF8L2s/D6KUzXcLakOmEOIvHMcxyk7\n1dQnVNNOyMy65tg3Fhgb15cCB+cwnUmYIyjbdlhi/T/Af0pSUMdxnBJSCVFvxVLTTqhSSPtNUkXP\nUdWwoiGd9lhaDbi0WnVpy7m8jTXg0rJs5fJUdm39N5FW/w3K+/dbypqQpOuAw4E5ZrZj3DcIuIrQ\nj74S+F8ze15hkrU/A4cCS4DhZvZCU/mvK31CjuM46wwNjXVFL0VwA2HYSpLfEfrZBwHnxW2ALwPb\nxmUE8LdCmbsTchzHqTFKOVjVzB4HsoehGNA9rvcA3ovrRwA3WeBZoGecyTovFemEJG0gaVJcPpA0\nO66bpC9lpT1L0lrethV0406TdFIaW8dxnLakOSHayTGNcRlRxCnOIgRvzSIMf/lZ3N8HmJVI927c\nl5eK7BMys3nAIAjSOsAiM/t9vDnHsWbQwHHAj3NkU2rduKvS2jqO47QlzQm9NrOrgaubeYrvAT80\ns7skfQP4B3BQM/MAKrQm1AR3AodJ6gAgqR+wGWHa71w0pRvXS9K9cXbUZyXtLKlO0kxJPRPpXpe0\ncVJnTtLWkh6UNFHSE5K2L/mVOo7jpKQNtONOBu6O63cAQ+P6bGCLRLrNKfDxX1VOKMrjPE/o/IJQ\nC/qn5Z8e9jbgOEmdCLpxzyWOXQC8aGY7Az8ntGM2AvcBRwJI+gLwtpl9mJXv1cDpZjYYOAf4a/aJ\nXbbHcZxyUeLAhFy8B+wX1w8AXo/rowljLyVpd2CBmb3fVEYV2RxXgFEE53Nf/P/UfAnNbEqsLeXS\njdsb+HpM92jsh+oO3E6I9rg+5n970khSV2BP4I4QjQjAWnMAJ6u47X16b8dx2pBSyvFIGgUMAzaU\n9C7wS+C7wJ8ltQM+I0TCQXjPHgq8QQjRPqVQ/tXohO4DLosyO13MbGKB9BnduGHABkXk/wywTVTK\n/hpwUdbxOmB+DE10HMepOEr51Wtmx+c5NDhHWgO+35z8q6o5DsDMFgGPEfTgsgVJc5FPN+4J4AQA\nScOAuWb2abyJ9wB/BF6OQRLJ838KzJB0TLSVpIEtuCTHcZyS4gKmrc8oYCBFOCEze9fMcunGnQ8M\nljQFuJjQ0ZbhduBbZDXFJTgBOFXSZGAaITbecRynIqgmAVPl79N3SkXaPqG2/mXqlO6BbPRnqOR0\nab9WN2NRLE8p97OysSGVXbVQDumstOdcsXx2iz3DE5scXXTR9/ngzrJ6omrsE3Icx3GaoKECajjF\n4k7IcRynxmhMXQ9re6qiT0jSZZLOSmz/V9K1ie0/SDq7mXmeJalLKcvpOI5TCRgqeik3VeGEgKcI\nY3OIk8htCHw+cXxP4Olm5nkW0CwnJKm+medwHMdpcxqbsZSbanFCTwN7xPXPAy8BCyWtL6kjsAPw\ngqT/kzQ+SvFcACBpPUn/ljRZ0kuSjpV0BkHu5zFJj8V0B0t6RtILku6Ig1KJMj6XSHoBOEbS2Lj9\nvKTXJO3TxvfCcRynSaqpJlQVfUJm9p6klZK2JNR6niEos+4BLACmEgajbkvQMBIwWtK+QG/gPTM7\nDEBSDzNbEJvv9jezuZI2BM4FDjKzxZJ+ApwNXBiLMM/Mdo32pwHtzGyopEMJo4fXEu6LYqsjAOrq\ne1BXt17pb4zjOE4O0sVIloeqcEKRpwkOaE/CQNI+cX0Bobnu4Li8GNN3JTilJ4A/SLoEuN/Mcomd\n7g4MAJ6KUjwdCI4uQ/Z4oYxw30SgX67CumyP4zjlohJqOMVSTU4o0y+0E6E5bhbwI+BTgs7bfsBv\nzezv2YZR4udQ4CJJY8zswuwkwMNNyFNkK5Bm5vxtoLruoeM46wAlnN271amWPiEINaHDgY/NrCEq\navckNMk9TZhj6NuJvpw+kjaStBmwxMxuAS4Fdo35LQS6xfVngb0kbRNt15PUv60uzHEcp5Q0oqKX\nclNNX/FTCVFxt2bt62pmc4GHJO0APBOb1BYRpHe2IcwA2AisIEzGBKGp7EFJ75nZ/pKGA6NioAOE\nPqLXWvmaHMdxSk41tf+7bE8b4LI9TnNx2Z7Ssq7J9ty9yTeLLvpRH9zqsj21zi82HZbK7r5lM1PZ\nzVv+aSq7rbtsksru/eXzU9lt1XHDVHYPfzgllR3AwZukEzwfN3d6KrvlK1eks0vpTBa+OzaV3db9\n21aDt0/ndL/9KwtmpbLbsHOPVHZLG5YVTpSHbu3KNxa+IeUHZTlwJ+Q4jlNjVMIg1GJxJ+Q4jlNj\nVFN0XM04IUkNhECF9oSxWjcBl5lZNX0UOI7jtJhKiHorlppxQsDSzJTbkjYiRNF1JygarEJSOzOr\npgHFjuM4zaKaQoWqaZxQ0ZjZHIJkzg/i9NvDJY2W9CgwRlJXSWOiTtxUSUcASOon6RVJN0RduJGS\nDpL0lKTXJQ2N6YZGnbkXJT0tabsyXq7jOM4aNKr4pdzUpBMCMLO3gHpgo7hrV+BoM9sP+Aw4MurB\n7U+Q9cn8HNsAfwC2j8s3gb2Bc4CfxzSvAPuY2S7AecBvss8vaYSkCZImTFz0RmtcouM4Tk4amrGU\nm1pqjivEw1FlAUII/2+iwGkjQYdu43hshplNBZA0DRhjZiZpKqt14noAN0rallDzbZ99sqR23Pl9\nT6im2rHjOFVOJdRwiqVma0KSPkdw9HPirqT+2wkEde3BsR/pQ6BTPJYcGNCY2G5ktdP+FfCYme0I\nfCVh6ziOU3aqaT6hmqwJSeoNXAVcGWsx2Ul6AHPMbIWk/YG+zTxFD2B2XB/ekrI6juOUmkpwLsVS\nS06os6RJrA7Rvpkw5UMuRgL/ik1sEwh9PM3hd4TmuHOBf6csr+M4TqtgVdQc59pxbUDnzn1T3eQV\nKaVb0lJfl651tqEx3XeXa9WVns269kpll6O1oCg++WxRKrslK9LL4VQL5dSOu3KLbxX9R/KDWbe4\ndpzjOI5TOqrpM82dkOM4To3h0XFtiKRFWdvDJV0Z10+TdFITtsMk7dnaZXQcx2lLShkdJ+k6SXMk\nvZTYd2kc2D9F0j2SeiaO/UzSG5JelfSlQvlXvRNqCjO7ysxuaiLJMMKU4UUjyWuPjuNUNCUO0b4B\nOCRr38PAjma2M2Hyz58BSBoAHAd8Ptr8VVJ9U5nXtBOSdL6kc+L6GZKmR899m6R+wGnADyVNkrRP\nlO15NKYZI2nLaHuDpKskPQf8Lkr49I7H6qLX712my3Qcx1kDa8ZSMC+zx4GPs/Y9lNDgfBbYPK4f\nAdxmZsvMbAbwBjC0qfxr4as+E5qdoRcwOke6nwJbmdkyST3NbL6kq4BFZvZ7AEn/Am40sxslfRu4\nHPhatN8c2NPMGiQtIAx4/RNwEDDZzD5KnkzSCIJ+He3a9aJdu64lu2DHcZymWNmMPqHkuypydVR8\nKZZvA7fH9T4Ep5Th3bgvL7XghFapZ0PoEwKG5Eg3BRgp6V7g3jx57QEcFddvJowHynCHmWWklq4D\n7iM4oW8D12dnlJTtSRui7TiOk4bmvHCS76rmIukXhHGZI9PYQ403x2VxGPAXgpDp+BR9O6tkf8xs\nFvChpAMIVc3/lKyUjuM4LaQRK3pJS/zgPxw4wVYPOJ0NbJFItjmr1WVysk44IUl1wBZm9hjwE4Ls\nTldgIdAtkfRpQqcahOa2J5rI9lrgFtasITmO45Sd1taOk3QI8GPgq2a2JHFoNHCcpI6StgK2BZ5v\nKq91wgkRpnS4Jcr0vAhcbmbzgX8BR2YCE4DTgVMkTQFOBM5sIs/RBEe2VlOc4zhOOSllYIKkUcAz\nwHaS3pV0KnAl4QP+4fj+vArAzKYB/wSmAw8C3y/0ke6yPSmRNIQwffg+hdK6bE9uXLan9LhsT+VQ\nTtme5kwfc/7bI122p9qQ9FPge4Qmu4oj7cu9S7uOqewWr/gslV05nIk7vtykdSa9O/csnCgHb6/4\nMJVdWsrxu9fXNTk8plVZqep5XteV5riSYmYXm1lfM3uy3GVxHMfJppTNca2N14Qcx3FqjGqaT2id\nrglJukzSWYnt/0q6NrH9B0lnl6d0juM46WiLEO1SsU47IeAponZcDOPekKB5lGFPQth2kyiwrt9L\nx3EqhGpqjlvXX5xPE1QSIDifl4CFktaX1BHYAZgedeRekDRV0hEAUWfuVUk3Rbstcp3AcRynrVmJ\nFb2Um3W6T8jM3pO0MgqV7kmIhe9DcEwLgKnAEuBIM/tU0obAs5Iy2nTbAieb2bPZebt2nOM45aL8\nrqV41mknFHma4ID2BP5IcEJ7EpzQU4Rw/99I2pfQ39cH2Djavp3LAYFrxzmOUz6qKTDBndDqfqGd\nCM1qs4AfAZ8S1BBOAHoDg81shaSZQKdou3it3BzHccqMVVFdaF3vE4JQEzoc+NjMGszsY6AnoUnu\naYLO3JzogPYH+pavqI7jOIVpbe24UuI1odDvsyFwa9a+rmY2V9JI4F9Rd24C8EoZyug4jlM0lRB6\nXSzrvBOK4nrds/YNT6zPZXUEXTY7FnOO9inlOxot3XdK945dUtlt03WzVHYzFn+Qyi6tbuG8pQtT\n2QHs2KtfKrvXFjSpRp+XZSuXp7JL+wrp03nDVHbvLfq4cKIcpJXfWXDusFR2x1z7SSq7OSvSPTO7\ndNwklR3APCufPl6DOyHHcRynXFRCM1uxuBNyHMepMTwwoZWQtImk2yS9KWmipAck9W+jc8+M44Qc\nx3EqGg9MaAUUJjy5B7jRzI6L+wYSxuy8Vs6yOY7jVBJeE2od9gdWmNlVmR1mNhl4UtKlkl6KsjrH\nAkgaJmmcpPskvSXpYkknSHo+pts6pust6S5J4+OyV9y/gaSHJE2LoqaK+y/MEj39taSmZmB1HMdp\nU6qpJlRNTmhHYGKO/UcBg4CBwEHApZI2jccGAqcRNOBOBPqb2VDgWsJU3gB/JsyQuhvw9XgM4JfA\nk2b2eUINbMu4/zrgJFglenoccEt2oSSNkDRB0oTlKz9NfdGO4zjNpcGs6KXcVE1zXBPsDYyKodYf\nShoH7EZQPBhvZu8DSHoTeCjaTCXUrCA4rgGJ6Y27S+oK7EtwcJjZvyV9EtdnSponaRdCU+CLZjYv\nu1BJ2Z7u632u/L+04zjrDD5OqHWYBhzdTJtkoH5jYruR1ddeB+xuZmvMUa2mpwO+FhgObEKoGTmO\n41QM3ifUOjwKdIzq1ABI2hmYDxwrqV5Sb0IN5vlm5PsQq5vmkDQorj4OfDPu+zKwfsLmHuAQQo3r\nv82/FMdxnNajmvqEqqYmZGYm6UjgT5J+AnwGzATOAroCkwkDzX9sZh9I2r7IrM8A/iJpCuF+PE7o\nR7oAGCVpGkFD7p1EWZZLegyYH5sBHcdxKoZqao5TWumUdZkYkPACcIyZvV4ofbsOfVLd5HYp5X5W\nNqbziz07rZfK7tNlS1LZNZbh2Wtfn+67K62EUkNj235rduvQOZXdwuVLS1ySpjl4k4Gp7L5Er1R2\nV32WfhTHV7psncrupcYFqeweeOeBJvsCiuGovl8t+o/r7rdHt/h8LaGamuMqAkkDgDeAMcU4IMdx\nqpe0DqjcmFnRS7mpmua4SsHMpgOfK3c5HMdx8lFNzXHuhBzHcWqMSgg4KJaKaI6T9DVJVkwwgaRr\nY5NYS8/ZT9I3E9tDJF3e0nwdx3HKjTXjX7mpCCcEHA88Gf9vEjP7TmwSayn9iCHYMd8JZnZGCfJ1\nHMcpK41Y0Uu5KbsTiuoEewOnEiRwMrpvYyXdKekVSSOjgClx/5C4vijqxk2T9IikofH4W5K+GtP0\nk/SEpBfismc89cXAPpImSfphPOf90aaXpHslTZH0bByPhKTzJV2XOIc7LcdxKo5Sy/ZI6pl4H78s\naY/4nnxY0uvx//UL57Q2ZXdCwBHAg2b2GjBP0uC4fxfCGKABhECAvXLYrgc8GvXdFgIXAV8EjgQu\njGnmAF80s12BY4FMk9tPgSfMbJCZXZaV7wUEOZ6dgZ8DNyWObQ98CRgK/FJS+1wXldSOa2xcXMx9\ncBzHKQmt0Bz3Z8J7enuCJufLhHfoGDPbFhgTt5tNJTih44Hb4vptrG6Se97M3jWzRmASofksm+XA\ng3F9KjDOzFbE9Uz69sA1kqYCdxCcWiH2Bm4GMLNHgQ0kZaYA/7eZLYvTfs8h6MethZldbWZDzGxI\nXV268TeO4zhpKGVznKQeBCWaf0AYrG9m8wkViBtjshuBr6Upa1mj4yT1Ag4AdpJkQD1B9eDfrKn7\n1kDusq6w1YHuq7ThzKxRUib9D4EPCd67jqC00BKKKZfjOE7ZaM74nyiFNiKx6+oowJxhK+Aj4Po4\nh9tE4Exg44xANPABeT7IC1HumtDRwM1m1tfM+pnZFsAMYJ8SnqMH8H6sUZ1IcHQQmu+65bF5AjgB\nQv8UMNfMfD4Gx3GqgubUhJKtNnG5Oiu7dsCuwN/MbBdgMVlNb7EykCrKodxO6HiCGGiSuygiSq4Z\n/BU4WdJkQn9OpoNmCtAgabKkH2bZnA8MjnpyFwMnl7A8juM4rUqDNRa9FMG7wLtm9lzcvpPglD7M\nzN0W/5+TpqyuHdcGbNxj+1Q3efGKZYUT5aBTu5yxEgUZ2nObVHavLvkgld2shame2RZpzm28Xs9U\ndnMWz09ll7akacW8+vXYJJXdjAXpfsO6pqc8ycvOvbZKZbe4Id3fxM0dNktl93Olb73fvX6DVHYX\nzhzZYi23ffocWPSj98TsMQXPJ+kJ4Dtm9qqk8wlBYQDzzOxiST8FepnZj5tbVu/PcBzHqTFaYfzP\n6cBISR2At4BTCC1p/5R0KvA28I00GbsTchzHqTFK7YTMbBIwJMehA1uad7n7hJqNpIY4wDSz9EuZ\nz1mSupS2dI7jOOXHVbRbl6VmNqhwsoKcBdwCpJsMx3Ecp0KpBDmeYqm6mlAu8knz5JP/iXI7mwGP\nxRlSkfS3qHAwTdIFibwvljQ9Svj8XlI3STMySgmSuie3Hcdxyk2jNRa9lJtqrAl1ljQprs8wsyNZ\nLc3zmaRtgVGsbr/cBfg88B7wFLCXmV0u6Wxg/6h8APALM/tYUj0wJurFzSZIAG0fpxfvaWYLJY0F\nDgPuJejd3R2VGlaRHADWrdPGdO6QLirLcRynuXhNqHVZGvXeBkUHBE1L8xQj/wPwDUkvAC8SnNYA\nYAFBYeEfko5iddPdtYToEOL/12dnlhwA5g7IcZy2pJr6hKrRCeUiKc0zBOiQOFZQZkfSVsA5wIFR\ntPTfQCczW0kQKr0TOJyoU2dmTwH9oppCvZm9VOoLchzHSYtP5dD25JPmaYqkbE93gpLCAkkbA1+G\nVdNM9DCzBwiObmDC/ibgVnLUghzHccpJNU1qV419Qrn4K3CXpJMItZVi5k64GnhQ0ntmtr+kF4FX\ngFmEviMITuo+SZ0Ig9jPTtiPJEwdMapE1+A4jlMSWqIq0ta4bE9KJB0NHGFmJxZK265Dn3TCfikl\nUdLSo1O6KSeWrVxROFEOlqaUJWrJE9u9Y7qhYSsbG1LZtfU1btI11bxifLDok5RnTMcpm+1ZOFEO\n1k/53TypId31/b8VXVPZAdzSKd3f799n3tHiP/wdNhpa9CP08pzn2/ZFk0Wt1ITaFElXEJrsDi13\nWRzHcbKphGa2YnEnlAIzO73cZXAcx8lHNTXH1UpgQsnkfLLyPC32MyHphtgE5ziOU9F4YEJ5KJWc\nzyrM7KpS5uc4jtMWeE2oQigg5zNO0n2S3orSPCdIel7SVElbx3TnSzonK88DJN2b2P6ipOyJ+RzH\nccpGozUUvZSbWnJCnRNNcRmnkJHz2RU4Frg8kX4gcBqwA2FsUX8zG0pQQ2iqz+cxYHtJveP2KcB1\nJbwOx3GcFlFNg1VrvTmuPXClpEEEtYT+iWPjzex9AElvAg/F/VOB/fOdJGrI3Qx8S9L1wB7ASdnp\nktpxqu9BXV268GfHcZzmUk1Db2rJCeUiKedTR9CBy5AcwNGY2G6k8H25HvhXzO+OKO+zBmZ2NWFA\nbOpxQo7jOGmohBpOsdS6E+oBvGtmjZJOpjg5n4KY2XuS3gPOBQ4qRZ6O4ziloppqQrXUJ5SLvwIn\nS5oMbE9xcj7FMhKYZWYvlzBPx3GcFtNoVvRSbmqmJmRma+lrmNnrwM6JXT+J+8cCYxPphiXWVx0z\ns/MT+4dnZb83cE2LCu04jtMKVMJkdcVSM06oLZE0kVCr+lEx6ffovX2q87y++L1UdguXL01ld1+X\nHVPZ3dox3WN000fjU9ktSanHBlBfl67yv3BZulng035nphXz6tYunTbeh6TTVquvS9fCPc/S/Yaz\nLV1jxu71G6Syu6V+eSo7gD/9fb/Uti3F+4RqHDMbXO4yOI7j5KOa+oTcCTmO49QYldDXUyxlD0yQ\n9DVJJilnm1VraLZFxYR0WvKO4zgVjk/v3TyOB56M/7cVw4BmOSFJXmt0HKcqqCbFhLI6oTh99t7A\nqcBxcZ8kXSnpVUmPABvF/YdIuiNhO0zS/XH9YEnPRH24O2K+SJop6YK4f6qk7aO69mnAD6PEzz7Z\ntS1JixLneELSaGB63PetqDE3SdLfJZVk7JHjOE6paGhsLHopN+WuCR0BPGhmrwHzJA0GjgS2AwYQ\n5HAyNZZHgC9IyujfHAvcJmlD4qDRqBE3gTWn4Z4b9/8NOMfMZgJXAZeZ2SAze6JAGXcFzjSz/pJ2\niOfdK0oENQAn5DKSNELSBEkTPlg8u+gb4jiO01J8KofiOR74c1y/LW63A0aZWQPwnqRHAcxspaQH\nga9IuhM4DPgxsB/BYT2lMB12B+CZxDnujv9PBI5KUcbnzWxGXD8QGAyMj+fqTBBJXYukbM8+fQ4s\n/y/tOM46QzUFJpTNCUnqBRwA7CTJCJI6BjQ1LcJtwA+Aj4EJZrZQwRs8bGb5+pQyAxIayH+9K4m1\nQkl1BEeWITkwQcCNZvazJsroOI5TVioh4KBYytkcdzRws5n1NbN+ZrYFMAOYBxwrqV7SpqypaD2O\n0Dz2XYJDAngW2EvSNgCS1pOUVMvOxUKgW2J7JqGGA/BVgvp2LsYAR0vK9FP1ktS38KU6juO0HaVs\njov98a9KekPST0td1nI6oeNZu9ZzF7Ap8DohEOAmEk1rsYnufuDL8X/M7CNgODBK0pSYvpBEwb+A\nIzOBCQT5nf2ixtwe5NGYM7PphP6nh+K5Ho7ldRzHqRgaGxuLXpoiBl79hfDOHQAcL2lAKctatuY4\nM1trzh4zuzxX2qw0PyA0ySX3PQrsliNtv8T6BEJoNjEQYues5Lsn1nNqzMV9twO3Fyqn4zhOuShh\nY9xQ4A0zewtA0m2EgLLpJTtDcwY1+VL6BRjRlnblOKfbVbddNZW11u1aYyFMvjkhsYxIHDsauDax\nfSJwZSnPX+4QbSfOvtqGduU4p9tVt105zul2bYSZXW1mQxLL1W15fndCjuM4Tj5mA1sktjeP+0qG\nOyHHcRwnH+OBbSVtJakDQdlmdClPUO7Bqk4c0NqGduU4p9tVt105zul2FYAFkYAfAP8ljOW8zsym\nlfIcip1NjuM4jtPmeHOc4ziOUzbcCTmO4zhlw51QGyKpTtI3yl0Ox3GcSsGdUBtiZo0E5e9USPpK\nFFiteCT1kbSnpH0zSzNsu6Q4X4vuTZpzpjhHf0ljJL0Ut3eWdG5rnzctkjZoZnpJ2qJwysrA5wKr\nDKrihVZjPCLpHElbRAHUXlFRvBiOBV6X9Lt806HnIorBPtbcgkrqLennkq6WdF1mKcLuEuApgs7e\n/8XlnCLs9pQ0HXglbg+U9Ncii5v23qQ6p6SHJfVMbK8v6b8FzK4BfgasADCzKcTJHIs4317xnK9J\nelTnOhIAAB+ySURBVEvSDElvFWG3raQ7JU2Pdm8VYxd5VmGSyEOjWn2TWIhyeqDIvLPLebqk9VPa\n1kvaTNKWmaVI09clXdpcLTRJd0s6rDkfPdFBf0vSeXF7S0lDm3PemqXckhHr2kJQCs9e3mqGfXfg\nfwjq4c8QRl53K8JuDNCjmWV9GrgE+Abw9cxShN2rQMcU9+Y5wsC4FxP7XmrNe5P2nMn0Te3LOj4+\nOx0wqchre4UgIrkRsEFmKcLuScI8WFOAvsD5wIVFnlPAF4FRwBvAb4D+BWxuBHZL8dtfFM/xT+AQ\nYuRuEXanA3OBacDUuEwp0rYbQZH/6fjMjAC6F2F3EDASeBO4GNiuCJu/EYRAX47b62eeh3V9KXsB\nfEnxo4UX0FmEKSj+Q1AdP72AzX3AO8A/gMszSwGbol6QOez+A3RNYfdc/D/5kp7cmvcm7TkJkyRu\nmdjuC7xQxH3ZOpOOoMv1n+bcmxT3dGL8f2r2vmbmsz9hpPx8wpQqe+RJ9wphfq43o+NrjlMQ8CXC\nNC0Zp7d1AZs3KMIZF3Hu/eL1LY6OdJsibHoApwGzoiM7BWifJ23mN0/9bNfq4oNV25jY93A24QU2\nQtK2hC+p+4uwPYIwbcU2hGkuhprZnJjndOCKJszvZvUss8Vyv6RDzay5TSxLgEmSxrB6UkHM7IwC\ndrMk7QmYpPbAmcDLxZywBfcm7Tl/ATwpaRzh5bnP/2/vTMMkq6p0/X5VzCCTgIjIqIIMMtsiqCCi\nIqKgIg3YoHYrfUUopka9DkwqIiAiXpBBQWQSmgZBZZa5xJahEPBagOBtwKEYC0QUKL77Y+2ojIyM\nOFNGRBZZ+32efDLjxFmxd5yMOGvvNVJeD2xvIklxbUmPELvgj1UYC+BaSUcT/8P2a3p7idw/ktno\nvpR0+AiwRJUBk0/oY0TRyr8Qu45LgA2BC4DVu4i9p8prd8O2Jf0Z+DOhyJYB/lPSVbZ7+VIfAmY3\nGS/5hLYnlMdqwLHEDudthFmxZ1+yjmtzR5LbEtiTVK2/gxfSeE7yywPFfRTmE3Ky6pCR9GNiFb2H\n7fXSTXK67Q0ryJ5BZCzf0OW5bWxfUyK/ECNfrJm2Xyg5/xlgceB5kh+DuFcsWSK3Z7fjtn9YIrcc\n0e79XcSN/Upgmu3Hi+SS7Bk0uDbjHHM5RlqA3GL7sTKZJLc4MMX2M1XOTzLdfHq2/c4Suc0Ipbo0\ncASxev+m7VsqjHkv8CPgdNsPdzz3OdtHFciuACzSNtH/KRlrGrAHYVo7DbjY9gstBWp7zR5y3wfW\nAn7GaOX8rZK3R/KNXQt83/b0jue+02vRJOmiNOaPgDNs/6ntuVttb9pFZnfCb7kxsdP6CPAl2xeU\nzXOyk5XQkGl9SCXdYXujdOxO2xuUyE0FrnaXPkwVx92K+PD/gbjZvhbYs9tNux/UVXjjHGtc16bm\nWGvb/p2kjbs9321nIuljts+SdEAPmcIbZroRf8T2+Y0m3YB0Tb9p+8Cach8gdhQrAbMIM+X/tb1u\nidxhxCLi/3V57o22u+5OJR3S7bjtwyrMdQnbfy07r4vc1rabBPqsTfjnBFzT6z3Nb2Rz3PB5XtKi\njGzL16RtBdcL23MkvSRpKdtNzA/HAu+2PTON+wbC4bxJkVC6qbTCq6+raDbcig6FJ6lU4UlanTD5\nrEbbZ9P2B4rkxnNtJP2Q2Pk8lR4vAxxr+5M9RA4gzG7HdpsK0G1nsnj6/Youz5Vi+yVJBxNO+1pI\n2pQwHa7K6Gva2dSxc8w5yUxZlyOI3eHVtjeStDUFJkeNRIYe3/G4NY8nim7WLWUjaYn0uI5S+Yqk\nrwLPAZcTjS73t31Wj7l+qNvfbXPpau5OCv0e22uTojAzI2QlNHwOIT7wr5V0NrAF4cuowl+BuyRd\nRVsL8gq+FgiH6cw2mXuTD6Qnkr5BdKw9Ox2aJmkL218oGauRwgMuJgInLqW+vbzptXlTSwGl85+U\ntFGvk21/Ov2uvOuyfXL680RHO/omXC3pIKKrb/v7e6JE7mwiRP4u6l/TGZIuIfw/7WMW+RZfsP24\nIjF7iu1rJX274PzbCOUtYBXgyfT30kQgTTe/01wkrUeYxZZNjx8jTN1Vimy+2/bBknYiFkwfAm4A\nuiohYIeC1zI9fK5Joc+UtEqZWXJ+JCuhIWP7Kkm3E6tFEavwSr4EmgUXtLhV0mmMfMF2J7ooFvE+\nYENHkm1r13AHketSRG2Fl/i7K7R470HTazNF0jK2n4S5K/HS74WknYHLbT+jSDjdGDjC9h0FYjdL\n+gOhSP6rNWZFdkm/9247ZmCNErlHbTctvb8I8Dijd3c9b7aJp9Ku5EbgbEmzaFNgndheHUDSqcBF\nrSAYSdsBO1aY4ynAAS3zWNqFnwpU2cW1PpPbAxfYnq2CdCjbn6jwmr1YBrhH0n8zWqEX7vLnB7JP\naEj08iG0qBDl1HqdRYnIupmlJ4+WW5i4gW2ZDt1IrMx7mgIl/QbYqrXaTjfo68pMOYqE1pcYrfCm\nFpi4WnK7Aa8nggPqRIA1RtIewP8mVvsiHMZfs/2jErnf2H6TpC2JHJejga/Y/qcSuTcTCao7ElF7\n5/Uy//QDSdsAuxJ5Yu3XtOliptc4+xFhyr8loiOnEP/3pYCzywI9JN1le/2yY13kxvhTq/hY03nf\nIP4PzwFvJnZfP+31PxyPb0/SO3rIXF82z8lOVkJDokd0U4vSKKf0GjsAxwAL2V5d0oZE4mHhairZ\npM+0vXvNOe9KJONdS9yg3w583vaPS+RqK7wkdyQR8vp7RkxHVa/N64EjgXUYHZVVtlNA0rpEHgzA\nL2z/toLMHcnncSSRg3NOe7BJBfnlgG8Bu9suLR+jhqH9ks4C1iaSOduvaeGCIMmuTIS2b5EO3Ujs\n3B/ucu4xxO5jbcL0dzOhlKZXMBmiqDZxI6MXLm+3XRjynSLVbidMchD+p01s71Q2ZpJfFpidTGaL\nEcmqf+5x7l62Tx5PMERmLFkJvYyQdBthGrnOI5F1d9ter4LsTcA7bT9fc8xXE34hgP/u9QXtB5Lu\nB9apO8ckexPhbzuOsN1/ggiD/koF2anAqxjtuC8LKf4pkXOzLWGKe464Pj1X4JKWBHYidkJrAhcB\n59u+rcIcG4X2S5ppe62y1+8hexVwDqNv8Lvb3rZAZiFgU0IhbZ5+nrJdWBonKYNDiIWOCd/M4WUK\nLAWSHMboBc+hVU2dyafUuXA5s4psHRTpDq2b7UKEKfBZl6Q7zA9kn9CQ6BZN005F88gLXezWVZ3N\nDxA+iUsYbZMeY0LQ2DDk1sp3JUkr9TKPSTrf9kcl3cXIF24uZWY84G7CJDKr/O2MYVHb10iSI8z3\n0KS0C5WQpH2Im99fgDnEjs9EpFQRHyXKyxxj+6mkrP+jROZOIvjicNu/LH1Ho1nT9i5pd4rtv6nI\ngTHCdEnrVNnddWF526e3PT4jmd2KWJQon7RU+vkjsTMqJCmbaZIWt93Th9RF7kmgSmDOGNKOZitC\nCf2cKIt0E5HsXCRXO4rT9tzIyPR/+yAjOWbzNVkJDY9GkTUd3JP8JlOTOWZfwuRRhd+nnymUhwo3\nCUOGqDYA8P6Kc+pkaeB3kn7NaP9FFedt08oA0wizVmlyagfLkQI7NFIwsyz8dg3blrSE6ueoNArt\nJ250MyQ9mM4XYY4rU7IAj0v6GBHZCOFb6nqdJJ0CrAs8Q9Tjmw58q8aO5K1EkuoSwCqSNgD2sv2Z\nHud/2/Z+ki6l+4KnymfmI8AGRCmdT0h6Fb0j49oZTxQnDvPTxUkJfr6u/GQjK6EhMc7Imhb7EDkf\n/yDMJFcQeRmFJHPTK2yXVrKGkTBkYDvbf+94rUW6iLTkWpnjn7H9uQ65o4DPjZUaRVdbe0WmAYsR\nivkIwsezRwW5pmVffsZIaPEiRCjxTOJG3It1JbXCiSXpUSJh+O4K4x3K2ND+Kp+p91Y4pxefJHxC\nxxHvdTq90wlWARYmavU9Quyen+pxbjeOI0r+XAJg+04Vt/9omQiPqTFGJ885crBeTKbSWUQSdxm1\nozg7LCFTCJPl33ucPl+RldCQGE9kTRvb2/4ioYhar7szEdnVk+R03aLonB5MJ/wdZcc62ZaxCme7\nLsdGMc5IodVs/5rIF/oEzL02vyqRewC4TlKtsi9dIrk2Brqu2tvoFk58ChXCiW1fmcyLtUL7k2kS\ndZTRqcjKnTuK9Dl6qMs4701mpnWJ93MgsJ6kJ4Bf2i5dYNh+qMPCOKfg3JYfbUPbx3fMcRpRZLWM\nWxXtOE4l/G1/Jaqvl3F82sXUieJst4S8SOQlfbDCWJOerISGx7iy5hNfYKzC6XasG5UTDyWtCLwG\nWFSRuNm6MyxJ7Da6Iul/ETfiNRXh3S1eQYHZUNJNtrfscN7CiOmoivO26bX5n/SzUPpphO3bJRWG\nZwOLu63ci+3rFHXkSpF0je1tiB1Y57Eiua5ldCjesbU4gbELjm7HgLlmprslPUXsLmcTptk3U77L\nbVpIdk9StYU2Pt7lWLf5thYN35N0OREZ95simcT6RBTnO2mLOKS3mRrgNNs3tx9ICr2J/3NSkZXQ\nkEihnVOBp20fV0dWkbj3PuA1ktrNAEsSq6oq1Ek8fA/xRV6ZuIG1lNDTRE5NL84h2hUcyWhb9zNF\nUU62t0y/ayvo8V6bpmG1HTvaKcSN+Y8lYg9I+jKjo80KG8wl8+diwHIpEqx9QfCaClOtVUYnjbk5\nsZtZvuN9Lgl0DSeXtG+SeStR7HZ6+vkBFQITiJYIxxPv6RFil7F3r5NTgMZuwOppcdXiFUBpSHh6\njblK3PYfOo8VsDPh36sTxVlLoc9PZCU0RJJZbFfC/l2HPxJO8A8QZoMWzwD7Vxy7sk/KUe36h5I+\nbPvCGnKzgdmSjgeecKoSLWlJSf9ku6dpTKPra9VhXNdGUVL/YGJn0B6mW5ab1K4wXyR2KGXX6pNE\nOHFL8d+YjhWxF9EfaSXi/bUvCL5bIgv1y+hA7AiXIO4P7e/zacKZ343ViF3n/m2+wcok02KdPLbp\nwJ+IAJH2AJpniD5GPemDYq8cxdlEoc9v5DyhISPpOCJHoLMGWGlVAEkLOlWjTl+e11Y0H9RKPGyT\n+TpRSbm9uOeBtr9UMtYdwMbJPEOKWrvVduGqT9JPiAZ0tetrtV+bmnJXEv+Lg4jV+J5EqZuyIIqh\nImkf20X9onrJXU1UBTiSuGHPIjqflvqhJK3a5lOaQjQqfLruHCrOs1Hx2oZjTWNEsT/CaMV+qu1C\n5S7pOiKEvzSKU1EpYSvis/W9tqeeAS61fV/T9zFZyEpoyKhhX5gkex2x4l+AWBXPIhIWq6z4myQe\njqkAIOn2CspkhjuSKJXK3JTI3QBsBNSur5Xs64cyUi265U8qrJgg6Tbbm7TPT9KvbW/W4/zCOmw9\nbkS1Zbq8RrdadV8tW7wkn9Nz1Cyjk2TPIW6ec4gb7pLA8baPLpOti6Q7ibDnUYVWy4JVJL2FWFy9\nkdjBTaViEug4FHvtEjztCj0zmmyOGzIeX8+bpWw/LenfiDI8h3QEABTRJPFwqqSFncrtKPJUFq4w\n1gPJR3BSevwZCnwfkl5HVCz4csdTbyNMLlX4PmF+u42CqKoutHZPf5K0PWHeW7bg/M2J6LBzici7\nKgmjTWQ6+bLtCxS16t5F1Ko7CegZDJFMnD9Nn7mXiPYadVgnfd52J3x9nyeub9+VEM2L136XqEBx\nARH2vAcFHVHbsX1CCoZYjdG7r8Jk1YZRnAsrcqk6xypdfE52shIaMoq6ah9m7Ifx8AriCygy8z9K\nW5h2RSonHrZxNnCNpNOJG+fHqXYj+3fgO8CXiOCHayhuff1t4Au2RzmwU3jv1wkFU8Zs25dVOK+T\nr0paiggpPoFY7RftLFckQtBbjvGfAee6uHVAE5lOWop1e+AU2z9T9MLpicffg2rBFKm2I/BdR6fT\nQZlOmoQ9t865X9JU23OA05M5uKzSO4qcrTWBGYxcX1NeMaHJ7usCwhx3GvUWSZOerISGz0+I0NXb\nqJbx3s7hRILqTbZ/LWkNIjmwCt0SDwuDFWwflcwk70oyVxDmrkJszyJWp1V5VacCSq9zl6TVKr7G\ntZKOJpz+lW9iHikAOpuRIqZF588hkkYvTwuKXYk8o8N6+RKayHThEUknE8rsqPQ6UyrIjacH1clE\nPsudwA2SViX8JoOgSdgzwN8U9epmSPomsXOucl0gdk7rtHyXNWiy+3rR9kkl58yXZJ/QkFHFgqM9\nZF9ZxZbfTxR5QrsRYakPAhf2unFKOtj2NyWdQPdSKl1vfJLus/36Hs/db/t1FebZyNfWxCGeFMD2\nhDJZjcjy/4HtR/op0yG/GFH94C7b96Ud8fq2ryyR27Pb8RQBWRtJC9iumhZQ53UbFa9NinEWEeyz\nP+HzOtH2/RVkLwD2rRvNJ+lW25t2+BELK6hLOjTN8yJGL5IqhZNPZvJOaPhMl7R+t5V/BW6RNAM4\nHbisygqul0Jo0U0xKDqh7pp+HiOix1TBn9VKLixrltfJrZI+ZfvUjnn8G6PDrnsyDl9brTpgks4E\n1iMKXh7mCiV3msh04ihYOouoFn0fERZeugu2/cMUho4rdnVVSXUPogVFv2lUvLbN2f8cEf5eh+WA\n3yoazdWpVdhk99VaDLQXua3SlHDSk3dCQ0LS3cRNbgGicdsD1CwoKUmEaeyTRHuF84EzbN9bINO+\nEj6Mjsz1bitiSS8RIdz/2lpRSnqgQqRZo1WyonDkRcDzjCidTQl7+06u0D4ivcbXgZVsbydpHWBz\n24X+JEm/ckkjuo7zX2LErFWpukMTmS6vcQhxTday/QZJKxHdQLuWY0qflUOAzxI3SBGK64Qy/6Mm\noG9OnbDnDrn3Ewm5nVGRVa5po0Zzaff1F+LzWWv3lRlLVkJDQtKTQM/eL3XDNxWZ72cR5YDuJJrN\nFda9KjMZtJ23I2Hz3oLwZZxHlB1ZvURubvi2pBNs71PpzYzIb03sGCASV39RQ/YyYof4RdsbSFqA\nqI5c1plz6N1cm5B2wBsBt3ukl1TPsPe0i9kO+LTtB9OxNYiIustds2rHoBmHQrgf+BBhphzozUzS\nKm6Qw5ZkGzUlnB/I5rjh8WBdRdOJpFcS+T3/QqzE9iF8CxsSTtJCJUGBWW7USfbFRKn5xYkii/sB\nK0g6CbiowA/RHnpcu2Cqo65aUQfaIpazfb6kL6TXelFSlSikpg7xYfO8bbei01Rec+5fgG3dVuTU\n9gMpQvJKCqp2aHT5ozFUDGqohe3r0262vYFiFdPcQ8DdTRRQgyi3i0lldiRdaPvDNYY7ndjlt5KE\nHyG+s1kJTfQE5iNWKLCxV62i/Usi2XRHj650cKuk7/WQaYyjudg5wDmKagk7E5WweymhidxWP5uU\ndOsm/RaqtWhoUgdsIjg/RcctLelThEn21ILzF3SXKtu2H01h10W0++HGmHAHgaSPEvlH1xGLmRMk\n/Yft/ywRPRj4uaTrqVEFPVE3yq19kVXXl9O0KeGkJyuh4TGVqMc1ng/eWr1WfLaP6nZcoytTLyap\nFWJbp0J1q4PlKemnF2srkmfF6EradRqpNeUAYle4pqSbgeXpXeesnfF0cx04ioTi6UQu1dZEiPRa\nwFdsX1UgWqRUCxVuu59Q0n5NI+lq8kWinNCsNO7ywNVAmRL6GhGGvggNqqC7Xo6Re/xdhaZNCSc9\nWQkNjz+VOYR7obayL90WT0XOWzeoTD0O3jjEsUbhaKXwDuIGLWCmq9WSG08310pI+j/AOe4o5V+R\nlQkFtDZR0uZmQimVRQ1u0LbgGDUd6vUVGtbudkqH+e1xquX7rOSGKQ/Uj3JrXVMRbU7qLOgOZWxT\nwo83nPekIgcmDImqQQE9ZB+loOxLmfN2fkBRomZ7xub7FJplmjrEa85tGmH2eTUR0Xiu7TtqvsZC\nhMnorUQZoM2Bp2yv06959hi3tFZgn8Y5moiOa1X02AX4jUsKySblcXWBn7JIdqhRbslc3GpKeEs3\nc+n8SFZCQ0LSsm6YmJZusK2yL2+iWdmXSY2knxPtkjsLYJaGEzd0iDeZ46qEMvpnYFHihntuUYh9\nm+xShOLZIv1emogI60fb+M6xRplwgb+1nqKGCbfiWK8jKmbcrGiBvWV66imi0OrvK8x1cWIX+0KV\nOY4nyq0pki4l/KuXJF9rJpGV0MsMjZR9OZpIfKxa9mVSUxSuXCLX6RB/G1DFIT4uFJUofgC8yXbP\nvjKKopfrEqX/fwXcQqyinxzk/IaFpJ/SvW7g+sDXbe/QXXJcY7anEtSNcms65juI3d32RC7UeURx\n2b8Peux5newTepmgsWVfvkMkeM6TJCfsKrZnDmnIyyS9u4FZpqlDvDYpd2k7Yie0DaH4Di0RW4Wo\nXH4fEdb7MLFLmCw0qhsoaW3bv5PUq9V4UZ7XeKLcGpHMu9cnq8Y7gU8Ri5C+7SpfrmQl9DJAfSj7\nMkwk7QAcQ9jaV5e0IXB4P539XbgFuEjRfK2SWSbR1CFeGUktU+r7iF5J5xFJpKVmGdvvTaG86xL+\noAOB9RQVxn9pe+Dh0wNm6YLnFi147gCiMvuxXZ4ry/MaT5RbY9LCbAdiR7Qx9VtrTEqyOe5lgPpQ\n9mWYSLqNuAlc15bdf1dZ9YJxjvkgkVhbK3O+h0P8LtsH93Fuc4g23ReOx4ym6I67BaGM3g+80nbR\nTXyeR9K5wC/cvW7gtrZ3GcCYc4jvkwhFNzCfV9uY5wNvJiLkfgxcb7u0VuH8QFZCmb4j6Rbbb2mP\nCGzqs6kx5g3AVk2+2B0O8Rtt99XMOc7IyH0JpfNWYoc3ve3nrpf7jUz9qRtYuzHdsJH0HiKKL/cS\n6iAroUzfkfR9opHd54kGfvsSGfz/PsAxzyDs+5dRP3O+/XWmALvaPruPc3uYgsrTRXOU9C1SbpBr\nthx4OaGGdQPVozGdB1BaqAlpgdMT2/81rLnMq2SfUGYQ7EM4/P9BhKVeARR2Ae0DD6afhaiQOS9p\nSWBv4DVEpYWr0uODiIKwfVNCRLWMRknDtnuWeppMuHndwKaN6YZFK7pvBWI321KuWxO72fleCeWd\nUGa+RNJPgCeJenzbEDcJAdNsz+jzWENJ+JwfUcPGdMNG0dl2j9Y8FU0Jz7D9nomd2cSTd0KZvpO+\ncDvbfio9XgY4b5BfOEVn1W7dXHtFSa3RCpSQdBpRsmWVAeVt5EKVg6NbYzrb/uAEzqkbK3coyr8Q\nPZDme7ISygyC5VoKCKL4qaQVBjzmQW1/L0L4oooa7M2tK2d7jqSHB5g4uM2AXjczOs+qlWz8zxMz\nlUKukXQFo6Mwa5camoxkJZQZBC+1l0ZJ5WoGave13VnQ8+a0Ou5Fe4HP9oKUfQ/TbVquKVOOow/R\nRsBuRFuOB4G+tzUZL7Y/K2kn4O3p0HRgxQmc0jxDVkKZQfBF4CZFj5fW6vTTgxxQ0rJtD6cAmxAF\nKbtSVConM+8j6Q1EAvCuwGNE7o1sbz2hEyvmD0RwQktZXjihs5lHyIEJmYEgaTmiYjAMoWJwSlY1\nofReJL7kh9u+aZDjZiaGlMB9I/CvrarXkh6wPZQyPFXpoSwPsp39QYm8E8oMioWBJ4jP2DqSsH3D\noAazXdbaPDO5+BDh+7lW0uVEKaR5MQDkd4SyfH+bstx/Yqc0b5F3Qpm+I+kowvF6DyNtFTzI2nGS\n9iZK/7dH5O1q+8RBjZmZeCQtTpRr2pUoFXUmcFGT/kKDQNKOhLLcgijZcx5wWl40jZCVUKbvSJpJ\ntCgYWvtiSTNsb9hxrHG5nMzLj7Tw2BnYxfY8FZE4ryvLiaSv1YIzmcQDwIJDHnNqqjYNzG0EWFo5\nITN5sP2k7VPmNQUEYPtZ2+ek/kgrA3cAhV1j5xfyTijTdyRdCGxA1I9rr+M2sHpeqRr2qsDJ6dBe\nwEO2DxzUmJlMZvxkJZTpO5L27Hbc9sD6p6TCo3sxkhh6FWF7z1WLM5l5mKyEMgNhAjqrImkhYC0i\nVHum7RdKRDKZzASTfUKZvpM6q84gooGQtKGkSwY85lZEC+zvAicC90p6e6FQJpOZcPJOKNN3enRW\nvdv2esWS4x5zt9bOKyUJnmt7k0GNmclkxk/eCWUGwQu2Z3ccG3QH0AXbTX+272X4EXqZTKYmuWJC\nZhDcI2k3Imz69URn1ekDHvPW1JLhrPR4d+DWAY+ZyWTGSTbHZfqOpMWIIqbvJkqpXAEcMcBWCUha\nmOiMumU6dCNw4jATZjOZTH2yEspMGiQtD2D70YmeSyaTqUZWQpm+IenbtveTdCndu5z2vXZcqpJw\nCPBZRnycc4ATbB/e7/EymUx/yT6hTD/5Ufp9zBDH3J8oDrmZ7QcBJK0BnCRpf9vHDXEumUymJnkn\nlOk7qVjjc7ZfSo+nAgvb/tsAxroD2LazX1EyzV2ZC5hmMvM2OUQ7MwiuARZre7wocPWAxlqwW8O8\n5BfKIdqZzDxOVkKZQbCI7b+2HqS/Fys4fzw83/C5TCYzD5B9QplB8KykjW3fDiBpE+C5AY21gaSn\nuxwXsMiAxsxkMn0i+4QyfUfSZkQHyT8SymBFotHYbRM6sUwmM8+RlVBmIEhakKhoDbmidSaT6UH2\nCWX6hqTNJK0IkJTOxsDXgGMlLTuhk8tkMvMkWQll+snJpGCA1EbhG8CZwGzglAmcVyaTmUfJgQmZ\nfjLV9hPp712AU2xfCFwoacYEziuTycyj5J1Qpp9MldRa2GwD/KLtubzgyWQyY8g3hkw/ORe4XtJj\nREj2jQCSXkeY5DKZTGYUOTou01ckvQV4NVEy59l07A3AEq28oUwmk2mRlVAmk8lkJozsE8pkMpnM\nhJGVUCaTyWQmjKyEMplMJjNhZCWUyWQymQnj/wMWumRuCEvqvQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11ff4bbd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = SpectralCoclustering(n_clusters=5)\n",
    "model.fit(visGrid)\n",
    "\n",
    "fit_data = visGrid[np.argsort(model.row_labels_)]\n",
    "fit_data = fit_data[:, np.argsort(model.column_labels_)]\n",
    "\n",
    "annot_lookup_sorted = []\n",
    "for i in np.argsort(model.row_labels_):\n",
    "    annot_lookup_sorted.append(Genre_ID_to_name[nr_ids[i]])\n",
    "    \n",
    "sns.heatmap(fit_data, xticklabels=annot_lookup_sorted, yticklabels=annot_lookup_sorted, annot=False)\n",
    "plt.title(\"After biclustering; rearranged to show biclusters\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the above figure, \"boxes\" or groups of movie genres automatically emerge! \n",
    "\n",
    "Intuitively - Crime, Sci-Fi, Mystery, Action, Horror, Drama, Thriller, etc co-occur. \n",
    "AND, Romance, Fantasy, Family, Music, Adventure, etc co-occur. \n",
    "\n",
    "That makes a lot of intuitive sense, right?\n",
    "\n",
    "One challenge is the broad range of the drama genre. It makes the two clusters highly overlapping. If we merge it together with action thriller, etc. We will end up with nearly all movies just having that label. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Based on playing around with the stuff above, we can sort the data into the following genre categories - \"Drama, Action, ScienceFiction, exciting(thriller, crime, mystery), uplifting(adventure, fantasy, animation, comedy, romance, family), Horror, History\"**\n",
    "\n",
    "\n",
    "Note: that this categorization is subjective and by no means the only right solution. One could also just stay with the original labels and only exclude the ones with not enough data. Such tricks are important to balance the dataset, it allows us to increase or decrease the strength of certain signals, making it possible to improve our inferences :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interesting Questions\n",
    "This really should be a place for you to get creative and hopefully come up with better questions than me. \n",
    "\n",
    "Here are some of my thoughts:\n",
    "- Which actors are bound to a genre, and which can easily hop genres?\n",
    "- Is there a trend in genre popularity over the years?\n",
    "- Can you use sound tracks to identify the genre of a movie?\n",
    "- Are top romance actors higher paid than top action actors?\n",
    "- If you look at release date vs popularity score, which movie genres have a longer shelf life?\n",
    "\n",
    "Ideas to explore specifically for feature correlations:\n",
    "- Are title length correlated with movie genre?\n",
    "- Are movie posters darker for horror than for romance end comedy?\n",
    "- Are some genres specifically released more often at a certain time of year? \n",
    "- Is the RPG rating correlated with the genre?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Based on this new category set, we will now pull posters from TMDB as our training data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting pulling movies from TMDB. If you want to debug, uncomment the print command. This will take a while, please wait...\n",
      "Pulled movies for genres - 12,14,16,18,27,28,35,36,37,53,80,99,878,9648,10402,10749,10751,10752,10770\n"
     ]
    }
   ],
   "source": [
    "# Done before, reading from pickle file now to maintain consistency of data!\n",
    "# We now sample 100 movies per genre. Problem is that the sorting is by popular movies, so they will overlap. \n",
    "# Need to exclude movies that were already sampled. \n",
    "movies = []\n",
    "baseyear = 2017\n",
    "\n",
    "print('Starting pulling movies from TMDB. If you want to debug, uncomment the print command. This will take a while, please wait...')\n",
    "done_ids=[]\n",
    "for g_id in nr_ids:\n",
    "    #print('Pulling movies for genre ID '+g_id)\n",
    "    baseyear -= 1\n",
    "    for page in xrange(1,6,1):\n",
    "        time.sleep(0.5)\n",
    "    \n",
    "        url = 'https://api.themoviedb.org/3/discover/movie?api_key=' + api_key\n",
    "        url += '&language=en-US&sort_by=popularity.desc&year=' + str(baseyear) \n",
    "        url += '&with_genres=' + str(g_id) + '&page=' + str(page)\n",
    "\n",
    "        data = urllib2.urlopen(url).read()\n",
    "\n",
    "        dataDict = json.loads(data)\n",
    "        movies.extend(dataDict[\"results\"])\n",
    "    done_ids.append(str(g_id))\n",
    "print(\"Pulled movies for genres - \"+','.join(done_ids))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f6=open(\"movies_for_posters\",'wb')\n",
    "pickle.dump(movies,f6)\n",
    "f6.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "f6=open(\"movies_for_posters\",'rb')\n",
    "movies=pickle.load(f6)\n",
    "f6.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's remove any duplicates that we have in the list of movies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "originally we had  1782  movies\n",
      "1700\n",
      "After removing duplicates we have  1700  movies\n"
     ]
    }
   ],
   "source": [
    "movie_ids = [m['id'] for m in movies]\n",
    "print \"originally we had \",len(movie_ids),\" movies\"\n",
    "movie_ids=np.unique(movie_ids)\n",
    "print len(movie_ids)\n",
    "seen_before=[]\n",
    "no_duplicate_movies=[]\n",
    "for i in range(len(movies)):\n",
    "    movie=movies[i]\n",
    "    id=movie['id']\n",
    "    if id in seen_before:\n",
    "        continue\n",
    "#         print \"Seen before\"\n",
    "    else:\n",
    "        seen_before.append(id)\n",
    "        no_duplicate_movies.append(movie)\n",
    "print \"After removing duplicates we have \",len(no_duplicate_movies), \" movies\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, let's remove movies for which we have no posters!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Total movies : ', 1782)\n",
      "Started downloading posters...\n",
      "0\n",
      "Downloaded first. Code is working fine. Please wait, this will take quite some time...\n",
      "10\n",
      "20\n",
      "30\n",
      "40\n",
      "50\n",
      "60\n",
      "70\n",
      "80\n",
      "90\n",
      "100\n",
      "110\n",
      "120\n",
      "130\n",
      "140\n",
      "150\n",
      "160\n",
      "170\n",
      "180\n",
      "190\n",
      "200\n",
      "210\n",
      "220\n",
      "230\n",
      "240\n",
      "250\n",
      "260\n",
      "270\n",
      "280\n",
      "290\n",
      "300\n",
      "Done with  300  movies!\n",
      "Trying to get poster for  The Dark Knight Rises\n",
      "310\n",
      "320\n",
      "330\n",
      "340\n",
      "350\n",
      "360\n",
      "370\n",
      "380\n",
      "390\n",
      "400\n",
      "410\n",
      "420\n",
      "430\n",
      "440\n",
      "450\n",
      "460\n",
      "470\n",
      "480\n",
      "490\n",
      "500\n",
      "510\n",
      "520\n",
      "530\n",
      "540\n",
      "550\n",
      "560\n",
      "570\n",
      "580\n",
      "590\n",
      "600\n",
      "Done with  600  movies!\n",
      "Trying to get poster for  Back to the Future\n",
      "610\n",
      "620\n",
      "630\n",
      "640\n",
      "650\n",
      "660\n",
      "670\n",
      "680\n",
      "690\n",
      "700\n",
      "710\n",
      "720\n",
      "730\n",
      "740\n",
      "750\n",
      "760\n",
      "770\n",
      "780\n",
      "790\n",
      "800\n",
      "810\n",
      "820\n",
      "830\n",
      "840\n",
      "850\n",
      "860\n",
      "870\n",
      "880\n",
      "890\n",
      "900\n",
      "Done with  900  movies!\n",
      "Trying to get poster for  Eastern Promises\n",
      "910\n",
      "920\n",
      "930\n",
      "940\n",
      "950\n",
      "960\n",
      "970\n",
      "980\n",
      "990\n",
      "1000\n",
      "1010\n",
      "1020\n",
      "1030\n",
      "1040\n",
      "1050\n",
      "1060\n",
      "1070\n",
      "1080\n",
      "1090\n",
      "1100\n",
      "1110\n",
      "1120\n",
      "1130\n",
      "1140\n",
      "1150\n",
      "1160\n",
      "1170\n",
      "1180\n",
      "1190\n",
      "1200\n",
      "Done with  1200  movies!\n",
      "Trying to get poster for  Good Boy!\n",
      "1210\n",
      "1220\n",
      "1230\n",
      "1240\n",
      "1250\n",
      "1260\n",
      "1270\n",
      "1280\n",
      "1290\n",
      "1300\n",
      "1310\n",
      "1320\n",
      "1330\n",
      "1340\n",
      "1350\n",
      "1360\n",
      "1370\n",
      "1380\n",
      "1390\n",
      "1400\n",
      "1410\n",
      "1420\n",
      "1430\n",
      "1440\n",
      "1450\n",
      "1460\n",
      "1470\n",
      "1480\n",
      "1490\n",
      "1500\n",
      "Done with  1500  movies!\n",
      "Trying to get poster for  Joe Dirt\n",
      "1510\n",
      "1520\n",
      "1530\n",
      "1540\n",
      "1550\n",
      "1560\n",
      "1570\n",
      "1580\n",
      "1590\n",
      "1600\n",
      "1610\n",
      "1620\n",
      "1630\n",
      "1640\n",
      "1650\n",
      "1660\n",
      "1670\n",
      "1680\n",
      "1690\n",
      "1700\n",
      "1710\n",
      "1720\n",
      "1730\n",
      "1740\n",
      "1750\n",
      "1760\n",
      "1770\n",
      "1780\n",
      "Done with all the posters!\n"
     ]
    }
   ],
   "source": [
    "poster_movies=[]\n",
    "counter=0\n",
    "movies_no_poster=[]\n",
    "print(\"Total movies : \",len(movies))\n",
    "print(\"Started downloading posters...\")\n",
    "for movie in movies:\n",
    "    if counter%10==0:\n",
    "        print(counter)\n",
    "    id=movie['id']\n",
    "    title=movie['title']\n",
    "    if counter==1:\n",
    "        print('Downloaded first. Code is working fine. Please wait, this will take quite some time...')\n",
    "    if counter%300==0 and counter!=0:\n",
    "        print \"Done with \",counter,\" movies!\"\n",
    "        print \"Trying to get poster for \",title\n",
    "    try:\n",
    "        #grab_poster_tmdb(title)\n",
    "        poster_movies.append(movie)\n",
    "    except:\n",
    "        try:\n",
    "            time.sleep(7)\n",
    "            grab_poster_tmdb(title)\n",
    "            poster_movies.append(movie)\n",
    "        except:\n",
    "            movies_no_poster.append(movie)\n",
    "    counter+=1\n",
    "print(\"Done with all the posters!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1782\n"
     ]
    }
   ],
   "source": [
    "print len(movies_no_poster)\n",
    "print len(poster_movies)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "f=open('poster_movies.pckl','w')\n",
    "pickle.dump(poster_movies,f)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "f=open('poster_movies.pckl','r')\n",
    "poster_movies=pickle.load(f)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "f=open('no_poster_movies.pckl','w')\n",
    "pickle.dump(movies_no_poster,f)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "f=open('no_poster_movies.pckl','r')\n",
    "movies_no_poster=pickle.load(f)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Congratulations, we are done scraping!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a dataset out of the scraped information!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This task is simple, but **extremely** important. It's basically what will set the stage for the whole project. Given that you have the freedom to cast their own project within the framework I am providing, there are many decisions that you must make to finalize **your own version** of the project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we are working on a **classification** problem, we need to make two decisions given the data at hand - \n",
    "* What do we want to predict, i.e. what's our Y?\n",
    "* What features to use for predicting this Y, i.e. what X should we use?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many different options possible, and it comes down to you to decide what's most exciting. I will be picking my own version for the example, **but it is imperative that you think this through, and come up with a version which excites you!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an example, here are some possible ways to frame Y, while still sticking to the problem of genre prediction -\n",
    "\n",
    "* Assume every movie can have multiple genres, and then it becomes a multi-label classification problem. For example, a movie can be Action, Horror and Adventure simultaneously. Thus, every movie can be more than one genre.\n",
    "\n",
    "* Make clusters of genres as we did in Milestone 1 using biclustering, and then every movie can have only 1 genre. This way, the problem becomes a simpler, multi-class problem. For example, a movie could have the class - Uplifting (refer Milestone 1), or Horror or History. No movie get's more than one class.\n",
    "\n",
    "For the purposes of this implementation, I'm going with the first case explained above - i.e. a multi-label classification problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, for designing our input features i.e. X, you may pick any features you think make sense, for example, the Director of a movie may be a good predictor for genre. OR, they may choose any features they design using algorithms like PCA. Given the richness of IMDB, TMDB and alternate sources like Wikipedia, there is a plethora of options available. **Be creative here!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another important thing to note is that in doing so, we must also make many more small implementation decisions on the way. For example, what genres are we going to include? what movies are we going to include? All these are open ended!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## My Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implementation decisions made - \n",
    "* The problem is framed here as a multi-label problem explained above. \n",
    "* We will try to predict multiple genres associated with a movie. This will be our Y.\n",
    "* We will use 2 different kinds of X - text and images. \n",
    "* For the text part - Input features being used to predict the genre is a form of the movie's plot available from TMDB using the property 'overview'. This will be our X.\n",
    "* For the image part - we will use the scraped poster images as our X. \n",
    "\n",
    "NOTE : We will first look at some conventional machine learning models, which were popular before the recent rise of neural networks and deep learning. For the poster image to genre prediction, I have avoided using this for the reason that conventional ML models are simply not used anymore without using deep learning for feature extraction (all discussed in detail ahead, don't be scared by the jargon). For the movie overview to genre prediction problem we will look at both conventional models and deep learning models. \n",
    "\n",
    "Now, let's build our X and Y!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's identify movies that have overviews. **Next few steps are going to be a good example on why data cleaning is important!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1668"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies_with_overviews=[]\n",
    "for i in range(len(no_duplicate_movies)):\n",
    "    movie=no_duplicate_movies[i]\n",
    "    id=movie['id']\n",
    "    overview=movie['overview']\n",
    "    \n",
    "    if len(overview)==0:\n",
    "        continue\n",
    "    else:\n",
    "        movies_with_overviews.append(movie)\n",
    "        \n",
    "len(movies_with_overviews)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's store the genre's for these movies in a list that we will later transform into a binarized vector. \n",
    "\n",
    "Binarized vector representation is a very common and important way data is stored/represented in ML. Essentially, it's a way to reduce a categorical variable with n possible values to n binary indicator variables. What does that mean? For example, let [(1,3),(4)] be the list saying that sample A has two labels 1 and 3, and sample B has one label 4. For every sample, for every possible label, the representation is simply 1 if it has that label, and 0 if it doesn't have that label. So the binarized version of the above list will be -\n",
    "~~~~~\n",
    "[(1,0,1,0]),\n",
    "(0,0,0,1])]\n",
    "~~~~~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# genres=np.zeros((len(top1000_movies),3))\n",
    "genres=[]\n",
    "all_ids=[]\n",
    "for i in range(len(movies_with_overviews)):\n",
    "    movie=movies_with_overviews[i]\n",
    "    id=movie['id']\n",
    "    genre_ids=movie['genre_ids']\n",
    "    genres.append(genre_ids)\n",
    "    all_ids.extend(genre_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MultiLabelBinarizer\n",
    "mlb=MultiLabelBinarizer()\n",
    "Y=mlb.fit_transform(genres)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[28, 12, 35]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "genres[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1668, 19)\n",
      "[349 253 256 671 226 450 439 135  47 432 241 132 277 187 137 235 294  99\n",
      " 125]\n"
     ]
    }
   ],
   "source": [
    "print Y.shape\n",
    "print np.sum(Y, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(list_of_genres)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is interesting. We started with only 19 genre labels if you remember. But the shape for Y is 1666,20 while it should be 1666,19 as there are only 19 genres? Let's explore."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's find genre IDs that are not present in our original list of genres!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tmdb genre object!\n",
    "genres=tmdb.Genres()\n",
    "# the list() method of the Genres() class returns a listing of all genres in the form of a dictionary.\n",
    "list_of_genres=genres.list()['genres']\n",
    "Genre_ID_to_name={}\n",
    "for i in range(len(list_of_genres)):\n",
    "    genre_id=list_of_genres[i]['id']\n",
    "    genre_name=list_of_genres[i]['name']\n",
    "    Genre_ID_to_name[genre_id]=genre_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in set(all_ids):\n",
    "    if i not in Genre_ID_to_name.keys():\n",
    "        print i"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, this genre ID wasn't given to us by TMDB when we asked it for all possible genres. How do we go about this now? We can either neglect all samples that have this genre. But if you look up you'll see there's too many of these samples. So, I googled more and went into their documentation and found that this ID corresponds to the genre \"Foreign\". So, we add it to the dictionary of genre names ourselves. Such problems are ubiquitous in machine learning, and it is up to us to diagnose and correct them. We must always make a decision about what to keep, how to store data and so on. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "Genre_ID_to_name[10769]=\"Foreign\" #Adding it to the dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(Genre_ID_to_name.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we turn to building the X matrix i.e. the input features! As described earlier, we will be using the overview of movies as our input vector! Let's look at a movie's overview for example!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The overview for the movie Zootopia  is - \n",
      "\n",
      "\n",
      "Determined to prove herself, Officer Judy Hopps, the first bunny on Zootopia's police force, jumps at the chance to crack her first case - even if it means partnering with scam-artist fox Nick Wilde to solve the mystery.\n"
     ]
    }
   ],
   "source": [
    "sample_movie=movies_with_overviews[5]\n",
    "sample_overview=sample_movie['overview']\n",
    "sample_title=sample_movie['title']\n",
    "print \"The overview for the movie\",sample_title,\" is - \\n\\n\"\n",
    "print sample_overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## So, how do we store this movie overview in a matrix?\n",
    "\n",
    "#### Do we just store the whole string? We know that we need to work with numbers, but this is all text. What do we do?!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The way we will be storing the X matrix is called a \"Bag of words\" representation. The basic idea of this representation in our context is that we can think of all the distinct words that are possible in the movies' reviews as a distinct object. And then every movie overview can be thought as a \"Bag\" containing a bunch of these possible objects.\n",
    "\n",
    "For example, in the case of Zootopia the movie above - The \"Bag\" contains the words (\"Determined\", \"to\", \"prove\", \"herself\"......\"the\", \"mystery\"). We make such lists for all movie overviews. Finally, we binarize again like we did above for Y. scikit-learn makes our job easy here by simply using a function CountVectorizer() because this representation is so often used in Machine Learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What this means is that, for all the movies that we have the data on, we will first count all the unique words. Say, there's 30,000 unique words. Then we can represent every movie overview as a 30000x1 vector, where each position in the vector corresponds to the presence or absence of a particular word. If the word corresponding to that position is present in the overview, that position will have 1, otherwise it will be 0. \n",
    "\n",
    "Ex - if our vocabular was 4 words - \"I\",\"am\",\"a\",\"good\",\"boy\", then the representation for the sentence \"I am a boy\" would be [1 1 1 0 1], and for the sentence \"I am good\" would be [1 1 0 1 0]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "content=[]\n",
    "for i in range(len(movies_with_overviews)):\n",
    "    movie=movies_with_overviews[i]\n",
    "    id=movie['id']\n",
    "    overview=movie['overview']\n",
    "    overview=overview.replace(',','')\n",
    "    overview=overview.replace('.','')\n",
    "    content.append(overview)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1668\n",
      "Following the events of Age of Ultron the collective governments of the world pass an act designed to regulate all superhuman activity This polarizes opinion amongst the Avengers causing two factions to side with Iron Man or Captain America which causes an epic battle between former allies\n",
      "1668\n"
     ]
    }
   ],
   "source": [
    "print len(movies_with_overviews)\n",
    "print content[0]\n",
    "print len(content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Are all words equally important?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### At the cost of sounding \"Animal Farm\" inspired, I would say not all words are equally important. \n",
    "\n",
    "For example, let's consider the overview for the Matrix - "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'Set in the 22nd century, The Matrix tells the story of a computer hacker who joins a group of underground insurgents fighting the vast and powerful computers who now rule the earth.'"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_movie_info_tmdb('The Matrix')['overview']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For \"The Matrix\" a word like \"computer\" is a stronger indicators of it being a Sci-Fi movie, than words like \"who\" or \"powerful\" or \"vast\". One way computer scientists working with natural language tackled this problem in the past (and it is still used very popularly) is what we call TF-IDF i.e. Term Frequence, Inverse Document Frequency. The basic idea here is that words that are strongly indicative of the content of a single document (every movie overview is a document in our case) are words that occur very frequently in that document, and very infrequently in all other documents. For example, \"Computer\" occurs twice here but probably will not in most other movie overviews. Hence, it is indicative. On the other hand, generic words like \"a\",\"and\",\"the\" will occur very often in all documents. Hence, they are not indicative. \n",
    "\n",
    "So, can we use this information to reduce our insanely high 30,000 dimensional vector representation to a smaller, more handle-able number? But first up, why should we even care? The answer is probably one of the most used phrases in ML - \"The Curse of Dimensionality\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Curse of Dimensionality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This section is strongly borrowing from one of the greatest <a href=\"https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf\">ML papers I've ever read.</a>\n",
    "\n",
    "This expression was coined by Bellman in 1961 to refer to the fact that many algorithms that work fine in low dimensions become intractable when the input is high-dimensional. The reason for them not working in high dimensions is very strongly linked to what we discussed earlier - having a representative dataset. Consider this, you have a function $f$ dependent only one dependent variable $x$, and $x$ can only integer values from 1 to 100. Since it's one dimensional, it can be plotted on a line. To get a representative sample, you'd need to sample something like - $f(1),f(20),f(40),f(60),f(80),f(100)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's increase the dimensionality i.e. number of dependent variables and see what happens. Say, we have 2 variables $x_1$ and $x_2$, same possible as before - integers between 1 and 100. Now, instead of a line, we'll have a plane with $x_1$ and $x_2$ on the two axes. The interesting bit is that instead of 100 possible values of dependent variables like before, we now have 100,000 possible values! Basically, we can make 100x100 table of possible values of $x_1$ and $x_2$. Wow, that increased exponentially. Not just figuratively, but mathematically exponentially. Needless to say, to cover 5% of the space like we did before, we'd need to sample $f$ at 5000 values. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For 3 variables, it would be 100,000,000, and we'd need to sample at 500,000 points. That's already more than the number of data points we have for most training problems we will ever come across."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basically, as the dimensionality (number of features) of the examples grows, because a fixed-size training set covers a dwindling fraction of the input space. Even with a moderate dimension of 100 and a huge training set of a trillion examples, the latter covers only a fraction of about $10^{−18}$ of the input space. This is what makes machine learning\n",
    "both necessary and hard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, yes, if some words are unimportant, we want to get rid of them and reduce the dimensionality of our X matrix. And the way we will do it is using TF-IDF to identify un-important words. Python let's us do this with just one line of code (And this is why you should spend more time reading maths, than coding!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The min_df paramter makes sure we exclude words that only occur very rarely\n",
    "# The default also is to exclude any words that occur in every movie description\n",
    "vectorize=CountVectorizer(max_df=0.95, min_df=0.005)\n",
    "X=vectorize.fit_transform(content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are excluding all words that occur in too many or too few documents, as these are very unlikely to be discriminative. Words that only occur in one document most probably are names, and words that occur in nearly all documents are probably stop words. Note that the values here were not tuned using a validation set. They are just guesses. It is ok to do, because we didn't evaluate the performance of these parameters. In a strict case, for example for a publication, it would be better to tune these as well. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1668, 19)"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape\n",
    "Y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, each movie's overview gets represented by a 1x1365 dimensional vector.\n",
    "\n",
    "Now, we are ready for the kill. Our data is cleaned, hypothesis is set (Overview can predict movie genre), and the feature/output vectors are prepped. Let's train some models!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "f4=open('X.pckl','wb')\n",
    "f5=open('Y.pckl','wb')\n",
    "pickle.dump(X,f4)\n",
    "pickle.dump(Y,f5)\n",
    "f6=open('Genredict.pckl','wb')\n",
    "pickle.dump(Genre_ID_to_name,f6)\n",
    "f4.close()\n",
    "f5.close()\n",
    "f6.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Congratulations, we have our data set ready!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A note : As we are building our own dataset, and I didn't want you to spend all your time waiting for poster image downloads to finish, I am working with an EXTREMELY small dataset. That is why, the results we will see for the deep learning portion will not be spectacular as compared to conventional machine learning methods. If you want to see the real power, you should spend some more time scraping something of the order of 100,000 images, as opposed to 1000 odd like I am doing here. Quoting the paper I mentioned above -  MORE DATA BEATS A CLEVERER ALGORITHM.\n",
    "\n",
    "#### As the TA, I saw that most teams working on the project had data of the order of 100,000 movies. So, if you want to extract the power of these models, consider scraping a larger dataset than me."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Section 5 - Non-deep, Conventional ML models with above data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a layout of what we will be doing - \n",
    "\n",
    "- We will implement two different models\n",
    "- We will decide a performance metric i.e. a quantitative method to be sure about how well difference models are doing. \n",
    "- Discussion of the differences between the models, their strengths, weaknesses, etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As discussed earlier, there are a LOT of implementation decisions to be made. Between feature engineering, hyper-parameter tuning, model selection and how interpretable do you want your model to be (Read : Bayesian vs Non-Bayesian approaches) a lot is to be decided. For example, some of these models could be: \n",
    "\n",
    "- Generalized Linear Models\n",
    "- SVM\n",
    "- Shallow (1 Layer, i.e. not deep) Neural Network\n",
    "- Random Forest\n",
    "- Boosting\n",
    "- Decision Tree\n",
    "\n",
    "Or go more bayesian:\n",
    "- Naive Bayes\n",
    "- Linear or Quadratic Discriminant Analysis\n",
    "- Bayesian Hierarchical models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The list is endless, and not all models will make sense for the kind of problem you have framed for yourself. ** Think about which model best fits for your purpose.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our purposes here, I will be showing the example of 2 very simple models, one picked from each category above - \n",
    "\n",
    "1. SVM\n",
    "2. Multinomial Naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A quick overview of the whole pipeline coming below: \n",
    "    \n",
    "- A little bit of feature engineering\n",
    "- 2 different Models \n",
    "- Evaluation Metrics chosen\n",
    "- Model comparisons"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's start with some feature engineering. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Engineering the right features depends on 2 key ideas. Firstly, what is it that you are trying to solve? For example, if you want to guess my music preferences and you try to train a super awesome model while giving it what my height is as input features, you're going to have no luck. On the other hand, giving it my Spotify playlist will solve the problem with any model. So, CONTEXT of the problem plays a role. \n",
    "\n",
    "Second, you can only represent based on the data at hand. Meaning, if you didn't have access to my Spotify playlist, but to my Facebook statuses - You know all my statuses about Harvard may not be useful. But if you represent me as my Facebook statuses which are YouTube links, that would also solve the problem. So, AVAILABILITY OF DATA at hand is the second factor. \n",
    "\n",
    "#### A nice way to think of it is to think that you start with the problem at hand, but design features constrained by the data you have available. If you have many independent features that each correlate well with the class, learning is easy. On the other hand, if the class is a very complex function of the features, you may not be able to learn it.\n",
    "\n",
    "\n",
    "In the context of this problem, we would like to predict the genre of a movie. what we have access to - movie overviews, which are text descriptions of the movie plot. The hypothesis makes sense, overview is a short description of the story and the story is clearly important in assigning genres to movies. \n",
    "\n",
    "So, let's improve our features by playing with the words in the overviews in our data. One interesting way to go back to what we discussed earlier - TF-IDF. We originally used it to filter words, but we can also assign the tf-idf values as \"importance\" values to words, as opposed to treating them all equally. Tf-idf simply tries to identify the assign a weightage to each word in the bag of words. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, the way it works is - Most movie descriptions have the word \"The\" in it. Obviously, it doesn't tell you anything special about it. So weightage should be inversely proportional to how many movies have the word in their description. This is the IDF part.\n",
    "\n",
    "On the other hand, for the movie interstellar, if the description has the word Space 5 times, and wormhole 2 times, then it's probably more about Space than about wormhole. Thus, space should have a high weightage. This is the TF part. \n",
    "\n",
    "We simply use TF-IDf to assign weightage to every word in the bag of words. Which makes sense, right? :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1668, 1238)"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_transformer = TfidfTransformer()\n",
    "X_tfidf = tfidf_transformer.fit_transform(X)\n",
    "X_tfidf.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's divide our X and Y matrices into train and test split. We train the model on the train split, and report the performance on the test split. Think of this like the questions you do in the problem sets v/s the exam. Of course, they are both (assumed to be) from the same population of questions. And doing well on Problem Sets is a good indicator that you'll do well in exams, but really, you must test before you can make any claims about you knowing the subject."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "msk = np.random.rand(X_tfidf.shape[0]) < 0.8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_tfidf=X_tfidf[msk]\n",
    "X_test_tfidf=X_tfidf[~msk]\n",
    "Y_train=Y[msk]\n",
    "Y_test=Y[~msk]\n",
    "positions=range(len(movies_with_overviews))\n",
    "# print positions\n",
    "test_movies=np.asarray(positions)[~msk]\n",
    "# test_movies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import make_scorer\n",
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OneVsRestClassifier(estimator=GridSearchCV(cv=None, error_score='raise',\n",
       "       estimator=SVC(C=1.0, cache_size=200, class_weight='balanced', coef0=0.0,\n",
       "  decision_function_shape=None, degree=3, gamma='auto', kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False),...refit=True, return_train_score=True,\n",
       "       scoring=make_scorer(f1_score, average=micro), verbose=0),\n",
       "          n_jobs=1)"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parameters = {'kernel':['linear'], 'C':[0.01, 0.1, 1.0]}\n",
    "gridCV = GridSearchCV(SVC(class_weight='balanced'), parameters, scoring=make_scorer(f1_score, average='micro'))\n",
    "classif = OneVsRestClassifier(gridCV)\n",
    "\n",
    "classif.fit(X_train_tfidf, Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(363, 1238) (363, 19) (363, 19) 4\n",
      "[u'Action', u'Adventure', u'Animation', u'Family']\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.41      0.67      0.51        72\n",
      "          1       0.36      0.51      0.42        59\n",
      "          2       0.43      0.50      0.46        64\n",
      "          3       0.53      0.45      0.49       154\n",
      "          4       0.32      0.49      0.38        41\n",
      "          5       0.55      0.67      0.60       100\n",
      "          6       0.43      0.55      0.49        96\n",
      "          7       0.32      0.36      0.34        36\n",
      "          8       0.33      0.10      0.15        10\n",
      "          9       0.24      1.00      0.39        88\n",
      "         10       0.48      0.61      0.53        51\n",
      "         11       0.57      0.55      0.56        29\n",
      "         12       0.36      0.43      0.39        60\n",
      "         13       0.30      0.45      0.36        38\n",
      "         14       0.78      0.41      0.54        34\n",
      "         15       0.00      0.00      0.00        57\n",
      "         16       0.49      0.56      0.52        71\n",
      "         17       0.00      0.00      0.00        21\n",
      "         18       0.11      0.12      0.12        32\n",
      "\n",
      "avg / total       0.40      0.51      0.43      1113\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predstfidf=classif.predict(X_test_tfidf)\n",
    "print X_test_tfidf.shape, Y_test.shape, predstfidf.shape, len(genre_names)\n",
    "print genre_names\n",
    "print classification_report(Y_test, predstfidf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the performance is by and large poorer for movies which are less represented like War and animation, and better for categories like Drama."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numbers aside, let's look at our model's predictions for a small sample of movies from our test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "genre_list=sorted(list(Genre_ID_to_name.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions=[]\n",
    "for i in range(X_test_tfidf.shape[0]):\n",
    "    pred_genres=[]\n",
    "    movie_label_scores=predstfidf[i]\n",
    "    #print len(movie_label_scores), movie_label_scores[18]\n",
    "#     print movie_label_scores\n",
    "    for j in range(len(movie_label_scores)):\n",
    "        #print j\n",
    "        if movie_label_scores[j]!=0:\n",
    "            genre=Genre_ID_to_name[genre_list[j]]\n",
    "            pred_genres.append(genre)\n",
    "    predictions.append(pred_genres)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "f=open('classifer_svc','wb')\n",
    "pickle.dump(classif,f)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MOVIE:  Alice Through the Looking Glass \tPREDICTION:  Adventure,Fantasy,Animation,Action,Comedy,Science Fiction,Family\n",
      "MOVIE:  The Hobbit: The Battle of the Five Armies \tPREDICTION:  Adventure,Horror,Thriller\n",
      "MOVIE:  Attack on Titan II: End of the World \tPREDICTION:  Action,Comedy,Crime,Science Fiction\n",
      "MOVIE:  Cloudy with a Chance of Meatballs 2 \tPREDICTION:  Adventure,Fantasy,Science Fiction\n",
      "MOVIE:  Zambezia \tPREDICTION:  Drama,Crime\n",
      "MOVIE:  The Bourne Supremacy \tPREDICTION:  Animation,Drama,Crime,Family\n"
     ]
    }
   ],
   "source": [
    "for i in range(X_test_tfidf.shape[0]):\n",
    "    if i%50==0 and i!=0:\n",
    "        print 'MOVIE: ',movies_with_overviews[i]['title'],'\\tPREDICTION: ',','.join(predictions[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try our second model? The naive bayes model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "classifnb = OneVsRestClassifier(MultinomialNB())\n",
    "classifnb.fit(X[msk].toarray(), Y_train)\n",
    "predsnb=classifnb.predict(X[~msk].toarray())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "f2=open('classifer_nb','wb')\n",
    "pickle.dump(classifnb,f2)\n",
    "f2.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictionsnb=[]\n",
    "for i in range(X_test_tfidf.shape[0]):\n",
    "    pred_genres=[]\n",
    "    movie_label_scores=predsnb[i]\n",
    "    for j in range(len(movie_label_scores)):\n",
    "        #print j\n",
    "        if movie_label_scores[j]!=0:\n",
    "            genre=Genre_ID_to_name[genre_list[j]]\n",
    "            pred_genres.append(genre)\n",
    "    predictionsnb.append(pred_genres)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MOVIE:  Alice Through the Looking Glass \tPREDICTION:  Adventure,Action,Science Fiction,Family\n",
      "MOVIE:  The Hobbit: The Battle of the Five Armies \tPREDICTION:  Adventure,Horror,Action,Thriller,Science Fiction\n",
      "MOVIE:  Attack on Titan II: End of the World \tPREDICTION:  Action,Thriller,Crime,Science Fiction\n",
      "MOVIE:  Cloudy with a Chance of Meatballs 2 \tPREDICTION:  Adventure,Fantasy,Action,Documentary,Science Fiction,Music\n",
      "MOVIE:  Zambezia \tPREDICTION:  Drama,Thriller\n",
      "MOVIE:  The Bourne Supremacy \tPREDICTION:  Drama,TV Movie\n"
     ]
    }
   ],
   "source": [
    "for i in range(X_test_tfidf.shape[0]):\n",
    "    if i%50==0 and i!=0:\n",
    "        print 'MOVIE: ',movies_with_overviews[i]['title'],'\\tPREDICTION: ',','.join(predictionsnb[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be seen above, the results seem promising, but how do we really compare the two models? We need to quantify our performance so that we can say which one's better. Takes us back to what we discussed right in the beginning - we're learning a function $g$ which can approximate the original unknown function $f$. For some values of $x_i$, the predictions will be wrong for sure, and we want to minimize it. \n",
    "\n",
    "For multi label systems, we often keep track of performance using \"Precision\" and \"Recall\". These are standard metrics, and you can google to read up more about them if you're new to these terms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the standard precision recall metrics for evaluating our system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def precision_recall(gt,preds):\n",
    "    TP=0\n",
    "    FP=0\n",
    "    FN=0\n",
    "    for t in gt:\n",
    "        if t in preds:\n",
    "            TP+=1\n",
    "        else:\n",
    "            FN+=1\n",
    "    for p in preds:\n",
    "        if p not in gt:\n",
    "            FP+=1\n",
    "    if TP+FP==0:\n",
    "        precision=0\n",
    "    else:\n",
    "        precision=TP/float(TP+FP)\n",
    "    if TP+FN==0:\n",
    "        recall=0\n",
    "    else:\n",
    "        recall=TP/float(TP+FN)\n",
    "    return precision,recall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4461090040736058 0.5394437420986093\n"
     ]
    }
   ],
   "source": [
    "precs=[]\n",
    "recs=[]\n",
    "for i in range(len(test_movies)):\n",
    "    if i%1==0:\n",
    "        pos=test_movies[i]\n",
    "        test_movie=movies_with_overviews[pos]\n",
    "        gtids=test_movie['genre_ids']\n",
    "        gt=[]\n",
    "        for g in gtids:\n",
    "            g_name=Genre_ID_to_name[g]\n",
    "            gt.append(g_name)\n",
    "#         print predictions[i],movies_with_overviews[i]['title'],gt\n",
    "        a,b=precision_recall(gt,predictions[i])\n",
    "        precs.append(a)\n",
    "        recs.append(b)\n",
    "\n",
    "print np.mean(np.asarray(precs)),np.mean(np.asarray(recs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.48468886079505547 0.5216392751790982\n"
     ]
    }
   ],
   "source": [
    "precs=[]\n",
    "recs=[]\n",
    "for i in range(len(test_movies)):\n",
    "    if i%1==0:\n",
    "        pos=test_movies[i]\n",
    "        test_movie=movies_with_overviews[pos]\n",
    "        gtids=test_movie['genre_ids']\n",
    "        gt=[]\n",
    "        for g in gtids:\n",
    "            g_name=Genre_ID_to_name[g]\n",
    "            gt.append(g_name)\n",
    "#         print predictions[i],movies_with_overviews[i]['title'],gt\n",
    "        a,b=precision_recall(gt,predictionsnb[i])\n",
    "        precs.append(a)\n",
    "        recs.append(b)\n",
    "\n",
    "print np.mean(np.asarray(precs)),np.mean(np.asarray(recs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The average precision and recall scores for our samples are pretty good! Models seem to be working! Also, we can see that the Naive Bayes performs outperforms SVM. **I strongly suggest you to go read about Multinomial Bayes and think about why it works so well for \"Document Classification\", which is very similar to our case as every movie overview can be thought of as a document we are assigning labels to.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 6 - Deep Learning : an intuitive overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above results were good, but it's time to bring out the big guns. So first and foremost, let's get a very short idea about what's deep learning. This is for peope who don't have background in this - it's high level and gives just the intuition. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As described above, the two most immportant concepts in doing good classification (or regression) are to 1) use the right representation which captures the right information about the data which is relevant to the problem at hand 2) Using the right model which has the capability of making sense of the representation fed to it. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While for the second part we have complicated and powerful models that we have studied at length, we don't seem to have a principled, mathematical way of doing the first part - i.e. representation. What we did above was to see \"What makes sense\", and go from there. That is not a good approach for complex data/ complex problems. Is there some way to automate this? Deep Learning, does just this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To just emphasize the importance of representation in the complex tasks we usually attempt with Deep Learning, let me talk about the original problem which made it famous. The paper is often reffered to as the \"Imagenet Challenge Paper\", and it was basically working on object recognition in images. Let's try to think about an algorithm that tries to detect a chair. \n",
    "\n",
    "## If I ask you to \"Define\" a chair, how would you? - Something with 4 legs?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"files/chair1.png\" height=\"400\" width=\"400\">\n",
    "<h3><center>All are chairs, none with 4 legs. (Pic Credit: Zoya Bylinskii)</center></h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How about some surface that we sit on then?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"files/chair2.png\" height=\"400\" width=\"400\">\n",
    "<h3><center>All are surfaces we sit on, none are chairs. (Pic Credit: Zoya Bylinskii)</center></h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, these definitions won't work and we need something more complicated. Sadly, we can't come up with a simple text rule that our computer can search for! And we take a more principled approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"Deep\" in the deep learning comes from the fact that it was conventionally applied to Neural Networks. Neural Networks, as we all know, are structures organized in layers. Layers of computations. Why do we need layers? Because these layers can be seen as sub-tasks that we do in the complicated task of identifying a chair. It can be thought as a heirarchical break down of a complicated job into smalled sub-tasks. \n",
    "\n",
    "Mathematically, each layer acts like a space transformation which takes the pixel values to a high dimensional space. When we start out, every pixel in the image is given equal importance in our matrix. With each layer, convolution operations give some parts more importance, and some lesser importance. In doing so, we transform our images to a space in which similar looking objects/object parts are closer (We are basically learning this space transformation in deep learning, nothing else)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What exactly was learnt by these neural networks is hard to know, and an active area of research. But one very crude way to visualize what it does is to think like - It starts by learning very generic features in the first layer. Something as simple as vertical and horizontal lines. In the next layer, it learns that if you combine the vectors representing vertical and horizontal vectors in different ratios, you can make all possible slanted lines. Next layer learns to combine lines to form curves - Say, something like the outline of a face. These curves come together to form 3D objects. And so on. Building sub-modules, combining them in the right way which can give it semantics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**So, in a nutshell, the first few layers of a \"Deep\" network learn the right representation of the data, given the problem (which is mathematically described by your objective function trying to minimize difference between ground truth and predicted labels). The last layer simply looks how close or far apart things are in this high dimensional space.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hence, we can give any kind of data a high dimensional representation using neural networks. Below we will see high dimensional representations of both words in overviews (text) and posters (image). Let's get started with the posters i.e. extracting visual features from posters using deep learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 7 - Deep Learning for predicting genre from poster\n",
    "\n",
    "Once again, we must make an implementation decision. This time, it has more to do with how much time are we willing to spend in return for added accuracy. We are going to use here a technique that is commonly referred to as Pre-Training in Machine Learning Literature. \n",
    "\n",
    "Instead of me trying to re-invent the wheel here, I am going to borrow this short section on pre-training from Stanford University's lecture on <a href='http://cs231n.github.io/transfer-learning/'> CNN's</a>. To quote - \n",
    "\n",
    "''In practice, very few people train an entire Convolutional Network from scratch (with random initialization), because it is relatively rare to have a dataset of sufficient size. Instead, it is common to pretrain a ConvNet on a very large dataset (e.g. ImageNet, which contains 1.2 million images with 1000 categories), and then use the ConvNet either as an initialization or a fixed feature extractor for the task of interest. ''\n",
    "\n",
    "There are three broad ways in which transfer learning or pre-training can be done. (The 2 concepts are different and to understand the difference clearly, I suggest you read the linked lecture thoroughly). The way we are going to about it is by using a pre-trained, released ConvNet as feature extractor. Take a ConvNet pretrained on ImageNet (a popular object detection dataset), remove the last fully-connected layer. After removing the last layer, what we have is just another neural network i.e. a stack of space tranformations. But, originally the output of this stack can be pumped into a single layer which can classify the image into categories like Car, Dog, Cat and so on.\n",
    "\n",
    "What this means, is that in the space this stack transforms the images to, all images which contain a \"dog\" are closer to each other, and all images containing a \"cat\" are closer. Thus, it is a meaningful space where images with similar objects are closer. \n",
    "\n",
    "Think about it, now if we pump our posters through this stack, it will embed them in a space where posters which contain similar objects are closer. This is a very meaningful feature engineering method! While this may not be ideal for genre prediction, it might be quite meaningful. For example, all posters with a gun or a car are probably action. While a smiling couple would point to romance or drama. The alternative would be to train the CNN from scratch which is fairly computationally intensive and involves a lot of tricks to get the CNN training to converge to the optimal space tranformation.\n",
    "\n",
    "This way, we can start off with something strong, and then build on top. We pump our images through the pre-trained network to extract the visual features from the posters. Then, using these features as descriptors for the image, and genres as the labels, we train a simpler neural network from scratch which learns to do simply classification on this dataset. These 2 steps are exactly what we are going to do for predicting genres from movie posters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Learning to extract visual features from posters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The basic problem here we are answering is that can we use the posters to predict genre. First check - Does this hypothesis make sense? Yes. Because that's what graphic designers do for a living. They leave visual cues to semantics. They make sure that when we look at the poster of a horror movie, we know it's not a happy image. Things like that. Can our deep learning system infer such subtleties? Let's find out!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Visual features, either we can train a deep neural network ourselves from scratch, or we can use a pre-trained one made available to us from the Visual Geometry Group at Oxford University, one of the most popular methods. This is called the VGG-net. Or as they call it, we will extract the VGG features of an image. Mathematically, as mentioned, it's just a space transformation in the form of layers. So, we simply need to perform this chain of transformations on our image, right? We will be using Pytorch in this notebook. OTher alternatives are Keras and Tensorflow.\n",
    "\n",
    "\n",
    "We will be working with Keras to keep things simple in code, so that we can spend more time understanding and less time coding. Some common ways people refer to this step are - \"Getting the VGG features of an image\", or \"Forward Propogating the image through VGG and chopping off the last layer\". In keras, this is as easy as writing 4 lines. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading the list of movies we had downloaded posters for eariler - \n",
    "import pickle\n",
    "f=open('poster_movies.pckl','r')\n",
    "poster_movies=pickle.load(f)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torchvision\n",
    "import torchvision.models as models\n",
    "import torch.nn as nn\n",
    "import numpy as np\n",
    "from torch.autograd import Variable\n",
    "import torchvision.transforms as transforms\n",
    "import pickle\n",
    "vgg16 = models.vgg16(pretrained=True)\n",
    "model = nn.Sequential(*list(vgg16.children())[:-1])\n",
    "model = nn.Sequential(*list(model.children())[:-2])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting extracting VGG features for scraped images. This will take time, Please be patient...\n",
      "Total images =  1591\n",
      "Working on Image :  1\n",
      "Working on Image :  25\n",
      "Working on Image :  50\n",
      "Working on Image :  100\n",
      "Working on Image :  125\n",
      "Working on Image :  200\n",
      "Working on Image :  250\n",
      "Working on Image :  325\n",
      "Working on Image :  350\n",
      "Working on Image :  375\n",
      "Working on Image :  400\n",
      "Working on Image :  450\n",
      "Working on Image :  475\n",
      "Working on Image :  500\n",
      "Working on Image :  525\n",
      "Working on Image :  550\n",
      "Working on Image :  600\n",
      "Working on Image :  625\n",
      "Working on Image :  650\n",
      "Working on Image :  750\n",
      "Working on Image :  775\n",
      "Working on Image :  825\n",
      "Working on Image :  850\n",
      "Working on Image :  900\n",
      "Working on Image :  950\n",
      "Working on Image :  975\n",
      "Working on Image :  1000\n",
      "Working on Image :  1075\n",
      "Working on Image :  1100\n",
      "Working on Image :  1150\n",
      "Working on Image :  1200\n",
      "Working on Image :  1225\n",
      "Working on Image :  1250\n",
      "Working on Image :  1275\n",
      "Working on Image :  1300\n",
      "Working on Image :  1350\n",
      "Working on Image :  1375\n",
      "Working on Image :  1450\n",
      "Working on Image :  1500\n",
      "Working on Image :  1575\n",
      "Working on Image :  1600\n",
      "Working on Image :  1625\n",
      "Working on Image :  1650\n",
      "Working on Image :  1700\n",
      "Working on Image :  1725\n",
      "Working on Image :  1750\n",
      "Done with all features, please pickle for future use!\n"
     ]
    }
   ],
   "source": [
    "from PIL import Image\n",
    "import os\n",
    "poster_folder='posters_final/'\n",
    "allnames=os.listdir(poster_folder)\n",
    "imnames=[j for j in allnames if j.endswith('.jpg')]\n",
    "feature_list=[]\n",
    "genre_list=[]\n",
    "file_order=[]\n",
    "print \"Starting extracting VGG features for scraped images. This will take time, Please be patient...\"\n",
    "print \"Total images = \",len(imnames)\n",
    "failed_files=[]\n",
    "succesful_files=[]\n",
    "i=0\n",
    "\n",
    "for mov in poster_movies:\n",
    "    i+=1\n",
    "    mov_name=mov['original_title']\n",
    "    mov_name1=mov_name.replace(':','/')\n",
    "    poster_name=mov_name.replace(' ','_')+'.jpg'\n",
    "    if poster_name in imnames:\n",
    "        \n",
    "        img_path=poster_folder+poster_name\n",
    "        try:\n",
    "            img = Image.open(img_path)\n",
    "            min_img_size = 224  # The min size, as noted in the PyTorch pretrained models doc, is 224 px.\n",
    "            transform_pipeline = transforms.Compose([transforms.Resize([min_img_size,min_img_size]),\n",
    "                                                     transforms.ToTensor(),\n",
    "                                                     transforms.Normalize(mean=[0.485, 0.456, 0.406],\n",
    "                                                                          std=[0.229, 0.224, 0.225])])\n",
    "            x = transform_pipeline(img)\n",
    "            img.close()\n",
    "            # PyTorch pretrained models expect the Tensor dims to be (num input imgs, num color channels, height, width).\n",
    "            # Currently however, we have (num color channels, height, width); let's fix this by inserting a new axis.\n",
    "#             print len(img)\n",
    "#             print img.shape\n",
    "#             print img\n",
    "#             break\n",
    "            x = x.unsqueeze(0)  # Insert the new axis at index 0 i.e. in front of the other axes/dims. \n",
    "            \n",
    "            \n",
    "            # Now that we have preprocessed our img, we need to convert it into a \n",
    "            # Variable; PyTorch models expect inputs to be Variables. A PyTorch Variable is a  \n",
    "            # wrapper around a PyTorch Tensor.\n",
    "            x = Variable(x)\n",
    "            succesful_files.append(poster_name)\n",
    "            \n",
    "            features = model(x)\n",
    "            \n",
    "            file_order.append(img_path)\n",
    "            feature_list.append(features)\n",
    "            genre_list.append(mov['genre_ids'])\n",
    "            max_prediction = features.data.numpy().argmax()  # Our prediction will be the index of the class label with the largest value.\n",
    "            if max_prediction==0.0:\n",
    "                print('problematic',i)\n",
    "            if i%25==0 or i==1:\n",
    "                print \"Working on Image : \",i\n",
    "        except:\n",
    "            failed_files.append(poster_name)\n",
    "            continue\n",
    "        \n",
    "    else:\n",
    "        continue\n",
    "print \"Done with all features, please pickle for future use!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1242"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(genre_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1242"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(feature_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 3, 224, 224])"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_list[0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Features dumped to pickle file\n"
     ]
    }
   ],
   "source": [
    "# Reading from pickle below, this code is not to be run.\n",
    "list_pickled=(feature_list,file_order,failed_files,succesful_files,genre_list)\n",
    "f=open('posters_new_features.pckl','wb')\n",
    "pickle.dump(list_pickled,f)\n",
    "f.close()\n",
    "print(\"Features dumped to pickle file\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "f7=open('posters_new_features.pckl','rb')\n",
    "list_pickled=pickle.load(f7)\n",
    "f7.close()\n",
    "# (feature_list2,file_order2)=list_pickled"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training a simple neural network model using these VGG features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "(feature_list,files,failed,succesful,genre_list)=list_pickled"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first get the labels on our 1342 samples first! As image download fails on a few instances, the best way to work with the right model is to read the poster names downloaded, and working from there. These posters cannot be uploaded to Github as they are too large, and so are being downloaded and read from my local computer. If you do re-do it, you might have to check and edit the paths in the code to make sure it runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "(a,b,c,d)=feature_list[0].shape\n",
    "feature_size=a*b*c*d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks odd, why are we re-running the loop we ran above again below? The reason is simple, the most important thing to know about numpy is that using vstack() and hstack() are highly sub-optimal. Numpy arrays when created, a fixed size is allocated in the memory and when we stack, a new one is copied and created in a new location. This makes the code really, really slow. The best way to do it (and this remains the same with MATLAB matrices if you work with them), is to create a numpy array of zeros, and over-write it row by row. The above code was just to see what size numpy array we will need!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final movie poster set for which we have all the information we need, is 1265 movies. In the above code we are making an X numpy array containing the visual features of one image per row. So, the VGG features are reshaped to be in the shape (1,25088) and we finally obtain a matrix of shape (1265,25088)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1242 150528\n"
     ]
    }
   ],
   "source": [
    "np_features=np.zeros((len(feature_list),feature_size))\n",
    "print len(feature_list),feature_size\n",
    "for i in range(len(feature_list)):\n",
    "    feat=feature_list[i]\n",
    "    reshaped_feat=feat.reshape(1,-1)\n",
    "    #print reshaped_feat.shape\n",
    "    np_features[i]=reshaped_feat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "# np_features[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "X=np_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MultiLabelBinarizer\n",
    "mlb=MultiLabelBinarizer()\n",
    "Y=mlb.fit_transform(genre_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1242, 19)"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our binarized Y numpy array contains the binarized labels corresponding to the genre IDs of the 1277 movies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "visual_problem_data=(X,Y)\n",
    "f8=open('visual_problem_data_clean.pckl','wb')\n",
    "pickle.dump(visual_problem_data,f8)\n",
    "f8.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "f8=open('visual_problem_data_clean.pckl','rb')\n",
    "visual_features=pickle.load(f8)\n",
    "f8.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X,Y)=visual_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1242, 150528)"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask = np.random.rand(len(X)) < 0.8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-2.10077929, -2.10077929, -2.11790395, ..., -1.80444443,\n",
       "        -1.80444443, -1.80444443],\n",
       "       [-0.85067213, -1.15891778, -1.56991184, ..., -1.54300654,\n",
       "        -1.54300654, -1.54300654],\n",
       "       [-1.72403467, -1.62128615, -1.46716332, ..., -1.43843138,\n",
       "        -1.43843138, -1.43843138],\n",
       "       ...,\n",
       "       [-2.11790395, -2.11790395, -2.11790395, ..., -1.80444443,\n",
       "        -1.80444443, -1.80444443],\n",
       "       [ 1.13579929,  1.17004883,  1.1871736 , ...,  0.14762534,\n",
       "         0.11276696,  0.09533776],\n",
       "       [ 2.21465874,  2.21465874,  2.21465874, ..., -0.95041388,\n",
       "        -0.07895417,  1.73368204]])"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train=X[mask]\n",
    "X_test=X[~mask]\n",
    "Y_train=Y[mask]\n",
    "Y_test=Y[~mask]\n",
    "X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(236, 19)"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we create our own keras neural network to use the VGG features and then classify movie genres. Keras makes this super easy. \n",
    "\n",
    "Neural network architectures have gotten complex over the years. But the simplest ones contain very standard computations organized in layers, as described above. Given the popularity of some of these, Keras makes it as easy as writing out the names of these operations in a sequential order. This way you can make a network while completely avoiding the Mathematics (HIGHLY RECOMMENDED SPENDING MORE TIME ON THE MATH THOUGH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sequential() allows us to make models the follow this sequential order of layers. Different kinds of layers like Dense, Conv2D etc can be used, and many activation functions like RELU, Linear etc are also available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Important Question : Why do we need activation functions?\n",
    "#### Copy pasting the answer I wrote for this question on <a href='https://www.quora.com/Why-do-neural-networks-need-an-activation-function/answer/Spandan-Madan?srid=5ydm'>Quora</a> Feel free to leave comments there.\n",
    "\n",
    "\"\"Sometimes, we tend to get lost in the jargon and confuse things easily, so the best way to go about this is getting back to our basics.\n",
    "\n",
    "Don’t forget what the original premise of machine learning (and thus deep learning) is - IF the input and output are related by a function y=f(x), then if we have x, there is no way to exactly know f unless we know the process itself. However, machine learning gives you the ability to approximate f with a function g, and the process of trying out multiple candidates to identify the function g best approximating f is called machine learning.\n",
    "\n",
    "Ok, that was machine learning, and how is deep learning different? Deep learning simply tries to expand the possible kind of functions that can be approximated using the above mentioned machine learning paradigm. Roughly speaking, if the previous model could learn say 10,000 kinds of functions, now it will be able to learn say 100,000 kinds (in actuality both are infinite spaces but one is larger than the other, because maths is cool that ways.)\n",
    "\n",
    "If you want to know the mathematics of it, go read about VC dimension and how more layers in a network affect it. But I will avoid the mathematics here and rely on your intuition to believe me when I say that not all data can be classified correctly into categories using a linear function. So, we need our deep learning model to be able to approximate more complex functions than just a linear function.\n",
    "\n",
    "Now, let’s come to your non linearity bit. Imagine a linear function y=2x+3, and another one y=4x+7. What happens if I pool them and take an average? I get another linear function y= 3x+5. So instead of doing those two computations separately and then averaging it out, I could have just used the single linear function y=3x+5. Obviously, this logic holds good if I have more than 2 such linear functions. This is exactly what will happen if you don’t have have non-linearities in your nodes, and also what others have written in their answers.\n",
    "\n",
    "It simply follows from the definition of a linear function -\n",
    "\n",
    "(i) If you take two linear functions, AND\n",
    "\n",
    "(ii)Take a linear combination of them (which is how we combine the outputs of multiple nodes of a network)\n",
    "\n",
    "You are BOUND to get a linear function because f(x)+g(x)=mx+b+nx+c=(m+n)x+(b+c)= say h(x).\n",
    "\n",
    "And you could in essence replace your whole network by a simple matrix transformation which accounts for all linear combinations and up/downsamplings.\n",
    "\n",
    "In a nutshell, you’ll only be trying to learn a linear approximation for original function f relating the input and the output. Which as we discussed above, is not always the best approximation. Adding non-linearities ensures that you can learn more complex functions by approximating every non-linear function as a LINEAR combination of a large number of non-linear functions.\n",
    "\n",
    "Still new to the field, so if there’s something wrong here please comment below! Hope it helps\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's train our model then, using the features we extracted from VGG net \n",
    "\n",
    "The model we will use has just 1 hidden layer between the VGG features and the final output layer. The simplest neural network you can get. An image goes into this network with the dimensions (1,25088), the first layer's output is 1024 dimensional. This hidden layer output undergoes a pointwise RELU activation. This output gets transformed into the output layer of 20 dimensions. It goes through a sigmoid.\n",
    "\n",
    "The sigmoid, or the squashing function as it is often called, is a function which squashes numbers between 0 and 1. What are you reminded of when you think of numebers between 0 and 1? Right, probability. \n",
    "\n",
    "By squashing the score of each of the 20 output labels between 0 and 1, sigmoid lets us interpret their scores as probabilities. Then, we can just pick the classes with the top 3 or 5 probability scores as the predicted genres for the movie poster! Simple! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import division\n",
    "import torch\n",
    "import time\n",
    "import copy\n",
    "\n",
    "\n",
    "class FeatureDataset():\n",
    "    def __init__(self, X_train, Y_train, transform=None):\n",
    "    \n",
    "        self.X_train = X_train\n",
    "        self.Y_train = Y_train\n",
    "        self.transform = transform\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.X_train)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        \n",
    "        #sample = {'image': self.X_train[idx], 'landmarks': self.Y_train[idx]}\n",
    "\n",
    "#         if self.transform:\n",
    "#             sample = self.transform(sample)\n",
    "\n",
    "        return X_train[idx].astype(float), Y_train[idx].astype(float)\n",
    "\n",
    "def train_model(model, criterion, optimizer, num_epochs):\n",
    "    \n",
    "    since = time.time()\n",
    "    USE_TENSORBOARD = 0\n",
    "    best_model = model\n",
    "    best_acc = 0.0\n",
    "    use_gpu = 0\n",
    "    \n",
    "    for epoch in range(num_epochs):\n",
    "        print('Epoch {}/{}'.format(epoch, num_epochs - 1))\n",
    "        print('-' * 10)\n",
    "\n",
    "        # Each epoch has a training and validation phase\n",
    "        for phase in ['train', 'test']:\n",
    "            if phase == 'train':\n",
    "                mode='train'\n",
    "                #optimizer = lr_scheduler(optimizer, epoch)\n",
    "                model.train()  # Set model to training mode\n",
    "                print \"TRAINING STARTED\"\n",
    "            else:\n",
    "                model.eval()\n",
    "                mode='val'\n",
    "                print \"TESTING STARTED\"\n",
    "\n",
    "            running_loss = 0.0\n",
    "            running_corrects = 0\n",
    "\n",
    "            counter=0\n",
    "            # Iterate over data.\n",
    "            answer = []\n",
    "            for data in dset_loaders[phase]:\n",
    "                inputs, labels = data \n",
    "                \n",
    "                # wrap them in Variable\n",
    "                if use_gpu:\n",
    "                    try:\n",
    "                        inputs, labels = Variable(inputs.float().cuda()),                             \n",
    "                        Variable(labels.long().cuda())\n",
    "                    except:\n",
    "                        print(inputs,labels)\n",
    "                else:\n",
    "                    inputs, labels = Variable(Variable(inputs).float()), Variable(Variable(labels).float())\n",
    "\n",
    "                # Set gradient to zero to delete history of computations in previous epoch. Track operations so that differentiation can be done automatically.\n",
    "                optimizer.zero_grad()\n",
    "                outputs = model(inputs)\n",
    "                for i in range(len(outputs.data)):\n",
    "                    answer.append(outputs.data[i])\n",
    "                _, preds = torch.max(outputs.data, 1)\n",
    "                #print outputs.data\n",
    "                #print _, preds\n",
    "                loss = criterion(outputs, labels)\n",
    "                print('loss done')                \n",
    "                # Just so that you can keep track that something's happening and don't feel like the program isn't running.\n",
    "                if counter%50==0:\n",
    "                    print(\"Reached iteration \",counter)\n",
    "                counter+=1\n",
    "\n",
    "                # backward + optimize only if in training phase\n",
    "                if phase == 'train':\n",
    "                    print('loss backward')\n",
    "                    loss.backward()\n",
    "                    print('done loss backward')\n",
    "                    optimizer.step()\n",
    "                    print('done optim')\n",
    "                # print evaluation statistics\n",
    "                try:\n",
    "                    running_loss += loss.item()\n",
    "\n",
    "                    print preds.shape, labels.data.shape\n",
    "                    for q in range(len(labels.data)):\n",
    "                        if labels.data[q][preds[q]]==1:\n",
    "                            running_corrects += 1\n",
    "                    \n",
    "                    #running_corrects += torch.sum(preds == labels.data.long())\n",
    "                except:\n",
    "                    print('unexpected error, could not calculate loss or do a sum.')\n",
    "            print('trying epoch loss')\n",
    "            epoch_loss = running_loss / dset_sizes[phase]\n",
    "            epoch_acc = running_corrects / dset_sizes[phase]\n",
    "            print phase, running_corrects, dset_sizes[phase]\n",
    "            print('{} Loss: {:.4f} Acc: {:.4f}'.format(\n",
    "                phase, epoch_loss, epoch_acc))\n",
    "\n",
    "\n",
    "            # deep copy the model\n",
    "            if phase == 'test':\n",
    "                if USE_TENSORBOARD:\n",
    "                    foo.add_scalar_value('epoch_loss',epoch_loss,step=epoch)\n",
    "                    foo.add_scalar_value('epoch_acc',epoch_acc,step=epoch)\n",
    "                if epoch_acc > best_acc:\n",
    "                    best_acc = epoch_acc\n",
    "                    best_model = copy.deepcopy(model)\n",
    "                    print('new best accuracy = ',best_acc)\n",
    "    time_elapsed = time.time() - since\n",
    "    print('Training complete in {:.0f}m {:.0f}s'.format(\n",
    "        time_elapsed // 60, time_elapsed % 60))\n",
    "    print('Best val Acc: {:4f}'.format(best_acc))\n",
    "    print('returning and looping back')\n",
    "    #torch.save(best_model.state_dict(), 'fine_tuned_best_model.pt')\n",
    "    return best_model, answer\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-2.10077929 -2.10077929 -2.11790395 ... -1.80444443 -1.80444443\n",
      "  -1.80444443]\n",
      " [-0.85067213 -1.15891778 -1.56991184 ... -1.54300654 -1.54300654\n",
      "  -1.54300654]\n",
      " [-1.72403467 -1.62128615 -1.46716332 ... -1.43843138 -1.43843138\n",
      "  -1.43843138]\n",
      " ...\n",
      " [-2.11790395 -2.11790395 -2.11790395 ... -1.80444443 -1.80444443\n",
      "  -1.80444443]\n",
      " [ 1.13579929  1.17004883  1.1871736  ...  0.14762534  0.11276696\n",
      "   0.09533776]\n",
      " [ 2.21465874  2.21465874  2.21465874 ... -0.95041388 -0.07895417\n",
      "   1.73368204]]\n"
     ]
    }
   ],
   "source": [
    "print(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/envs/deeplearningproject/lib/python2.7/site-packages/torchvision-0.2.1-py2.7.egg/torchvision/transforms/transforms.py:563: UserWarning: The use of the transforms.RandomSizedCrop transform is deprecated, please use transforms.RandomResizedCrop instead.\n",
      "/anaconda3/envs/deeplearningproject/lib/python2.7/site-packages/torchvision-0.2.1-py2.7.egg/torchvision/transforms/transforms.py:188: UserWarning: The use of the transforms.Scale transform is deprecated, please use transforms.Resize instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'test': 236, 'train': 1006}\n",
      "Epoch 0/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/envs/deeplearningproject/lib/python2.7/site-packages/ipykernel_launcher.py:93: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 332 1006\n",
      "train Loss: 0.0996 Acc: 0.3300\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 92 236\n",
      "test Loss: 0.0958 Acc: 0.3898\n",
      "('new best accuracy = ', 0.3898305084745763)\n",
      "Epoch 1/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 479 1006\n",
      "train Loss: 0.0638 Acc: 0.4761\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 156 236\n",
      "test Loss: 0.0334 Acc: 0.6610\n",
      "('new best accuracy = ', 0.6610169491525424)\n",
      "Epoch 2/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 587 1006\n",
      "train Loss: 0.0218 Acc: 0.5835\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 171 236\n",
      "test Loss: 0.0127 Acc: 0.7246\n",
      "('new best accuracy = ', 0.7245762711864406)\n",
      "Epoch 3/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 701 1006\n",
      "train Loss: 0.0100 Acc: 0.6968\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 196 236\n",
      "test Loss: 0.0074 Acc: 0.8305\n",
      "('new best accuracy = ', 0.8305084745762712)\n",
      "Epoch 4/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 799 1006\n",
      "train Loss: 0.0061 Acc: 0.7942\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 215 236\n",
      "test Loss: 0.0041 Acc: 0.9110\n",
      "('new best accuracy = ', 0.9110169491525424)\n",
      "Epoch 5/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 859 1006\n",
      "train Loss: 0.0042 Acc: 0.8539\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 206 236\n",
      "test Loss: 0.0041 Acc: 0.8729\n",
      "Epoch 6/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 882 1006\n",
      "train Loss: 0.0034 Acc: 0.8767\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 210 236\n",
      "test Loss: 0.0034 Acc: 0.8898\n",
      "Epoch 7/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 923 1006\n",
      "train Loss: 0.0025 Acc: 0.9175\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 227 236\n",
      "test Loss: 0.0025 Acc: 0.9619\n",
      "('new best accuracy = ', 0.961864406779661)\n",
      "Epoch 8/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 953 1006\n",
      "train Loss: 0.0019 Acc: 0.9473\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 224 236\n",
      "test Loss: 0.0016 Acc: 0.9492\n",
      "Epoch 9/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 965 1006\n",
      "train Loss: 0.0015 Acc: 0.9592\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 224 236\n",
      "test Loss: 0.0018 Acc: 0.9492\n",
      "Epoch 10/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 962 1006\n",
      "train Loss: 0.0014 Acc: 0.9563\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 235 236\n",
      "test Loss: 0.0011 Acc: 0.9958\n",
      "('new best accuracy = ', 0.9957627118644068)\n",
      "Epoch 11/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 968 1006\n",
      "train Loss: 0.0012 Acc: 0.9622\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 230 236\n",
      "test Loss: 0.0012 Acc: 0.9746\n",
      "Epoch 12/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 984 1006\n",
      "train Loss: 0.0010 Acc: 0.9781\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 221 236\n",
      "test Loss: 0.0021 Acc: 0.9364\n",
      "Epoch 13/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 956 1006\n",
      "train Loss: 0.0014 Acc: 0.9503\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 231 236\n",
      "test Loss: 0.0011 Acc: 0.9788\n",
      "Epoch 14/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 984 1006\n",
      "train Loss: 0.0009 Acc: 0.9781\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 235 236\n",
      "test Loss: 0.0007 Acc: 0.9958\n",
      "Epoch 15/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 996 1006\n",
      "train Loss: 0.0006 Acc: 0.9901\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 235 236\n",
      "test Loss: 0.0006 Acc: 0.9958\n",
      "Epoch 16/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 999 1006\n",
      "train Loss: 0.0004 Acc: 0.9930\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 235 236\n",
      "test Loss: 0.0005 Acc: 0.9958\n",
      "Epoch 17/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1000 1006\n",
      "train Loss: 0.0004 Acc: 0.9940\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 236 236\n",
      "test Loss: 0.0005 Acc: 1.0000\n",
      "('new best accuracy = ', 1.0)\n",
      "Epoch 18/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 992 1006\n",
      "train Loss: 0.0005 Acc: 0.9861\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 235 236\n",
      "test Loss: 0.0006 Acc: 0.9958\n",
      "Epoch 19/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 997 1006\n",
      "train Loss: 0.0004 Acc: 0.9911\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 236 236\n",
      "test Loss: 0.0004 Acc: 1.0000\n",
      "Epoch 20/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 990 1006\n",
      "train Loss: 0.0005 Acc: 0.9841\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 229 236\n",
      "test Loss: 0.0024 Acc: 0.9703\n",
      "Epoch 21/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 867 1006\n",
      "train Loss: 0.0058 Acc: 0.8618\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 222 236\n",
      "test Loss: 0.0037 Acc: 0.9407\n",
      "Epoch 22/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 963 1006\n",
      "train Loss: 0.0026 Acc: 0.9573\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 233 236\n",
      "test Loss: 0.0011 Acc: 0.9873\n",
      "Epoch 23/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 998 1006\n",
      "train Loss: 0.0005 Acc: 0.9920\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 235 236\n",
      "test Loss: 0.0004 Acc: 0.9958\n",
      "Epoch 24/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1002 1006\n",
      "train Loss: 0.0003 Acc: 0.9960\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 235 236\n",
      "test Loss: 0.0004 Acc: 0.9958\n",
      "Epoch 25/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1003 1006\n",
      "train Loss: 0.0002 Acc: 0.9970\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 233 236\n",
      "test Loss: 0.0005 Acc: 0.9873\n",
      "Epoch 26/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1002 1006\n",
      "train Loss: 0.0002 Acc: 0.9960\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 235 236\n",
      "test Loss: 0.0003 Acc: 0.9958\n",
      "Epoch 27/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1004 1006\n",
      "train Loss: 0.0002 Acc: 0.9980\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 235 236\n",
      "test Loss: 0.0003 Acc: 0.9958\n",
      "Epoch 28/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 997 1006\n",
      "train Loss: 0.0005 Acc: 0.9911\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 228 236\n",
      "test Loss: 0.0016 Acc: 0.9661\n",
      "Epoch 29/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 975 1006\n",
      "train Loss: 0.0015 Acc: 0.9692\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 227 236\n",
      "test Loss: 0.0021 Acc: 0.9619\n",
      "Epoch 30/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 973 1006\n",
      "train Loss: 0.0022 Acc: 0.9672\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 226 236\n",
      "test Loss: 0.0018 Acc: 0.9576\n",
      "Epoch 31/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 996 1006\n",
      "train Loss: 0.0007 Acc: 0.9901\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 235 236\n",
      "test Loss: 0.0004 Acc: 0.9958\n",
      "Epoch 32/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1004 1006\n",
      "train Loss: 0.0002 Acc: 0.9980\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 236 236\n",
      "test Loss: 0.0003 Acc: 1.0000\n",
      "Epoch 33/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1004 1006\n",
      "train Loss: 0.0001 Acc: 0.9980\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 235 236\n",
      "test Loss: 0.0004 Acc: 0.9958\n",
      "Epoch 34/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1003 1006\n",
      "train Loss: 0.0001 Acc: 0.9970\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 235 236\n",
      "test Loss: 0.0003 Acc: 0.9958\n",
      "Epoch 35/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1004 1006\n",
      "train Loss: 0.0001 Acc: 0.9980\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 236 236\n",
      "test Loss: 0.0003 Acc: 1.0000\n",
      "Epoch 36/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1004 1006\n",
      "train Loss: 0.0001 Acc: 0.9980\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 236 236\n",
      "test Loss: 0.0002 Acc: 1.0000\n",
      "Epoch 37/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 965 1006\n",
      "train Loss: 0.0015 Acc: 0.9592\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 224 236\n",
      "test Loss: 0.0040 Acc: 0.9492\n",
      "Epoch 38/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 964 1006\n",
      "train Loss: 0.0027 Acc: 0.9583\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 233 236\n",
      "test Loss: 0.0008 Acc: 0.9873\n",
      "Epoch 39/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 998 1006\n",
      "train Loss: 0.0005 Acc: 0.9920\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 236 236\n",
      "test Loss: 0.0005 Acc: 1.0000\n",
      "Epoch 40/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1003 1006\n",
      "train Loss: 0.0002 Acc: 0.9970\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 236 236\n",
      "test Loss: 0.0002 Acc: 1.0000\n",
      "Epoch 41/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1003 1006\n",
      "train Loss: 0.0001 Acc: 0.9970\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 236 236\n",
      "test Loss: 0.0002 Acc: 1.0000\n",
      "Epoch 42/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1005 1006\n",
      "train Loss: 0.0001 Acc: 0.9990\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 235 236\n",
      "test Loss: 0.0003 Acc: 0.9958\n",
      "Epoch 43/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1004 1006\n",
      "train Loss: 0.0001 Acc: 0.9980\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 235 236\n",
      "test Loss: 0.0002 Acc: 0.9958\n",
      "Epoch 44/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1005 1006\n",
      "train Loss: 0.0001 Acc: 0.9990\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 236 236\n",
      "test Loss: 0.0004 Acc: 1.0000\n",
      "Epoch 45/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 999 1006\n",
      "train Loss: 0.0006 Acc: 0.9930\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 227 236\n",
      "test Loss: 0.0020 Acc: 0.9619\n",
      "Epoch 46/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 942 1006\n",
      "train Loss: 0.0043 Acc: 0.9364\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 213 236\n",
      "test Loss: 0.0045 Acc: 0.9025\n",
      "Epoch 47/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 990 1006\n",
      "train Loss: 0.0013 Acc: 0.9841\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 236 236\n",
      "test Loss: 0.0004 Acc: 1.0000\n",
      "Epoch 48/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1003 1006\n",
      "train Loss: 0.0002 Acc: 0.9970\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 235 236\n",
      "test Loss: 0.0004 Acc: 0.9958\n",
      "Epoch 49/49\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([46]) torch.Size([46, 19])\n",
      "trying epoch loss\n",
      "train 1004 1006\n",
      "train Loss: 0.0001 Acc: 0.9980\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([64]) torch.Size([64, 19])\n",
      "loss done\n",
      "torch.Size([44]) torch.Size([44, 19])\n",
      "trying epoch loss\n",
      "test 236 236\n",
      "test Loss: 0.0003 Acc: 1.0000\n",
      "Training complete in 35m 4s\n",
      "Best val Acc: 1.000000\n",
      "returning and looping back\n"
     ]
    }
   ],
   "source": [
    "dsets = {}\n",
    "dsets['train'] = FeatureDataset(X_train, Y_train)\n",
    "dsets['test'] = FeatureDataset(X_test, Y_test)\n",
    "dset_loaders = {x: torch.utils.data.DataLoader(dsets[x], batch_size=64,\n",
    "                                               shuffle=True, num_workers=25)\n",
    "                for x in ['train', 'test']}\n",
    "\n",
    "dset_sizes = {x: len(dsets[x]) for x in ['train', 'test']}\n",
    "print dset_sizes\n",
    "model_visual = nn.Sequential(\n",
    "    nn.Linear(X.shape[1],1024),  #150528\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(1024,256),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(256,Y_test.shape[1]),\n",
    "    nn.Sigmoid()\n",
    ")\n",
    "# #opt = optimizers.rmsprop(lr=0.0001, decay=1e-6)\n",
    "optimizer = torch.optim.RMSprop(model_visual.parameters(), lr=0.0001, weight_decay=1e-6)\n",
    "criterion = nn.BCELoss()\n",
    "\n",
    "#model_visual.load_state_dict(torch.load('fine_tuned_best_model.pt'))\n",
    "model_ft, Y_preds = train_model(model_visual, criterion, optimizer, num_epochs=50)\n",
    "\n",
    "# Save model\n",
    "torch.save(model_ft.state_dict(), 'fine_tuned_best_model.pt')\n",
    "#model_ft.save_state_dict('fine_tuned_best_model.pt')\n",
    "# #sgd = optimizers.SGD(lr=0.05, decay=1e-6, momentum=0.4, nesterov=False)\n",
    "# model_visual.compile(optimizer=opt,\n",
    "#               loss='binary_crossentropy',\n",
    "#               metrics=['accuracy'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We train the model using the fit() function. The parameters it takes are - training features and training labels, epochs, batch_size and verbose. \n",
    "\n",
    "Simplest one - verbose. 0=\"dont print anything as you work\", 1=\"Inform me as you go\". \n",
    "\n",
    "Often the data set is too large to be loaded into the RAM. So, we load data in batches. For batch_size=32 and epochs=10, the model starts loading rows from X in batches of 32 everytime it calculates the loss and updates the model. It keeps on going till it has covered all the samples 10 times. \n",
    "\n",
    "So, the no. of times model is updated = (Total Samples/Batch Size) * (Epochs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(774.7288)"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(sum(Y_preds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's look at some of our predictions? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "f6=open('Genredict.pckl','rb')\n",
    "Genre_ID_to_name=pickle.load(f6)\n",
    "f6.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(4.9819)"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(Y_preds[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(3.0128)"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(Y_preds[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [],
   "source": [
    "genre_list=sorted(list(Genre_ID_to_name.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<type 'list'>\n",
      "Predicted:  Action,Thriller,Drama  Actual:  Adventure,Fantasy,Action\n",
      "Predicted:  Animation,Adventure,Family  Actual:  Horror,Thriller\n",
      "Predicted:  Fantasy,Adventure,Action  Actual:  Drama,History,Documentary\n",
      "Predicted:  Science Fiction,Action,Adventure  Actual:  Action,Science Fiction\n",
      "Predicted:  War,Comedy,Drama  Actual:  Fantasy,Animation,Comedy,Family\n"
     ]
    }
   ],
   "source": [
    "precs=[]\n",
    "recs=[]\n",
    "print type(Y_preds)\n",
    "for i in range(len(Y_preds)):\n",
    "    row=Y_preds[i]\n",
    "    gt_genres=Y_test[i]\n",
    "    gt_genre_names=[]\n",
    "    for j in range(len(gt_genres)):\n",
    "        if gt_genres[j]==1:\n",
    "            gt_genre_names.append(Genre_ID_to_name[genre_list[j]])\n",
    "    top_3=np.argsort(row)[-3:]\n",
    "    predicted_genres=[]\n",
    "    for genre in top_3:\n",
    "        predicted_genres.append(Genre_ID_to_name[genre_list[genre]])\n",
    "    (precision,recall)=precision_recall(gt_genre_names,predicted_genres)\n",
    "    precs.append(precision)\n",
    "    recs.append(recall)\n",
    "    if i%50==0:\n",
    "        print \"Predicted: \",','.join(predicted_genres),\" Actual: \",','.join(gt_genre_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.2217514124293785 0.216454802259887\n"
     ]
    }
   ],
   "source": [
    "print np.mean(np.asarray(precs)),np.mean(np.asarray(recs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, even with just the poster i.e. visual features we are able to make great predictions! Sure, text outperforms the visual features, but the important thing is that it still works. In more complicated models, we can combine the two to make even better predictions. That is precisely what I work on in my research."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These models were trained on CPU's, and a simple 1 layer model was used to show that there is a lot of information in this data that the models can extract. With a larger dataset, and more training I was able to bring these numbers to as high as 70%, which is the similar to textual features. Some teams in my class outperformed this even more. More data is the first thing you should try if you want better results. Then, you can start playing with training on GPUs, learning rate schedules and other hyperparameters. Finally, you can consider using ResNet, a much more powerful neural network model than VGG. All of these can be tried once you have a working knowledge of machine learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 8 - Deep Learning to get Textual Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do the same thing as above with text now?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use an off the shelf representation for words - Word2Vec model. Just like VGGnet before, this is a model made available to get a meaningful representation. As the total number of words is small, we don't even need to forward propagate our sample through a network. Even that has been done for us, and the result is stored in the form of a dictionary. We can simply look up the word in the dictionary and get the Word2Vec features for the word."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can download the dictionary from here - https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit <br>\n",
    "Download it to the directory of this tutorial i.e. in the same folder as this ipython notebook.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from gensim import models\n",
    "# model2 = models.Word2Vec.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True) \n",
    "model2 = models.KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can simply look up for a word in the above loaded model. For example, to get the Word2Vec representation of the word \"King\" we just do - model2['king']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(300,)\n",
      "(300,)\n"
     ]
    }
   ],
   "source": [
    "print model2['king'].shape\n",
    "print model2['dog'].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This way, we can represent the words in our overviews using this word2vec model. And then, we can use that as our X representations. So, instead of count of words, we are using a representation which is based on the semantic representation of the word. Mathematically, each word went from 3-4 dimensional (the length) to 300 dimensions!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "For the same set of movies above, let's try and predict the genres from the deep representation of their overviews!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1668"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_movies_set = movies_with_overviews\n",
    "len(final_movies_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import RegexpTokenizer\n",
    "from stop_words import get_stop_words\n",
    "tokenizer = RegexpTokenizer(r'\\w+')\n",
    "\n",
    "# create English stop words list\n",
    "en_stop = get_stop_words('en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1668, 300)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_mean_wordvec=np.zeros((len(final_movies_set),300))\n",
    "movie_mean_wordvec.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Text needs some pre-processing before we can train the model. The only preprocessing we do here is - we delete commonly occurring words which we know are not informative about the genre. Think of it as the clutter in some sense. These words are often removed and are referred to as \"stop words\". You can look them up online. These include simple words like \"a\", \"and\", \"but\", \"how\", \"or\" and so on. They can be easily removed using the python package NLTK.\n",
    "\n",
    "From the above dataset, movies with overviews which contain only stop words, or movies with overviews containing no words with word2vec representation are neglected. Others are used to build our Mean word2vec representation. Simply, put for every movie overview - \n",
    "\n",
    "* Take movie overview\n",
    "* Throw out stop words\n",
    "* For non stop words:\n",
    "    - If in word2vec - take it's word2vec representation which is 300 dimensional\n",
    "    - If not - throw word\n",
    "* For each movie, calculate the arithmetic mean of the 300 dimensional vector representations for all words in the overview which weren't thrown out\n",
    "\n",
    "This mean becomes the 300 dimensional representation for the movie. For all movies, these are stored in a numpy array. So the X matrix becomes (1263,300). And, Y is (1263,20) i.e. binarized 20 genres, as before"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Why do we take the arithmetic mean?**\n",
    "If you feel that we should have kept all the words separately - Then you're thinking correct, but sadly we're limited by the way current day neural networks work. I will not mull over this for the fear of stressing too much on an otherwise irrelevant detail. But if you're interested, read this awesome paper - \n",
    "https://jiajunwu.com/papers/dmil_cvpr.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "genres=[]\n",
    "rows_to_delete=[]\n",
    "for i in range(len(final_movies_set)):\n",
    "    mov=final_movies_set[i]\n",
    "    movie_genres=mov['genre_ids']\n",
    "    genres.append(movie_genres)\n",
    "    overview=mov['overview']\n",
    "    tokens = tokenizer.tokenize(overview)\n",
    "    stopped_tokens = [k for k in tokens if not k in en_stop]\n",
    "    count_in_vocab=0\n",
    "    s=0\n",
    "    if len(stopped_tokens)==0:\n",
    "        rows_to_delete.append(i)\n",
    "        genres.pop(-1)\n",
    "#         print overview\n",
    "#         print \"sample \",i,\"had no nonstops\"\n",
    "    else:\n",
    "        for tok in stopped_tokens:\n",
    "            if tok.lower() in model2.vocab:\n",
    "                count_in_vocab+=1\n",
    "                s+=model2[tok.lower()]\n",
    "        if count_in_vocab!=0:\n",
    "            movie_mean_wordvec[i]=s/float(count_in_vocab)\n",
    "        else:\n",
    "            rows_to_delete.append(i)\n",
    "            genres.pop(-1)\n",
    "#             print overview\n",
    "#             print \"sample \",i,\"had no word2vec\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1668"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(genres)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask2=[]\n",
    "for row in range(len(movie_mean_wordvec)):\n",
    "    if row in rows_to_delete:\n",
    "        mask2.append(False)\n",
    "    else:\n",
    "        mask2.append(True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "X=movie_mean_wordvec[mask2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1668, 300)"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y=mlb.fit_transform(genres)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1668, 19)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "textual_features=(X,Y)\n",
    "f9=open('textual_features.pckl','wb')\n",
    "pickle.dump(textual_features,f9)\n",
    "f9.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "# textual_features=(X,Y)\n",
    "f9=open('textual_features.pckl','rb')\n",
    "textual_features=pickle.load(f9)\n",
    "f9.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "(X,Y)=textual_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1668, 300)"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1668, 19)"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask_text=np.random.rand(len(X))<0.8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train=X[mask_text]\n",
    "Y_train=Y[mask_text]\n",
    "X_test=X[~mask_text]\n",
    "Y_test=Y[~mask_text]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, we use a very similar, super simple architecture as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'test': 373, 'train': 1295}\n",
      "Epoch 0/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 323 1295\n",
      "train Loss: 0.0088 Acc: 0.2494\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 106 373\n",
      "test Loss: 0.0148 Acc: 0.2842\n",
      "('new best accuracy = ', 0.28418230563002683)\n",
      "Epoch 1/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 260 1295\n",
      "train Loss: 0.0132 Acc: 0.2008\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 147 373\n",
      "test Loss: 0.0071 Acc: 0.3941\n",
      "('new best accuracy = ', 0.3941018766756032)\n",
      "Epoch 2/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 317 1295\n",
      "train Loss: 0.0029 Acc: 0.2448\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 86 373\n",
      "test Loss: 0.0013 Acc: 0.2306\n",
      "Epoch 3/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 341 1295\n",
      "train Loss: 0.0011 Acc: 0.2633\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 96 373\n",
      "test Loss: 0.0013 Acc: 0.2574\n",
      "Epoch 4/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 402 1295\n",
      "train Loss: 0.0011 Acc: 0.3104\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 125 373\n",
      "test Loss: 0.0014 Acc: 0.3351\n",
      "Epoch 5/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 589 1295\n",
      "train Loss: 0.0011 Acc: 0.4548\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 190 373\n",
      "test Loss: 0.0013 Acc: 0.5094\n",
      "('new best accuracy = ', 0.5093833780160858)\n",
      "Epoch 6/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 702 1295\n",
      "train Loss: 0.0010 Acc: 0.5421\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 146 373\n",
      "test Loss: 0.0012 Acc: 0.3914\n",
      "Epoch 7/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 613 1295\n",
      "train Loss: 0.0010 Acc: 0.4734\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 159 373\n",
      "test Loss: 0.0014 Acc: 0.4263\n",
      "Epoch 8/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 702 1295\n",
      "train Loss: 0.0010 Acc: 0.5421\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 241 373\n",
      "test Loss: 0.0011 Acc: 0.6461\n",
      "('new best accuracy = ', 0.646112600536193)\n",
      "Epoch 9/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 808 1295\n",
      "train Loss: 0.0009 Acc: 0.6239\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 230 373\n",
      "test Loss: 0.0011 Acc: 0.6166\n",
      "Epoch 10/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 852 1295\n",
      "train Loss: 0.0009 Acc: 0.6579\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 236 373\n",
      "test Loss: 0.0011 Acc: 0.6327\n",
      "Epoch 11/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 854 1295\n",
      "train Loss: 0.0009 Acc: 0.6595\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 234 373\n",
      "test Loss: 0.0011 Acc: 0.6273\n",
      "Epoch 12/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 885 1295\n",
      "train Loss: 0.0009 Acc: 0.6834\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 259 373\n",
      "test Loss: 0.0011 Acc: 0.6944\n",
      "('new best accuracy = ', 0.6943699731903485)\n",
      "Epoch 13/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 901 1295\n",
      "train Loss: 0.0009 Acc: 0.6958\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 219 373\n",
      "test Loss: 0.0012 Acc: 0.5871\n",
      "Epoch 14/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 902 1295\n",
      "train Loss: 0.0009 Acc: 0.6965\n",
      "TESTING STARTED\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 277 373\n",
      "test Loss: 0.0011 Acc: 0.7426\n",
      "('new best accuracy = ', 0.7426273458445041)\n",
      "Epoch 15/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 977 1295\n",
      "train Loss: 0.0009 Acc: 0.7544\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 276 373\n",
      "test Loss: 0.0010 Acc: 0.7399\n",
      "Epoch 16/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 960 1295\n",
      "train Loss: 0.0009 Acc: 0.7413\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 268 373\n",
      "test Loss: 0.0011 Acc: 0.7185\n",
      "Epoch 17/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 976 1295\n",
      "train Loss: 0.0009 Acc: 0.7537\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 279 373\n",
      "test Loss: 0.0011 Acc: 0.7480\n",
      "('new best accuracy = ', 0.7479892761394102)\n",
      "Epoch 18/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 976 1295\n",
      "train Loss: 0.0009 Acc: 0.7537\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 285 373\n",
      "test Loss: 0.0010 Acc: 0.7641\n",
      "('new best accuracy = ', 0.7640750670241286)\n",
      "Epoch 19/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1015 1295\n",
      "train Loss: 0.0008 Acc: 0.7838\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 267 373\n",
      "test Loss: 0.0010 Acc: 0.7158\n",
      "Epoch 20/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 961 1295\n",
      "train Loss: 0.0009 Acc: 0.7421\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 280 373\n",
      "test Loss: 0.0011 Acc: 0.7507\n",
      "Epoch 21/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1025 1295\n",
      "train Loss: 0.0008 Acc: 0.7915\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 282 373\n",
      "test Loss: 0.0010 Acc: 0.7560\n",
      "Epoch 22/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1010 1295\n",
      "train Loss: 0.0008 Acc: 0.7799\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 297 373\n",
      "test Loss: 0.0010 Acc: 0.7962\n",
      "('new best accuracy = ', 0.7962466487935657)\n",
      "Epoch 23/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1027 1295\n",
      "train Loss: 0.0008 Acc: 0.7931\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 298 373\n",
      "test Loss: 0.0010 Acc: 0.7989\n",
      "('new best accuracy = ', 0.7989276139410187)\n",
      "Epoch 24/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1049 1295\n",
      "train Loss: 0.0008 Acc: 0.8100\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 308 373\n",
      "test Loss: 0.0010 Acc: 0.8257\n",
      "('new best accuracy = ', 0.8257372654155496)\n",
      "Epoch 25/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1054 1295\n",
      "train Loss: 0.0008 Acc: 0.8139\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 308 373\n",
      "test Loss: 0.0010 Acc: 0.8257\n",
      "Epoch 26/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1054 1295\n",
      "train Loss: 0.0008 Acc: 0.8139\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 302 373\n",
      "test Loss: 0.0010 Acc: 0.8097\n",
      "Epoch 27/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1060 1295\n",
      "train Loss: 0.0008 Acc: 0.8185\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 315 373\n",
      "test Loss: 0.0010 Acc: 0.8445\n",
      "('new best accuracy = ', 0.8445040214477212)\n",
      "Epoch 28/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1055 1295\n",
      "train Loss: 0.0008 Acc: 0.8147\n",
      "TESTING STARTED\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 279 373\n",
      "test Loss: 0.0010 Acc: 0.7480\n",
      "Epoch 29/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1017 1295\n",
      "train Loss: 0.0008 Acc: 0.7853\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 314 373\n",
      "test Loss: 0.0010 Acc: 0.8418\n",
      "Epoch 30/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1092 1295\n",
      "train Loss: 0.0008 Acc: 0.8432\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 304 373\n",
      "test Loss: 0.0010 Acc: 0.8150\n",
      "Epoch 31/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1062 1295\n",
      "train Loss: 0.0008 Acc: 0.8201\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 327 373\n",
      "test Loss: 0.0009 Acc: 0.8767\n",
      "('new best accuracy = ', 0.8766756032171582)\n",
      "Epoch 32/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1078 1295\n",
      "train Loss: 0.0008 Acc: 0.8324\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 312 373\n",
      "test Loss: 0.0010 Acc: 0.8365\n",
      "Epoch 33/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1090 1295\n",
      "train Loss: 0.0008 Acc: 0.8417\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 314 373\n",
      "test Loss: 0.0009 Acc: 0.8418\n",
      "Epoch 34/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1103 1295\n",
      "train Loss: 0.0008 Acc: 0.8517\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 300 373\n",
      "test Loss: 0.0010 Acc: 0.8043\n",
      "Epoch 35/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1093 1295\n",
      "train Loss: 0.0008 Acc: 0.8440\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 311 373\n",
      "test Loss: 0.0009 Acc: 0.8338\n",
      "Epoch 36/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1091 1295\n",
      "train Loss: 0.0008 Acc: 0.8425\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 312 373\n",
      "test Loss: 0.0010 Acc: 0.8365\n",
      "Epoch 37/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1123 1295\n",
      "train Loss: 0.0008 Acc: 0.8672\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 328 373\n",
      "test Loss: 0.0009 Acc: 0.8794\n",
      "('new best accuracy = ', 0.8793565683646113)\n",
      "Epoch 38/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1127 1295\n",
      "train Loss: 0.0008 Acc: 0.8703\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 332 373\n",
      "test Loss: 0.0009 Acc: 0.8901\n",
      "('new best accuracy = ', 0.8900804289544236)\n",
      "Epoch 39/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1120 1295\n",
      "train Loss: 0.0008 Acc: 0.8649\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 323 373\n",
      "test Loss: 0.0009 Acc: 0.8660\n",
      "Epoch 40/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1122 1295\n",
      "train Loss: 0.0007 Acc: 0.8664\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 325 373\n",
      "test Loss: 0.0009 Acc: 0.8713\n",
      "Epoch 41/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1106 1295\n",
      "train Loss: 0.0008 Acc: 0.8541\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 331 373\n",
      "test Loss: 0.0009 Acc: 0.8874\n",
      "Epoch 42/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1143 1295\n",
      "train Loss: 0.0007 Acc: 0.8826\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 320 373\n",
      "test Loss: 0.0009 Acc: 0.8579\n",
      "Epoch 43/99\n",
      "----------\n",
      "TRAINING STARTED\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1122 1295\n",
      "train Loss: 0.0008 Acc: 0.8664\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 315 373\n",
      "test Loss: 0.0009 Acc: 0.8445\n",
      "Epoch 44/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1126 1295\n",
      "train Loss: 0.0007 Acc: 0.8695\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 339 373\n",
      "test Loss: 0.0009 Acc: 0.9088\n",
      "('new best accuracy = ', 0.9088471849865952)\n",
      "Epoch 45/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1159 1295\n",
      "train Loss: 0.0007 Acc: 0.8950\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 333 373\n",
      "test Loss: 0.0009 Acc: 0.8928\n",
      "Epoch 46/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1153 1295\n",
      "train Loss: 0.0007 Acc: 0.8903\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 339 373\n",
      "test Loss: 0.0009 Acc: 0.9088\n",
      "Epoch 47/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1153 1295\n",
      "train Loss: 0.0007 Acc: 0.8903\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 334 373\n",
      "test Loss: 0.0009 Acc: 0.8954\n",
      "Epoch 48/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1172 1295\n",
      "train Loss: 0.0007 Acc: 0.9050\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 345 373\n",
      "test Loss: 0.0009 Acc: 0.9249\n",
      "('new best accuracy = ', 0.9249329758713136)\n",
      "Epoch 49/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1176 1295\n",
      "train Loss: 0.0007 Acc: 0.9081\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 328 373\n",
      "test Loss: 0.0009 Acc: 0.8794\n",
      "Epoch 50/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1146 1295\n",
      "train Loss: 0.0007 Acc: 0.8849\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 338 373\n",
      "test Loss: 0.0009 Acc: 0.9062\n",
      "Epoch 51/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1169 1295\n",
      "train Loss: 0.0007 Acc: 0.9027\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 335 373\n",
      "test Loss: 0.0009 Acc: 0.8981\n",
      "Epoch 52/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1179 1295\n",
      "train Loss: 0.0007 Acc: 0.9104\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 342 373\n",
      "test Loss: 0.0009 Acc: 0.9169\n",
      "Epoch 53/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1179 1295\n",
      "train Loss: 0.0007 Acc: 0.9104\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 337 373\n",
      "test Loss: 0.0009 Acc: 0.9035\n",
      "Epoch 54/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1197 1295\n",
      "train Loss: 0.0007 Acc: 0.9243\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 344 373\n",
      "test Loss: 0.0009 Acc: 0.9223\n",
      "Epoch 55/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1163 1295\n",
      "train Loss: 0.0007 Acc: 0.8981\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 347 373\n",
      "test Loss: 0.0009 Acc: 0.9303\n",
      "('new best accuracy = ', 0.9302949061662198)\n",
      "Epoch 56/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1193 1295\n",
      "train Loss: 0.0007 Acc: 0.9212\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 352 373\n",
      "test Loss: 0.0008 Acc: 0.9437\n",
      "('new best accuracy = ', 0.9436997319034852)\n",
      "Epoch 57/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1207 1295\n",
      "train Loss: 0.0007 Acc: 0.9320\n",
      "TESTING STARTED\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 328 373\n",
      "test Loss: 0.0009 Acc: 0.8794\n",
      "Epoch 58/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1160 1295\n",
      "train Loss: 0.0007 Acc: 0.8958\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 351 373\n",
      "test Loss: 0.0008 Acc: 0.9410\n",
      "Epoch 59/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1204 1295\n",
      "train Loss: 0.0007 Acc: 0.9297\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 347 373\n",
      "test Loss: 0.0008 Acc: 0.9303\n",
      "Epoch 60/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1212 1295\n",
      "train Loss: 0.0007 Acc: 0.9359\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 352 373\n",
      "test Loss: 0.0008 Acc: 0.9437\n",
      "Epoch 61/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1222 1295\n",
      "train Loss: 0.0007 Acc: 0.9436\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 353 373\n",
      "test Loss: 0.0008 Acc: 0.9464\n",
      "('new best accuracy = ', 0.9463806970509383)\n",
      "Epoch 62/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1212 1295\n",
      "train Loss: 0.0007 Acc: 0.9359\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 342 373\n",
      "test Loss: 0.0009 Acc: 0.9169\n",
      "Epoch 63/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1206 1295\n",
      "train Loss: 0.0007 Acc: 0.9313\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 348 373\n",
      "test Loss: 0.0008 Acc: 0.9330\n",
      "Epoch 64/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1191 1295\n",
      "train Loss: 0.0007 Acc: 0.9197\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 347 373\n",
      "test Loss: 0.0008 Acc: 0.9303\n",
      "Epoch 65/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1218 1295\n",
      "train Loss: 0.0007 Acc: 0.9405\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 354 373\n",
      "test Loss: 0.0008 Acc: 0.9491\n",
      "('new best accuracy = ', 0.9490616621983914)\n",
      "Epoch 66/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1197 1295\n",
      "train Loss: 0.0007 Acc: 0.9243\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 351 373\n",
      "test Loss: 0.0008 Acc: 0.9410\n",
      "Epoch 67/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1216 1295\n",
      "train Loss: 0.0007 Acc: 0.9390\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 357 373\n",
      "test Loss: 0.0008 Acc: 0.9571\n",
      "('new best accuracy = ', 0.9571045576407506)\n",
      "Epoch 68/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1222 1295\n",
      "train Loss: 0.0007 Acc: 0.9436\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 355 373\n",
      "test Loss: 0.0008 Acc: 0.9517\n",
      "Epoch 69/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1236 1295\n",
      "train Loss: 0.0007 Acc: 0.9544\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 358 373\n",
      "test Loss: 0.0008 Acc: 0.9598\n",
      "('new best accuracy = ', 0.9597855227882037)\n",
      "Epoch 70/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1250 1295\n",
      "train Loss: 0.0007 Acc: 0.9653\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 359 373\n",
      "test Loss: 0.0008 Acc: 0.9625\n",
      "('new best accuracy = ', 0.9624664879356568)\n",
      "Epoch 71/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1231 1295\n",
      "train Loss: 0.0007 Acc: 0.9506\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 343 373\n",
      "test Loss: 0.0009 Acc: 0.9196\n",
      "Epoch 72/99\n",
      "----------\n",
      "TRAINING STARTED\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1207 1295\n",
      "train Loss: 0.0007 Acc: 0.9320\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 354 373\n",
      "test Loss: 0.0008 Acc: 0.9491\n",
      "Epoch 73/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1234 1295\n",
      "train Loss: 0.0006 Acc: 0.9529\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 351 373\n",
      "test Loss: 0.0008 Acc: 0.9410\n",
      "Epoch 74/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1230 1295\n",
      "train Loss: 0.0007 Acc: 0.9498\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 346 373\n",
      "test Loss: 0.0008 Acc: 0.9276\n",
      "Epoch 75/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1216 1295\n",
      "train Loss: 0.0007 Acc: 0.9390\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 353 373\n",
      "test Loss: 0.0008 Acc: 0.9464\n",
      "Epoch 76/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1233 1295\n",
      "train Loss: 0.0007 Acc: 0.9521\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 357 373\n",
      "test Loss: 0.0008 Acc: 0.9571\n",
      "Epoch 77/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1243 1295\n",
      "train Loss: 0.0006 Acc: 0.9598\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 362 373\n",
      "test Loss: 0.0008 Acc: 0.9705\n",
      "('new best accuracy = ', 0.9705093833780161)\n",
      "Epoch 78/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1252 1295\n",
      "train Loss: 0.0006 Acc: 0.9668\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 358 373\n",
      "test Loss: 0.0008 Acc: 0.9598\n",
      "Epoch 79/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1254 1295\n",
      "train Loss: 0.0006 Acc: 0.9683\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 357 373\n",
      "test Loss: 0.0008 Acc: 0.9571\n",
      "Epoch 80/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1259 1295\n",
      "train Loss: 0.0006 Acc: 0.9722\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 362 373\n",
      "test Loss: 0.0008 Acc: 0.9705\n",
      "Epoch 81/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1264 1295\n",
      "train Loss: 0.0006 Acc: 0.9761\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 363 373\n",
      "test Loss: 0.0008 Acc: 0.9732\n",
      "('new best accuracy = ', 0.9731903485254692)\n",
      "Epoch 82/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1248 1295\n",
      "train Loss: 0.0006 Acc: 0.9637\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 359 373\n",
      "test Loss: 0.0008 Acc: 0.9625\n",
      "Epoch 83/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1249 1295\n",
      "train Loss: 0.0006 Acc: 0.9645\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 358 373\n",
      "test Loss: 0.0008 Acc: 0.9598\n",
      "Epoch 84/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1254 1295\n",
      "train Loss: 0.0006 Acc: 0.9683\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 362 373\n",
      "test Loss: 0.0008 Acc: 0.9705\n",
      "Epoch 85/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1266 1295\n",
      "train Loss: 0.0006 Acc: 0.9776\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 362 373\n",
      "test Loss: 0.0008 Acc: 0.9705\n",
      "Epoch 86/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1252 1295\n",
      "train Loss: 0.0006 Acc: 0.9668\n",
      "TESTING STARTED\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 354 373\n",
      "test Loss: 0.0008 Acc: 0.9491\n",
      "Epoch 87/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1244 1295\n",
      "train Loss: 0.0006 Acc: 0.9606\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 364 373\n",
      "test Loss: 0.0008 Acc: 0.9759\n",
      "('new best accuracy = ', 0.9758713136729222)\n",
      "Epoch 88/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1263 1295\n",
      "train Loss: 0.0006 Acc: 0.9753\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 358 373\n",
      "test Loss: 0.0008 Acc: 0.9598\n",
      "Epoch 89/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1233 1295\n",
      "train Loss: 0.0006 Acc: 0.9521\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 354 373\n",
      "test Loss: 0.0008 Acc: 0.9491\n",
      "Epoch 90/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1242 1295\n",
      "train Loss: 0.0006 Acc: 0.9591\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 362 373\n",
      "test Loss: 0.0008 Acc: 0.9705\n",
      "Epoch 91/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1266 1295\n",
      "train Loss: 0.0006 Acc: 0.9776\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 365 373\n",
      "test Loss: 0.0008 Acc: 0.9786\n",
      "('new best accuracy = ', 0.9785522788203753)\n",
      "Epoch 92/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1276 1295\n",
      "train Loss: 0.0006 Acc: 0.9853\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 365 373\n",
      "test Loss: 0.0008 Acc: 0.9786\n",
      "Epoch 93/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1281 1295\n",
      "train Loss: 0.0006 Acc: 0.9892\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 366 373\n",
      "test Loss: 0.0008 Acc: 0.9812\n",
      "('new best accuracy = ', 0.9812332439678284)\n",
      "Epoch 94/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1261 1295\n",
      "train Loss: 0.0006 Acc: 0.9737\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 353 373\n",
      "test Loss: 0.0008 Acc: 0.9464\n",
      "Epoch 95/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1253 1295\n",
      "train Loss: 0.0006 Acc: 0.9676\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 355 373\n",
      "test Loss: 0.0008 Acc: 0.9517\n",
      "Epoch 96/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1254 1295\n",
      "train Loss: 0.0006 Acc: 0.9683\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 365 373\n",
      "test Loss: 0.0008 Acc: 0.9786\n",
      "Epoch 97/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1270 1295\n",
      "train Loss: 0.0006 Acc: 0.9807\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 364 373\n",
      "test Loss: 0.0008 Acc: 0.9759\n",
      "Epoch 98/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1264 1295\n",
      "train Loss: 0.0006 Acc: 0.9761\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 363 373\n",
      "test Loss: 0.0008 Acc: 0.9732\n",
      "Epoch 99/99\n",
      "----------\n",
      "TRAINING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([500]) torch.Size([500, 19])\n",
      "loss done\n",
      "loss backward\n",
      "done loss backward\n",
      "done optim\n",
      "torch.Size([295]) torch.Size([295, 19])\n",
      "trying epoch loss\n",
      "train 1272 1295\n",
      "train Loss: 0.0006 Acc: 0.9822\n",
      "TESTING STARTED\n",
      "loss done\n",
      "('Reached iteration ', 0)\n",
      "torch.Size([373]) torch.Size([373, 19])\n",
      "trying epoch loss\n",
      "test 363 373\n",
      "test Loss: 0.0008 Acc: 0.9732\n",
      "Training complete in 1m 19s\n",
      "Best val Acc: 0.981233\n",
      "returning and looping back\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "dsets = {}\n",
    "dsets['train'] = FeatureDataset(X_train, Y_train)\n",
    "dsets['test'] = FeatureDataset(X_test, Y_test)\n",
    "dset_loaders = {x: torch.utils.data.DataLoader(dsets[x], batch_size=500,\n",
    "                                               shuffle=True, num_workers=25)\n",
    "                for x in ['train', 'test']}\n",
    "\n",
    "dset_sizes = {x: len(dsets[x]) for x in ['train', 'test']}\n",
    "print dset_sizes\n",
    "model_textual = nn.Sequential(\n",
    "    nn.Linear(X.shape[1],1024),  #300\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(1024,Y.shape[1]),\n",
    "    nn.Softmax(dim=1)\n",
    ")\n",
    "\n",
    "optimizer = torch.optim.RMSprop(model_textual.parameters())\n",
    "criterion = nn.BCELoss()\n",
    "textual_model, Y_preds = train_model(model_textual, criterion, optimizer, num_epochs=100)\n",
    "\n",
    "# Save model\n",
    "torch.save(textual_model.state_dict(), 'text_fine_tuned_best_model.pt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "genre_list.append(10769)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Our predictions for the movies are - \n",
      "\n",
      "Predicted:  [u'Mystery', u'Thriller', u'Horror']  Actual:  [u'Adventure', u'Fantasy', u'Action', u'Science Fiction']\n",
      "Predicted:  [u'Family', u'Adventure', u'Fantasy']  Actual:  [u'Animation', u'Drama', u'Action', u'Science Fiction']\n",
      "Predicted:  [u'Comedy', u'Animation', u'Family']  Actual:  [u'Fantasy', u'Horror', u'Science Fiction', u'Mystery']\n",
      "Predicted:  [u'Comedy', u'Family', u'Fantasy']  Actual:  [u'Drama', u'History', u'Thriller']\n",
      "Predicted:  [u'Family', u'Fantasy', u'Comedy']  Actual:  [u'Documentary', u'Science Fiction']\n",
      "Predicted:  [u'Comedy', u'Animation', u'Family']  Actual:  [u'Drama', u'Crime', u'Mystery']\n",
      "Predicted:  [u'Comedy', u'Family', u'Animation']  Actual:  [u'Adventure', u'Fantasy', u'Comedy', u'Family']\n"
     ]
    }
   ],
   "source": [
    "print \"Our predictions for the movies are - \\n\"\n",
    "precs=[]\n",
    "recs=[]\n",
    "for i in range(len(Y_preds)):\n",
    "    row=Y_preds[i]\n",
    "    gt_genres=Y_test[i]\n",
    "    gt_genre_names=[]\n",
    "    for j in range(len(gt_genres)):\n",
    "        if gt_genres[j]==1:\n",
    "            gt_genre_names.append(Genre_ID_to_name[genre_list[j]])\n",
    "    top_3=np.argsort(row)[-3:]\n",
    "    predicted_genres=[]\n",
    "    for genre in top_3:\n",
    "        predicted_genres.append(Genre_ID_to_name[genre_list[genre]])\n",
    "    (precision,recall)=precision_recall(gt_genre_names,predicted_genres)\n",
    "    precs.append(precision)\n",
    "    recs.append(recall)\n",
    "    if i%50==0:\n",
    "        print \"Predicted: \",predicted_genres,\" Actual: \",gt_genre_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.21298949379178606 0.19942693409742118\n"
     ]
    }
   ],
   "source": [
    "print np.mean(np.asarray(precs)),np.mean(np.asarray(recs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even without much tuning of the above model, these results are able to beat our previous results. \n",
    "\n",
    "Note - I got accuracies as high as 78% when doing classification using plots scraped from Wikipedia. The large amount of information was very suitable for movie genre classification with a deep model. Strongly suggest you to try playing around with architectures."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 9 - Upcoming Tutorials and Acknowledgements\n",
    "\n",
    "Congrats! This is the end of our pilot project! Needless to say, a lot of the above content may be new to you, or may be things that you know very well. If it's the former, I hope this tutorial would have helped you. If it is the latter and you think I wrote something incorrect or that my understanding can be improved, feel free to create a github issue so that I can correct it! \n",
    "\n",
    "Writing tutorials can take a lot of time, but it is a great learning experience. I am currently working on a tutorial focussing on word embeddings, which will explore word2vec and other word embeddings in detail. While it will take some time to be up, I will post a link to it's repository on the README for this project so that interested readers can find it.\n",
    "\n",
    "I would like to thank a few of my friends who had an indispensible role to play in me making this tutorial. Firstly, Professor Hanspeter Pfister and Verena Kaynig at Harvard, who helped guide this tutorial/project and scope it. Secondly, my friends Sahil Loomba and Matthew Tancik for their suggestions and editing the material and the presentation of the storyline. Thirdly, Zoya Bylinskii at MIT for constantly motivating me to put in my effort into this tutorial. Finally, all others who helped me feel confident enough to take up this task and to see it till the end. Thanks all of you!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "deeplearningproject",
   "language": "python",
   "name": "deeplearningproject"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}