06. Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification\n",
    "Put simply, classification is the task of predicting a label for a given observation. For example: you are given certain physical descriptions of an animal, and your taks is to classify them as either a dog or a cat. Here, we will classify iris flowers.\n",
    "\n",
    "As we will see later, we will use different classifiers and at the end of this notebook, we will compare them. We will define our accuracy function right now to get it out of the way. We will use a simple accuracy function that returns the ratio of the number of correctly classified observations to the total number of predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "findaccuracy (generic function with 1 method)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "findaccuracy(predictedvals,groundtruthvals) = sum(predictedvals.==groundtruthvals)/length(groundtruthvals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "using GLMNet\n",
    "using RDatasets\n",
    "using MLBase\n",
    "using Plots\n",
    "using DecisionTree\n",
    "using Distances\n",
    "using NearestNeighbors\n",
    "using Random\n",
    "using LinearAlgebra\n",
    "using DataStructures\n",
    "using LIBSVM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the data first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>SepalLength</th><th>SepalWidth</th><th>PetalLength</th><th>PetalWidth</th><th>Species</th></tr><tr><th></th><th>Float64</th><th>Float64</th><th>Float64</th><th>Float64</th><th>Cat…</th></tr></thead><tbody><p>150 rows × 5 columns</p><tr><th>1</th><td>5.1</td><td>3.5</td><td>1.4</td><td>0.2</td><td>setosa</td></tr><tr><th>2</th><td>4.9</td><td>3.0</td><td>1.4</td><td>0.2</td><td>setosa</td></tr><tr><th>3</th><td>4.7</td><td>3.2</td><td>1.3</td><td>0.2</td><td>setosa</td></tr><tr><th>4</th><td>4.6</td><td>3.1</td><td>1.5</td><td>0.2</td><td>setosa</td></tr><tr><th>5</th><td>5.0</td><td>3.6</td><td>1.4</td><td>0.2</td><td>setosa</td></tr><tr><th>6</th><td>5.4</td><td>3.9</td><td>1.7</td><td>0.4</td><td>setosa</td></tr><tr><th>7</th><td>4.6</td><td>3.4</td><td>1.4</td><td>0.3</td><td>setosa</td></tr><tr><th>8</th><td>5.0</td><td>3.4</td><td>1.5</td><td>0.2</td><td>setosa</td></tr><tr><th>9</th><td>4.4</td><td>2.9</td><td>1.4</td><td>0.2</td><td>setosa</td></tr><tr><th>10</th><td>4.9</td><td>3.1</td><td>1.5</td><td>0.1</td><td>setosa</td></tr><tr><th>11</th><td>5.4</td><td>3.7</td><td>1.5</td><td>0.2</td><td>setosa</td></tr><tr><th>12</th><td>4.8</td><td>3.4</td><td>1.6</td><td>0.2</td><td>setosa</td></tr><tr><th>13</th><td>4.8</td><td>3.0</td><td>1.4</td><td>0.1</td><td>setosa</td></tr><tr><th>14</th><td>4.3</td><td>3.0</td><td>1.1</td><td>0.1</td><td>setosa</td></tr><tr><th>15</th><td>5.8</td><td>4.0</td><td>1.2</td><td>0.2</td><td>setosa</td></tr><tr><th>16</th><td>5.7</td><td>4.4</td><td>1.5</td><td>0.4</td><td>setosa</td></tr><tr><th>17</th><td>5.4</td><td>3.9</td><td>1.3</td><td>0.4</td><td>setosa</td></tr><tr><th>18</th><td>5.1</td><td>3.5</td><td>1.4</td><td>0.3</td><td>setosa</td></tr><tr><th>19</th><td>5.7</td><td>3.8</td><td>1.7</td><td>0.3</td><td>setosa</td></tr><tr><th>20</th><td>5.1</td><td>3.8</td><td>1.5</td><td>0.3</td><td>setosa</td></tr><tr><th>21</th><td>5.4</td><td>3.4</td><td>1.7</td><td>0.2</td><td>setosa</td></tr><tr><th>22</th><td>5.1</td><td>3.7</td><td>1.5</td><td>0.4</td><td>setosa</td></tr><tr><th>23</th><td>4.6</td><td>3.6</td><td>1.0</td><td>0.2</td><td>setosa</td></tr><tr><th>24</th><td>5.1</td><td>3.3</td><td>1.7</td><td>0.5</td><td>setosa</td></tr><tr><th>25</th><td>4.8</td><td>3.4</td><td>1.9</td><td>0.2</td><td>setosa</td></tr><tr><th>26</th><td>5.0</td><td>3.0</td><td>1.6</td><td>0.2</td><td>setosa</td></tr><tr><th>27</th><td>5.0</td><td>3.4</td><td>1.6</td><td>0.4</td><td>setosa</td></tr><tr><th>28</th><td>5.2</td><td>3.5</td><td>1.5</td><td>0.2</td><td>setosa</td></tr><tr><th>29</th><td>5.2</td><td>3.4</td><td>1.4</td><td>0.2</td><td>setosa</td></tr><tr><th>30</th><td>4.7</td><td>3.2</td><td>1.6</td><td>0.2</td><td>setosa</td></tr><tr><th>&vellip;</th><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccc}\n",
       "\t& SepalLength & SepalWidth & PetalLength & PetalWidth & Species\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64 & Float64 & Float64 & Cat…\\\\\n",
       "\t\\hline\n",
       "\t1 & 5.1 & 3.5 & 1.4 & 0.2 & setosa \\\\\n",
       "\t2 & 4.9 & 3.0 & 1.4 & 0.2 & setosa \\\\\n",
       "\t3 & 4.7 & 3.2 & 1.3 & 0.2 & setosa \\\\\n",
       "\t4 & 4.6 & 3.1 & 1.5 & 0.2 & setosa \\\\\n",
       "\t5 & 5.0 & 3.6 & 1.4 & 0.2 & setosa \\\\\n",
       "\t6 & 5.4 & 3.9 & 1.7 & 0.4 & setosa \\\\\n",
       "\t7 & 4.6 & 3.4 & 1.4 & 0.3 & setosa \\\\\n",
       "\t8 & 5.0 & 3.4 & 1.5 & 0.2 & setosa \\\\\n",
       "\t9 & 4.4 & 2.9 & 1.4 & 0.2 & setosa \\\\\n",
       "\t10 & 4.9 & 3.1 & 1.5 & 0.1 & setosa \\\\\n",
       "\t11 & 5.4 & 3.7 & 1.5 & 0.2 & setosa \\\\\n",
       "\t12 & 4.8 & 3.4 & 1.6 & 0.2 & setosa \\\\\n",
       "\t13 & 4.8 & 3.0 & 1.4 & 0.1 & setosa \\\\\n",
       "\t14 & 4.3 & 3.0 & 1.1 & 0.1 & setosa \\\\\n",
       "\t15 & 5.8 & 4.0 & 1.2 & 0.2 & setosa \\\\\n",
       "\t16 & 5.7 & 4.4 & 1.5 & 0.4 & setosa \\\\\n",
       "\t17 & 5.4 & 3.9 & 1.3 & 0.4 & setosa \\\\\n",
       "\t18 & 5.1 & 3.5 & 1.4 & 0.3 & setosa \\\\\n",
       "\t19 & 5.7 & 3.8 & 1.7 & 0.3 & setosa \\\\\n",
       "\t20 & 5.1 & 3.8 & 1.5 & 0.3 & setosa \\\\\n",
       "\t21 & 5.4 & 3.4 & 1.7 & 0.2 & setosa \\\\\n",
       "\t22 & 5.1 & 3.7 & 1.5 & 0.4 & setosa \\\\\n",
       "\t23 & 4.6 & 3.6 & 1.0 & 0.2 & setosa \\\\\n",
       "\t24 & 5.1 & 3.3 & 1.7 & 0.5 & setosa \\\\\n",
       "\t25 & 4.8 & 3.4 & 1.9 & 0.2 & setosa \\\\\n",
       "\t26 & 5.0 & 3.0 & 1.6 & 0.2 & setosa \\\\\n",
       "\t27 & 5.0 & 3.4 & 1.6 & 0.4 & setosa \\\\\n",
       "\t28 & 5.2 & 3.5 & 1.5 & 0.2 & setosa \\\\\n",
       "\t29 & 5.2 & 3.4 & 1.4 & 0.2 & setosa \\\\\n",
       "\t30 & 4.7 & 3.2 & 1.6 & 0.2 & setosa \\\\\n",
       "\t$\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "150×5 DataFrame\n",
       "│ Row │ SepalLength │ SepalWidth │ PetalLength │ PetalWidth │ Species   │\n",
       "│     │ \u001b[90mFloat64\u001b[39m     │ \u001b[90mFloat64\u001b[39m    │ \u001b[90mFloat64\u001b[39m     │ \u001b[90mFloat64\u001b[39m    │ \u001b[90mCat…\u001b[39m      │\n",
       "├─────┼─────────────┼────────────┼─────────────┼────────────┼───────────┤\n",
       "│ 1   │ 5.1         │ 3.5        │ 1.4         │ 0.2        │ setosa    │\n",
       "│ 2   │ 4.9         │ 3.0        │ 1.4         │ 0.2        │ setosa    │\n",
       "│ 3   │ 4.7         │ 3.2        │ 1.3         │ 0.2        │ setosa    │\n",
       "│ 4   │ 4.6         │ 3.1        │ 1.5         │ 0.2        │ setosa    │\n",
       "│ 5   │ 5.0         │ 3.6        │ 1.4         │ 0.2        │ setosa    │\n",
       "│ 6   │ 5.4         │ 3.9        │ 1.7         │ 0.4        │ setosa    │\n",
       "│ 7   │ 4.6         │ 3.4        │ 1.4         │ 0.3        │ setosa    │\n",
       "│ 8   │ 5.0         │ 3.4        │ 1.5         │ 0.2        │ setosa    │\n",
       "│ 9   │ 4.4         │ 2.9        │ 1.4         │ 0.2        │ setosa    │\n",
       "│ 10  │ 4.9         │ 3.1        │ 1.5         │ 0.1        │ setosa    │\n",
       "⋮\n",
       "│ 140 │ 6.9         │ 3.1        │ 5.4         │ 2.1        │ virginica │\n",
       "│ 141 │ 6.7         │ 3.1        │ 5.6         │ 2.4        │ virginica │\n",
       "│ 142 │ 6.9         │ 3.1        │ 5.1         │ 2.3        │ virginica │\n",
       "│ 143 │ 5.8         │ 2.7        │ 5.1         │ 1.9        │ virginica │\n",
       "│ 144 │ 6.8         │ 3.2        │ 5.9         │ 2.3        │ virginica │\n",
       "│ 145 │ 6.7         │ 3.3        │ 5.7         │ 2.5        │ virginica │\n",
       "│ 146 │ 6.7         │ 3.0        │ 5.2         │ 2.3        │ virginica │\n",
       "│ 147 │ 6.3         │ 2.5        │ 5.0         │ 1.9        │ virginica │\n",
       "│ 148 │ 6.5         │ 3.0        │ 5.2         │ 2.0        │ virginica │\n",
       "│ 149 │ 6.2         │ 3.4        │ 5.4         │ 2.3        │ virginica │\n",
       "│ 150 │ 5.9         │ 3.0        │ 5.1         │ 1.8        │ virginica │"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iris = dataset(\"datasets\", \"iris\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "150-element CategoricalArray{String,1,UInt8}:\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " ⋮\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\""
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = Matrix(iris[:,1:4])\n",
    "irislabels = iris[:,5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "150×4 Array{Float64,2}:\n",
       " 5.1  3.5  1.4  0.2\n",
       " 4.9  3.0  1.4  0.2\n",
       " 4.7  3.2  1.3  0.2\n",
       " 4.6  3.1  1.5  0.2\n",
       " 5.0  3.6  1.4  0.2\n",
       " 5.4  3.9  1.7  0.4\n",
       " 4.6  3.4  1.4  0.3\n",
       " 5.0  3.4  1.5  0.2\n",
       " 4.4  2.9  1.4  0.2\n",
       " 4.9  3.1  1.5  0.1\n",
       " 5.4  3.7  1.5  0.2\n",
       " 4.8  3.4  1.6  0.2\n",
       " 4.8  3.0  1.4  0.1\n",
       " ⋮              \n",
       " 6.0  3.0  4.8  1.8\n",
       " 6.9  3.1  5.4  2.1\n",
       " 6.7  3.1  5.6  2.4\n",
       " 6.9  3.1  5.1  2.3\n",
       " 5.8  2.7  5.1  1.9\n",
       " 6.8  3.2  5.9  2.3\n",
       " 6.7  3.3  5.7  2.5\n",
       " 6.7  3.0  5.2  2.3\n",
       " 6.3  2.5  5.0  1.9\n",
       " 6.5  3.0  5.2  2.0\n",
       " 6.2  3.4  5.4  2.3\n",
       " 5.9  3.0  5.1  1.8"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "150-element Array{Int64,1}:\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " ⋮\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "irislabelsmap = labelmap(irislabels)\n",
    "y = labelencode(irislabelsmap, irislabels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In classification, we often want to use some of the data to fit a model, and the rest of the data to validate (commonly known as `training` and `testing` data). We will get this data ready now so that we can easily use it in the rest of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "perclass_splits (generic function with 1 method)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function perclass_splits(y,at)\n",
    "    uids = unique(y)\n",
    "    keepids = []\n",
    "    for ui in uids\n",
    "        curids = findall(y.==ui)\n",
    "        rowids = randsubseq(curids, at) \n",
    "        push!(keepids,rowids...)\n",
    "    end\n",
    "    return keepids\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "search: \u001b[0m\u001b[1mr\u001b[22m\u001b[0m\u001b[1ma\u001b[22m\u001b[0m\u001b[1mn\u001b[22m\u001b[0m\u001b[1md\u001b[22m\u001b[0m\u001b[1ms\u001b[22m\u001b[0m\u001b[1mu\u001b[22m\u001b[0m\u001b[1mb\u001b[22m\u001b[0m\u001b[1ms\u001b[22m\u001b[0m\u001b[1me\u001b[22m\u001b[0m\u001b[1mq\u001b[22m \u001b[0m\u001b[1mr\u001b[22m\u001b[0m\u001b[1ma\u001b[22m\u001b[0m\u001b[1mn\u001b[22m\u001b[0m\u001b[1md\u001b[22m\u001b[0m\u001b[1ms\u001b[22m\u001b[0m\u001b[1mu\u001b[22m\u001b[0m\u001b[1mb\u001b[22m\u001b[0m\u001b[1ms\u001b[22m\u001b[0m\u001b[1me\u001b[22m\u001b[0m\u001b[1mq\u001b[22m! \u001b[0m\u001b[1mR\u001b[22m\u001b[0m\u001b[1ma\u001b[22m\u001b[0m\u001b[1mn\u001b[22m\u001b[0m\u001b[1md\u001b[22mom\u001b[0m\u001b[1mS\u001b[22m\u001b[0m\u001b[1mu\u001b[22m\u001b[0m\u001b[1mb\u001b[22m St\u001b[0m\u001b[1mr\u001b[22m\u001b[0m\u001b[1ma\u001b[22mtifiedRa\u001b[0m\u001b[1mn\u001b[22m\u001b[0m\u001b[1md\u001b[22mom\u001b[0m\u001b[1mS\u001b[22m\u001b[0m\u001b[1mu\u001b[22m\u001b[0m\u001b[1mb\u001b[22m\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/latex": [
       "\\begin{verbatim}\n",
       "randsubseq([rng=GLOBAL_RNG,] A, p) -> Vector\n",
       "\\end{verbatim}\n",
       "Return a vector consisting of a random subsequence of the given array \\texttt{A}, where each element of \\texttt{A} is included (in order) with independent probability \\texttt{p}. (Complexity is linear in \\texttt{p*length(A)}, so this function is efficient even if \\texttt{p} is small and \\texttt{A} is large.) Technically, this process is known as \"Bernoulli sampling\" of \\texttt{A}.\n",
       "\n",
       "\\section{Examples}\n",
       "\\begin{verbatim}\n",
       "julia> rng = MersenneTwister(1234);\n",
       "\n",
       "julia> randsubseq(rng, collect(1:8), 0.3)\n",
       "2-element Array{Int64,1}:\n",
       " 7\n",
       " 8\n",
       "\\end{verbatim}\n"
      ],
      "text/markdown": [
       "```\n",
       "randsubseq([rng=GLOBAL_RNG,] A, p) -> Vector\n",
       "```\n",
       "\n",
       "Return a vector consisting of a random subsequence of the given array `A`, where each element of `A` is included (in order) with independent probability `p`. (Complexity is linear in `p*length(A)`, so this function is efficient even if `p` is small and `A` is large.) Technically, this process is known as \"Bernoulli sampling\" of `A`.\n",
       "\n",
       "# Examples\n",
       "\n",
       "```jldoctest\n",
       "julia> rng = MersenneTwister(1234);\n",
       "\n",
       "julia> randsubseq(rng, collect(1:8), 0.3)\n",
       "2-element Array{Int64,1}:\n",
       " 7\n",
       " 8\n",
       "```\n"
      ],
      "text/plain": [
       "\u001b[36m  randsubseq([rng=GLOBAL_RNG,] A, p) -> Vector\u001b[39m\n",
       "\n",
       "  Return a vector consisting of a random subsequence of the given array \u001b[36mA\u001b[39m,\n",
       "  where each element of \u001b[36mA\u001b[39m is included (in order) with independent probability\n",
       "  \u001b[36mp\u001b[39m. (Complexity is linear in \u001b[36mp*length(A)\u001b[39m, so this function is efficient even\n",
       "  if \u001b[36mp\u001b[39m is small and \u001b[36mA\u001b[39m is large.) Technically, this process is known as\n",
       "  \"Bernoulli sampling\" of \u001b[36mA\u001b[39m.\n",
       "\n",
       "\u001b[1m  Examples\u001b[22m\n",
       "\u001b[1m  ≡≡≡≡≡≡≡≡≡≡\u001b[22m\n",
       "\n",
       "\u001b[36m  julia> rng = MersenneTwister(1234);\u001b[39m\n",
       "\u001b[36m  \u001b[39m\n",
       "\u001b[36m  julia> randsubseq(rng, collect(1:8), 0.3)\u001b[39m\n",
       "\u001b[36m  2-element Array{Int64,1}:\u001b[39m\n",
       "\u001b[36m   7\u001b[39m\n",
       "\u001b[36m   8\u001b[39m"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "?randsubseq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "51-element Array{Int64,1}:\n",
       "   2\n",
       "   4\n",
       "  10\n",
       "  12\n",
       "  13\n",
       "  17\n",
       "  21\n",
       "  22\n",
       "  23\n",
       "  25\n",
       "  26\n",
       "  27\n",
       "  28\n",
       "   ⋮\n",
       " 111\n",
       " 124\n",
       " 125\n",
       " 126\n",
       " 127\n",
       " 129\n",
       " 133\n",
       " 136\n",
       " 140\n",
       " 144\n",
       " 146\n",
       " 148"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainids = perclass_splits(y,0.7)\n",
    "testids = setdiff(1:length(y),trainids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will need one more function, and that is the function that will assign classes based on the predicted values when the predicted values are continuous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "assign_class (generic function with 1 method)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assign_class(predictedvalue) = argmin(abs.(predictedvalue .- [1,2,3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 1: Lasso"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Least Squares GLMNet Cross Validation\n",
       "71 models for 4 predictors in 10 folds\n",
       "Best λ 0.001 (mean loss 0.057, std 0.005)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = glmnet(X[trainids,:], y[trainids])\n",
    "cv = glmnetcv(X[trainids,:], y[trainids])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# choose the best lambda to predict with.\n",
    "path = glmnet(X[trainids,:], y[trainids])\n",
    "cv = glmnetcv(X[trainids,:], y[trainids])\n",
    "mylambda = path.lambda[argmin(cv.meanloss)]\n",
    "\n",
    "path = glmnet(X[trainids,:], y[trainids],lambda=[mylambda]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "51×1 Array{Float64,2}:\n",
       " 1.0137504950307894\n",
       " 1.0148017024933282\n",
       " 0.9520491218707511\n",
       " 0.9866846373551885\n",
       " 0.9509979144082121\n",
       " 0.9923630858100112\n",
       " 1.0023199811180667\n",
       " 1.0528020459364464\n",
       " 0.8637043021772398\n",
       " 1.0335906686438234\n",
       " 1.045021182556546\n",
       " 1.112189798600343\n",
       " 0.9564651572919708\n",
       " ⋮\n",
       " 2.692636392862999\n",
       " 2.608781225593785\n",
       " 2.8346168997625067\n",
       " 2.7078493254837492\n",
       " 2.578561745530567\n",
       " 2.8919022375013252\n",
       " 2.9546548181239025\n",
       " 3.0664158449601926\n",
       " 2.8168791410745504\n",
       " 3.005976884833757\n",
       " 2.9256977510942876\n",
       " 2.737440009226556"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q = X[testids,:];\n",
    "predictions_lasso = GLMNet.predict(path,q)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions_lasso = assign_class.(predictions_lasso)\n",
    "findaccuracy(predictions_lasso,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 2: Ridge\n",
    "We will use the same function but set alpha to zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# choose the best lambda to predict with.\n",
    "path = glmnet(X[trainids,:], y[trainids],alpha=0);\n",
    "cv = glmnetcv(X[trainids,:], y[trainids],alpha=0)\n",
    "mylambda = path.lambda[argmin(cv.meanloss)]\n",
    "path = glmnet(X[trainids,:], y[trainids],alpha=0,lambda=[mylambda]);\n",
    "q = X[testids,:];\n",
    "predictions_ridge = GLMNet.predict(path,q)\n",
    "predictions_ridge = assign_class.(predictions_ridge)\n",
    "findaccuracy(predictions_ridge,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 3: Elastic Net\n",
    "We will use the same function but set alpha to 0.5 (it's the combination of lasso and ridge)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# choose the best lambda to predict with.\n",
    "path = glmnet(X[trainids,:], y[trainids],alpha=0.5);\n",
    "cv = glmnetcv(X[trainids,:], y[trainids],alpha=0.5)\n",
    "mylambda = path.lambda[argmin(cv.meanloss)]\n",
    "path = glmnet(X[trainids,:], y[trainids],alpha=0.5,lambda=[mylambda]);\n",
    "q = X[testids,:];\n",
    "predictions_EN = GLMNet.predict(path,q)\n",
    "predictions_EN = assign_class.(predictions_EN)\n",
    "findaccuracy(predictions_EN,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 4: Decision Trees\n",
    "We will use the package `DecisionTree`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier\n",
       "max_depth:                2\n",
       "min_samples_leaf:         1\n",
       "min_samples_split:        2\n",
       "min_purity_increase:      0.0\n",
       "pruning_purity_threshold: 1.0\n",
       "n_subfeatures:            0\n",
       "classes:                  [1, 2, 3]\n",
       "root:                     Decision Tree\n",
       "Leaves: 3\n",
       "Depth:  2"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = DecisionTreeClassifier(max_depth=2)\n",
    "DecisionTree.fit!(model, X[trainids,:], y[trainids])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9607843137254902"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q = X[testids,:];\n",
    "predictions_DT = DecisionTree.predict(model, q)\n",
    "findaccuracy(predictions_DT,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 5: Random Forests\n",
    "The `RandomForestClassifier` is available through the `DecisionTree` package as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier\n",
       "n_trees:             20\n",
       "n_subfeatures:       -1\n",
       "partial_sampling:    0.7\n",
       "max_depth:           -1\n",
       "min_samples_leaf:    1\n",
       "min_samples_split:   2\n",
       "min_purity_increase: 0.0\n",
       "classes:             [1, 2, 3]\n",
       "ensemble:            Ensemble of Decision Trees\n",
       "Trees:      20\n",
       "Avg Leaves: 6.15\n",
       "Avg Depth:  4.2"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = RandomForestClassifier(n_trees=20)\n",
    "DecisionTree.fit!(model, X[trainids,:], y[trainids])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9803921568627451"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q = X[testids,:];\n",
    "predictions_RF = DecisionTree.predict(model, q)\n",
    "findaccuracy(predictions_RF,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 6: Using a Nearest Neighbor method\n",
    "We will use the `NearestNeighbors` package here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KDTree{StaticArrays.SArray{Tuple{4},Float64,1,4},Euclidean,Float64}\n",
       "  Number of points: 99\n",
       "  Dimensions: 4\n",
       "  Metric: Euclidean(0.0)\n",
       "  Reordered: true"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xtrain = X[trainids,:]\n",
    "ytrain = y[trainids]\n",
    "kdtree = KDTree(Xtrain')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "51×4 Array{Float64,2}:\n",
       " 4.9  3.0  1.4  0.2\n",
       " 4.6  3.1  1.5  0.2\n",
       " 4.9  3.1  1.5  0.1\n",
       " 4.8  3.4  1.6  0.2\n",
       " 4.8  3.0  1.4  0.1\n",
       " 5.4  3.9  1.3  0.4\n",
       " 5.4  3.4  1.7  0.2\n",
       " 5.1  3.7  1.5  0.4\n",
       " 4.6  3.6  1.0  0.2\n",
       " 4.8  3.4  1.9  0.2\n",
       " 5.0  3.0  1.6  0.2\n",
       " 5.0  3.4  1.6  0.4\n",
       " 5.2  3.5  1.5  0.2\n",
       " ⋮              \n",
       " 6.5  3.2  5.1  2.0\n",
       " 6.3  2.7  4.9  1.8\n",
       " 6.7  3.3  5.7  2.1\n",
       " 7.2  3.2  6.0  1.8\n",
       " 6.2  2.8  4.8  1.8\n",
       " 6.4  2.8  5.6  2.1\n",
       " 6.4  2.8  5.6  2.2\n",
       " 7.7  3.0  6.1  2.3\n",
       " 6.9  3.1  5.4  2.1\n",
       " 6.8  3.2  5.9  2.3\n",
       " 6.7  3.0  5.2  2.3\n",
       " 6.5  3.0  5.2  2.0"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "queries = X[testids,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([[20, 27, 18, 21, 2], [28, 17, 18, 2, 27], [20, 18, 30, 17, 27], [17, 6, 18, 5, 30], [27, 20, 18, 2, 28], [8, 29, 14, 4, 10], [19, 16, 8, 23, 29], [14, 12, 3, 29, 1], [5, 2, 22, 24, 28], [17, 18, 6, 15, 23]  …  [81, 93, 73, 67, 96], [85, 86, 81, 73, 68], [84, 92, 97, 53, 88], [67, 72, 77, 91, 73], [67, 72, 73, 93, 77], [86, 68, 83, 71, 79], [73, 94, 81, 93, 77], [81, 96, 93, 67, 73], [94, 73, 76, 93, 81], [72, 77, 76, 91, 49]], [[0.14142135623730964, 0.14142135623730986, 0.24494897427831822, 0.3, 0.30000000000000016], [0.14142135623730964, 0.17320508075688815, 0.22360679774997916, 0.24494897427831802, 0.26457513110645925], [0.1, 0.1732050807568881, 0.26457513110645875, 0.26457513110645925, 0.2645751311064593], [0.22360679774997858, 0.22360679774997916, 0.2999999999999998, 0.3000000000000002, 0.30000000000000027], [0.19999999999999998, 0.20000000000000037, 0.244948974278318, 0.26457513110645897, 0.3000000000000002], [0.3464101615137753, 0.3605551275463989, 0.38729833462074226, 0.3999999999999999, 0.46904157598234253], [0.282842712474619, 0.36055512754639907, 0.3605551275463991, 0.3605551275463995, 0.3741657386773945], [0.14142135623730928, 0.244948974278318, 0.26457513110645897, 0.2828427124746191, 0.30000000000000016], [0.45825756949558405, 0.5099019513592785, 0.5099019513592788, 0.5196152422706636, 0.5656854249492379], [0.3741657386773937, 0.4242640687119283, 0.4472135954999579, 0.47958315233127186, 0.49999999999999983]  …  [0.2999999999999998, 0.374165738677394, 0.374165738677394, 0.3872983346207416, 0.3999999999999999], [0.3464101615137756, 0.46904157598234325, 0.6557438524301997, 0.7348469228349537, 0.8062257748298546], [0.24494897427831838, 0.2828427124746193, 0.38729833462074154, 0.4242640687119284, 0.4358898943540672], [0.31622776601683805, 0.374165738677394, 0.3872983346207416, 0.4358898943540675, 0.45825756949558355], [0.3000000000000001, 0.43588989435406733, 0.46904157598234253, 0.4690415759823429, 0.469041575982343], [0.5385164807134504, 0.5477225575051661, 0.7000000000000004, 0.806225774829855, 0.8944271909999164], [0.1732050807568879, 0.36055512754639935, 0.37416573867739383, 0.4123105625617657, 0.5196152422706635], [0.22360679774997935, 0.31622776601683794, 0.346410161513776, 0.3872983346207417, 0.4898979485566359], [0.24494897427831822, 0.3741657386773937, 0.374165738677394, 0.42426406871192807, 0.574456264653803], [0.34641016151377513, 0.3605551275463988, 0.3872983346207415, 0.3872983346207415, 0.4123105625617663]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idxs, dists = knn(kdtree, queries', 5, true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = ytrain[hcat(idxs...)]\n",
    "possible_labels = map(i->counter(c[:,i]),1:size(c,2))\n",
    "predictions_NN = map(i->parse(Int,string(argmax(DataFrame(possible_labels[i])[1,:]))),1:size(c,2))\n",
    "findaccuracy(predictions_NN,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 7: Support Vector Machines\n",
    "We will use the `LIBSVM` package here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "99-element Array{Int64,1}:\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " ⋮\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xtrain = X[trainids,:]\n",
    "ytrain = y[trainids]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LIBSVM.SVM{Int64}(SVC, LIBSVM.Kernel.RadialBasis, nothing, 4, 3, [1, 2, 3], Int32[1, 2, 3], Float64[], Int32[], LIBSVM.SupportVectors{Int64,Float64}(36, Int32[4, 16, 16], [1, 1, 1, 1, 2, 2, 2, 2, 2, 2  …  3, 3, 3, 3, 3, 3, 3, 3, 3, 3], [4.4 4.3 … 6.3 5.9; 2.9 3.0 … 2.5 3.0; 1.4 1.1 … 5.0 5.1; 0.2 0.1 … 1.9 1.8], Int32[7, 9, 11, 15, 31, 33, 34, 35, 38, 41  …  84, 85, 87, 88, 89, 92, 94, 95, 97, 99], LIBSVM.SVMNode[LIBSVM.SVMNode(1, 4.4), LIBSVM.SVMNode(1, 4.3), LIBSVM.SVMNode(1, 5.7), LIBSVM.SVMNode(1, 5.1), LIBSVM.SVMNode(1, 7.0), LIBSVM.SVMNode(1, 6.5), LIBSVM.SVMNode(1, 5.7), LIBSVM.SVMNode(1, 5.0), LIBSVM.SVMNode(1, 6.1), LIBSVM.SVMNode(1, 5.6)  …  LIBSVM.SVMNode(1, 6.1), LIBSVM.SVMNode(1, 7.2), LIBSVM.SVMNode(1, 7.9), LIBSVM.SVMNode(1, 6.3), LIBSVM.SVMNode(1, 6.1), LIBSVM.SVMNode(1, 6.0), LIBSVM.SVMNode(1, 6.9), LIBSVM.SVMNode(1, 5.8), LIBSVM.SVMNode(1, 6.3), LIBSVM.SVMNode(1, 5.9)]), 0.0, [0.6758338451531314 0.4655767891335155; 0.0 0.46483582345841046; … ; -0.0 -1.0; -0.0 -1.0], Float64[], Float64[], [0.12481585335305392, 0.21468698168850497, 0.2342190706882077], 3, 0.25, 200.0, 0.001, 1.0, 0.5, 0.1, true, false)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = svmtrain(Xtrain', ytrain)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions_SVM, decision_values = svmpredict(model, X[testids,:]')\n",
    "findaccuracy(predictions_SVM,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Putting all the results together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7×2 Array{Any,2}:\n",
       " \"lasso\"  1.0\n",
       " \"ridge\"  1.0\n",
       " \"EN\"     1.0\n",
       " \"DT\"     0.960784\n",
       " \"RF\"     0.980392\n",
       " \"kNN\"    1.0\n",
       " \"SVM\"    1.0"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "overall_accuracies = zeros(7)\n",
    "methods = [\"lasso\",\"ridge\",\"EN\", \"DT\", \"RF\",\"kNN\", \"SVM\"]\n",
    "ytest = y[testids]\n",
    "overall_accuracies[1] = findaccuracy(predictions_lasso,ytest)\n",
    "overall_accuracies[2] = findaccuracy(predictions_ridge,ytest)\n",
    "overall_accuracies[3] = findaccuracy(predictions_EN,ytest)\n",
    "overall_accuracies[4] = findaccuracy(predictions_DT,ytest)\n",
    "overall_accuracies[5] = findaccuracy(predictions_RF,ytest)\n",
    "overall_accuracies[6] = findaccuracy(predictions_NN,ytest)\n",
    "overall_accuracies[7] = findaccuracy(predictions_SVM,ytest)\n",
    "hcat(methods, overall_accuracies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finally...\n",
    "After finishing this notebook, you should be able to:\n",
    "- [ ] split your data into training and testing data to test the effectiveness of a certain method\n",
    "- [ ] apply a simple accuracy function to test the effectiveness of a certain method\n",
    "- [ ] run multiple classification algorithms:\n",
    "    - [ ] LASSO\n",
    "    - [ ] Ridge\n",
    "    - [ ] ElasticNet\n",
    "    - [ ] Decision Tree\n",
    "    - [ ] Random Forest\n",
    "    - [ ] Nearest Neighbors\n",
    "    - [ ] Support Vector Machines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 🥳 One cool finding\n",
    "\n",
    "We used multiple methods to run classification on the `iris` dataset which is a dataset of flowers and there are three types of iris flowers in it. We split the data into training and testing and ran our methods. Here is the scoreboard:\n",
    "\n",
    "| method | accuracy score |\n",
    "|---|---|\n",
    "| lasso  |1.0|\n",
    "| ridge  |1.0|\n",
    "| EN     |1.0|\n",
    "| DT     |0.960784|\n",
    "| RF     |0.980392|\n",
    "| kNN    |1.0|\n",
    "| SVM    |1.0|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.4.0",
   "language": "julia",
   "name": "julia-1.4"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.4.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}