3.1_accuracy.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "e462b3b0-f88e-4099-90ee-901df0654a3e"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'use strict'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "// boring imports\n",
    "var {loadUnlabelledWine, grid2} = require('./utils')\n",
    "var Plot = require('plotly-notebook-js');\n",
    "var table = require('text-table');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "d68070d3-5c5e-48ef-83bf-ae4ae2267562"
    }
   },
   "source": [
    "### Accuracy\n",
    "\n",
    "Ok, we have classified some data and calculated how confident we are with each of the classifications. \n",
    "\n",
    "But how do we know whether our classification is right? We'll in many applications of unsupervised learning, we don't know for sure as we typically use this type of technique when we don't know the expected outputs before hand. \n",
    "\n",
    "However, in this example dataset, we do have training labels available but we've just not loaded them. So let's reload those and measure the accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "b2085534-e8f0-46dd-8b68-e03c2043f76d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "our dataset has 178 rows and  14 columns\n",
      "Alcohol | Malic Acid | Ash | Alcalinity of ash | Magnesium | Total phenols | Flavanoids | Nonflavanoid phenols | Proanthocyanins | Color intensity | Hue | OD280/OD315 of diluted wines | Proline | Class\n",
      "---------------------------\n",
      "14.23 | 1.71 | 2.43 | 15.6 | 127 | 2.8  | 3.06 | 0.28 | 2.29 | 5.64 | 1.04 | 3.92 | 1065 | 1\n",
      " 13.2 | 1.78 | 2.14 | 11.2 | 100 | 2.65 | 2.76 | 0.26 | 1.28 | 4.38 | 1.05 | 3.4  | 1050 | 1\n",
      "13.16 | 2.36 | 2.67 | 18.6 | 101 | 2.8  | 3.24 | 0.3  | 2.81 | 5.68 | 1.03 | 3.17 | 1185 | 1\n",
      "14.37 | 1.95 | 2.5  | 16.8 | 113 | 3.85 | 3.49 | 0.24 | 2.18 | 7.8  | 0.86 | 3.45 | 1480 | 1\n",
      "13.24 | 2.59 | 2.87 | 21   | 118 | 2.8  | 2.69 | 0.39 | 1.82 | 4.32 | 1.04 | 2.93 | 735  | 1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'use strict'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var {loadLabelledWine} = require('./utils')\n",
    "var labelledDataset = loadLabelledWine({ verbose: true }).dataset;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "d8218bf7-cb02-400d-976e-c7976ae562c7"
    }
   },
   "source": [
    "Notice the additional (last) column containing the known class index. We need to note that they are coulding forom `1`, whilst our class labels are from `0`, we'll need to compensate for that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "de004a34-a3f6-4ead-90b5-da8d1ab8413a"
    }
   },
   "source": [
    "### Confusion Matrix\n",
    "\n",
    "With mljs we can use the confusion matrix package to compute a suite of different measures to determine the performance on our clustering and classifer.\n",
    "\n",
    "Dig into [the docs](https://mljs.github.io/confusion-matrix/) and compute:\n",
    "\n",
    " - overall `accuracy`\n",
    " - the `F1 score` for each class label. \n",
    " - the average `F! score`\n",
    " \n",
    "These metrics are in the range [0,1] or [appaling, 100% match]. print them out on the console.\n",
    "\n",
    "NB: with this library we can compute a full suite of diagnisoc measures, see the table [here](https://en.wikipedia.org/wiki/F1_score#Diagnostic_testing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'use strict'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var ConfusionMatrix = require('ml-confusion-matrix');\n",
    "var actuals = labelledDataset.map(d => d[13]-1);\n",
    "var predicted = clusters.map(d => d);\n",
    "\n",
    "var C = ConfusionMatrix.fromLabels(actuals, predicted)\n",
    "\n",
    "// compute and print out some metrics here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So accuracy might not have been as good as we hoped for.\n",
    "\n",
    "Why? let's dig deeper and look at the confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "d54579fb-5451-403f-a9eb-dd3a3d5e308a"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class='plotly-plot'><div id='notebook-plot-1510329310983'></div><script>function plot(){Plotly.plot('notebook-plot-1510329310983',[{\"x\":[0,1,2],\"y\":[0,1,2],\"z\":[[0,58,1],[55,6,10],[0,1,47]],\"type\":\"heatmap\",\"showscale\":false,\"colorscale\":[[0,\"#3D9970\"],[100,\"#001f3f\"]]}],{\"xaxis\":{\"title\":\"predicted\",\"side\":\"top\"},\"yaxis\":{\"title\":\"actuals\",\"nticks\":6,\"autosize\":false,\"autorange\":\"reversed\"},\"annotations\":[{\"x\":0,\"y\":0,\"text\":0,\"font\":{\"family\":\"Arial\",\"size\":12,\"color\":\"white\"},\"showarrow\":false},{\"x\":1,\"y\":0,\"text\":58,\"font\":{\"family\":\"Arial\",\"size\":12,\"color\":\"white\"},\"showarrow\":false},{\"x\":2,\"y\":0,\"text\":1,\"font\":{\"family\":\"Arial\",\"size\":12,\"color\":\"white\"},\"showarrow\":false},{\"x\":0,\"y\":1,\"text\":55,\"font\":{\"family\":\"Arial\",\"size\":12,\"color\":\"white\"},\"showarrow\":false},{\"x\":1,\"y\":1,\"text\":6,\"font\":{\"family\":\"Arial\",\"size\":12,\"color\":\"white\"},\"showarrow\":false},{\"x\":2,\"y\":1,\"text\":10,\"font\":{\"family\":\"Arial\",\"size\":12,\"color\":\"white\"},\"showarrow\":false},{\"x\":0,\"y\":2,\"text\":0,\"font\":{\"family\":\"Arial\",\"size\":12,\"color\":\"white\"},\"showarrow\":false},{\"x\":1,\"y\":2,\"text\":1,\"font\":{\"family\":\"Arial\",\"size\":12,\"color\":\"white\"},\"showarrow\":false},{\"x\":2,\"y\":2,\"text\":47,\"font\":{\"family\":\"Arial\",\"size\":12,\"color\":\"white\"},\"showarrow\":false}],\"width\":500,\"height\":500});}if(window.Plotly){plot();}else if(!window.require){var head = document.head || document.getElementsByTagName('head')[0];var s = document.createElement('script');s.src = 'https://cdn.plot.ly/plotly-latest.min.js';s.type = 'text/javascript';s.async = false;s.onreadystatechange = s.onload = plot;head.appendChild(s);}else{require(['/notebooks/node_modules/plotly-notebook-js/build/plotly.min.js'], function(Plotly){window.Plotly = Plotly;plot();});}</script></div>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var M = C.getMatrix();\n",
    "\n",
    "var trace = { \n",
    "    x: [0,1,2],\n",
    "    y: [0,1,2],\n",
    "    z: M,\n",
    "    type: 'heatmap',\n",
    "    showscale: false,\n",
    "    colorscale:[[0, '#3D9970'], [100, '#001f3f']]\n",
    "};\n",
    "\n",
    "var annotations = [];\n",
    "\n",
    "M.map((a,y) => {\n",
    "    a.map((b,x) => {\n",
    "        annotations.push(\n",
    "            {\n",
    "                x: x,\n",
    "                y: y,\n",
    "                text: M[y][x],\n",
    "                font: {\n",
    "                    family: 'Arial',\n",
    "                    size: 12,\n",
    "                    color: 'white'\n",
    "                  },\n",
    "                showarrow: false\n",
    "            }\n",
    "        )\n",
    "    })\n",
    "})\n",
    "\n",
    "var layout = { \n",
    "    xaxis: { title: \"predicted\", side: 'top' },\n",
    "    yaxis: { title: \"actuals\", nticks: 6, autosize: false, autorange: 'reversed' },\n",
    "    annotations,\n",
    "    width: 500, height: 500};\n",
    "\n",
    "$$html$$ = Plot.createPlot([trace], layout).render();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy 0.29775280898876405\n",
      "F1 Class 1 0\n",
      "F1 Class 2 0.08823529411764706\n",
      "F1 Class 3 0.8867924528301887\n"
     ]
    }
   ],
   "source": [
    "console.log(\"Accuracy\", C.getAccuracy())\n",
    "console.log(\"F1 Class 1\", C.getF1Score(0))\n",
    "console.log(\"F1 Class 2\", C.getF1Score(1))\n",
    "console.log(\"F1 Class 3\", C.getF1Score(2))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also plot the original labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class='plotly-plot'><div id='notebook-plot-1510329330339'></div><script>function plot(){Plotly.plot('notebook-plot-1510329330339',[{\"x\":[14.23,13.2,13.16,14.37,13.24,14.2,14.39,14.06,14.83,13.86,14.1,14.12,13.75,14.75,14.38,13.63,14.3,13.83,14.19,13.64,14.06,12.93,13.71,12.85,13.5,13.05,13.39,13.3,13.87,14.02,13.73,13.58,13.68,13.76,13.51,13.48,13.28,13.05,13.07,14.22,13.56,13.41,13.88,13.24,13.05,14.21,14.38,13.9,14.1,13.94,13.05,13.83,13.82,13.77,13.74,13.56,14.22,13.29,13.72,12.37,12.33,12.64,13.67,12.37,12.17,12.37,13.11,12.37,13.34,12.21,12.29,13.86,13.49,12.99,11.96,11.66,13.03,11.84,12.33,12.7,12,12.72,12.08,13.05,11.84,12.67,12.16,11.65,11.64,12.08,12.08,12,12.69,12.29,11.62,12.47,11.81,12.29,12.37,12.29,12.08,12.6,12.34,11.82,12.51,12.42,12.25,12.72,12.22,11.61,11.46,12.52,11.76,11.41,12.08,11.03,11.82,12.42,12.77,12,11.45,11.56,12.42,13.05,11.87,12.07,12.43,11.79,12.37,12.04,12.86,12.88,12.81,12.7,12.51,12.6,12.25,12.53,13.49,12.84,12.93,13.36,13.52,13.62,12.25,13.16,13.88,12.87,13.32,13.08,13.5,12.79,13.11,13.23,12.58,13.17,13.84,12.45,14.34,13.48,12.36,13.69,12.85,12.96,13.78,13.73,13.45,12.82,13.58,13.4,12.2,12.77,14.16,13.71,13.4,13.27,13.17,14.13],\"y\":[3.92,3.4,3.17,3.45,2.93,2.85,3.58,3.58,2.85,3.55,3.17,2.82,2.9,2.73,3,2.88,2.65,2.57,2.82,3.36,3.71,3.52,4,3.63,3.82,3.2,3.22,2.77,3.4,3.59,2.71,2.88,2.87,3,2.87,3.47,2.78,2.51,2.69,3.53,3.38,3,3.56,3,3.35,3.33,3.44,3.33,2.75,3.1,2.91,3.37,3.26,2.93,3.2,3.03,3.31,2.84,2.87,1.82,1.67,1.59,2.46,2.87,2.23,2.3,3.18,3.48,1.93,3.07,1.82,3.16,2.78,3.5,3.13,2.14,2.48,2.52,2.31,3.13,3.12,3.14,2.72,2.01,3.08,3.16,2.26,3.21,2.75,3.21,2.27,2.65,2.06,3.3,2.96,2.63,2.26,2.74,2.77,2.83,2.96,2.77,3.38,2.44,3.57,3.3,3.17,2.42,3.02,3.26,2.81,2.78,2.5,2.31,3.19,2.87,3.33,2.96,2.12,3.05,3.39,3.69,3.12,3.1,3.64,3.28,2.84,2.44,2.78,2.57,1.29,1.42,1.36,1.29,1.51,1.58,1.27,1.69,1.82,2.15,2.31,2.47,2.06,2.05,2,1.68,1.33,1.86,1.62,1.33,1.3,1.47,1.33,1.51,1.55,1.48,1.64,1.73,1.96,1.78,1.58,1.82,2.11,1.75,1.68,1.75,1.56,1.75,1.8,1.92,1.83,1.63,1.71,1.74,1.56,1.56,1.62,1.6],\"mode\":\"markers\",\"marker\":{\"color\":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2],\"size\":8,\"colorbar\":{\"xpad\":100}},\"type\":\"scatter\"},{\"x\":[12.079818181818181],\"y\":[2.9029090909090898],\"mode\":\"markers\",\"marker\":{\"size\":20,\"line\":{\"width\":2,\"color\":\"#000\"},\"opacity\":1},\"opacity\":0.3,\"type\":\"scatter\"},{\"x\":[13.714153846153845],\"y\":[3.1453846153846152],\"mode\":\"markers\",\"marker\":{\"size\":20,\"line\":{\"width\":2,\"color\":\"#000\"},\"opacity\":1},\"opacity\":0.3,\"type\":\"scatter\"},{\"x\":[13.074137931034484],\"y\":[1.7374137931034477],\"mode\":\"markers\",\"marker\":{\"size\":20,\"line\":{\"width\":2,\"color\":\"#000\"},\"opacity\":1},\"opacity\":0.3,\"type\":\"scatter\"}],{\"width\":800,\"height\":700,\"xaxis\":{\"title\":\"Alcohol\"},\"yaxis\":{\"title\":\"OD280/OD315 of diluted wines\"}});}if(window.Plotly){plot();}else if(!window.require){var head = document.head || document.getElementsByTagName('head')[0];var s = document.createElement('script');s.src = 'https://cdn.plot.ly/plotly-latest.min.js';s.type = 'text/javascript';s.async = false;s.onreadystatechange = s.onload = plot;head.appendChild(s);}else{require(['/notebooks/node_modules/plotly-notebook-js/build/plotly.min.js'], function(Plotly){window.Plotly = Plotly;plot();});}</script></div>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var trace = { \n",
    "    x: input.map(d => d[0]),\n",
    "    y: input.map(d => d[1]),\n",
    "    mode: 'markers',\n",
    "    marker: { \n",
    "        color: actuals, // <- here are the true labels\n",
    "        size: 8,\n",
    "        colorbar: {\n",
    "            xpad: 100\n",
    "        }\n",
    "    },\n",
    "    type: 'scatter'\n",
    "};\n",
    "\n",
    "var layout = { width: 800, height: 700, xaxis: { title: features[0] }, yaxis: { title: features[11] }};\n",
    "\n",
    "$$html$$ = Plot.createPlot([trace, ...centroidsTraces], layout).render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on the attributes you chose, we can see some gross misclassifications here. In high accuracy cases the main diagonal contains the most weight.\n",
    "\n",
    "So why do you thing this has done so poorly? (if it has)\n",
    "\n",
    "Try running the whole notebook again (Cell > Run All), a few times and watch the score and the confusion matrix, does it change? \n",
    "\n",
    "Any idea what is happening?\n",
    "\n",
    "Any idea how to fix it?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hey, What about preprocessing and normalisation?\n",
    "\n",
    "We've gone ahead and worked on the raw values here. Which is working, for now, but:\n",
    "\n",
    " - is it the best thing to do?\n",
    " - Do you think normalisation would affec our result might in this case?\n",
    " - why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "72409247-a808-4f09-9310-78163fdb78f0"
    }
   },
   "source": [
    "#### Further Reading\n",
    "\n",
    "More sophisticated techniques can produce different\n",
    "\n",
    " - bayes learning and 2nd order bayes classifers\n",
    " - gaussian mixture modelling\n",
    " - measuring performance in undersuipervised learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "jp-Babel (Node.js)",
   "language": "babel",
   "name": "babel"
  },
  "language_info": {
   "file_extension": ".js",
   "mimetype": "application/javascript",
   "name": "javascript",
   "version": "8.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}