CI_Workshop_Teplitskiy_Classification.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:6cb82d3bd23627f843d520cbd67b75cc4a902633b7bc755611565ee14d6b20b0"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Classification: Building a Language Detector\n",
      "\n",
      "\n",
      "- inspired by http://bugra.github.io/work/notes/2014-12-26/language-detector-via-scikit-learn/"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Overfitting\n",
      "\n",
      "### = Big difference between social science stats and machine learning\n",
      "\n",
      "<img src=http://pingax.com/wp-content/uploads/2014/05/underfitting-overfitting.png>\n",
      "\n",
      "###Solution: Split data into training part and testing part\n",
      "\n",
      "- \"testing\" set also called \"validation set,\" \"held-out set\"\n",
      "\n",
      "###Result: 2 sets of accuracies, 2 sets of errors\n",
      "- One for training set <--- no one cares about\n",
      "- One for test set <--- everyone cares about, also called \"generalization error\"\n",
      "\n",
      "<img src=https://raw.githubusercontent.com/tijptjik/DS_assets/master/overfitting.png>"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%matplotlib inline"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import matplotlib.pyplot as plt\n",
      "import pandas as pd\n",
      "import numpy as np"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Data Description"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "European Parliament Proceedings corpus\n",
      "- https://language-detection.googlecode.com/git-history/packages/packages/europarl-test.zip\n",
      "- 21 languages, 1000 sentences each \n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Import data and put it in pandas dataframe"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import codecs\n",
      "lines = codecs.open('europarl.txt', 'r', 'utf-8').readlines()\n",
      "lines = [l.split('\\t') for l in lines]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df = pd.DataFrame(lines, columns=['language', 'text'])\n",
      "df.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>language</th>\n",
        "      <th>text</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> bg</td>\n",
        "      <td> \"\u0415\u0432\u0440\u043e\u043f\u0430 2020\" \u043d\u0435 \u0442\u0440\u044f\u0431\u0432\u0430 \u0434\u0430 \u0441\u0442\u0430\u0440\u0442\u0438\u0440\u0430 \u043d\u043e\u0432 \u043a\u043e\u043d\u043a\u0443\u0440...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> bg</td>\n",
        "      <td> (CS) \u041d\u0430\u0439-\u0433\u043e\u043b\u044f\u043c\u0430\u0442\u0430 \u043d\u0435\u0441\u043f\u0440\u0430\u0432\u0435\u0434\u043b\u0438\u0432\u043e\u0441\u0442 \u043d\u0430 \u0441\u0435\u0433\u0430\u0448\u043d\u0430\u0442\u0430...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> bg</td>\n",
        "      <td> (DE) \u0413-\u0436\u043e \u043f\u0440\u0435\u0434\u0441\u0435\u0434\u0430\u0442\u0435\u043b, \u0433-\u043d \u0447\u043b\u0435\u043d \u043d\u0430 \u041a\u043e\u043c\u0438\u0441\u0438\u044f\u0442\u0430, ...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> bg</td>\n",
        "      <td> (DE) \u0413-\u043d \u043f\u0440\u0435\u0434\u0441\u0435\u0434\u0430\u0442\u0435\u043b, \u0431\u0438\u0445 \u0438\u0441\u043a\u0430\u043b \u0434\u0430 \u0437\u0430\u043f\u043e\u0447\u043d\u0430 \u0441 \u043a...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> bg</td>\n",
        "      <td> (DE) \u0413-\u043d \u043f\u0440\u0435\u0434\u0441\u0435\u0434\u0430\u0442\u0435\u043b, \u0432\u044a\u043f\u0440\u043e\u0441\u044a\u0442 \u0437\u0430 \u043f\u0440\u0430\u0432\u0430\u0442\u0430 \u043d\u0430 \u0447...</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "  language                                               text\n",
        "0       bg  \"\u0415\u0432\u0440\u043e\u043f\u0430 2020\" \u043d\u0435 \u0442\u0440\u044f\u0431\u0432\u0430 \u0434\u0430 \u0441\u0442\u0430\u0440\u0442\u0438\u0440\u0430 \u043d\u043e\u0432 \u043a\u043e\u043d\u043a\u0443\u0440...\n",
        "1       bg  (CS) \u041d\u0430\u0439-\u0433\u043e\u043b\u044f\u043c\u0430\u0442\u0430 \u043d\u0435\u0441\u043f\u0440\u0430\u0432\u0435\u0434\u043b\u0438\u0432\u043e\u0441\u0442 \u043d\u0430 \u0441\u0435\u0433\u0430\u0448\u043d\u0430\u0442\u0430...\n",
        "2       bg  (DE) \u0413-\u0436\u043e \u043f\u0440\u0435\u0434\u0441\u0435\u0434\u0430\u0442\u0435\u043b, \u0433-\u043d \u0447\u043b\u0435\u043d \u043d\u0430 \u041a\u043e\u043c\u0438\u0441\u0438\u044f\u0442\u0430, ...\n",
        "3       bg  (DE) \u0413-\u043d \u043f\u0440\u0435\u0434\u0441\u0435\u0434\u0430\u0442\u0435\u043b, \u0431\u0438\u0445 \u0438\u0441\u043a\u0430\u043b \u0434\u0430 \u0437\u0430\u043f\u043e\u0447\u043d\u0430 \u0441 \u043a...\n",
        "4       bg  (DE) \u0413-\u043d \u043f\u0440\u0435\u0434\u0441\u0435\u0434\u0430\u0442\u0435\u043b, \u0432\u044a\u043f\u0440\u043e\u0441\u044a\u0442 \u0437\u0430 \u043f\u0440\u0430\u0432\u0430\u0442\u0430 \u043d\u0430 \u0447..."
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# how many of each language\n",
      "df.language.value_counts()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 10,
       "text": [
        "hu    1000\n",
        "nl    1000\n",
        "fi    1000\n",
        "ro    1000\n",
        "de    1000\n",
        "da    1000\n",
        "pt    1000\n",
        "pl    1000\n",
        "it    1000\n",
        "bg    1000\n",
        "sv    1000\n",
        "cs    1000\n",
        "sk    1000\n",
        "sl    1000\n",
        "lv    1000\n",
        "lt    1000\n",
        "el    1000\n",
        "en    1000\n",
        "et    1000\n",
        "es    1000\n",
        "fr    1000\n",
        "dtype: int64"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# let's consider just two: english (en) and french (fr)\n",
      "df[df.language=='en'].head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>language</th>\n",
        "      <th>text</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>5000</th>\n",
        "      <td> en</td>\n",
        "      <td>                   (BG) Thank you, Mr President.\\n</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5001</th>\n",
        "      <td> en</td>\n",
        "      <td> (EL) Madam President, I agree and recognise Tu...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5002</th>\n",
        "      <td> en</td>\n",
        "      <td> (FI) Madam President, firstly, I would like to...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5003</th>\n",
        "      <td> en</td>\n",
        "      <td> (FI) Mr President, the Treaty of Lisbon will r...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5004</th>\n",
        "      <td> en</td>\n",
        "      <td> (FR) Madam President, one of the priorities of...</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 11,
       "text": [
        "     language                                               text\n",
        "5000       en                    (BG) Thank you, Mr President.\\n\n",
        "5001       en  (EL) Madam President, I agree and recognise Tu...\n",
        "5002       en  (FI) Madam President, firstly, I would like to...\n",
        "5003       en  (FI) Mr President, the Treaty of Lisbon will r...\n",
        "5004       en  (FR) Madam President, one of the priorities of..."
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_sub = df[df.language.isin(('lt', 'lv'))]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 12
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Build classifier"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\n",
      "from sklearn.linear_model import LogisticRegression\n",
      "from sklearn.cross_validation import train_test_split"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###1. Engineer features we will use to predict"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# vectorizer = TfidfVectorizer(ngram_range=(1,3),\n",
      "# #                              analyzer='char',\n",
      "#                              max_features=50)\n",
      "# #                             use_idf=False)\n",
      "\n",
      "vectorizer = CountVectorizer(max_features=50,\n",
      "                             analyzer='char',\n",
      "                                )\n",
      "\n",
      "\n",
      "X_features = vectorizer.fit_transform(df_sub.text)  # fit_transform() is like calling fit() and then predict()\n",
      "print X_features.shape, type(X_features)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "(2000, 50) <class 'scipy.sparse.csr.csr_matrix'>\n"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###2. Split into train and test sets"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "y = df_sub.language.values\n",
      "X_train, X_test, y_train, y_test = train_test_split(X_features, y, test_size=0.2)  \n",
      "#setting random_state=0 to make #sure we all get the same answer"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#composition of train and test sets\n",
      "print 'Composition of train set:', np.unique(y_train, return_counts=True)\n",
      "print 'Composition of test set:', np.unique(y_test, return_counts=True)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Composition of train set: (array([u'lt', u'lv'], dtype=object), array([812, 788]))\n",
        "Composition of test set: (array([u'lt', u'lv'], dtype=object), array([188, 212]))\n"
       ]
      }
     ],
     "prompt_number": 18
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###3. Train model"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "clf = LogisticRegression()\n",
      "clf.fit(X_train, y_train)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 19,
       "text": [
        "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
        "          intercept_scaling=1, penalty='l2', random_state=None, tol=0.0001)"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###4. Evaluate model\n",
      "\n",
      "*Test it on the held-out test set*\n",
      "\n",
      "* **accuracy**: percent correct\n",
      "\n",
      "\n",
      "* When especially interested in a particular class, say \"positive,\"\n",
      "    - **precision**: of the things you called \"positive,\" what percent were correct?\n",
      "    - **recall**: of all positive cases, what percent did you find?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "y_predicted = clf.predict(X_test)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 20
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn import metrics\n",
      "print 'Accuracy:', metrics.accuracy_score(y_test, y_predicted)\n",
      "print\n",
      "print metrics.classification_report(y_test, y_predicted)\n",
      "print\n",
      "print 'confusion matrix'\n",
      "print\n",
      "print pd.DataFrame(metrics.confusion_matrix(y_test, y_predicted))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Accuracy: 0.955\n",
        "\n",
        "             precision    recall  f1-score   support\n",
        "\n",
        "         lt       0.91      1.00      0.95       188\n",
        "         lv       1.00      0.92      0.96       212\n",
        "\n",
        "avg / total       0.96      0.95      0.96       400\n",
        "\n",
        "\n",
        "confusion matrix\n",
        "\n",
        "     0    1\n",
        "0  188    0\n",
        "1   18  194\n"
       ]
      }
     ],
     "prompt_number": 21
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###Out of curiousity, how well did we do on the training set?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print 'Accuracy:', metrics.accuracy_score(y_train, clf.predict(X_train))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Accuracy: 0.9575\n"
       ]
      }
     ],
     "prompt_number": 22
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##ROC curve\n",
      "\n",
      "x-axis: What percent of negative things did you falsely call positive?\n",
      "\n",
      "y-axis: Of the positive examples, what percent did you find?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.metrics import roc_curve, roc_auc_score\n",
      "\n",
      "y_label_test = np.asarray(y_test == 'lv', dtype=int)\n",
      "proba = clf.predict_proba(X_test)\n",
      "proba_label = proba[:,1]\n",
      "fpr, tpr, roc_thresholds = roc_curve(y_label_test, proba_label)\n",
      "\n",
      "plt.plot(fpr, tpr, '-', linewidth=5)\n",
      "plt.plot([0, 1], [0, 1], 'k--')\n",
      "plt.xlabel('False Positive Rate (\"Cost\")')\n",
      "plt.ylabel('True Positive Rate (\"Benefit\")')\n",
      "plt.title('Receiver operating characteristic example')\n",
      "plt.legend(loc=\"lower right\")   \n",
      "    "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "C:\\WinPython27\\python-2.7.5\\lib\\site-packages\\matplotlib\\axes.py:4752: UserWarning: No labeled objects found. Use label='...' kwarg on individual plots.\n",
        "  warnings.warn(\"No labeled objects found. \"\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEZCAYAAACTsIJzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuclGX9//HXh+WkAp7QVFBMJUvzmKAm6pokqFmWp68o\niJn2rewgeEj7qnwz7Nc3PKQoiiZ0MDHTb1/AA5m5ZXlKdlcwgdQixWOiIYm6Ap/fH9c1cO84O3vv\nsjP3zM77+XjsY+ee+/SZe2buz1z3dd3XZe6OiIhITo+sAxARkcqixCAiIq0oMYiISCtKDCIi0ooS\ng4iItKLEICIirSgxVDEze8rMDsk6jqyZ2TQz+68y73OmmV1Wzn2WipmdYmbzOrlut/0MmtlaM9sp\n6ziyYLqPoWuY2VJga2AN8DZwP/A1d38ry7i6GzMbD5zh7gdnHMcM4AV3vyTjOCYBO7v72DLsaybh\nNV9c6n1VAjNbC+zi7n/LOpZyU4mh6zjwGXfvD+wF7AGU9VdsVzCznrW47yyZWV0t7lsqmLvrrwv+\ngL8Dn0pM/w9wd2L6AOBh4E2gGTg0MW8LYAbwIvAG8L+JeZ+Jy78J/AnYIzFvKfApYDtgFbB5Yt4+\nwD+Bujj9ReDpuP37gB0Sy64Fvgo8AzzXxuv7LPCXGMeDwEfz4vh2nP8GcAvQpwOv4XxgAfAOUBe3\n9SzwVtzmsXHZj8VlVgMrgTfi8zOBy+LjemAZMAF4FXgJGJ/Y35bAHGAF8DjwPeChIu/riMT79jww\nLj4/A5gKzI1xPgrslFjvR3H5FcATwIjEvEnAr4CfxflfBIYBj8T9vARcC/RKrLM7oRS6HHgFuBAY\nBbwHtMTj0RSX3RT4cdzOMuAyoEecNz6+B1cCr8d543PHADDgqnjsVsT3ZXfgrLif9+K+/i/x/h0e\nH9cBFyXeuyeAwW0c14LfB+CThM/t4Di9F+Ez9ZE4XfCzUeC1vRmX+yRwenwvXs29f4nPzQ3Ab+L2\nGvjg92Kn+LgPMAX4Rzz+04C+WZ93SnY+yzqA7vJHSAy5L8jg+IW6JE4Pil/C0XF6ZJzeMk7fDdwW\nv9A9gYPj8/vED/Ow+IUdF/fTK7HPT8XHDwBfSsTzQ+D6+PhzhJP+roRS4neAPyWWXQvMAzYjcUJP\nzP8I8G/g8PjlPy9ur2ecvzS+3kHA5sAfWX+ibu81LAUa47p94nPHA9vExyfGfX8oTp9G3omccJL+\nbnxcD7xPOPnWAUcSLu1tGufPAn4B9CUkmueBP7Txng6JJ4yT4ra2APaK82bG93C/OO/nwG2JdU+J\nx6IHIUm9DPSO8yYRTrKfjdN9gX2B4XH5IYQk/s04v39c/xygN9APGB7nXQr8NC/u/yWcuDYCtgIe\nA86K88bH4/O1uK++tE4Mowgn9AFxetfEe7HuOOd97nOfwfPi52BonN4D2KLAcW3v+/A9wud5I2Ah\n8NXEusU+G7nXdhrhs3YZITFeC/QCPh3fz40T7+FbhOTfG7iaxGeL1onhKuDXhO9IP2A2cHnW552S\nnc+yDqC7/BFOcCvjB21t/HLmfqVdUODLex/hJLktoV5i0wLbnFbgi7iY9Ykj+aU8A3ggPjbCCW9E\nnL4X+GJiGz0IJ8vt4/RaoL7Ia7sYmJWYtviFOyQRx1mJ+UcCz3bgNYxv59g2sf4kOp7CiSFZYliV\nO/bxuVcJJ906wgl5aGLeZfnbS8y7ELizjXkzgOl5r3lRkdfwBrGkREgMDe285m8Bd8XHJwPz21hu\nEvCzxPSHgHdJ/JqN6/8ucfz+kbeNdceUUAJdAuyfPIb5xznxXPIzuAQ4JsV3pc3vQ3zck5CcFgL3\ndPCz8dfEvD3iZ3urxHOvA3vGxzOBXyTmbUIojQ5KfC92Inze/03rEuGBwN/ae63V+qc6hq7jwOfc\nfQDh5PQpwq9JCL8ATzCzN3N/wEHANsD2hEsiKwpscwgwMW+9wYRLR/nuAg40s22AQ4C17v7HxHZ+\nlNjG8vj8oMT6LxR5bdsSEk14oeGb8UKR9Z9PxJjmNbTat5mNM7OmxPIfJ1wCSmu5u69NTK8i/Mrb\ninDSSe5vWZHtDAaKVTy+mnj8TtwHAGZ2rpk9bWb/iq9hU2BgW/s1s4+Y2Vwze9nMVgCTWf+at28n\njqQhhF/HLyeO3w2E157T5nvt7r8jXCK7DnjVzG40s/4p9z0YeC5ljG19H3D31cBPCJewrkiumOKz\nkf+e4O7/zHsu9z45iffB3d8mJPD879dWwMbA/MR+76X1+9mtKDGUgLv/gVB8/UF86nnCr7rNE3/9\n3f1/CF/SLcxs0wKbeh6YnLdeP3e/vcA+3yRcKz0JGEO4NJXczll529nE3R9NbqLIS3qJ8GUGwMyM\ncLJ6MbHMDnmPc/PSvIZ1+zazIcB0wqWOLdx9c+Apwq+2YnEWiz/nn4RfhNsnntu+jWUhvDc7p9hu\nK2Z2MOGyygnuvll8DStY/xrgg/FOI1w+2sXdNyVc7st9P58n/HItZG3e9AuEeoAtE8d7U3ffo8i+\nW3H3a919P2A3wmXE89KsF/e9SzvLQPHvA2Y2CLiEUFd1pZn1js+399noqNznmLj9foTLhS/lLfc6\nIaHsloh3s/gjsFtSYiidq4HhZrY/4frzMWZ2hJnVmVlfM6s3s0Hu/jLh18f1ZraZmfVKtAu/CfhP\nMxtuwSZmdnT8ABfyC8L11ePi45wbgIvMbDcAM9vUzE7owGv5JXC0mX3KzHoBEwmXKx6O8w34qpkN\nMrMtCCe13Im/o69hE8IJ6HWgh5mdTvhVmPMqMDjGkWOkODm4+xpCyWqSmW1kZh8FxtL2Ce9WYKSZ\nnWBmPc1sSzPbK7HPtvQnJKDXzay3mV0CtHcS6Ue4FLkqxvWVxLy7gW3N7Jtm1sfM+pvZ8DjvVWDH\nmKyJn6ffEE6o/c2sh5ntnPZeAzPbz8z2j8d3FeF9XpPYV7F2/TcDl5nZLvG93jN+HvK1+X2Ir2Mm\ncLO7f4lQt5K7X6S9z0ZnHGVmB8XkcxnwiLsnf/AQS583AVeb2VYQkpeZHbGB+65YSgwl4u6vE4rD\nF7j7MkIF8EXAa4RfTBNZf/zHEirNFhO+fN+I25gPnEko2r9BqPAdR9snstmEX2wvu/vCRCy/JpRe\nZsXLFAsJlYzrFmnntfwVOJVQCvoncDThWvLqxPq/IJyQnotxfq8zr8HdnyZcPniE0Prj44TK7JwH\nCK1RXjGz1xL7T26v2Os5m3BZ5xXC+3Mbod6hUCwvAEcR3qvlhOvZe7axz+R+74t/fyXUPb1D4lJc\nG+ueSyjpvUX4VTwrt4y7ryRUnB5DOFH+lXC5EuCO+H+5mT0RH48jVKbmWqHdQbxMUyTu3HMD4v7f\niLG/TmjIAKGl027xcspdfNCVhB8RvyGUkG4iVG633lnx78M3CJdocvdKnA6cbmYHpfhsFHtPCsl9\nbi8lvL/7ED7nhda9gNDK6dH4HbqfUJrqlkp6g5uZ3UI4ibyWV5RNLnMNoeJuFaESsqlkAUlJmNnf\nCTed/S7rWDrKzH4AbO3up2cdi5SXhZsUl3mN3LDXEaUuMcwARrc108yOIlxTHUpoJz2txPFIjTOz\nXeMlDouXY75IaEEmtaezdRPdXkkTg7s/RLjRpC2fJRTncffHgM3M7EOljElqXn/gTkLzw1nAFHef\nnW1IkpFCl56E0HQvS4P4YNPBwbRuciYVzt0/nHUMabn7E8DQrOOQ7OnyYdsqofI5vzinDC4ikqGs\nSwwv0rod+WBat40HYJdddvHnnktz34yIiCQ85+5p7i1pJesSw2xC0zrM7ADgX+7+gctIzz33XElu\n+37/fefOO539989daqyGv0srIIZK+dOx0LHQsVj/10hoTX004fe1Qydu0IQSlxjM7DbgUGCgmb1A\neMd6Abj7je5+j5kdZWbPEvruKcs1vzfegJtvhuuug+efb3/5zTeHb3wDxo2Dvh9olV1eV1wBEydm\nG0Ol0LFYT8divVo8FtOnX8XUqd/n4ouncPzxY4n3OzJoUDsrtqGkicHdT06xzNmljCFp4UK49lr4\n+c/hnXfaX3677cIH7KyzoF9b9+mWWf/+IS7RsUjSsVivFo/FyJHDOOusZrbroheedR1Dya1ZA3Pn\nwjXXwO9S3n61yy5wwQUwdiz06VPa+Dqqvr4+6xAqho7FejoW69XisRgxYkSXbq8qhvY0M+9onG++\nCbfcAlOnwtKl6dbZe2+48EI47jio07hWIlLlzAx37/CNfN2uxPD00+Fy0U9/CqtWtb+8GXzmM6EO\n4fDDw7SISKVpaWlh8uTJbLbZZpxzzjkl3Ve3SAxr1sA994TLRb/9bbp1BgyAM86Ar30Ndu5Uvb2I\nSHk0NTUxfvx4tt9+e6ZPn17y/VV1YlixAmbMCCWEv6UcxmTXXde3MKqUCmURkUJypYRp06YxZcoU\nxo5d3+KolKomMTQ2hmamEEoIc+bAzJnw9tvp1j/66JAQRo6EHlnfvSEiksK3vvUtnn/+eZqbu67F\nURpVU/kcb9bokP794fTT4eyzYah6xxGRKrNy5Ur69evX6VJCZyufu2ViGDoUvv51OO20UJcgIlKL\n1CoJGD06XC4aNUqXi0SkerS0tLBy5Uq23HLLrEMBsu8raYP16xcuFS1eDPfeC0ceqaQgItWjqamJ\nYcOGcf3112cdyjpVWWLo2zeUDg47LFwu2nTTrCMSEemYQi2OKkVVJoY//hE+8YmsoxAR6ZzkfQnl\nbnGURtVVPm+3HTz3XPa9nIqIdNaVV17JwIEDS35fQk20SurVC+6/Hw49NOuIREQqX2cTQ1VV0557\nrpKCiEipVVViqJCWXCIiqTQ1NfHggw9mHUaHVVViEBGpBi0tLVx66aWMGjWK5cuXZx1Oh1VlqyQR\nkUpV6S2O0lCJQUSki0ydOpVRo0YxceJE5syZU5VJAVRiEBHpMgceeGDVlhKSlBhERLrIJ7rJnbe6\nlCQiIq2oxCAi0gG5Po569OjBpZdemnU4JaESg4hISrmeUOfPn8+ZZ56ZdTglo8QgItKO5H0J1d7i\nKA1dShIRacd3vvMdFi1a1C1aHKWhxCAi0o7vfve79O3bt6Q9oVYSJQYRkXZstNFGWYdQVqpjEBGJ\nWlpaeOWVV7IOI3NKDCIirG9xdM0112QdSuaUGESkpuW3OJo8eXLWIWVOdQwiUrO6Q0+opaDEICI1\na/78+UycOLHkYy9XGyUGEalZX/rSl7IOoSKpjkFERFpRYhCRbq+pqYm5c+dmHUbVSJ0YzKyvmfXp\nyMbNbLSZLTazZ8zsggLzB5rZfWbWbGZPmdn4jmxfRKSYZIujt99+O+twqkabdQxm1gM4FjgZ+CQh\niZiZrQEeAW4Ffu3u3sb6dcBUYCTwIvBnM5vt7osSi50NNLn7hWY2EFhiZj9399Vd8NpEpIapxVHn\nFSsxNACfAKYAO7n7tu6+DbBTfG4Y8Psi6w8HnnX3pe7+PjAL+FzeMi8DA+LjAcByJQUR2VDTp0+v\nmZ5QS6FYq6RPu/t7+U/G5x4FHm3n0tIg4IXE9DJg/7xlbgJ+Z2YvAf2BE1NFLSJSxIgRI1RK2ABt\nJoZCSaGDyxS8xJTnIqDZ3evNbGfgfjPby91XfnDRScybBytXQn19PfX19Sk2LyK1aLfddss6hEw0\nNDTQ0NCwwduxNqoIMLOlhJP7a+6e/0u//Q2bHQBMcvfRcfpCYK27/yCxzD3AZHf/U5x+ALjA3Z/I\n25aDM2UKTJzY0UhEpDtzd92c1gYzw907fHCKXUo6NP5f07mQeAIYamY7Ai8BJxEqspMWEyqn/2Rm\nHwJ2Bf7Wyf2JSA3Jjb28cuVKrrzyyqzD6VaKJYaZ8f8bwHEd3bC7rzazs4F5QB3wY3dfZGZfjvNv\nBC4HZpjZk4SK8PPd/Y2O7ktEakuyxdH06dOzDqfbKVbHcBiE+xfy55lZX3d/t72Nu/u9wL15z92Y\nePw6cExHAhaR2pUrJUybNo0pU6aoj6MSSdNX0sPAvimeExEpqcsvv5z58+erxVGJFbvBbVtgO2Bj\nM9sXMEJl9ABg4/KEJyKy3kUXXUSvXr1USiixYiWGUcBphPsRrkg8v5LQzFREpKx69+6ddQg1oVgd\nw0xgppkd5+53li8kEal1ubGXd9hhh6xDqUltdolhZmPjwx3NbELib6KZTShTfCJSY3JjL1999dVZ\nh1Kzil1KytUj9CfdXcwiIp1WqMWRZKNYYtg5/n/a3X9ZjmBEpDapJ9TKUqx31aMsVP1fWK5gRKQ2\nPfPMM+oJtYIUKzHcC7wJ9DOz/E7t3N0HFFhHRKTDTjxRHStXkjZLDO5+nrtvBtzj7v3z/pQURES6\nqXaH9nT3z5rZEDMbCWBmG5tZ/9KHJiLdTVNTE7Nmzco6DGlHu4nBzM4CfgXk+jgaDPy6lEGJSPeS\nHHt57dq1WYcj7UjTV9LXCMN0Pgrg7n81s61LGpWIdBtqcVR92i0xAO8lR2ozs57ovgYRSWHmzJka\ne7kKpSkx/N7MvkPoTO/TwFeBOaUNS0S6g0MOOUSlhCqUJjF8GzgDWAh8GbgHuLmUQYlI97DTTjtl\nHYJ0QruJwd3XANPjn4hIQRp7uftI0ypphJndb2bPmNnf45/GZRYRYH2LozPPPDPrUKSLpLmU9GPg\nW0AjsKa04YhINdHYy91TmsTwrzh2s4gIoLGXu7s0ieFBM/shcBewrtmquzeWLCoRqWjXXnutxl7u\nxsy9+C0JZtZAgfsW3P2wEsVUKAYHZ8oUmDixXHsVkbasXr2auro6lRIqnJnh7h1+k9K0SqrvVEQi\n0m317JnmYoNUqzStkrYxsx+b2X1xejczO6P0oYlI1lpaWnjmmWeyDkPKLE2XGDOB3wC5C4nPAOeU\nKiARqQy5sZevuuqqrEORMkuTGAa6++3Epqru/j6wuqRRiUhmWlpauOSSSxg1ahTnnnsu1113XdYh\nSZmluVD4bzPbMjdhZgcAK0oXkohkpampidNOO40hQ4aoxVENS5MYJhI6zdvJzB4GtgKOL2lUIpKJ\nV155hfPOO49TTz1VLY5qWJpWSfPN7FBg1/jUkng5SUS6mSOPPDLrEKQCFK1jiEN6DoyJoD9wJPCZ\nskQmIiKZaDMxmNklwO+Ax8zse8BVwJbAN8zsR2WKT0RKoLGxkZtvVu/5UlixEsPJwG7AfsA3gcPd\n/dvAp4GRZYhNRLpYrsXR6NGj2WijjbIORypUsTqGd+OQnu+Z2bPu/jaAu682s5byhCciXaWxsZHx\n48erxZG0q1hi2NTMvgBY4jG56ZJHJiJd5tZbb+Wcc87hiiuuUIsjaVexxPAH4JgCjwF+X7KIRKTL\nHXbYYSolSGptJgZ3H7+hGzez0cDVQB1ws7v/oMAy9YSK7V7A6+q0T6TrKSFIR7SZGMzstPhwlbvf\n0dENm1kdMJVQUf0i8Gczm+3uixLLbAZcB4xy92VmNrCj+xGR1tauXUuPHml6uxEprNinZ8f4t0Mn\ntz0ceNbdl8b7IGYBn8tbZgxwp7svA3D31zu5L5Gal2txNGbMmKxDkSpXyp8Vg4AXEtPL4nNJQ4Et\nzOxBM3vCzMaWMB6RbquxsZH99tuPxsZGrrzyyqzDkSpXrPL5H4SR297p5LaLDw0X9AL2BQ4HNgYe\nMbNH3V0dwIukoLGXpRSKVT7P3MBtvwhsn5jenlBqSHqBUOH8DvCOmf0B2Isw5kOeScybBytXQn19\nPfX19RsYnkj1u+WWWzT2sqzT0NBAQ0PDBm+nzTGfzewWYJq7/7mN+fsD/+nup7cxvyewhFAaeAl4\nHDg5r/L5o4QK6lFAH+Ax4CR3fzpvWxrzWaSAtWvXYmYqJUhBpRjz+SrgvDj+whLgZcLNbdsQelp9\nGJjS1srxDumzgXmE5qo/dvdFZvblOP9Gd18chwxdAKwFbspPCiLSNrU+klJos8SwbgGzPsA+wBBC\nvcE/gCfd/d3Sh7cuBpUYpKblxl7efffdsw5FqkhnSwzt/txw9/fc/VF3v93df+nuj5UzKYjUuubm\nZoYPH67WRlI2KoeKVKiWlhYuvfRSjjjiCCZMmKBusqVs0gztKSJltmDBAsaNG8fgwYPV4kjKLnVi\nMLON3X1VKYMRkWDFihVMmDBB9yVIJtq9lGRmnzSzpwktkzCzvc3s+pJHJlLDDj74YMaNG6ekIJlI\nU8dwNTAaeB3A3ZuBQ0sZlIiIZCdV5bO7P5/31OoSxCJSc5qbm7n66quzDkOklTSJ4XkzOwjAzHqb\n2bnAonbWEZEiki2Ottxyy6zDEWklTeXzV4AfEXpGfRH4DfC1UgYl0p01Nzczfvx4tTiSipWmxPAR\ndx/j7lu7+1bufgrw0VIHJtId3XnnnevuS5gzZ46SglSkNCWGqYQuMdp7TkTaUV9fr1KCVLxiQ3se\nCHwS2MrMJhA60APoj+6YFukU1SdINShWYuhNSAJ18X/OW8DxpQxKpDtYs2YNdXV1WYch0mHFBur5\nPfB7M5vp7kvLF5JIdcuNqvbEE09w9913Zx2OSIelqWNYZWZTgN2AjeJz7u6fKl1YItUp2eLopptu\nyjockU5JU1dwK7AY2AmYBCwFnihdSCLVJ78nVLU4kmqWpsSwpbvfbGbfSFxeUmIQSbjjjjs09rJ0\nG2kSQ0v8/4qZfYYwfvPmpQtJpPqMGTOGMWPGqNM76RbSJIbJZrYZMBG4FhgAnFPSqESqjBKCdCdp\nhvac4+7/cveF7l7v7vsCr5QhNpGK09LSQmNjY9ZhiJRUm4nBzHqY2XFmdr6ZHRWf28/MfgNML1uE\nIhVCYy9LrShWYpgOfJVQn/BfZnYn8BPgetQdhtSQ/BZHP/vZz7IOSaSkitUxHADs6e5rzawv4fLR\nzu6+vDyhiWRv4cKFjB07Vj2hSk0plhjed/e1AO7+rpn9XUlBas2aNWs09rLUnGKJ4aNmtjAxvXNi\n2t19zxLGJVIR9t57b/bee++swxApq2KJ4WNli0JERCpGsU70lpYxDpFMNTc3M2fOHC6++OKsQxHJ\nnMZVkJqWbHE0ZMiQrMMRqQhp7nwW6ZY09rJIYalKDGa2sZntWupgRMrl7rvvVk+oIm1ot8RgZp8F\nfgj0AXY0s32A/3b3z5Y6OJFSOeSQQ1RKEGlDmhLDJGB/4E0Ad28ijM0gUrX69++vpCDShjSJ4X13\n/1fec2tLEYxIKbz//vtZhyBSVdIkhr+Y2SlATzMbambXAg+XOC6RDZZrcTRy5EjcPetwRKpGmsTw\ndWB34D3gNuAt4FulDEpkQ+V6Qp0/fz633XaburMQ6YA0zVV3dfeLgItKHYzIhmppaWHy5MlMmzaN\nKVOmqI8jkU5IU2K40swWm9llZvbxjmzczEbHdZ8xswuKLDfMzFab2Rc6sn2RfPPmzVs39vK4ceOU\nFEQ6wdJcezWzbYET498A4Jfuflk769QBS4CRwIvAn4GT3X1RgeXuB1YBM9z9zgLbcnCmTIGJE1O9\nLqlRuc+zEoJI+B64e4e/DKlucHP3l939R8B/Ak8Cl6RYbTjwrLsvdff3gVnA5wos93XgV8A/04Us\n0jYzU1IQ2UDtJgYz283MJpnZU8BUQoukQSm2PQh4ITG9LH89MxtESBbT4lNqOiKptLS08PDDahwn\nUgppKp9vIfzaH+XuL3Zg22lO8lcD33Z3t/AzTz/1pF25Po6GDh3KgQceqBKCSBdrNzG4+wGd3PaL\nwPaJ6e0JpYakTwCz4hd7IHCkmb3v7rM/uLlJzJsHK1dCfX099fX1nQxLqpVaHIkU19DQQENDwwZv\np83KZzO7w91PyBvFLafdEdzMrCeh8vlw4CXgcQpUPieWnwHMcfe7CsxT5XONe/rppxkzZgyDBw9m\n+vTp6s5CJIXOVj4XKzF8M/7/DB+8xNPuZSJ3X21mZwPzgDrgx+6+yMy+HOff2NFgpXb17t1bYy+L\nlEm7zVXN7AfufkF7z5WSSgwiIh1XyuaqRxR47qiO7khERKpDm4nBzL4S6xd2NbOFib+lwIKyRSg1\npbm5mfPOO0+d3olkqFiJ4RfAMcBsQj3DMfHvE+5+ShlikxqSHHt5jz32yDockZpWrPLZ3X2pmX2N\nvMpmM9vC3d8obWhSKzT2skhlKZYYbgOOBuZTuBXSh0sSkdSUBx54gJNPPln3JYhUkDYTg7sfHf/v\nWLZopOaMGDFCpQSRCpOmr6SDzKxffDzWzK40syGlD01qQZ8+fZQURCpMmuaqNwCrzGwvYALwN+Cn\nJY1KuqV333036xBEJIU0iWG1u68FjgWuc/epQP/ShiXdSa7F0YgRI9QMVaQKpEkMK83sIuBUYG4c\nWKdXacOS7iI59vLs2bNVuSxSBdIkhpOA94AvuvsrhDEVfljSqKTqJe9LmDBhAnPmzFFdgkiVSNPt\n9stmdiswzMw+Azzu7qpjkKIeeeQRGhsb1eJIpAqlaZV0IvAYcAJhzOfHzeyEUgcm1e3QQw9VKUGk\nSqUZwe2/gGHu/hqAmW0FPADcUcrAREQkG2nqGAz4Z2J6ORqCU6KWlhYeeOCBrMMQkS6UJjHcB8wz\ns/FmdjpwD3BvacOSapBrcTR16lTWrl2bdTgi0kXaTQzufh7hJrc9gT2AG939/FIHJpUrv8XRXXfd\nRY8eaX5jiEg1aLOOwcw+QmiWugth/IXz3H1ZuQKTyrR48WL+4z/+Qz2hinRjxX7m3QLMBY4DGoFr\nyhKRVLQBAwYwceJEtTgS6caKtUrq5+43xceLzaypHAFJZdtuu+0YO3Zs1mGISAkVSwx9zWzf+NiA\njeK0EQbxaSx5dCIiUnbFEsMrwBVFpg8rSURSEZqbm7nhhhu4/vrrVbEsUmOKDdRTX8Y4pEK0tLQw\nefJkpk2bxpQpU9TpnUgNSnPns9QIjb0sIpDuBjepAQ8//LB6QhURQCUGifbff38WLFjANttsk3Uo\nIpKxNL2r9ohjPV8Sp3cws+GlD03Kqa6uTklBRIB0l5KuBw4ExsTpf8fnpEq9/fbbWYcgIhUsTWLY\n392/CrwD4O5voKE9q1Kuj6Phw4ezZs2arMMRkQqVJjG0xHGegXXjMagrzSqTHHv5/vvvp66urv2V\nRKQmpUnrnoHRAAAO9UlEQVQM1wL/C2xtZpcDfwK+X9KopMto7GUR6ag0Yz7/3MzmA4fHpz7n7otK\nG5Z0lYULF9Lc3Kz7EkQkNXP34guY7ZB7GP87gLs/X8K48mNwcKZMgYkTy7VXEZHqZma4e4e7L0hz\nH8M9xGQA9AU+DCwBdu/ozkREpPKlGcHt4+6+R/wbCgwHHi19aNIRLS0tzJ07N+swRKQb6HCXGLG7\n7f1LEIt0Uq7F0fTp01m9enXW4YhIlWv3UpKZJa/q9wD2BV5MuwMzGw1cDdQBN7v7D/LmnwKcT6jD\nWAl8xd0XpN1+LcvvCXXs2LHqDVVENliaOoZ+icerCcN93plm4/H+h6nASEIy+bOZzc5r1fQ34BB3\nXxGTyHTggDTbr2XPPvssxx9/vHpCFZEuVzQxxBP7AHfvbFug4cCz7r40bm8W8DlgXWJw90cSyz8G\nDO7kvmrKlltuyfnnn8/JJ5+sUoKIdKk26xjMrKe7rwEOss6feQYBLySml8Xn2nIGoRWUtGPzzTdn\nzJgxSgoi0uWKlRgeJ9QnNAP/Z2Z3AKviPHf3u1Jsv/hNEglmdhjwReCgwktMYt48WLkS6uvrqa+v\nT7tpEZGa0NDQQENDwwZvp80b3Mysyd33MbOZFDjBu/vp7W7c7ABgkruPjtMXAmsLVEDvCdwFjHb3\nZwtsp2ZvcGtubmbKlCnMmDGDXr3Ud6GIpNfZG9yKNVfdyswmAAuBpwr8pfEEMNTMdjSz3sBJwOy8\nwHcgJIVTCyWFWpXs4+iII46gZ0+NqSQi5VHsbFMH9N+Qjbv7ajM7G5gXt/djd19kZl+O828ELgE2\nB6bF6+Xvu3tNDwSksZdFJEvtXkoqczwF1dKlpKamJkaNGqX7EkRkg5WyryQpo7333pu//OUvbLXV\nVlmHIiI1qlgdw8iyRSHrmJmSgohkqs3E4O7LyxlILVqxYkXWIYiIfECHO9GTDZdrcbTvvvvS0tKS\ndTgiIq0oMZRZU1MTw4YNY/78+Tz00EP07t0765BERFpRYiiTXClh1KhRnHvuuRp7WUQqllollclz\nzz3HU089pfsSRKTiKTGUycc+9jHuvDNVb+UiIpnSpSQREWlFiaGLtbS0cMcdd2QdhohIpykxdKFc\ni6Of/vSnvPfee1mHIyLSKUoMXSC/xdHs2bPp06dP1mGJiHSKKp830N///neOPfZYdthhB7U4EpFu\nQYlhA2299dZcdNFFnHjiieoJVUS6BSWGDbTJJptw0kknZR2GiEiXUR2DiIi0osSQUlNTE1/4whd4\n9913sw5FRKSklBjakWxx9PnPf16tjUSk21MdQxFNTU2MHz9eLY5EpKYoMbRhyZIljBo1iiuuuIJT\nTz1VLY5EpGYoMbRh1113ZcmSJWy++eZZhyIiUlaqYyhCSUFEapESA7B8uYa3FhHJqenEkGtxtM8+\n+7Bq1aqswxERqQg1mxhyPaE2Njby6KOPsvHGG2cdkohIRai5xFCoJ1Q1QxURWa/mWiW9/PLLLF68\nWPcliIi0oeYSw5AhQ7j99tuzDkNEpGLV3KUkEREprtsmhpaWFn7yk5/g7lmHIiJSVbplYsi1OPrV\nr36lZqgiIh3UrRJDoRZHm2yySdZhiYhUlaqpfP74x2HgwLbnL1u2jKOPPlo9oYqIbCCrhmvwZubt\nxdnS0sLcuXP5/Oc/r55QRUQAM8PdO3xC7DaJQUREWutsYihpHYOZjTazxWb2jJld0MYy18T5T5rZ\nPqWMR0RE2leyxGBmdcBUYDSwG3CymX0sb5mjgF3cfShwFjCtve02NTVx5JFH8tZbb5Ug6srX0NCQ\ndQgVQ8diPR2L9XQsNlwpSwzDgWfdfam7vw/MAj6Xt8xngZ8AuPtjwGZm9qFCG0u2OBozZgz9+/cv\nYeiVSx/69XQs1tOxWE/HYsOVslXSIOCFxPQyYP8UywwGXs3f2LBhw9TiSESkDEqZGNLWFudXjBRc\nb+LEiYwdO1YtjkRESqxkrZLM7ABgkruPjtMXAmvd/QeJZW4AGtx9VpxeDBzq7q/mbetZYOeSBCoi\n0n095+67dHSlUpYYngCGmtmOwEvAScDJecvMBs4GZsVE8q/8pADQmRcmIiKdU7LE4O6rzexsYB5Q\nB/zY3ReZ2Zfj/Bvd/R4zOyqWCN4GTi9VPCIikk5V3OAmIiLlU1Gd6OmGuPXaOxZmdko8BgvM7E9m\ntmcWcZZDms9FXG6Yma02sy+UM75ySfn9qDezJjN7yswayhxi2aT4fgw0s/vMrDkei/EZhFkWZnaL\nmb1qZguLLNOx86a7V8Qf4XLTs8COQC+gGfhY3jJHAffEx/sDj2Ydd4bH4kBg0/h4dC0fi8RyvwPm\nAsdlHXdGn4nNgL8Ag+P0wKzjzvBYTAK+nzsOwHKgZ9axl+h4HAzsAyxsY36Hz5uVVGLo0hviqly7\nx8LdH3H3FXHyMcL9H91Rms8FwNeBXwH/LGdwZZTmOIwB7nT3ZQDu/nqZYyyXNMfiZWBAfDwAWO7u\nq8sYY9m4+0PAm0UW6fB5s5ISQ6Gb3QalWKY7nhDTHIukM4B7ShpRdto9FmY2iHBiyHWp0h0rztJ8\nJoYCW5jZg2b2hJmNLVt05ZXmWNwE7G5mLwFPAt8sU2yVqMPnzUoaj6FLb4ircqlfk5kdBnwROKh0\n4WQqzbG4Gvi2u7uFOyC7412QaY5DL2Bf4HBgY+ARM3vU3Z8paWTll+ZYXAQ0u3u9me0M3G9me7n7\nyhLHVqk6dN6spMTwIrB9Ynp7QmYrtszg+Fx3k+ZYECucbwJGu3uxomQ1S3MsPkG4FwbC9eQjzex9\nd59dnhDLIs1xeAF43d3fAd4xsz8AewHdLTGkORafBCYDuPtzZvZ3YFfC/VW1psPnzUq6lLTuhjgz\n6024IS7/iz0bGAfr7qwueENcN9DusTCzHYC7gFPd/dkMYiyXdo+Fu+/k7h929w8T6hm+0s2SAqT7\nfvwfMMLM6sxsY0JF49NljrMc0hyLxcBIgHg9fVfgb2WNsnJ0+LxZMSUG1w1x66Q5FsAlwObAtPhL\n+X13H55VzKWS8lh0eym/H4vN7D5gAbAWuMndu11iSPmZuByYYWZPEn4An+/ub2QWdAmZ2W3AocBA\nM3sBuJRwWbHT503d4CYiIq1U0qUkERGpAEoMIiLSihKDiIi0osQgIiKtKDGIiEgrSgwiItKKEoN0\nipmtid075/52KLLsv7tgfzPN7G9xX/PjjTod3cZNZvbR+PiivHl/2tAY43Zyx2WBmd1lZv3aWX4v\nMzuyE/vZ2szujo/rzWyGmZ1mZpcmlhlnZgtjLI1mNrET+znUzA5MTF8a9zPDzA6Nz/3SzD7c0W1L\n5VJikM5a5e77JP6eL7JsV9ws48C57r4P8G2gwze2ufuZ7r44Tl6YN6+r+prKHZc9gbeAL7ez/D6E\nbpE76mxgZnz8geMbk803gU/HWA4AVuQvl8JhhO4lirkJOKcT25YKpcQgXcLMNjGz38Zf8wvM7LMF\nltnWzP4Qf1EvNLMR8fkjzOzhuO4vzWyTtnYT/z8E7BLXnRC3tdDMvpmI5e44SMtCMzshPt9gZp8w\ns/8HbBTj+Fmc9+/4f5aZrTtRx5LKF8ysh5n90MwetzDYyVkpDssjwM5xO8Pja2y0MLDSR2J3Dt8F\nToqxnBBjv8XMHovLfuA4RscDd8fHLYST/jtArpO4C4GJ7v4KgLu3uPvNMZa9zezR+DruMrPN4vPf\nMLO/xOd/YWZDCIntnBjfCODfcT8rgPfivhroXHKTSpX1IBP6q84/YDXQFP/uJHRN0D/OGwg8k1h2\nZfw/EbgoPu4B9IvL/h7YKD5/AXBxgf3NIA7AA5xAOOnuS+j+YSNgE+ApYG/gOGB6Yt0B8f+DwL7J\nmArEeCwwMz7uDTwP9AHOAr4Tn+8D/BnYsUCcue3UxePy1TjdH6iLj0cCv4qPTwOuSax/OXBKfLwZ\nsATYOG8f29DGoCyJZZbn3o8C8xYAB8fH/w1cFR+/CPTKO2aXAhNSfB5+T4EBlPRXnX8V01eSVJ13\nPFzWAcDMegHfN7ODCf30bGdmW7v7a4l1Hgduicv+2t2fNLN6YDfg4djnU2/g4QL7M+CHZvZfwGuE\nMSg+DdzloTdRzOwuwmhW9wFTYslgrrv/sQOv6z7gR/HX/JHA7939PTM7AtjDzI6Pyw0glFqW5q2/\nkZk1EfrAXwrcEJ/fDPipme1CuPST++7ldxN+BHCMmZ0bp/sQesZcklhmCGEgmg4zs00JI/89FJ/6\nCXBHfLwA+IWZ/Rr4dXK1FJt+iTCi2qLOxCWVRZeSpKucQvj1v29MGK8BfZMLxJPRwYRfpjNt/UAy\n9/v6uord3f3MAttfV8fg7qM8dA7ntD5pWdiNP0Mc6hD4npldnPZFuPu7hEsjo4ATgdsTs89OxLmz\nu/+2wCZyCXMI8C7rRxa7DHjA3fcAjiGUctryhcR+dnT3JQWWae9k/Rdgv3aWyd/O0cB1hJLYn82s\nLsX6ye2s7cDyUsGUGKSrDABec/c1FgYPGpK/QGy59E8P17pvJpy8HwUOsjCYSq5+YGgb+8g/GT4E\nHGtmG8V6iWOBh8xsW+Bdd78VmBL3k+99M2urxHw7YfCjXOkDQk+eX82tE+sINm5jfWIp5hvAZAtF\noQGEX9XQunfLtwiXmXLmxfWI+ykU+z8Il5OK+T6hhPWhuJ3eZnaGh+Fg38zV7wBjgYYY4w7u3kCo\n3N+UcKlvZV58bdk2xiXdgBKDdFZ+S5hbgf3MbAHhZLOowLKHAc1m1kj4Nf4jD+MSjwdus9BF8sOE\nvvPb3ae7NxFa5jxOSDA3ufuTwB7AY/GSziXA9wpsazqwIFf5nLft3wCHEEoyuXGCbyaMbdBoZgsJ\nw4gWSizrtuPuzYRB608E/odwqa2RUP+QW+5BYLdc5TOhZNErVuA/RagDaL2DUKHcs0glPe5+LzAV\n+G3cznzWn+BPIySNJ4E9CRXgPYGfxfevkfDerADmAJ+P8RVsuRUvDQ729S2+pMqp222RKmRmk4BF\n7n57e8uWIZYjgKPdvZbHVe5WVGIQqU7XEX75V4IvAVdlHYR0HZUYRESkFZUYRESkFSUGERFpRYlB\nRERaUWIQEZFWlBhERKQVJQYREWnl/wOXz5j71NmaVgAAAABJRU5ErkJggg==\n",
       "text": [
        "<matplotlib.figure.Figure at 0x786f9f0>"
       ]
      }
     ],
     "prompt_number": 24
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Examine the coefficients"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pd.DataFrame(zip(vectorizer.get_feature_names(), np.exp(clf.coef_[0]))).sort(1)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>0</th>\n",
        "      <th>1</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>47</th>\n",
        "      <td>        yra</td>\n",
        "      <td>  0.132292</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>36</th>\n",
        "      <td>         su</td>\n",
        "      <td>  0.137260</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>37</th>\n",
        "      <td>        tai</td>\n",
        "      <td>  0.145047</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>10</th>\n",
        "      <td>    europos</td>\n",
        "      <td>  0.166072</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>7 </th>\n",
        "      <td>        d\u0117l</td>\n",
        "      <td>  0.174451</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>16</th>\n",
        "      <td>        kad</td>\n",
        "      <td>  0.214095</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>24</th>\n",
        "      <td>        mes</td>\n",
        "      <td>  0.219679</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>12</th>\n",
        "      <td>         i\u0161</td>\n",
        "      <td>  0.230045</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5 </th>\n",
        "      <td>       b\u016bti</td>\n",
        "      <td>  0.236922</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>17</th>\n",
        "      <td>       kaip</td>\n",
        "      <td>  0.237122</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>34</th>\n",
        "      <td> pirmininke</td>\n",
        "      <td>  0.240614</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>38</th>\n",
        "      <td>       taip</td>\n",
        "      <td>  0.241801</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4 </th>\n",
        "      <td>       buvo</td>\n",
        "      <td>  0.248400</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>35</th>\n",
        "      <td>       savo</td>\n",
        "      <td>  0.299503</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>0 </th>\n",
        "      <td>       apie</td>\n",
        "      <td>  0.334180</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>42</th>\n",
        "      <td>       turi</td>\n",
        "      <td>  0.341146</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>28</th>\n",
        "      <td>       m\u016bs\u0173</td>\n",
        "      <td>  0.389030</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>6 </th>\n",
        "      <td>       b\u016bt\u0173</td>\n",
        "      <td>  0.394295</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>40</th>\n",
        "      <td>        tik</td>\n",
        "      <td>  0.459583</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>20</th>\n",
        "      <td>   komisija</td>\n",
        "      <td>  0.531496</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>33</th>\n",
        "      <td>        pat</td>\n",
        "      <td>  0.739714</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3 </th>\n",
        "      <td>        bet</td>\n",
        "      <td>  0.855985</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>11</th>\n",
        "      <td>         ir</td>\n",
        "      <td>  0.984499</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>30</th>\n",
        "      <td>         ne</td>\n",
        "      <td>  1.088736</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>41</th>\n",
        "      <td>         to</td>\n",
        "      <td>  1.269904</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>14</th>\n",
        "      <td>         jo</td>\n",
        "      <td>  1.299818</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1 </th>\n",
        "      <td>         ar</td>\n",
        "      <td>  1.669845</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>18</th>\n",
        "      <td>        kas</td>\n",
        "      <td>  1.879318</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>25</th>\n",
        "      <td>       mums</td>\n",
        "      <td>  2.019102</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>19</th>\n",
        "      <td>         ko</td>\n",
        "      <td>  2.301956</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>13</th>\n",
        "      <td>         ja</td>\n",
        "      <td>  2.590359</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9 </th>\n",
        "      <td>         es</td>\n",
        "      <td>  3.274294</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>29</th>\n",
        "      <td>        nav</td>\n",
        "      <td>  3.316575</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>27</th>\n",
        "      <td>       m\u016bsu</td>\n",
        "      <td>  3.328093</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>23</th>\n",
        "      <td>        lai</td>\n",
        "      <td>  3.810891</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>39</th>\n",
        "      <td>        tas</td>\n",
        "      <td>  3.846516</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>48</th>\n",
        "      <td>       \u013coti</td>\n",
        "      <td>  3.864085</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>46</th>\n",
        "      <td>        vai</td>\n",
        "      <td>  4.200415</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2 </th>\n",
        "      <td>        ar\u012b</td>\n",
        "      <td>  4.425187</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>49</th>\n",
        "      <td>         \u0161o</td>\n",
        "      <td>  4.436995</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>43</th>\n",
        "      <td>         t\u0101</td>\n",
        "      <td>  5.567311</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>22</th>\n",
        "      <td>         k\u0101</td>\n",
        "      <td>  5.645506</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>31</th>\n",
        "      <td>         no</td>\n",
        "      <td>  6.474617</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>21</th>\n",
        "      <td>      kungs</td>\n",
        "      <td>  6.480887</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>45</th>\n",
        "      <td>         uz</td>\n",
        "      <td>  7.243644</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>8 </th>\n",
        "      <td>    eiropas</td>\n",
        "      <td>  9.303418</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>26</th>\n",
        "      <td>        m\u0113s</td>\n",
        "      <td>  9.333379</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>15</th>\n",
        "      <td>         ka</td>\n",
        "      <td> 11.846539</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>32</th>\n",
        "      <td>        par</td>\n",
        "      <td> 16.106159</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>44</th>\n",
        "      <td>         un</td>\n",
        "      <td> 20.055701</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 26,
       "text": [
        "             0          1\n",
        "47         yra   0.132292\n",
        "36          su   0.137260\n",
        "37         tai   0.145047\n",
        "10     europos   0.166072\n",
        "7          d\u0117l   0.174451\n",
        "16         kad   0.214095\n",
        "24         mes   0.219679\n",
        "12          i\u0161   0.230045\n",
        "5         b\u016bti   0.236922\n",
        "17        kaip   0.237122\n",
        "34  pirmininke   0.240614\n",
        "38        taip   0.241801\n",
        "4         buvo   0.248400\n",
        "35        savo   0.299503\n",
        "0         apie   0.334180\n",
        "42        turi   0.341146\n",
        "28        m\u016bs\u0173   0.389030\n",
        "6         b\u016bt\u0173   0.394295\n",
        "40         tik   0.459583\n",
        "20    komisija   0.531496\n",
        "33         pat   0.739714\n",
        "3          bet   0.855985\n",
        "11          ir   0.984499\n",
        "30          ne   1.088736\n",
        "41          to   1.269904\n",
        "14          jo   1.299818\n",
        "1           ar   1.669845\n",
        "18         kas   1.879318\n",
        "25        mums   2.019102\n",
        "19          ko   2.301956\n",
        "13          ja   2.590359\n",
        "9           es   3.274294\n",
        "29         nav   3.316575\n",
        "27        m\u016bsu   3.328093\n",
        "23         lai   3.810891\n",
        "39         tas   3.846516\n",
        "48        \u013coti   3.864085\n",
        "46         vai   4.200415\n",
        "2          ar\u012b   4.425187\n",
        "49          \u0161o   4.436995\n",
        "43          t\u0101   5.567311\n",
        "22          k\u0101   5.645506\n",
        "31          no   6.474617\n",
        "21       kungs   6.480887\n",
        "45          uz   7.243644\n",
        "8      eiropas   9.303418\n",
        "26         m\u0113s   9.333379\n",
        "15          ka  11.846539\n",
        "32         par  16.106159\n",
        "44          un  20.055701"
       ]
      }
     ],
     "prompt_number": 26
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Exercise\n",
      "\n",
      "##Create a classifier for *all* 21 languages\n",
      "i.e. Given a sentence, output its most probable language\n",
      "\n",
      "**hint**: Create 21 classifiers which classify *langauge x* vs. *all other languages* and choose langauge with highest probability"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}