logistic_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Theano example: logistic regression\n",
    "\n",
    "This notebook is inspired from [the tutorial on logistic regression](http://deeplearning.net/tutorial/logreg.html) on [deeplearning.net](http://deeplearning.net).\n",
    "\n",
    "In this notebook, we show how Theano can be used to implement the most basic classifier: the **logistic regression**. We start off with a quick primer of the model, which serves both as a refresher but also to anchor the notation and show how mathematical expressions are mapped onto Theano graphs.\n",
    "\n",
    "In the deepest of machine learning traditions, this tutorial will tackle the exciting problem of MNIST digit classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the data\n",
    "This will load a pre-packaged version of MNIST, each split as NumPy ndarrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import gzip\n",
    "import six\n",
    "from six.moves import cPickle\n",
    "\n",
    "with gzip.open('data/mnist.pkl.gz', 'rb') as data_file:\n",
    "    if six.PY3:\n",
    "        train_set, valid_set, test_set = cPickle.load(data_file, encoding='latin1')\n",
    "    else:\n",
    "        train_set, valid_set, test_set = cPickle.load(data_file)\n",
    "\n",
    "train_set_x, train_set_y = train_set\n",
    "valid_set_x, valid_set_y = valid_set\n",
    "test_set_x, test_set_y = test_set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The model\n",
    "Logistic regression is a probabilistic, linear classifier. It is parametrized\n",
    "by a weight matrix $W$ and a bias vector $b$. Classification is\n",
    "done by projecting an input vector onto a set of hyperplanes, each of which\n",
    "corresponds to a class. The distance from the input to a hyperplane reflects\n",
    "the probability that the input is a member of the corresponding class.\n",
    "\n",
    "Mathematically, the probability that an input vector $x$ is a member of a\n",
    "class $i$, a value of a stochastic variable $Y$, can be written as:\n",
    "\n",
    "$$P(Y=i|x, W,b) = softmax_i(W x + b) = \\frac {e^{W_i x + b_i}} {\\sum_j e^{W_j x + b_j}}$$\n",
    "\n",
    "The model's prediction $y_{pred}$ is the class whose probability is maximal, specifically:\n",
    "\n",
    "$$  y_{pred} = {\\rm argmax}_i P(Y=i|x,W,b)$$\n",
    "\n",
    "Now, let us define our input variables. First, we need to define the dimension of our tensors:\n",
    "- `n_in` is the length of each training vector,\n",
    "- `n_out` is the number of classes.\n",
    "\n",
    "Our variables will be:\n",
    "- `x` is a matrix, where each row contains a different example of the dataset. Its shape is `(batch_size, n_in)`, but `batch_size` does not have to be specified in advance, and can change during training.\n",
    "- `W` is a shared matrix, of shape `(n_in, n_out)`, initialized with zeros. Column `k` of `W` represents the separation hyperplane for class `k`.\n",
    "- `b` is a shared vector, of length `n_out`, initialized with zeros. Element `k` of `b` represents the free parameter of hyperplane `k`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy\n",
    "import theano\n",
    "from theano import tensor\n",
    "\n",
    "# Size of the data\n",
    "n_in = 28 * 28\n",
    "# Number of classes\n",
    "n_out = 10\n",
    "\n",
    "x = tensor.matrix('x')\n",
    "W = theano.shared(value=numpy.zeros((n_in, n_out), dtype=theano.config.floatX),\n",
    "                  name='W',\n",
    "                  borrow=True)\n",
    "b = theano.shared(value=numpy.zeros((n_out,), dtype=theano.config.floatX),\n",
    "                  name='b',\n",
    "                  borrow=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can build a symbolic expression for the matrix of class-membership probability (`p_y_given_x`), and for the class whose probability is maximal (`y_pred`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "p_y_given_x = tensor.nnet.softmax(tensor.dot(x, W) + b)\n",
    "y_pred = tensor.argmax(p_y_given_x, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining a loss function\n",
    "Learning optimal model parameters involves minimizing a loss function. In the\n",
    "case of multi-class logistic regression, it is very common to use the negative\n",
    "log-likelihood as the loss. This is equivalent to maximizing the likelihood of the\n",
    "data set $\\cal{D}$ under the model parameterized by $\\theta$. Let\n",
    "us first start by defining the likelihood $\\cal{L}$ and loss\n",
    "$\\ell$:\n",
    "\n",
    "$$\\mathcal{L} (\\theta=\\{W,b\\}, \\mathcal{D}) =\n",
    "     \\sum_{i=0}^{|\\mathcal{D}|} \\log(P(Y=y^{(i)}|x^{(i)}, W,b)) \\\\\n",
    "   \\ell (\\theta=\\{W,b\\}, \\mathcal{D}) = - \\mathcal{L} (\\theta=\\{W,b\\}, \\mathcal{D})\n",
    "$$\n",
    "\n",
    "Again, we will express those expressions using Theano. We have one additional input, the actual target class `y`:\n",
    "- `y` is an input vector of integers, of length `batch_size` (which will have to match the length of `x` at runtime). The length of `y` can be symbolically expressed by `y.shape[0]`.\n",
    "- `log_prob` is a `(batch_size, n_out)` matrix containing the log probabilities of class membership for each example.\n",
    "- `arange(y.shape[0])` is a symbolic vector which will contain `[0,1,2,... batch_size-1]`\n",
    "- `log_likelihood` is a vector containing the log probability of the target, for each example.\n",
    "- `loss` is the mean of the negative `log_likelihood` over the examples in the minibatch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y = tensor.lvector('y')\n",
    "log_prob = tensor.log(p_y_given_x)\n",
    "log_likelihood = log_prob[tensor.arange(y.shape[0]), y]\n",
    "loss = - log_likelihood.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training procedure\n",
    "This notebook will use the method of stochastic gradient descent with mini-batches (MSGD) to find values of `W` and `b` that minimize the loss.\n",
    "\n",
    "We can let Theano compute symbolic expressions for the gradient of the loss wrt `W` and `b`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "g_W, g_b = theano.grad(cost=loss, wrt=[W, b])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`g_W` and `g_b` are symbolic variables, which can be used as part of a computation graph. In particular, let us define the expressions for one step of gradient descent for `W` and `b`, for a fixed learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learning_rate = numpy.float32(0.13)\n",
    "new_W = W - learning_rate * g_W\n",
    "new_b = b - learning_rate * g_b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then define **update expressions**, or pairs of (shared variable, expression for its update), that we will use when compiling the Theano function. The updates will be performed each time the function gets called.\n",
    "\n",
    "The following function, `train_model`, returns the loss on the current minibatch, then changes the values of the shared variables according to the update rules. It needs to be passed `x` and `y` as inputs, but not the shared variables, which are implicit inputs.\n",
    "\n",
    "The entire learning algorithm thus consists in looping over all examples in the dataset, considering all the examples in one minibatch at a time, and repeatedly calling the `train_model` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_model = theano.function(inputs=[x, y],\n",
    "                              outputs=loss,\n",
    "                              updates=[(W, new_W),\n",
    "                                       (b, new_b)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing the model\n",
    "When testing the model, we are interested in the number of misclassified examples (and not only in the likelihood). Here, we build a symbolic expression for retrieving the number of misclassified examples in a minibatch.\n",
    "\n",
    "This will also be useful to apply on the validation and testing sets, in order to monitor the progress of the model during training, and to do early stopping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "misclass_nb = tensor.neq(y_pred, y)\n",
    "misclass_rate = misclass_nb.mean()\n",
    "\n",
    "test_model = theano.function(inputs=[x, y],\n",
    "                             outputs=misclass_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the model\n",
    "Here is the main training loop of the algorithm:\n",
    "- For each *epoch*, or pass through the training set\n",
    "  - split the training set in minibatches, and call `train_model` on each minibatch\n",
    "  - split the validation set in minibatches, and call `test_model` on each minibatch to measure the misclassification rate\n",
    "  - if the misclassification rate has not improved in a while, stop training\n",
    "- Measure performance on the test set\n",
    "\n",
    "The **early stopping procedure** is what decide whether the performance has improved enough. There are many variants, and we will not go into the details of this one here.\n",
    "\n",
    "We first need to define a few parameters for the training loop and the early stopping procedure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "## Define a couple of helper variables and functions for the optimization\n",
    "batch_size = 500\n",
    "# compute number of minibatches for training, validation and testing\n",
    "n_train_batches = train_set_x.shape[0] // batch_size\n",
    "n_valid_batches = valid_set_x.shape[0] // batch_size\n",
    "n_test_batches = test_set_x.shape[0] // batch_size\n",
    "\n",
    "def get_minibatch(i, dataset_x, dataset_y):\n",
    "    start_idx = i * batch_size\n",
    "    end_idx = (i + 1) * batch_size\n",
    "    batch_x = dataset_x[start_idx:end_idx]\n",
    "    batch_y = dataset_y[start_idx:end_idx]\n",
    "    return (batch_x, batch_y)\n",
    "\n",
    "## early-stopping parameters\n",
    "# maximum number of epochs\n",
    "n_epochs = 1000\n",
    "# look as this many examples regardless\n",
    "patience = 5000\n",
    "# wait this much longer when a new best is found\n",
    "patience_increase = 2\n",
    "# a relative improvement of this much is considered significant\n",
    "improvement_threshold = 0.995\n",
    "\n",
    "# go through this many minibatches before checking the network on the validation set;\n",
    "# in this case we check every epoch\n",
    "validation_frequency = min(n_train_batches, patience / 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 1, minibatch 100/100, validation error 11.890000 %\n",
      "     epoch 1, minibatch 100/100, test error of  best model 12.170000 %\n",
      "epoch 2, minibatch 100/100, validation error 10.520000 %\n",
      "     epoch 2, minibatch 100/100, test error of  best model 10.940000 %\n",
      "epoch 3, minibatch 100/100, validation error 9.940000 %\n",
      "     epoch 3, minibatch 100/100, test error of  best model 10.210000 %\n",
      "epoch 4, minibatch 100/100, validation error 9.480000 %\n",
      "     epoch 4, minibatch 100/100, test error of  best model 9.820000 %\n",
      "epoch 5, minibatch 100/100, validation error 9.230000 %\n",
      "     epoch 5, minibatch 100/100, test error of  best model 9.510000 %\n",
      "epoch 6, minibatch 100/100, validation error 9.100000 %\n",
      "     epoch 6, minibatch 100/100, test error of  best model 9.280000 %\n",
      "epoch 7, minibatch 100/100, validation error 8.820000 %\n",
      "     epoch 7, minibatch 100/100, test error of  best model 9.100000 %\n",
      "epoch 8, minibatch 100/100, validation error 8.750000 %\n",
      "     epoch 8, minibatch 100/100, test error of  best model 8.890000 %\n",
      "epoch 9, minibatch 100/100, validation error 8.630000 %\n",
      "     epoch 9, minibatch 100/100, test error of  best model 8.780000 %\n",
      "epoch 10, minibatch 100/100, validation error 8.490000 %\n",
      "     epoch 10, minibatch 100/100, test error of  best model 8.620000 %\n",
      "epoch 11, minibatch 100/100, validation error 8.440000 %\n",
      "     epoch 11, minibatch 100/100, test error of  best model 8.570000 %\n",
      "epoch 12, minibatch 100/100, validation error 8.380000 %\n",
      "     epoch 12, minibatch 100/100, test error of  best model 8.530000 %\n",
      "epoch 13, minibatch 100/100, validation error 8.310000 %\n",
      "     epoch 13, minibatch 100/100, test error of  best model 8.480000 %\n",
      "epoch 14, minibatch 100/100, validation error 8.250000 %\n",
      "     epoch 14, minibatch 100/100, test error of  best model 8.400000 %\n",
      "epoch 15, minibatch 100/100, validation error 8.270000 %\n",
      "epoch 16, minibatch 100/100, validation error 8.220000 %\n",
      "     epoch 16, minibatch 100/100, test error of  best model 8.300000 %\n",
      "epoch 17, minibatch 100/100, validation error 8.160000 %\n",
      "     epoch 17, minibatch 100/100, test error of  best model 8.210000 %\n",
      "epoch 18, minibatch 100/100, validation error 8.100000 %\n",
      "     epoch 18, minibatch 100/100, test error of  best model 8.200000 %\n",
      "epoch 19, minibatch 100/100, validation error 8.070000 %\n",
      "     epoch 19, minibatch 100/100, test error of  best model 8.190000 %\n",
      "epoch 20, minibatch 100/100, validation error 8.080000 %\n",
      "epoch 21, minibatch 100/100, validation error 8.030000 %\n",
      "     epoch 21, minibatch 100/100, test error of  best model 8.160000 %\n",
      "epoch 22, minibatch 100/100, validation error 8.020000 %\n",
      "     epoch 22, minibatch 100/100, test error of  best model 8.180000 %\n",
      "epoch 23, minibatch 100/100, validation error 7.980000 %\n",
      "     epoch 23, minibatch 100/100, test error of  best model 8.110000 %\n",
      "epoch 24, minibatch 100/100, validation error 7.970000 %\n",
      "     epoch 24, minibatch 100/100, test error of  best model 8.130000 %\n",
      "epoch 25, minibatch 100/100, validation error 7.950000 %\n",
      "     epoch 25, minibatch 100/100, test error of  best model 8.110000 %\n",
      "epoch 26, minibatch 100/100, validation error 7.940000 %\n",
      "     epoch 26, minibatch 100/100, test error of  best model 8.090000 %\n",
      "epoch 27, minibatch 100/100, validation error 7.890000 %\n",
      "     epoch 27, minibatch 100/100, test error of  best model 8.060000 %\n",
      "epoch 28, minibatch 100/100, validation error 7.880000 %\n",
      "     epoch 28, minibatch 100/100, test error of  best model 8.030000 %\n",
      "epoch 29, minibatch 100/100, validation error 7.830000 %\n",
      "     epoch 29, minibatch 100/100, test error of  best model 7.970000 %\n",
      "epoch 30, minibatch 100/100, validation error 7.800000 %\n",
      "     epoch 30, minibatch 100/100, test error of  best model 7.930000 %\n",
      "epoch 31, minibatch 100/100, validation error 7.780000 %\n",
      "     epoch 31, minibatch 100/100, test error of  best model 7.900000 %\n",
      "epoch 32, minibatch 100/100, validation error 7.760000 %\n",
      "     epoch 32, minibatch 100/100, test error of  best model 7.850000 %\n",
      "epoch 33, minibatch 100/100, validation error 7.750000 %\n",
      "     epoch 33, minibatch 100/100, test error of  best model 7.840000 %\n",
      "epoch 34, minibatch 100/100, validation error 7.760000 %\n",
      "epoch 35, minibatch 100/100, validation error 7.780000 %\n",
      "epoch 36, minibatch 100/100, validation error 7.750000 %\n",
      "epoch 37, minibatch 100/100, validation error 7.730000 %\n",
      "     epoch 37, minibatch 100/100, test error of  best model 7.810000 %\n",
      "epoch 38, minibatch 100/100, validation error 7.720000 %\n",
      "     epoch 38, minibatch 100/100, test error of  best model 7.810000 %\n",
      "epoch 39, minibatch 100/100, validation error 7.700000 %\n",
      "     epoch 39, minibatch 100/100, test error of  best model 7.780000 %\n",
      "epoch 40, minibatch 100/100, validation error 7.690000 %\n",
      "     epoch 40, minibatch 100/100, test error of  best model 7.800000 %\n",
      "epoch 41, minibatch 100/100, validation error 7.670000 %\n",
      "     epoch 41, minibatch 100/100, test error of  best model 7.790000 %\n",
      "epoch 42, minibatch 100/100, validation error 7.690000 %\n",
      "epoch 43, minibatch 100/100, validation error 7.690000 %\n",
      "epoch 44, minibatch 100/100, validation error 7.690000 %\n",
      "epoch 45, minibatch 100/100, validation error 7.680000 %\n",
      "epoch 46, minibatch 100/100, validation error 7.660000 %\n",
      "     epoch 46, minibatch 100/100, test error of  best model 7.820000 %\n",
      "epoch 47, minibatch 100/100, validation error 7.640000 %\n",
      "     epoch 47, minibatch 100/100, test error of  best model 7.810000 %\n",
      "epoch 48, minibatch 100/100, validation error 7.600000 %\n",
      "     epoch 48, minibatch 100/100, test error of  best model 7.790000 %\n",
      "epoch 49, minibatch 100/100, validation error 7.600000 %\n",
      "epoch 50, minibatch 100/100, validation error 7.590000 %\n",
      "     epoch 50, minibatch 100/100, test error of  best model 7.800000 %\n",
      "epoch 51, minibatch 100/100, validation error 7.600000 %\n",
      "epoch 52, minibatch 100/100, validation error 7.580000 %\n",
      "     epoch 52, minibatch 100/100, test error of  best model 7.810000 %\n",
      "epoch 53, minibatch 100/100, validation error 7.570000 %\n",
      "     epoch 53, minibatch 100/100, test error of  best model 7.820000 %\n",
      "epoch 54, minibatch 100/100, validation error 7.560000 %\n",
      "     epoch 54, minibatch 100/100, test error of  best model 7.820000 %\n",
      "epoch 55, minibatch 100/100, validation error 7.550000 %\n",
      "     epoch 55, minibatch 100/100, test error of  best model 7.820000 %\n",
      "epoch 56, minibatch 100/100, validation error 7.520000 %\n",
      "     epoch 56, minibatch 100/100, test error of  best model 7.830000 %\n",
      "epoch 57, minibatch 100/100, validation error 7.490000 %\n",
      "     epoch 57, minibatch 100/100, test error of  best model 7.810000 %\n",
      "epoch 58, minibatch 100/100, validation error 7.480000 %\n",
      "     epoch 58, minibatch 100/100, test error of  best model 7.810000 %\n",
      "epoch 59, minibatch 100/100, validation error 7.450000 %\n",
      "     epoch 59, minibatch 100/100, test error of  best model 7.810000 %\n",
      "epoch 60, minibatch 100/100, validation error 7.450000 %\n",
      "epoch 61, minibatch 100/100, validation error 7.430000 %\n",
      "     epoch 61, minibatch 100/100, test error of  best model 7.810000 %\n",
      "epoch 62, minibatch 100/100, validation error 7.430000 %\n",
      "epoch 63, minibatch 100/100, validation error 7.410000 %\n",
      "     epoch 63, minibatch 100/100, test error of  best model 7.810000 %\n",
      "epoch 64, minibatch 100/100, validation error 7.400000 %\n",
      "     epoch 64, minibatch 100/100, test error of  best model 7.790000 %\n",
      "epoch 65, minibatch 100/100, validation error 7.410000 %\n",
      "epoch 66, minibatch 100/100, validation error 7.410000 %\n",
      "epoch 67, minibatch 100/100, validation error 7.390000 %\n",
      "     epoch 67, minibatch 100/100, test error of  best model 7.770000 %\n",
      "epoch 68, minibatch 100/100, validation error 7.380000 %\n",
      "     epoch 68, minibatch 100/100, test error of  best model 7.770000 %\n",
      "epoch 69, minibatch 100/100, validation error 7.390000 %\n",
      "epoch 70, minibatch 100/100, validation error 7.370000 %\n",
      "     epoch 70, minibatch 100/100, test error of  best model 7.750000 %\n",
      "epoch 71, minibatch 100/100, validation error 7.370000 %\n",
      "epoch 72, minibatch 100/100, validation error 7.380000 %\n",
      "epoch 73, minibatch 100/100, validation error 7.380000 %\n",
      "epoch 74, minibatch 100/100, validation error 7.380000 %\n",
      "epoch 75, minibatch 100/100, validation error 7.350000 %\n",
      "     epoch 75, minibatch 100/100, test error of  best model 7.780000 %\n",
      "epoch 76, minibatch 100/100, validation error 7.330000 %\n",
      "     epoch 76, minibatch 100/100, test error of  best model 7.800000 %\n",
      "epoch 77, minibatch 100/100, validation error 7.320000 %\n",
      "     epoch 77, minibatch 100/100, test error of  best model 7.780000 %\n",
      "epoch 78, minibatch 100/100, validation error 7.310000 %\n",
      "     epoch 78, minibatch 100/100, test error of  best model 7.780000 %\n",
      "epoch 79, minibatch 100/100, validation error 7.310000 %\n",
      "epoch 80, minibatch 100/100, validation error 7.330000 %\n",
      "epoch 81, minibatch 100/100, validation error 7.300000 %\n",
      "     epoch 81, minibatch 100/100, test error of  best model 7.770000 %\n",
      "epoch 82, minibatch 100/100, validation error 7.290000 %\n",
      "     epoch 82, minibatch 100/100, test error of  best model 7.750000 %\n",
      "epoch 83, minibatch 100/100, validation error 7.290000 %\n",
      "epoch 84, minibatch 100/100, validation error 7.290000 %\n",
      "epoch 85, minibatch 100/100, validation error 7.300000 %\n",
      "epoch 86, minibatch 100/100, validation error 7.280000 %\n",
      "     epoch 86, minibatch 100/100, test error of  best model 7.710000 %\n",
      "epoch 87, minibatch 100/100, validation error 7.280000 %\n",
      "epoch 88, minibatch 100/100, validation error 7.260000 %\n",
      "     epoch 88, minibatch 100/100, test error of  best model 7.690000 %\n",
      "epoch 89, minibatch 100/100, validation error 7.260000 %\n",
      "epoch 90, minibatch 100/100, validation error 7.250000 %\n",
      "     epoch 90, minibatch 100/100, test error of  best model 7.700000 %\n",
      "epoch 91, minibatch 100/100, validation error 7.250000 %\n",
      "epoch 92, minibatch 100/100, validation error 7.230000 %\n",
      "     epoch 92, minibatch 100/100, test error of  best model 7.680000 %\n",
      "epoch 93, minibatch 100/100, validation error 7.230000 %\n",
      "epoch 94, minibatch 100/100, validation error 7.230000 %\n",
      "epoch 95, minibatch 100/100, validation error 7.220000 %\n",
      "     epoch 95, minibatch 100/100, test error of  best model 7.690000 %\n",
      "Optimization complete with best validation score of 7.220000 %, with test performance 7.690000 %\n",
      "The code ran for 96 epochs, with 2.282257 epochs/sec\n"
     ]
    }
   ],
   "source": [
    "import timeit\n",
    "from six.moves import xrange\n",
    "\n",
    "best_validation_loss = numpy.inf\n",
    "test_score = 0.\n",
    "start_time = timeit.default_timer()\n",
    "\n",
    "done_looping = False\n",
    "epoch = 0\n",
    "while (epoch < n_epochs) and (not done_looping):\n",
    "    epoch = epoch + 1\n",
    "    for minibatch_index in xrange(n_train_batches):\n",
    "        minibatch_x, minibatch_y = get_minibatch(minibatch_index, train_set_x, train_set_y)\n",
    "        minibatch_avg_cost = train_model(minibatch_x, minibatch_y)\n",
    "\n",
    "        # iteration number\n",
    "        iter = (epoch - 1) * n_train_batches + minibatch_index\n",
    "        if (iter + 1) % validation_frequency == 0:\n",
    "            # compute zero-one loss on validation set\n",
    "            validation_losses = []\n",
    "            for i in xrange(n_valid_batches):\n",
    "                valid_xi, valid_yi = get_minibatch(i, valid_set_x, valid_set_y)\n",
    "                validation_losses.append(test_model(valid_xi, valid_yi))\n",
    "            this_validation_loss = numpy.mean(validation_losses)\n",
    "            print('epoch %i, minibatch %i/%i, validation error %f %%' %\n",
    "                  (epoch,\n",
    "                   minibatch_index + 1,\n",
    "                   n_train_batches,\n",
    "                   this_validation_loss * 100.))\n",
    "\n",
    "            # if we got the best validation score until now\n",
    "            if this_validation_loss < best_validation_loss:\n",
    "                # improve patience if loss improvement is good enough\n",
    "                if this_validation_loss < best_validation_loss * improvement_threshold:\n",
    "                    patience = max(patience, iter * patience_increase)\n",
    "\n",
    "                best_validation_loss = this_validation_loss\n",
    "\n",
    "                # test it on the test set\n",
    "                test_losses = []\n",
    "                for i in xrange(n_test_batches):\n",
    "                    test_xi, test_yi = get_minibatch(i, test_set_x, test_set_y)\n",
    "                    test_losses.append(test_model(test_xi, test_yi))\n",
    "\n",
    "                test_score = numpy.mean(test_losses)\n",
    "                print('     epoch %i, minibatch %i/%i, test error of  best model %f %%' %\n",
    "                      (epoch,\n",
    "                       minibatch_index + 1,\n",
    "                       n_train_batches,\n",
    "                       test_score * 100.))\n",
    "\n",
    "                # save the best parameters\n",
    "                numpy.savez('logreg_best_model.npz', W=W.get_value(), b=b.get_value())\n",
    "\n",
    "        if patience <= iter:\n",
    "            done_looping = True\n",
    "            break\n",
    "\n",
    "end_time = timeit.default_timer()\n",
    "print('Optimization complete with best validation score of %f %%, '\n",
    "      'with test performance %f %%' %\n",
    "      (best_validation_loss * 100., test_score * 100.))\n",
    "\n",
    "print('The code ran for %d epochs, with %f epochs/sec' %\n",
    "      (epoch, 1. * epoch / (end_time - start_time)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization\n",
    "You can visualize the columns of `W`, which correspond to the separation hyperplanes for each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-0.5, 143.5, 56.5, -0.5)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2QAAAFpCAYAAADz6es/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3UmMX1de9/+T7iS247k8lV2ex9hOHGdOOjOkExAJNBIN\nEhJqRBYsWgjEBhaPWPBsYA3sEBLQaiEBEgIEiAUhNOkMdpzEjud5rCrP5YrHDDyb///RI93P26qT\n/lVOVfJ+Lb85fX/33jPce1K5n77jf/7nf4okSZIk6cv3jdYnIEmSJElfV27IJEmSJKkRN2SSJEmS\n1IgbMkmSJElqxA2ZJEmSJDXihkySJEmSGnFDJkmSJEmN3DneP/D973//f4/3b0hj8ad/+qex/k//\n9E9f8plIXa+++mqs/+Ef/uGXfCaTwx133BHr/n9rjp8/+IM/iPU/+ZM/+ZLPROr6rd/6rVj/4z/+\n4y/5TKTo1u///u/jnmjcN2SllP/1JfyGJEmSJE1EH5dScEPmf7IoSZIkSY24IZMkSZKkRtyQSZIk\nSVIjbsgkSZIkqZEvI9Tja4GSvT777LNY//zzz2P9zju7XUJpYnQM+s1vfCPvv9NvSgmNc6rXjHN9\n9dDaNZ7HTnUah59++mms37p1K9anT58+xrNjpkBKX1/f/OY3q+okrV30jlf77lfz3nrXXXfRKUY3\nbtyI9Rbr4kRbi/0LmSRJkiQ14oZMkiRJkhpxQyZJkiRJjbghkyRJkqRG3JBJkiRJUiMTLuqsF6kn\nvUj2qj1GrxIMU9IOpdjQOVJCTk0SHqWS1aZJavxQKhONxRo3b96M9atXr1b95ujoaKxPnTq1U5s1\na1ZsWzsXxzPZ7+uO7m3N2kLJhjSeqf0nn3wS6zX9T8em9Y9SFu+5555YT9dUmzJG51Kbspv0Yq2Y\nzGqSOkvhPqKxmNS+E9DzP/U/nXft3Kp9/usnR/1M4yX1BT1DqT9rn7nXrl0bc1s670uXLsU6rYtp\nbtW+49ZqNc6/3quxJEmSJDXkhkySJEmSGnFDJkmSJEmNuCGTJEmSpEbckEmSJElSIxMuZbFGbboL\nJUql41BqEh27NpWIjkPpdgklJ1GdUnz6+vo6Nbq3dH6U1pRSeahOyTl0b79qahOy7r777p+4TnPi\nypUrVedCYyulKZaS58Xs2bOrfrM2ZS8dh45B9V6kjE6kpLLaNDmq0/z/+OOPO7VTp07Ftrdu3Yp1\nSuWi9mltWb16dWw7Z86cWK8dF+vXr4/1mTNndmq0nlGCKfURnUs6zmROwaUxl8ZWKXme0/ik+0Lr\nFv3m8PBwp3bhwoUxn18ppZw5cybWaYymZE96r6AU0P7+/lgnK1eujPVp06Z1avTsr00Z/aqpTTal\nZNe0tlBbeiegZy6NucuXL3dqtG5dv3491mn9o/ty8eLFMR+bngk0L6ZMmRLrrZKa/QuZJEmSJDXi\nhkySJEmSGnFDJkmSJEmNuCGTJEmSpEbckEmSJElSIxMuZbEm3aQ2ZZHaUwJTQimDlBxDCWHz5s2L\n9ZQSM2PGjNiWUpwoOefs2bNj/s358+fHtpSEQ3VK30tpQJR4RYk6Nf02GdQmGNL10/hPqWwpHasU\nTrw6ffp0rNM51sxRSjaj6zl37lys0xxNx6H0KUqrmjVrVqxTulNKFGuV4JR+t3ZNpIQ0Wv/S3E1J\nXaVwEhwlftE9X7t27ZjO43Zo/atN/KxJN1ywYEGs03pO9zHNi5GRkTGfx3irTSqlezs0NBTrqa8p\nNfbo0aOxTulrdBzqi4TGIiWB0lxM7xYLFy6MbSkdks6FUkNpjKbnCL0THD58ONZrEqZbSWO3NpG2\n9nlOa3Sq03OL1i1aQykJN10TnR+NOarTuaS1i9Yzeg7RsVNSZSl8H9MzqpfPc/9CJkmSJEmNuCGT\nJEmSpEbckEmSJElSI27IJEmSJKkRN2SSJEmS1MiES1msUZtKR0krKTmmNsGO0hTpNylp5siRI50a\nJcr09fXFem0qXUpDSkllpZSybNmyWKckLOqLlDJJ501pRVevXo31iYTOPY0Lakvpg5S+RvMi3V/q\nH7q3lD5GSUPnz5+P9fvuu69To2QjOhdqT3NrcHCwU6OUxblz58Y6pThRWlOaW3SMFuhcKPGO1kVK\nmRsdHe3UKKmN1hBKgqPEy4TGLV3niRMnYr12vixatKhTozFHfTEwMBDrdP0plZfuIT2fWqB7SM/W\nQ4cOjbk9jdsNGzbEOvXRxYsXYz09iykdlMYQJSRSEuSZM2c6taeffjq2pTWR3i0o2ZmeOelaaTxT\n2jP1UU1Saa/UJCfSddL1UHt6/6FnaErZpv6kdwiydOnSWE9zLq3xpfCYo3W+pj2949C8pftCY5He\nf+mdo1f8C5kkSZIkNeKGTJIkSZIacUMmSZIkSY24IZMkSZKkRiZ1qEf6YL6UUm7evBnr9JFlqtNH\nkPSxP300SB8ZUn3atGmdGoV3UJ0+SKcPIW/cuNGp0ceO9OHpsWPHYp3CHh5//PFOjT6wJvSxb4sP\n1ekjYJI+gqaPvelDavrwmu55ui+p70vhwIBNmzZV/Sade+o7GltDQ0OxnuZKKflj51JyOAL1Gx2b\n5i1df+246AU6l7Re0ofU165di3VaW+lD7TTOaR2i+bx8+fJYp7GV1nNan+n6+/v7Y33+/PmxTvcl\nhRfRGKJQC+rPmnAUelaOJxr7FGpA4QAUpEFBLWlNe+SRR2JbWnNPnToV63Qf77nnnk6N1pDt27fH\n+oMPPhjrJB0/BX2UUsrp06dj/dlnn431FC5WCq+tW7du7dToXtWGN1EgzURBc5+uh9Y/ul/0LprG\nHN0rCoyhdyWaF+kdjdb+2nWO3s/379/fqdEaQtdP85nelVesWBHraU2j5xa9z9yOfyGTJEmSpEbc\nkEmSJElSI27IJEmSJKkRN2SSJEmS1IgbMkmSJElqZFKkLFLKFNUp9YRSXFKKzZIlS2JbSmupTc6h\nc0zJiXSdlJBH6VOUBpQSdShlh469evXqWKeEvJScM3PmzNiW0iQpxYbuSy9QX9S2T/eRrp/SilLK\n0u2kMUrnd+nSpVinlFFKN6JzTO0p8YmOQUmg1D6lktH8pDFH6ZOU1kf18UTrXOprSlOkOUQpa5Ti\nlRK4UvJgKZxKR2sIXWc6R0okpXWLrp+SYCmtK91zGkM0ns+fPx/ra9eujfV0rfR86lU6aGpPSWh0\nLoQSAimVcc2aNZ0ara2UVLlo0aJYp+Ok6z969GhsS0m1NIYoIfS5557r1Gjdvvfee2P9o48+inWa\nz3SclJxHzwo69uXLl2N9IqUspjFNc4WeZ7QW1aQp0u/Sc4vWSkqwpbmb1m56l6U69SelNaY0Xbrn\n9K5EY44SH+nc0zs0raGmLEqSJEnSJOKGTJIkSZIacUMmSZIkSY24IZMkSZKkRtyQSZIkSVIjkzpl\nkVJMKGmFEtVSGhClo9Ex6DfpHM+cORPrKd1wcHAwtqWEmF27dsU6JSSldDNK66H0GbpOSnEbHh7u\n1JYuXRrbzp8/P9Zrk8DGE43RmkSxXiSblcJjMSUtUX9SQhSNRUrfO3nyZKynvj506FBsSwlJc+bM\niXVKTkt9RAlWKdmpFL5fNM5boDUqXX9NImMp3BeUypruL/3mtm3bYp3WIkrCHRgY6NRofB45ciTW\n6Tr37t0b64sXL4719957r1OjBD+at5Rsm5JqSyll2bJlYz5Gr8ZtGi+09tG6ReOWxgulVabn+fTp\n02Pb2neF9NwqJT9DKWWQ0vToeU5zK6VMUpIyjVs6F0qTputPz//XXnsttq19h2ih5p2zNpGanjl0\nnP7+/lin9NWExhYlHlLiZZpH9K5A108JtpSmmuYi3Stac+hZQWOO6il9ktaWL/J+6l/IJEmSJKkR\nN2SSJEmS1IgbMkmSJElqxA2ZJEmSJDXihkySJEmSGmkWa1OTQEKJR5SoQgkp9Jspxak2fenixYux\nTulblLRz/PjxTm3FihWxLSWBkZTKVEopIyMjnRol+GzZsiXWKSGR0m127NgxpvMopS6psJSJlb5I\naUBUTyhljJKTapIA6RiUMkXpW8eOHatqf99993VqNFfoXlH6Jo2XVKckKJrndL+oj5JejU/qo5o6\ntaX1bP369bFOiZdpLd6zZ09sS8mWtM5RElhKwqLroVS62mS/S5cuxXpa/2h80likxN+ahFRq2wKd\nS80cKqWUtWvXxnp6dtM9pLWFxuKTTz4Z62k9o/FJ4//tt9+O9QMHDsT60NBQp0bPYRq3lD5Kz2Ia\noyllktZ+Sp+sTXztxTpKxyYpUY/e5dI9KYXHFqUVUhJqun46Bq1zBw8ejPUZM2bEekp8TO+spfB4\npmcopS+m9bwW9QU9z+hdJK3ztPZTX9yOfyGTJEmSpEbckEmSJElSI27IJEmSJKkRN2SSJEmS1Igb\nMkmSJElqpFnKIkkJTJRKRSk7lG5Cx0kpNnQMSoKhtJbh4eGqc0mJipTKMzAwEOvPPvtsrL/55pux\nnhKVahKsSinlzJkzsX7o0KFYT6lnlFZDKTu1CUm9QGOOkj1rEqJq0yQprYjGbkrZ/OSTT2JbStk8\nfPhwrL/11luxvn///lhPSVuUbJSSrUrhtDZqn1K86Pop8ev69euxTilWKa1xvBPvalIWFyxYENvS\nnKPrT+lbpeS19emnnx7z+ZVSytSpU2OdkjDT2KJkO5q3tG5TchylyaY5TfOTzoUSwqiPUrpfbYLZ\neK6tdGw6xwcffDDWH3jggTH/JiWyUoLxsmXLYp3WlrSeU6rx66+/Hut0jpRseOXKlU6NxhYlPtKY\nI8uXL4/1NBfT+ZXC/f9FUunGC92X1Be0nlOyJR2bxj+9F6U1itJEqU7PrV/+5V+O9fQuSgnLlGxL\ndZpbaR5R8i7d276+vlhftWpVrNNzLr0v0Dw3ZVGSJEmSJhE3ZJIkSZLUiBsySZIkSWrEDZkkSZIk\nNdIs1IM+7EwfQVOQAn3sXBs8kT6mpA/G6cNbCmSgQA76gDH5qZ/6qVhfuXJlrNMHnOlj91LyB7/0\n8XoKACmllBMnTsQ6fdib+m5oaCi2pY9AJ9JHwPRxPIVGpHO/6667YluaK7VBImle3HPPPbEt9TMF\niezbt6/qXNJHtvPmzYtt6aNm+lCXxlzqCwrvoLFF/Ul1GhfjidbL1NczZsyIbSnsg8IOKNQirXO0\nbtNaWbOGlJLXuXXr1sW2FN7xwQcfxPrWrVtjncborl27OrWlS5fGttRv9DyjZ87Zs2c7NQrpoflf\nGzCU0PUQGov03KIP8hctWtSpUWAQrbkUdkDjJb1bnDt3LraltbU2MCeFd1FgSG0IAq1/jz76aKyn\n+0VjKwX9lMJraO04qkHHpve/dJ0U3kH9Se85FLxC4+jo0aOdGt1bGv8XLlyI9d/93d+N9fSuTIFe\n77zzTqzTGkrrf3rO05pQu1eg5zO9F6RnEYWxfBH+hUySJEmSGnFDJkmSJEmNuCGTJEmSpEbckEmS\nJElSI27IJEmSJKmRZimLJKWyUVIbJeFQKt/g4GCspwQWSl+iFBtKd6G0opSEVUpOA6NUrpQmVUop\nU6ZMqWqf7hcl3k2dOjXWKZWG0r3Sb1KCH6Vj0nW2QGk9VE/ji9K3aJxTUif1RepTOgYlPtE4P3ny\nZKx/+9vfjvWUkkSJT5RiRfeFUo9S6hXdK6pTf45nEhih36xZLylNcfXq1bG+Zs2aWKeEvLTmUr9R\nyhq1pz5Kv0nrNq1zmzZtinVKfKTrT+dI/UbXQ4lilEp2/PjxTo2S3aj/ac7VqElSLqWUgYGBWKdk\nY1r/07nTvKVxsXv37linRLmU+Lhz587YlvqCkj2fe+65WE/P1vvuuy+2pXei+++/P9YplfHxxx+P\n9bR20/OM1vnPPvss1scTrS0kjTlK8KPrp/fQY8eOxTo9i1P6KvUzjWd65/q3f/u3WE/9T4nclDJJ\nayX1f01qLKWjUsokPXNojNI19Yp/IZMkSZKkRtyQSZIkSVIjbsgkSZIkqRE3ZJIkSZLUiBsySZIk\nSWpk3FMWa9PHUjITHYMSTyjdZv/+/WP+zZSaVEopo6OjsU5pNZTAQ2mFKSWpNmWK0q1IOk5fX19s\nW5uQQ+d+7ty5To0S/yg5h5IwW6gd55SollA/U0IYJWQtWbKkU6OUKRq3lEq0fv36qnNJfUrpUw88\n8ECs07nTHE1qk+CoPd2XiSQlzc2ZMye2TWOlFJ7/H3/8caxfunRpTLVSSjlw4ECsU+IVzYuUytrf\n3x/b0tiiZwil41Iq4caNG8d8bFrPaJynNMVScuIpJbXRPB/PtbU2NXnmzJmxTv2f1lZKO6aEuPR8\nKqWUhx56KNbT85L658c//nGsU+Lh888/H+tpvtB7xcGDB2Od5jmdC/Vdes7TO1GLNMValMqZxhal\nZtI9P3r0aKxTgitJ9/HOO/MrPa3P1M/Lli2L9ZQESusZja1t27bFOiXeJvTuQ4m0NOZoDaEE2/Ru\nUZt2fDv+hUySJEmSGnFDJkmSJEmNuCGTJEmSpEbckEmSJElSI27IJEmSJKmRcU9ZpKSRmlS6u+++\nO9YpOeX999+PdUrrS4ktlOxE6TuUVkVJkMuXL4/1lDRFiW+U7HbPPfeM+dil5Gui9BlKDqLktIGB\ngTEfh65zMqcp1iTw0D2nOvU/3a/0mzSeKcWI5jOlzNG8SF544YVYX7RoUazTOdKcSwlUtIZQv1E6\nJrWvHS81alOcUl9TgiElZ+3duzfW33333VhP6yjdQ0oCpHtIKW6HDx8e829SaiRdP6VvUSpfSvej\n66GUMZrnNP7Tc46efXTsmnlbSr6m2vQxmovnz5+PdVoXatBaSfeFnlEpxY5S8x555JFYpz7as2dP\nrKfkUDq/1atXxzrdcxrPlFSd0qFpftJ11ryflNKbtZWOQb+Z+pTuCaVs0jy/ePFirFNyYHrPo3cF\nGov0Hnrjxo1YT+Oc5uGuXbtiPaXAllLKE088Eetp/q9Zsya2paRumhe1Y67mHaImSfv/nk/1/0KS\nJEmS1BNuyCRJkiSpETdkkiRJktSIGzJJkiRJasQNmSRJkiQ1Mu4pi6QmIYzaUnLM6dOnY33Lli2x\n/tFHH3VqlPg1derUWKe0ssceeyzWN27cGOsprYiSwOj6z549G+uUHJkSpegYp06dinVKwhwdHY31\ndE2U1kPJQS3UJttRik8aR5SOWZtWRr+ZkuPOnDkT2w4NDcU6jQuqL1u2LNbTta5YsSK2rb1OGi8p\n9YjSlKhe2//jic6Fzj31/3vvvRfb7tixI9ZpXFASGKW1JZR4l5KtSuHkrNSeEt/oeuhcKMGTngup\nPSWE0b2iY5N9+/Z1aikdrZRS5s+fH+u9SLCjcVjbfnBwMNbpvqQUP0qB7evri/UNGzbEOp3jzp07\nOzVKX6MEOxrnlPi5du3aTo2ew0uXLo31119/PdZr179t27Z1apQmSHNrwYIFsU6pjL1A10kJeak9\nvZ+tWrUq1lPyainc/5T4mdIKadzSWrl48eJYJ2kdofdtSlOk+0LP/7RGUTokXSe9KxOaL8nw8HDV\nsW/Hv5BJkiRJUiNuyCRJkiSpETdkkiRJktSIGzJJkiRJaqRZqEcN+vCSPpolx48fj/X+/v5O7fz5\n87EtfRxNH0HTudOH3elc6INECmQ4cuRIrNMH/JcuXerU6ON1+jiePmylj4DTh530QSqFXdA51n5M\n3gt0nVRPH59S6AqFV9B10hhNwRs0bgcGBmJ9zpw5sU59tHnz5ljftGlTp0bXf+3atVinD6/pmtJ6\nQW1r73lN//ciMOGLHCcFD9E6R+EA9IE59X8KdTlw4EBsS2EHdJ20nqf17OGHH45tKWCA1lBaiykc\nIoVJUEhBCpcqhT/2p7GYrp/CmB566KFYH0/0DKF1i55zFI6QnkU0P2fPnh3rNC5oLKbgAQojOXbs\nWKw/9dRTsU4BAytXruzUtm/fHtu+8847sU7zn9Z/WnPTmKNgtPSOc7vfpH7uxTpK44LmXHpG0XsL\nhVpQCAaNF5oXKQRj/fr1sS29b9JcTCE1pZTy4YcfdmoffPBBbEtrIoVu0fqf7jndKxpbdP00zylI\nJz0Xp0+fHtvSvb0d/0ImSZIkSY24IZMkSZKkRtyQSZIkSVIjbsgkSZIkqRE3ZJIkSZLUyIRLWaTU\nm6Q2IY0SwlKiDiX+UHIOpRXROVKKS0qaoWSjlJpWSinvvvturB86dCjW586d26mlFMBSOAlv2rRp\nsZ6S/UopZe3atZ0aJfhRmhjd216l2PUCpTWlZB5K66HxTClWNM6HhoY6NUrwohQjGotbt26N9TS2\nSillyZIlnRqdN6Usjo6OxjqlNaXj02/Wji2aL3S/eoHOhX4z1Y8ePRrbptS4UkqZOnVq1W9evHix\nU6OUwTQ+S+EkLBr/KX2O0vQoTZHWucWLF8c6JcGlVK7apEpqT+tCGot0D+k6e4Ge5bSenzhxItYp\nZXHhwoWxPjw83KnR85YS3ygdlsZ/Sg1et25dbPuzP/uzsb5gwYJYT9dTSilvvPFGp7Zt27bYlt5n\n0vwsJaejlsJJoOn9h57nDz74YKxTsiulDI4nmltpjtJ4pjWU5nNKTSyFx24a/5R2TfOfnpW7du2K\n9R07dnRqlKZI83P58uWxTs+Q9K5A94TeQ2ks1jwrCbWt2cv8//wLmSRJkiQ14oZMkiRJkhpxQyZJ\nkiRJjbghkyRJkqRG3JBJkiRJUiMTLmUxoSQkSrehtBpKt0nJNJTsRskxV69erfpNSkhLyXGU4nLg\nwIFYpxQzSrdJKK2GkoPOnTsX69R3KfWMUgZrk+ooIakFSjdKaU2zZs2Kbak/KWWQkrNSihfNIRq3\nKU2sFE4ZpFS69LuUYErXQ8l2V65cifU0pql/auct9VELdE1pfNG6ReOC+vPHP/5xrL/44oudGo0V\nureU4kUpi2nNobmS0l5L4WS/48ePxzqlqaaU0Xnz5sW21G80nvfu3RvraR5t2rQptqVzqU2qTYli\ntA5T+hhdD43F3bt3j/Hs8jgspZTTp0/HOqXVUSptWosoNZHeLeg6f/jDH8b69u3bx3x+tLZS+iit\nCzT+01x8+umnY9tFixbFOqWsTqTU5PRcoH6jfqbkYUr2pL5LY3TNmjWxLa1nlA5OczclZFI6LK3z\nL7zwQqzXJKFS8ia9b9JcpPfWmvd5eg/5IuN24ry5SpIkSdLXjBsySZIkSWrEDZkkSZIkNeKGTJIk\nSZIacUMmSZIkSY2Me8piL9KaKJWJUvnmzJkT68PDw7GeEtIoZY3SWs6ePRvrO3fujHVKjktpRXSd\n169fj3VK66NkmnStlCZJ501JiNQXKZlnIqUp1aIxSulrKQ2I0tQoTY76nxKCUsrgpUuXYtv+/v5Y\np1QmSvA8dOhQrL/11ludGqXpURISJRtSclzqC+o3Ssiie17b/y3MnDmzU1uyZElsS2ltn332WaxT\nitexY8c6tRs3boz5/ErhdYvap76j1Dxan6dOnRrrlBxG7VOiGqWmUcpcuoelcBLuli1bOjVKtqP+\nHM+kWnqeUUIajZcPP/ww1tPatWPHjtiWrpOe55RWuWzZsk6Nkj2pn/fv3x/r9Dxfv359p3bkyJHY\n9u677451Sk2le07tU/oevUPQ2kq/OZGkZzSNW1q36NlK9/axxx6LdZpHCSUhbtiwIdaPHj0a6wcP\nHuzUaq+zdv6n5zmNrdr0RVpDae6m4/fyvdW/kEmSJElSI27IJEmSJKkRN2SSJEmS1IgbMkmSJElq\nxA2ZJEmSJDXSLAKMkklSnZKgKGWNjk2pLwklvlHKFiXbUbIjpZLdunWrU3viiSdiW7ovtSk2KYGL\nkp3I559/HuuUvpjOkdpS4mULlKZHaV10TSnFh9IUR0ZGYn1wcDDW77rrrlhP6V5pvJXCKXsrVqyI\ndZqLdF8uXLjQqZ0+fTq2pcSjlGBXCidKpYQsGuc0t2icU5piWotoDPUKzZeUEElrAqVJ1kppdbNn\nz45taawcP3481tetWxfrae2m36RULhrP8+fPj3Uau++9916nRvecxgVdJz1zkpQCWAqnjI3nGKVj\n01yk+b969epYT+OF5i2Ni927d8c6nXtaL+h9g9IEKfGW0vfS82LlypWxLaUj05yjxEcaRwMDA50a\nJZvSM5HW3BbpyzRe0vsfjQlK5KZ3S0oOpOdcOke6V3SONOYofTE9WyiRlH6TkkDpnSPNF3o+Ub3m\nfbsUTgKlsdsr/oVMkiRJkhpxQyZJkiRJjbghkyRJkqRG3JBJkiRJUiNuyCRJkiSpkXFPWaxNa0rt\nKdnk7rvvjnVKGaRElZRWQ+lLtal0p06divV58+bFekrxopQ9SiWi5KS1a9fGekIpQ3RsSuuqSauj\n36xNcKTEu/FEyVmU+rNv374xH4NSE+m+ULpTSmB67LHHYltKyKIUJ0rCoxS7lDRFiU80hiiVjdLn\nUkIWJV7RPac+qp0vvUB9UTPnaB2i66H1fOvWrbGe1jNKvKL1nI5N5576n1ITUzpcKTyeKSGMngsp\ngWzLli2xLa1blGJG53jvvfd2arQ+T6S1csmSJbFOfZeus5Q8L2g80zz/pV/6papzSelzNFboXNas\nWRPrlKab1mhK8KTkVUqTe/jhh2Od+ii9c1FqYG3KYgu0tvbivWXmzJmxTmvryZMnYz3dRxrPtObW\ntk8JxvQcpkRySvyk99yU+En3durUqbFO85bOkcYo/W6v+BcySZIkSWrEDZkkSZIkNeKGTJIkSZIa\ncUMmSZIkSY18+V/0/n/oA8b0MSV9YEkfdY+Ojsb6+fPnYz192EdhHHTe9GEjBWnQx4Rvvvlmp/bI\nI4/EtvQBI92vXbt2xXr6+JY+Aqd67UfDKeyCAhDoesYzMKEWnQuFeuzZs6dTow+yV65cGesUSEF9\nlMIRVq1aFdtSqAddD32oTx9qp49saX5S8AB9YEvXn+4XfdRcGyRUG4IxnmhtSddK10/3nAJTaM29\ncuVKp0b3loKUasfcnDlzOrX0MXopHIBD43bu3LmxnkJ6Simlv7+/U6NnC81/CgxZt25drKfnIo3b\nFmso/SaQyOYAAAAgAElEQVT1//Lly2Odnn9pztF6RmvrwYMHY52kPt28eXNsSyEt9Jyj9mm80Bii\ne0hBSrSG1oQA0TyntWK8AxN6IfVRbbgaPecojI7WufTORWsF9TOtf9RHaR2tfYbSuyKti0ePHu3U\nKBiFUOge3XOaizX7ky/y7J84b7SSJEmS9DXjhkySJEmSGnFDJkmSJEmNuCGTJEmSpEbckEmSJElS\nI81SFmtQWktKaiuFk2Mo9eTixYudGqUMpcSXUkoZGRmJdTp3Sv1Kv3v8+PHYdtGiRbFOSUiUBjR/\n/vxOLSWVlcIJOXSdNel7dH6kRYIdJepQihmdY+o7Gs80VijxjfooOXfuXKzv378/1mlsUeLjhQsX\nYj3dR5pzS5YsiXVKWqK0tpSEWpvgWZtKl47fq3FLx6GUxXR/6d5SWhvNUUpIS+sCHYOSHT/++ONY\np3Gezp2SGilNrDbZk/oipZvRfKZzobWY5mLqZzq/Fmso/SaltdG6eObMmTEfn+55evaVwuvi5cuX\nYz2Nl40bN8a2lCaXUnBL4WdLOvdt27bFtocOHYr1hx56KNYplY7uY7rn1G+U1EprMc25Fgmh6Vxo\nPaM1h66f+pnerWqSqmmtpHOkenqe03mn9+pSuD9pzqUkRDo/Sk2kRNZaNEZ7xb+QSZIkSVIjbsgk\nSZIkqRE3ZJIkSZLUiBsySZIkSWrEDZkkSZIkNTIpUhYplSUlWJXCSTOUVrV8+fJOjZKgKJWIEoUo\nleXw4cNjPk5KhyuFE6IWLlwY65SclhJorl27FttSamJtWl1K/aF+Ji0SwmpRAtGmTZvGfAzqTxrP\nNBZTAhGNCUrTorQiSnFau3ZtrKfrpwRTSp+r7f90HDpGrxI/xzuVKaE5l1LsKKmSEvyofc3cpXtF\nc+Xs2bNV9TT+qe2tW7dineYFjUVao9etWxfrCaVJUvoo3fO0RteOz/FcWynBktYWSg2l+7V9+/ZO\njfqTUmDpXI4dOxbr6fhXrlyJbSnx8I033oj1zZs3x3rqI0qY/d73vjfmY5RSyvnz52Od5lE6Tk3y\nailtxmKtdI70rKB3KFqf6TqpfU3iY+250DqfxjnNT0pZXL16ddVvpmRT+k26HlrnaS1q9Tz3L2SS\nJEmS1IgbMkmSJElqxA2ZJEmSJDXihkySJEmSGnFDJkmSJEmNTIqURULJbpS0QsksKYGGEl+WLVsW\n65SydfXq1VinlL2UBjNjxozYlhJfKAmM0o1GRkY6NUqwIpTsV5uoNFnRdVKKT0rDorZUpxSvmuRE\nSh+iuUVziFByXhovlOxI6J5T0lRtimdSO25bJITROab7QveQUtYolY7Wy7Qu0vik/qHER0ofvH79\n+pjPj9YtmnNLly6NdUrlS/V0fqVwamptWl26L70Y+7Vo7NO97e/vj3XqO1r/Uvrw8ePHY9uPPvoo\n1tMzsZSc+FZKXrt27NgR29Lz/Nlnn411GlsJpSzu3Lkz1impetGiRbGeknpL4XmUTIY1tAZdDz37\nqP9p/lM9PaPpeU6J3LSG0jtkWkcowZHecem+0G+m+zs8PFx1DFor6H7VjNFejk//QiZJkiRJjbgh\nkyRJkqRG3JBJkiRJUiNuyCRJkiSpETdkkiRJktTIpE5ZJJSQcuPGjVhPaTWUjnjx4sWq36R0K0qC\nu+uuuzo1SqWiY1C95hxrEywpaYau/6uWskhq+ojuIaWyETpOSiCitlSnlEW6ThpHaVzQMUjt2Pq6\njDmS7m+v0vdojJ47d65To36oTeWjc09rFP0mjWc6FzoOPS9SKl9tKlcv1tCJlFRXe28pCY7S2vr6\n+jq1zZs3V50L1WvWqNpxS9dD9yWlbNJ503Ob0vfoOJSm+HVfWxPqZ7qHlHhZk8pKiZyUJkhozqU6\nnR9dPyUh0rtCSkikY9A7fu07MRnvddS/kEmSJElSI27IJEmSJKkRN2SSJEmS1IgbMkmSJElq5GsV\n6lEbvJHQB5m9kj6+rP2QcDxDDeheTaSPxieSmnvei/F5O+kj8NqABdKLsdXiN7/uenUP6aPpXqwL\ntWEvXxeTNdSjVm3AENV7IYVu9UptH43nuRiMNHZ0Tyikguo0bkdGRmJ9cHCwU6PwFjrH2vZpzNUG\nZrR4D50s49a/kEmSJElSI27IJEmSJKkRN2SSJEmS1IgbMkmSJElqxA2ZJEmSJDVyx3inj/zPZIk3\n0VfeZE/g0Veb41MTnWNUE5njUxPcx3fcccdM+of+hUySJEmSGnFDJkmSJEmNuCGTJEmSpEbckEmS\nJElSI27IJEmSJKmRO8f7B/75n/95vH9CGpNXXnkl1v/sz/7sSz6TiY9SqT777LNYv/POcV9KvvK+\n//3vx/rv/d7vfclnoq87Sqv7oz/6o1j/y7/8y/E8HWlMvve978X6X/zFX3zJZyJlr732Gv4z/0Im\nSZIkSY24IZMkSZKkRtyQSZIkSVIjbsgkSZIkqRE3ZJIkSZLUiNFoUgMpxfCTTz6JbalOSYh33XVX\nrKeExM8//zy2/fTTT2Od0tfuvvvuWP/GN/K/8/nmN7855nOh36xtT/WEUiNrjiFNVrS2SNL/i57x\n9KyktYWe518n/oVMkiRJkhpxQyZJkiRJjbghkyRJkqRG3JBJkiRJUiNuyCRJkiSpEVMWK1FyDCXN\n1NbT8aktpdLcuHEj1lPKXu1vUvocJefQb37V1KaSpfbUn9evX4916mc6l7Nnz3ZqNJ4pZbG2n2fO\nnBnraRzR9demL82aNSvWp06d2qlROmRqW0op06ZNi/XaRCl1tUiwrO03+/ProRfPcxpbtQmuNe8c\nlMh78+bNWE9pt6XUr//Oiy9fGhe1Y6gXicS1KYs176Gl1L0rTXb+hUySJEmSGnFDJkmSJEmNuCGT\nJEmSpEbckEmSJElSI1/JUA/6yJA+PO3Fx5H0oS6FBtCHjencKbzh2rVrsV77oe6VK1fG3JaOXXv9\nU6ZMGfOxqd9qPybthV6Ed1D91q1bsS19wEr3i8ZFjdmzZ8d6GiulcP9TCEYKzaAP0k+fPh3rFHZC\n9yUdn+4tnfeMGTNiff78+WP+TRq3kxnNudQXtWso1anvauY/zc/agJma4/QqvEZZTUhVbf2uu+6K\n9bSe0VpBx6BnJa1z6TpHR0djWxqffX19sX758uVYT8FQpeRronU4PftL4XeOr4tehLrUvp/VjGdq\nT79J77j0Pnv16tVYT9dPx6YxNFnG1lfvzUCSJEmSJgk3ZJIkSZLUiBsySZIkSWrEDZkkSZIkNeKG\nTJIkSZIamdQpi5TuQgkxNUlglJBESViUEEN1Sre55557OjVKtrtw4UKsU4oRJdNcunSpU+vv749t\nFy5cGOuDg4OxTol/6dypfygJKt2r2x2nF+jYtWltKfWIxhylb1FaEfVR6udNmzbFtrNmzYr1oaGh\nWCfUR/v27evU6F7de++9sX7q1KlYpzmX5hGlklFqItVrEv8mQ5per1Jm0zpH6zPVCf1mOg6tiYT6\niJJQKQkv1W/evFl1bDoXSg6rTYKd6MZzzNE9p5RB6udz5851akeOHIltKcH28OHDsU7XP3fu3E5t\nzZo1sS2Nf1qfP/7441hfsWJFrC9durRTo3FOqbk1Kdil8PvfRFG7htL10PM/9V1NqnUpnCZM9Zkz\nZ3Zq06dPj20pqZTeZ0dGRmI9zVEan5TUTO3p/bRVEq5/IZMkSZKkRtyQSZIkSVIjbsgkSZIkqRE3\nZJIkSZLUiBsySZIkSWpkUqQs1qRpfZF6Sh+k9DVKX6LEO0q3oUSdlAZDqUxz5syJdUqfo6SZdI4p\nTaeUUk6cODHmY5SSk/1KyUk7lOxD94p+czxTFgklClGyZUrrozFE6VOUsllzHzdv3hzbUprihg0b\nYp36mebuY4891qnR2KLxTOOCfjONCxrnlKZ08eLFWK9JNp3o6WClcIIfzS26X+n66dg0h2j9o75L\n6XOExkrtmkPpXmnuUsoYJdXWPnNSuh3d84mE+p+S4yjdLaUs0hpKSYA0zx955JFY/7u/+7tOjRIc\nUyJjKZy8vGrVqlhPx9+2bVts+93vfjfWU9ptKZy+SOeY+mjRokWxLY3FQ4cOxTrNfxovXzZaK+he\n0XsorWe05qTjU8Lwf/3Xf1WdCx3niSee6NQoHXrdunVVv0n3K50LjU9636LnE73n07tVeofu5do6\nMUa0JEmSJH0NuSGTJEmSpEbckEmSJElSI27IJEmSJKkRN2SSJEmS1EizlEVKPUoofYUSsujYlISV\nEpUokZCOQUlQdC6U4rRw4cJO7eDBg7EtpS+lY5TC9zGd4+HDh2NbSvCjxEdKDkqpfLt3745t169f\nH+uU1kPnMp5q09oSSpOiBMN58+ZVncv999/fqVGCW6+S8CjFKPURpePdd999sX79+vVYp7S6kZGR\nTo2uh+Y5pTjR3Er9T8eoRWtLzdpKY+XatWuxTusi/Wa6VlpDjh8/HuuUStjX1xfra9as6dRqky0p\nfY/6jpLG0hpFz4o0Pm+H7mNSO85rxlCv1KbS0X2cNWtWp0bjmdYQerbSuvj44493apR4SP22dOnS\nWP+bv/mbWH/55Zc7NbqeFStWxDql723cuDHWz549G+spCe/YsWOx7ZIlS2KdnkX0PtOrdbRGSpml\n/qTrpIRAGuc0dtNzjtbK2hRoSt88cOBAp5YSo0sp5fTp07FO7y10Hx944IFOjd7xBgYGYn3BggWx\nTsehtSjdc0qk/CLpi/6FTJIkSZIacUMmSZIkSY24IZMkSZKkRtyQSZIkSVIjbsgkSZIkqZFmKYsp\nrYbqlJBFCSmff/55rFMCUUrUOnXqVGxLqYEXLlyIdUrOoRSXdO6UPkNpPYcOHRrzsUvJCXmUhEOJ\nMkNDQ7FOUnLQ8PBwbEv9T4k6NLbGE53jrVu3Yj2NORpbU6ZMiXVKdqP2CaV90tyiY1OaIo3/NOYo\n8YnGOaWs0X1JY5f6jeYKpY/SOab7QsfuFbqPaSzSOkTznFLWbty4EespIS0ldZXCqVQ0L2jspuuk\nlDkat7SeHTlyJNZpnK9bt27Mv0mpbDTn6LmQ5kVt8vB4ovWZ5kVt4mWa09RvlOy7cuXKWKf0ueee\ne65TW716dWxL7wqUbPjRRx/Feur/3/iN34ht//3f/z3Wn3766Vin1FxKZdy8eXOn9uGHH8a2lL6Y\nUoBL4ZRBWv9r0FikVMI0z/v7+2PbRYsWxTrNRTqXc+fOxXpKvKVkQ3oPOXr0aKxv37491k+ePNmp\n0bpN6Zg05iitMT1z6f2EUoOnT58e6zXvJ6XkPQTdW1MWJUmSJGkScUMmSZIkSY24IZMkSZKkRtyQ\nSZIkSVIjbsgkSZIkqZFmKYs1KPGGks0oTZFSeWbNmtWpUSoLJV5REhilj82dOzfWU+oZJYTR9VNa\nF6VYpbQeSp+jhCBKt6EEmpTWdvDgwdiWkuAofasFureUBJauie4hpTjRb1K6T+o7GrfUz5S+R4mX\ndPyE7tXy5ctjneYW3ZeUbkbzmfqCku0uXboU6+k+joyMxLaE+oL6mebc5cuXx3xsOgal8lESWOpT\nurdLly6NdUpfpeS8rVu3dmqUVEfHphQvSp+lsZieRXQPaSzSOkdrNKWMJi0SaWnM0T2k66S1aNu2\nbZ0apc/Rs5LSFPv6+mI9JYfu2LEjtqXUwGXLlsX6r/7qr8b63//933dqa9asiW0JPXNpPae5u3Pn\nzk6N5hytLSmRtJS8bpVSN84JXQ+9K6b+p/cwep7VJnLTfVy8eHGnRsmOa9eujfVnnnkm1umdc9eu\nXZ0aPePpHZeOfeXKlVhP7z90byk1lc4lvfuXUp+E2Sv+hUySJEmSGnFDJkmSJEmNuCGTJEmSpEbc\nkEmSJElSIxMu1CN9TEcf0tHH0fSx++joaKynjyzpA0P62JM+1D98+HCs0wfsK1euHHNb+vB89uzZ\nsU4fx6aACTo2fQS5Z8+eWKcPmNOH6sePH49tKWCFPg6mDzJ7ofajTgokSddE550CUG5nwYIFsZ6O\nTx8104f0NLYoSIfGS/oQuPYjbfqwl+Z/+miaPsimcUv3i64zrTm9Gp90/VRPwSN0LvSBOa0L9KF2\nClihe0jHpiAVGqMJhTTQsY8ePRrrFDBFYR+Dg4Od2vz582PbY8eOxTrNOQqYSOOfnpW1YUS9ULuG\nptCpUko5f/58rKc5R4Ep9E5A6xndxxS8sHHjxtiWxhwFMjz00EOxnp4hKdCklFIefPDBWH/jjTdi\n/eLFi7FO7z8p1IfWlhdeeCHWaS2m49B6UYPGYs1zkY5x5syZWP/www9jndZcGi9pntO6ReOf3q0o\nqCONUXpXfvLJJ2Od5hDNuXQu9C6bAp1K4T6isUVBSuncexn04V/IJEmSJKkRN2SSJEmS1IgbMkmS\nJElqxA2ZJEmSJDXihkySJEmSGplwKYsJJUFR0golpFASYkofo8SXU6dOxTq1pyQwSjFKiVLp/Eop\nZc2aNbFOiV+U7pUSeCipjhJ/1q9fH+uUhJXSJ+leUb9RyiIlofUCJerQuVNyUkrrofSh4eHhWKeE\nIJovaV6sWrUqtqWUqZMnT8Z6f39/rFOKU7qPlNZE10n9f/Xq1VhP10QJbpRsSf1M8yL1Ra9Smeg4\ndF9Sct6NGzeqfpMSX6mf586d26nRekbrNqXPUcrg+++/36lROlxKQSyF+//ee++NdZq7aS2iJDQa\nW7TO0zqX1vPafm6xhtJauXfv3livWedobFH6XEpNvN1vpj6ilL0XX3wx1nfv3h3r9NxOyaZ0b//h\nH/4h1mn8032pSYilZEe6/oGBgVin+UJJuDVqx3l6L6LnDSW7Hjp0KNYXLlwY6zWpufQ8O3DgQKxv\n2LAh1l966aVYT0mYNG/pHZLQe2t6V6a18siRI7FOydP0bKF07NQXNCe+yHPev5BJkiRJUiNuyCRJ\nkiSpETdkkiRJktSIGzJJkiRJasQNmSRJkiQ1MilSFinFhBJlKH2H0upSAhelclGyF6WyjI6Oxjol\n85w+fbpTW7ZsWWxL94WSDSkJMknpaKVwKt/q1atjnZJmUrrPpk2bYltKq6JUovFEKVt0nZSQlFLc\n6Njnzp2L9ZkzZ8b6zp07Y52ShhJKAaW0MkplonmR0kop8YoS7Citj9JXU0IkrRUnTpyIdUoOo7Sq\ndP01SWW3Q2OO0q3SfaT1idYcSl+j48yYMaNTo/FM6ZAp2asUXnPWrVvXqdEYojRFWv8orZDmy1NP\nPdWpUUIYjX/q59r6T9q2V2ido/6n+ZKelaXk+7t9+/bYtraf09gqJc8XSqT97d/+7Vinefvnf/7n\nsf7cc891apTI+O1vfzvWly9fHuu16/+//Mu/dGqUjjw0NBTrP/dzPxfrNEYXL14c6zXo+UzPrTQW\nqZ8pqbM2TfjYsWOx/sADD3Rq9K5E95DWeXr+pfdWSsGk9/Ndu3bFOiXYpvvy5ptvxrY0P++///5Y\np3TwlStXxnp6jtC8/SLPef9CJkmSJEmNuCGTJEmSpEbckEmSJElSI27IJEmSJKkRN2SSJEmS1Mik\nSFmkZB+qU1oTpb7MmjWrUxsYGIhtKZVp7969sU7pXpR6k5Lz5syZE9tSWhUl59BvpuSc2jSxefPm\nxTolpD399NOd2saNG2Pbt99+O9ap/1skh1GiDiWnpXQ/Sjai66SxRQmeaWxR+hgde+nSpbFO93z/\n/v2xvm/fvk6N5tb06dOr6pQmmhKoKCGKUpzovixZsiTWe5WoWHNsqqf1gtpSyhilz1Ky6wcffNCp\nURJWSmQshdNEX3755VhP62VKgSullL/927+N9T179sQ6XX9/f/+Y63Q9NJ4pCZTSGtPaTccgLdZQ\nej7TujA4OBjrKQn5kUceiW0ffvjhWKcEx5//+Z+P9ZQc+tZbb8W2hJ5/hw8fjvX0XKB+o/F54MCB\nWKeERGqfUP/Q+Kf3Ger/O+/8yV9fa5NK09x69913Y1tKO6akbnpXWrFiRawfOXKkU6P3DUoBpvGf\n1u1ScuLjhQsXYlt6VtL7OaWgp3tO75uUvEzPJ3rO0TOK7m+v+BcySZIkSWrEDZkkSZIkNeKGTJIk\nSZIacUMmSZIkSY24IZMkSZKkRiZFyiKlslCdUuYWLVoU6ymVjZLtKAmK0soorZCS4FLiI6XVUFrP\nlClTYn10dHTMdTo/SqWh66RzSclJn376aWxL6WukRUIYqUmlO3ToUGy7fv36WKf0TRq7KQmM+odQ\nuhElalH6aOojStOi66EkNErrGhoaGlOtlFJOnToV6ympsRRei9K5UDper0ydOjXWU99Rsh/NIeqL\n5cuXx3oaozQnaP4/++yzsX7//ffHejr3adOmxbb33ntvrM+fPz/WFyxYEOuUvpV+lxLs6BlC6x8l\nJ9LzL6GkuvFcQ+k6adxSgimtRQkl2NGaQ8emZ+6PfvSjTo2S6ijx8LXXXov1H/zgB7H+zDPPdGq/\n9mu/Ftt++OGHsU7zn8bzb/7mb8Z6Shp8/vnnY1uaW1Snc+xFgi09K+j9J73n7Nq1K7al8bx27dpY\np7RCWrtSKiddz8qVK2Od7iGlD6d3UVpDKKmW3sPpXSQ9i1evXh3bElrnn3jiiVin+Z+e81euXKk6\nl9vxL2SSJEmS1IgbMkmSJElqxA2ZJEmSJDXihkySJEmSGhn3UA/6aLAXHw3Th6cLFy6Mdfr4MH1M\nnkIXSuEQAPoIdOvWrbE+PDwc67Nnz+7U6ENy+giePjKk4I3UR/SxM314eunSpVinEJT00SR9kEr9\nTHU6znii36TgjXTu9PE+fRxMY4j6KH2QeuLEidh2y5YtsU4fux4/fjzWKcAgXX8a+6XwOCc0R9N9\npICNNWvWxDqtLZcvX471dPza6yF07vRxdC8CNmiMUj2FANHaQoFBNObo2ZICZmhNpMAIOjaFvbz/\n/vuxnp4LGzZsiG0pGIXW1pkzZ8b60qVLOzV69rVAz366HgoBWLZsWayn8VKzDpfCgSnUz//xH//R\nqVEADq1Pb731VqxTIEMKtaHwjoGBgVinvnjyySdjnebFU0891alRiBg9nyjUajyf85999lmsU5Ba\nuqZVq1bFtmkelsJr0eLFi2Od5kX6XepPeibQOwSt0WnO7dixI7al8BoKb6F5vmnTpk6N3ivpvO+7\n775Yp7lF7/MpMKlXz/NS/AuZJEmSJDXjhkySJEmSGnFDJkmSJEmNuCGTJEmSpEbckEmSJElSIxMn\neuk2KE2M0ncorY0SdaZPn96pUcoOpa9QQg6lwdC5pISoe+65J7al66d0p5rkMErCoWNQyhwlBB47\ndmzMbSkhqBdJnb1CaXWUkJQSiM6cORPbUuInjVH6zZMnT3ZqlL5GqXmp30op5fTp07E+bdq0WF+7\ndm2ndvjw4diWxkVKPCqFU+nSuWzcuDG2pcQ7So6ipKV07rXjltrTukhjMaF+7uvri3W653SclMpF\n85nWREJpdWkcUVItzS1Kpfvrv/7rWKd1MR2HUuYofS09n0rhFLfU/5RgSc+KFmicU52euWnO0XWm\n5NlSSjl69Gis03r2O7/zO53aX/3VX8W2H3zwQaz/53/+55iPXUoeF3R+9K6QkhpL4cQ7WnPoPSeh\ndEQa5zS3qO9q0LFJmlv0rKBnKyX+0fOc1qiUKHjkyJHYluYQPc8efvjhWJ81a1anRu8t+/bti3V6\nh6Y03fQson5bvXp1rNNaQfeFnkWpPfVz7fOsFP9CJkmSJEnNuCGTJEmSpEbckEmSJElSI27IJEmS\nJKkRN2SSJEmS1MikSFmkVB5KSKG0GkogSkkzlGy4fv36WKf0MTqXkZGRWE8pNpQ+RMmG1J7OMSXQ\nUBIYJRtRchqlL6XjUFIdpdjUJiSNJ0qxojGaxiIdg1IDKdkupSmWkhOYBgcHY9tXX3011m/cuBHr\ntWmNaX5RsueOHTtine7LunXrYj2NORqfNLboeih9LP1mr9JB6TjUFykhjNYnGouU1peSWkvJazcl\nHq5ZsybWaW3ZtWtXrJ89e7ZTo/6h86b78sADD8Q6pdKlMUr9RmOLEh9pvqQ++iKJX+OlNk2xNmV3\neHi4U6N1bvfu3bFOzzkauwcPHuzU9uzZE9t+5zvfifUlS5bEOs25d955p1P7mZ/5mdi2Nn2u9p0r\nreeUvkfHoHt77ty5WB9PdP1pHUlph6XUPxPpmZPGcyn5Pl67di22pec2JURSUnN6b6V0RBrPlIJO\nv/nMM890ajQ/6dh0b+m5UPOM7uXa6l/IJEmSJKkRN2SSJEmS1IgbMkmSJElqxA2ZJEmSJDXihkyS\nJEmSGhn3lMXaRLGU7kWJKjdv3oz1EydOxDol0KRkIkpZpPStlI5YCqf1UKJYOv6VK1eqzuXw4cOx\nTimLFy9e7NQoIYfOhRJyKIEo9RGdHx17IqG0Hkq8XLhwYadGCZ7vv/9+rNcm5KXEN0q2pKRGSiql\n61y5cmWsp+QwSs1L47MUTkik9MX0m7S2UH/WzvOUwNQqHTT1Ed1Dup40bkvhtLa0RtcmXqU0uVJK\n2bdvX6yn5ET6TUrIonXr8ccfj3V6tqxdu7ZTo+dWbZog1dPxaZzTfaE1ZDxRKh3Vqe+uX7/eqVGa\nIiUhUrIdpRW+/vrrnRqlINeMlVJK2bx585h/k9YneobSHKJzpPU/tad7SM+tFStWxPr58+djfTzV\npElSvxGai/v37491eramdY7WEOq306dPxzqtxSk58+23345t77///linlMmUdl5Kfuek/qH1jMY/\ntaf3opRWSc/KL8K/kEmSJElSI27IJEmSJKkRN2SSJEmS1IgbMkmSJElqxA2ZJEmSJDUy7imLtVJi\nCaUpUYoR1Y8dOxbrKbHl0UcfjW0pIYZSqSiBhZJ2UjIRJRvWJs1QctLSpUs7NUo8JHSdlHiX0mpS\nCl4p3P+t0upq0H0ZGBjo1GjcUuIhpVVRQtKWLVs6taGhodj2Bz/4Qaz/4i/+YqwvW7Ys1inxM9Vp\nDgg1ljcAAA3sSURBVE2dOjXWKZWSEqVGR0c7NUqworFF50LzJR2nNnmW0DlSPY1Fuh5KmU1rxe2k\ntCrqn5SOV0opZ8+ejXU6ThpHlD62YMGCWF+1alWs03yZM2dOrKeEQJqfhJ45d999d6ynVNILFy7E\nthNpDaWEwL6+vlinFL+0XtKzhe5tSpMrhZNgU9Leq6++GtvSWKHkZUpC/da3vtWp0bz96KOPYj2t\niaXw/aI5l+4LPftoztH7CSXejSc69/S8oERW6gsaW5TWmN6VSsn3hY5N69/LL78c6/QucujQoU5t\nw4YNse1jjz0W6/Q+u2jRolhP7+00bintl9Y5eg+nc0zjopdrqH8hkyRJkqRG3JBJkiRJUiNuyCRJ\nkiSpETdkkiRJktRIs1AP+hAuBRVQeAF97EgfMM6fP3/Mv/mjH/0otqWP2tPH26WUcvPmzVifOXNm\nrKcPNekDW/o4nD5Upt8cHBwc82/Sx650nVSv+fCe1IaajCf68JjGaBov9BEwBWZQCMKRI0difWRk\npFPbuXNnbLtw4cJYJ/v37491mhdpfFFIAX3sS0Ea9BF8+oCX5gr9JgWP0Mfu6RwnUpAC9Q/VaS2m\noII0RukY1J90bOrn1Kc0D5csWRLrFPZCc5HmSwqeoLYUsEJrC33An66Vxm2LsUjrNs0h+vB+z549\nsZ7uC91zClKgMCp6hn73u9/t1GgdpjCiefPmxfrp06djPYVgvP/++7EthRTQvKBwhPPnz8f6vn37\nOjUatxT2UBsk1kK6JnoO0xpKz3maixQakuo0nmncUmAajYvvfOc7nRqF8dBYqV1zU5ASvSsQ6qMU\ngFRKKefOnYv1dL96+R7qX8gkSZIkqRE3ZJIkSZLUiBsySZIkSWrEDZkkSZIkNeKGTJIkSZIaaZay\nSFIaFKUJzp49O9Yp3aW/vz/WUwLT0aNHY1tKa1mwYEGspySkUjhpJ6USUooNJRhS+halvqR0K2pL\nKZN0zylpamBgYEznUQonO06ktDo6R0orSmOXxif1J6U40XxJ7V988cXYdu/evbH+0Ucfxfrhw4dj\nfevWrbG+YcOGTo3mxKlTp2J9+fLlsU73PKWYUZoipYxSKhUlhKWEuF6NW0p3okS91J7WUJqLtUmo\nKVGR0hEpOYvGMyXHpTW3tj8pfZPGaF9fX6yn+0hpYrWJZ7T+03ESGkPjiX6T1jNCfZFSCf/7v/87\ntv3Wt74V608++WSs0/MsPaNp3NJaSePivffei/Vf+ZVf6dS2b98e246OjsY69cW7774b6zTnUl/Q\nWkFrDqVpjqdejH9az+iZQGvLihUrYp3WrjReKKmVjkHPipq5WPtMpGcLPRfTexElplOy5+XLl2M9\nJTiWwsmm6b7UJj7ejn8hkyRJkqRG3JBJkiRJUiNuyCRJkiSpETdkkiRJktSIGzJJkiRJaqRZymJN\n0hIlwYyMjMR6Sk0shVN8UkoOpdVQogqlL1LSDJ17ui+UMkfnQulTvUAJURcuXIh1uo8J9fNESlMk\nNLZoXMycObNTo+QgSgL84IMPYp3SvVLKIKVvDQ8PxzolSlFyFCWEpuPUpklSchTdx8WLF3dqlBpG\nc4uS7SjdqcXYpfuV5hetw3SdtUlog4ODnRqNT7pXly5dinW6zvvuu2/Mv0nr8KFDh2Kdkj3p+On+\n0jOBklrpntckxE2GNZTWIlpD5syZE+snT57s1Cjxb9myZbFO698LL7wQ62ntSuOwlFJ++MMfxjqh\n9T8lQadU01L4nYieTzTnaJ1P6+XGjRtjW3o/oSS8mtTQWjQvqJ7mKCVp07sPHZvSVGldOHv2bKdG\n/UP9f/HixVinc0x9QWOOksfp+Uz9nMYRPZ/oPZTWeWpPfZHWHFqHv8ia61/IJEmSJKkRN2SSJEmS\n1IgbMkmSJElqxA2ZJEmSJDXihkySJEmSGmmWskhSYgml71Fy0ty5c2N9+vTpsZ7SXSiVhdBvUnLM\n0NBQrKfkOErrmT17dqxTchylG6VrpdQ4OkZKDSyFk5OoTycrShqitKqUykbJfitXrox1Sis6fvx4\nrKdULkpluvfee2OdEt/oXFISVCk5rY7S1Pr7+2OdxhAlMKVkR/pNSrai+UztW6BzSXVa56ifKZWL\n6mlM0z2fNm1arKd00FI4lS+NRUqqPXfuXKzTOdJ6Ruee7iOt2zRuKcWtJgluIqHzpucW3a8VK1bE\n+ksvvdSpUTow9dvDDz8c67t37471tLbSHKJnxTvvvBPra9eujfWE3hVoDNWu/3QfV61a1anR+xbN\nIbpfEykhNKVSUtprbdo1JV6msVVKHrspYbQUHrc05wYGBmI99d39998f29L10LOCxsuSJUs6tQ8/\n/DC2pfWc1n9KjaYE17Q/6eV6+9V6K5YkSZKkScQNmSRJkiQ14oZMkiRJkhpxQyZJkiRJjbghkyRJ\nkqRGJlzKYkrUoZQdSqWj5BRqn5KJKPGH0gcpTS+lJpbCKS4pUY6SYyitieqUHJZS/Og3a1IDb/eb\nyURKU6pF53716tVYT+OLUokoZZDS56j++OOPd2qUVDg8PFxVr00aSuOf7iH9Jt0vmucplYr6h+YQ\nzf/JMHbTNVH6GqVvUUIaJR4++uijnRrdc0oro/V8wYIFsX7hwoUxH4PQuk3nfv78+Vjv6+vr1GgM\n0TOH+ojmXJrTKR2slDbjls6b7gvNxUWLFsX6r//6r3dqlFT3j//4j7FOfZH6s5RSzpw5M+bfpKRC\nSqv713/911hPY+6VV16JbWn807igNZTmXEoUpHcISsemcdFi7NKx01ikdYvGM70rUfL2vn37Yv3J\nJ5/s1GjcUvomrXM195zeCendgtZQcvDgwTHVSqlP06X5TPV0TaYsSpIkSdJXgBsySZIkSWrEDZkk\nSZIkNeKGTJIkSZIacUMmSZIkSY1M6pTF6dOnxzql2FDqCyWHJYODg7FOaTWUwHPjxo1YT8lJdGxC\nCY6nT5+O9XRNtak8dM8p9Yfuy1cNJS199tlnnRoltVEqEfXz7NmzYz0lYdEcojStgYGBWKekITr3\nlOJHqYk111MKz63UF7VJpb1MVBovdI7p/tI9p7Xi4sWLsU6pbDt37uzUaGzR+Ke1gtbzs2fPdmo0\nVlLyZimcHJaOXUopN2/ejPX0bKHrqR1baQ0phRPSJjq6L/TMobU1PbsoeZbGRUpNLIXXhSNHjnRq\ntPb99E//dKxT/9MzdPfu3Z0a9T2l0tG4femll2J9xYoVsZ7WaDoXWp/pPW8iJYGmMUprBSXV0rp1\n/PjxWKeEzDTmaN1ev359rC9dujTWaV6k9nv37o1tqd8ofZTWs5SQSMmTtJ5TvTYFnfou+SLvCv6F\nTJIkSZIacUMmSZIkSY24IZMkSZKkRtyQSZIkSVIjkyJZgT4OrP2omdqnDxjpYz/6UI8+9r127Vqs\nU5hCOsdZs2bFtvQx6cjISKzTNc2dO3fMbalOHwdT2MfXXRrTNIbo43X62PvChQtjPg+aKzTnKLyB\n+rlm7tYGadB8rq0nLT4k7xW6X6lOY2vhwoWx3t/fH+ujo6OxnkIwTpw4Edv29fXFevqou5RSpkyZ\nEuvp3A8cOBDbUtgDhRrRnEsf2JeSA0zoHtaGGtCcS6EJk2E807OSQreOHTsW62lNo3XrlVdeGfMx\nSillaGgo1lNQBwUA0LPyvffei3UKWHjggQc6NRq3v/ALvxDrtFbQ+wmNoxT2U7tuT4bApHTutCYM\nDw/HOoV0UKgR3fP0vKRnP51L7bty+k0aKxSAR+Of6ikc59SpU7EthZesW7cu1tO77+2k52UvQ5T8\nC5kkSZIkNeKGTJIkSZIacUMmSZIkSY24IZMkSZKkRtyQSZIkSVIjEy5lMSWWUFpRrZoUH/rN+fPn\nxzolxKWEmFI4OSclKt15Z+4mSl+i9C1qn45fm/hH6YuUbtXLZJqvitqUtZSmVqu2H2p/sxfJWbUJ\ncZMhUW48UZ+mdaE2ZYrm+fXr12N9zpw5nRqljJGBgYFYp3UxpZUtWbIktqUEP0r8ot+kxMd0X+iZ\nQHOF+pPW1sk6/um8KWWT+i6NUXom0nOe+nnatGmxnsYLHYM8/PDDsU7nnsYWzS16ntP101ik53+q\n16TaThbpvlA6KN1b6gt6P6N5kRJsaW2hekrHLIWv6fDhw53a888/H9vS9VCd7svmzZs7tS1btsS2\nM2bMiHVK8KV1u1Uis38hkyRJkqRG3JBJkiRJUiNuyCRJkiSpETdkkiRJktSIGzJJkiRJamTCpSwm\ntUlwlHpCySnjmQZEiTIkXStdDyU+UXtKSOrF9dcmYZqy+JPrRbpP7TF6kZqoNmrWFnLr1q0xH7uU\nUmbPnt2pzZo1K7alsVWbBJeSs2iNowRHQverJsWvV6mhdE1ftbW19rndi/RZUjMWJ0M/TNZEzlZS\nn06fPj22pfqCBQtind7PaMyleu3zmX4zrdul5LlFa19tsinN23Sc2ndWGucTbY76FzJJkiRJasQN\nmSRJkiQ14oZMkiRJkhpxQyZJkiRJjXwZoR47voTfkCRJkqSJ6Nrt/uEdpuxIkiRJUhv+J4uSJEmS\n1IgbMkmSJElqxA2ZJEmSJDXihkySJEmSGnFDJkmSJEmNuCGTJEmSpEbckEmSJElSI27IJEmSJKkR\nN2SSJEmS1IgbMkmSJElqxA2ZJEmSJDXihkySJEmSGnFDJkmSJEmNuCGTJEmSpEbckEmSJElSI27I\nJEmSJKkRN2SSJEmS1IgbMkmSJElqxA2ZJEmSJDXihkySJEmSGvk/Xv4WTUJQZe4AAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fa7f1e13978>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from convnet.utils import tile_raster_images\n",
    "\n",
    "plt.clf()\n",
    "\n",
    "# Increase the size of the figure\n",
    "plt.gcf().set_size_inches(15, 10)\n",
    "\n",
    "plot_data = tile_raster_images(W.get_value(borrow=True).T,\n",
    "                               img_shape=(28, 28), tile_shape=(2, 5), tile_spacing=(1, 1))\n",
    "plt.imshow(plot_data, cmap='Greys', interpolation='none')\n",
    "plt.axis('off')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}