From a9c52a7ea3170d26fac1b8a81b1579cbefc217d3 Mon Sep 17 00:00:00 2001 From: Lokesh Baviskar Date: Mon, 17 Apr 2023 12:57:30 +0530 Subject: [PATCH] Created using Colaboratory --- 01_pytorch_workflow.ipynb | 2418 +++++++++++++++++++++++++++++++++++++ 1 file changed, 2418 insertions(+) create mode 100644 01_pytorch_workflow.ipynb diff --git a/01_pytorch_workflow.ipynb b/01_pytorch_workflow.ipynb new file mode 100644 index 0000000..3c1c71a --- /dev/null +++ b/01_pytorch_workflow.ipynb @@ -0,0 +1,2418 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "view-in-github", + "colab_type": "text" + }, + "source": [ + "\"Open" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OgYkrRCRec0r" + }, + "source": [ + "# 01. PyTorch Workflow Fundamentals\n", + "\n", + "The essence of machine learning and deep learning is to take some data from the past, build an algorithm (like a neural network) to discover patterns in it and use the discoverd patterns to predict the future.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "51Ug7Ug123Ip" + }, + "source": [ + "## What we're going to cover\n", + "\n", + "In this module we're going to cover a standard PyTorch workflow (it can be chopped and changed as necessary but it covers the main outline of steps).\n", + "\n", + "\"a\n", + "\n", + "For now, we'll use this workflow to predict a simple straight line but the workflow steps can be repeated and changed depending on the problem you're working on.\n", + "\n", + "Specifically, we're going to cover:\n", + "\n", + "| **Topic** | **Contents** |\n", + "| ----- | ----- |\n", + "| **1. Getting data ready** | Data can be almost anything but to get started we're going to create a simple straight line |\n", + "| **2. Building a model** | Here we'll create a model to learn patterns in the data, we'll also choose a **loss function**, **optimizer** and build a **training loop**. | \n", + "| **3. Fitting the model to data (training)** | We've got data and a model, now let's let the model (try to) find patterns in the (**training**) data. |\n", + "| **4. Making predictions and evaluating a model (inference)** | Our model's found patterns in the data, let's compare its findings to the actual (**testing**) data. |\n", + "| **5. Saving and loading a model** | You may want to use your model elsewhere, or come back to it later, here we'll cover that. |\n", + "| **6. Putting it all together** | Let's take all of the above and combine it. |" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "L9EOt5cbod6l" + }, + "source": [ + "And now let's import what we'll need for this module.\n", + "\n", + "We're going to get `torch`, `torch.nn` (`nn` stands for neural network and this package contains the building blocks for creating neural networks in PyTorch) and `matplotlib`." + ] + }, + { + "cell_type": "code", + "source": [ + "import torch\n", + "from torch import nn # nn contains all PyTorch's building blocks for neural networks\n", + "import matplotlib.pyplot as plt\n", + "# Check PyTorch Version\n", + "torch.__version__" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 35 + }, + "id": "irwAzjKh40kb", + "outputId": "101d60a8-f1c9-4727-a0f0-795e48c65984" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'1.13.1+cu116'" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 1 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ci_-geIdec0w" + }, + "source": [ + "## 1. Data (preparing and loading)\n", + "\n", + "I want to stress that \"data\" in machine learning can be almost anything you can imagine. A table of numbers (like a big Excel spreadsheet), images of any kind, videos (YouTube has lots of data!), audio files like songs or podcasts, protein structures, text and more.\n", + "\n", + "\n", + "Let's create our data as a straight line.\n", + "\n", + "We'll use [linear regression](https://en.wikipedia.org/wiki/Linear_regression) to create the data with known **parameters** (things that can be learned by a model) and then we'll use PyTorch to see if we can build model to estimate these parameters using [**gradient descent**](https://en.wikipedia.org/wiki/Gradient_descent).\n", + "\n", + "Don't worry if the terms above don't mean much now, we'll see them in action and I'll put extra resources below where you can learn more.\n", + "\n" + ] + }, + { + "cell_type": "code", + "source": [ + "# Create known parameters\n", + "weight = 0.7\n", + "bias = 0.3\n", + "\n", + "# Create data\n", + "start = 0\n", + "end = 1\n", + "step = 0.02\n", + "X = torch.arange(start,end,step).unsqueeze(dim = 1)\n", + "y = weight * X + bias\n", + "X[:10], y[:10]" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "QqRnkCY955Mi", + "outputId": "55871dea-b708-4efe-9b37-ec23c7ed9961" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "(tensor([[0.0000],\n", + " [0.0200],\n", + " [0.0400],\n", + " [0.0600],\n", + " [0.0800],\n", + " [0.1000],\n", + " [0.1200],\n", + " [0.1400],\n", + " [0.1600],\n", + " [0.1800]]), tensor([[0.3000],\n", + " [0.3140],\n", + " [0.3280],\n", + " [0.3420],\n", + " [0.3560],\n", + " [0.3700],\n", + " [0.3840],\n", + " [0.3980],\n", + " [0.4120],\n", + " [0.4260]]))" + ] + }, + "metadata": {}, + "execution_count": 2 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dzNigr8dtW2Y" + }, + "source": [ + "Beautiful! Now we're going to move towards building a model that can learn the relationship between `X` (**features**) and `y` (**labels**). " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "YApM7diprjP0" + }, + "source": [ + "### Split data into training and test sets \n", + "\n", + "One of most important steps in a machine learning project is creating a training and test set (and when required, a validation set).\n", + "\n", + "Each split of the dataset serves a specific purpose:\n", + "\n", + "| Split | Purpose | Amount of total data | How often is it used? |\n", + "| ----- | ----- | ----- | ----- |\n", + "| **Training set** | The model learns from this data (like the course materials you study during the semester). | ~60-80% | Always |\n", + "| **Validation set** | The model gets tuned on this data (like the practice exam you take before the final exam). | ~10-20% | Often but not always |\n", + "| **Testing set** | The model gets evaluated on this data to test what it has learned (like the final exam you take at the end of the semester). | ~10-20% | Always |\n", + "\n", + "For now, we'll just use a training and test set, this means we'll have a dataset for our model to learn on as well as be evaluated on.\n", + "\n", + "We can create them by splitting our `X` and `y` tensors.\n", + "\n", + "> **Note:** When dealing with real-world data, this step is typically done right at the start of a project (the test set should always be kept separate from all other data). We want our model to learn on training data and then evaluate it on test data to get an indication of how well it **generalizes** to unseen examples.\n" + ] + }, + { + "cell_type": "code", + "source": [ + "# Create train/test split\n", + "train_split = int(0.8 * len(X)) # 80% data fro training and 20% for testing\n", + "X_train, y_train = X[:train_split], y[:train_split]\n", + "X_test, y_test = X[train_split:], y[train_split:]\n", + "\n", + "len(X_train), len(y_train), len(X_test), len(y_test)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "wUq3AiYo86qq", + "outputId": "a19f7c76-87de-4e78-e9b1-6ea0b3afaaa6" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "(40, 40, 10, 10)" + ] + }, + "metadata": {}, + "execution_count": 3 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ua1y5hFjtLxC" + }, + "source": [ + "Wonderful, we've got 40 samples for training (`X_train` & `y_train`) and 10 samples for testing (`X_test` & `y_test`).\n", + "\n", + "The model we create is going to try and learn the relationship between `X_train` & `y_train` and then we will evaluate what it learns on `X_test` and `y_test`.\n", + "\n", + "But right now our data is just numbers on a page.\n", + "\n", + "Let's create a function to visualize it." + ] + }, + { + "cell_type": "code", + "source": [ + "def plot_predictions(train_data = X_train,\n", + " train_labels = y_train,\n", + " test_data = X_test,\n", + " test_labels = y_test,\n", + " predictions = None):\n", + " \"\"\" Function to plot training data, testing data and predications\n", + " \"\"\"\n", + " plt.figure(figsize = (10,7))\n", + "\n", + " # Plot training data in blue dot\n", + " plt.scatter(train_data, train_labels, c= 'b', s = 4, label =\"Training data\")\n", + " \n", + " # plot test data in green dot\n", + " plt.scatter(test_data,test_labels, c= 'g', s = 4, label ='Testing data')\n", + "\n", + " if predictions is not None:\n", + " # plot predications in red colour dot\n", + " plt.scatter(test_data, predictions, c= 'r', s = 4, label ='Predications')\n", + " # show the legend\n", + " plt.legend(prop = {'size':14});" + ], + "metadata": { + "id": "8AnpxtRE-RZB" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 428 + }, + "id": "xTaIwydGec0z", + "outputId": "0f378798-135e-4772-c83a-c677a0b704dc" + }, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAGbCAYAAADgEhWsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAAsTAAALEwEAmpwYAAAXjElEQVR4nO3df4hl93ke8Of1btRA4sQlmhajlS2Rrmm2jnHsQXXpHza1XSQbpLQOQVtC4uBYKkiJmzqhCg2OUQglaetAytSR0grHAVtRTQhbrKKW2MY02EYj7KiRhNStklgrBzzxr/5hYnnVt3/MtRmPZvfe2XN/388Hhplzzpd7XjjM7jPnnnmmujsAAFyZFy16AACAVSZMAQAMIEwBAAwgTAEADCBMAQAMcHJRJ7766qv7uuuuW9TpAQAm9sgjj/xVd28ddWxhYeq6667L7u7uok4PADCxqvqLSx3zNh8AwADCFADAAMIUAMAAwhQAwADCFADAAGPDVFXdV1VfrKo/vcTxqqrfqqrzVfVoVb1m+mMCACynSe5MfSDJjZc5flOS06OP25K8f/hYAACrYWyY6u5PJvnyZZbckuSDve/TSV5SVS+d1oAAAMtsGs9MXZPkmQPbF0b7AADW3lwfQK+q26pqt6p29/b25nlqAICZmEaYejbJtQe2T432vUB339vd2929vbV15J+3AQBYKdMIU+eS/OTot/pel+Rr3f2XU3hdAIClN/YPHVfVh5O8IcnVVXUhya8k+a4k6e7fTvJgkrckOZ/k60l+elbDAgAsm7FhqrvPjjneSe6Y2kQAACtEAzoAwADCFACwsu746B05effJ3PHRxb1JJkwBACvrnkfuyfP9fO555J6FzSBMAQAr6/bX3p4TdSK3v/b2hc1Q+8+Pz9/29nbv7u4u5NwAAMdRVY909/ZRx9yZAgAYQJgCABhAmAIAGECYAgCWyjLUHRyHMAUALJVlqDs4DmEKAFgqy1B3cByqEQAAxlCNAAAwI8IUAMAAwhQAwADCFAAwF6tWeTApYQoAmItVqzyYlDAFAMzFqlUeTEo1AgDAGKoRAABmRJgCABhAmAIAGECYAgAGWdfKg0kJUwDAIOtaeTApYQoAGGRdKw8mpRoBAGAM1QgAADMiTAEADCBMAQAMIEwBAC+w6XUHxyFMAQAvsOl1B8chTAEAL7DpdQfHoRoBAGAM1QgAADMiTAEADCBMAQAMIEwBwAZReTB9whQAbBCVB9MnTAHABlF5MH2qEQAAxlCNAAAwI8IUAMAAE4Wpqrqxqp6sqvNVddcRx19eVX9UVY9W1Seq6tT0RwUAWD5jw1RVnUiyk+SmJGeSnK2qM4eW/bskH+zuVyW5O8m/mfagAMClqTxYnEnuTN2Q5Hx3P93dzyW5P8kth9acSfKx0dcfP+I4ADBDKg8WZ5IwdU2SZw5sXxjtO+hPkvzT0df/JMmLq+oHDr9QVd1WVbtVtbu3t3cl8wIAR1B5sDjTegD9F5K8vqo+m+T1SZ5N8vzhRd19b3dvd/f21tbWlE4NAOy8dScX33MxO2/dWfQoG+fkBGueTXLtge1To33f1t1fyOjOVFV9b5K3dfdXpzQjAMDSmuTO1MNJTlfV9VV1VZJbk5w7uKCqrq6qb73WLyW5b7pjAgAsp7FhqrsvJrkzyUNJnkjyQHc/VlV3V9XNo2VvSPJkVT2V5G8n+bUZzQsAsFQmemaqux/s7ld09w9296+N9r2nu8+Nvv5Id58erfmZ7v7GLIcGgE2g7mA1aEAHgCWl7mA1CFMAsKTUHayG6u6FnHh7e7t3d3cXcm4AgOOoqke6e/uoY+5MAQAMIEwBAAwgTAEADCBMAcCcqTxYL8IUAMyZyoP1IkwBwJypPFgvqhEAAMZQjQAAMCPCFADAAMIUAMAAwhQATInKg80kTAHAlKg82EzCFABMicqDzaQaAQBgDNUIAAAzIkwBAAwgTAEADCBMAcBl3HFHcvLk/mc4ijAFAJdxzz3J88/vf4ajCFMAcBm3356cOLH/GY6iGgEAYAzVCAAAMyJMAQAMIEwBAAwgTAGwkVQeMC3CFAAbSeUB0yJMAbCRVB4wLaoRAADGUI0AADAjwhQAwADCFADAAMIUAGtD3QGLIEwBsDbUHbAIwhQAa0PdAYugGgEAYAzVCAAAMyJMAQAMIEwBAAwwUZiqqhur6smqOl9Vdx1x/GVV9fGq+mxVPVpVb5n+qABsKpUHLLOxD6BX1YkkTyV5c5ILSR5Ocra7Hz+w5t4kn+3u91fVmSQPdvd1l3tdD6ADMKmTJ/crD06cSC5eXPQ0bKKhD6DfkOR8dz/d3c8luT/JLYfWdJLvG339/Um+cKXDAsBhKg9YZicnWHNNkmcObF9I8vcPrXlvkv9eVT+b5HuSvOmoF6qq25LcliQve9nLjjsrABtqZ2f/A5bRtB5AP5vkA919KslbkvxeVb3gtbv73u7e7u7tra2tKZ0aAGBxJglTzya59sD2qdG+g96R5IEk6e5PJfnuJFdPY0AAgGU2SZh6OMnpqrq+qq5KcmuSc4fWfD7JG5Okqn4o+2Fqb5qDAgAso7FhqrsvJrkzyUNJnkjyQHc/VlV3V9XNo2XvTvLOqvqTJB9O8vZe1N+pAWBlqDxgHfjbfAAsjMoDVoW/zQfAUlJ5wDpwZwoAYAx3pgAAZkSYAgAYQJgCABhAmAJgqtQdsGmEKQCm6p579usO7rln0ZPAfAhTAEyVugM2jWoEAIAxVCMAAMyIMAUAMIAwBQAwgDAFADCAMAXARPRHwdGEKQAmoj8KjiZMATAR/VFwND1TAABj6JkCAJgRYQoAYABhCgBgAGEKYMOpPIBhhCmADafyAIYRpgA2nMoDGEY1AgDAGKoRAABmRJgCABhAmAIAGECYAlhD6g5gfoQpgDWk7gDmR5gCWEPqDmB+VCMAAIyhGgEAYEaEKQCAAYQpAIABhCmAFaLyAJaPMAWwQlQewPIRpgBWiMoDWD6qEQAAxlCNAAAwI8IUAMAAwhQAwADCFMASUHkAq2uiMFVVN1bVk1V1vqruOuL4b1bV50YfT1XVV6c+KcAaU3kAq2tsmKqqE0l2ktyU5EySs1V15uCa7v757n51d786yX9I8gczmBVgbak8gNU1yZ2pG5Kc7+6nu/u5JPcnueUy688m+fA0hgPYFDs7ycWL+5+B1TJJmLomyTMHti+M9r1AVb08yfVJPnaJ47dV1W5V7e7t7R13VgCApTPtB9BvTfKR7n7+qIPdfW93b3f39tbW1pRPDQAwf5OEqWeTXHtg+9Ro31Fujbf4AIANMkmYejjJ6aq6vqquyn5gOnd4UVX93SR/M8mnpjsiwGpSdwCbYWyY6u6LSe5M8lCSJ5I80N2PVdXdVXXzgaW3Jrm/F/XH/gCWjLoD2AwnJ1nU3Q8mefDQvvcc2n7v9MYCWH23374fpNQdwHqrRd1I2t7e7t3d3YWcGwDgOKrqke7ePuqYPycDADCAMAUAMIAwBQAwgDAFcEwqD4CDhCmAY1J5ABwkTAEc0+23JydOqDwA9qlGAAAYQzUCAMCMCFMAAAMIUwAAAwhTACMqD4ArIUwBjKg8AK6EMAUwovIAuBKqEQAAxlCNAAAwI8IUAMAAwhQAwADCFLDW1B0AsyZMAWtN3QEwa8IUsNbUHQCzphoBAGAM1QgAADMiTAEADCBMAQAMIEwBK0nlAbAshClgJak8AJaFMAWsJJUHwLJQjQAAMIZqBACAGRGmAAAGEKYAAAYQpoClovIAWDXCFLBUVB4Aq0aYApaKygNg1ahGAAAYQzUCAMCMCFMAAAMIUwAAAwhTwMypOwDWmTAFzJy6A2CdTRSmqurGqnqyqs5X1V2XWPPjVfV4VT1WVR+a7pjAKlN3AKyzsdUIVXUiyVNJ3pzkQpKHk5zt7scPrDmd5IEk/6i7v1JVf6u7v3i511WNAACsiqHVCDckOd/dT3f3c0nuT3LLoTXvTLLT3V9JknFBCgBgXUwSpq5J8syB7QujfQe9IskrquqPq+rTVXXjUS9UVbdV1W5V7e7t7V3ZxAAAS2RaD6CfTHI6yRuSnE3yO1X1ksOLuvve7t7u7u2tra0pnRoAYHEmCVPPJrn2wPap0b6DLiQ5193f7O4/y/4zVqenMyKwrFQeAEwWph5Ocrqqrq+qq5LcmuTcoTV/mP27Uqmqq7P/tt/T0xsTWEYqDwAmCFPdfTHJnUkeSvJEkge6+7Gquruqbh4teyjJl6rq8SQfT/KL3f2lWQ0NLAeVBwATVCPMimoEAGBVDK1GAADgEoQpAIABhCkAgAGEKeA7qDsAOB5hCvgO6g4AjkeYAr6DugOA41GNAAAwhmoEAIAZEaYAAAYQpgAABhCmYEOoPACYDWEKNoTKA4DZEKZgQ6g8AJgN1QgAAGOoRgAAmBFhCgBgAGEKAGAAYQpWnMoDgMUSpmDFqTwAWCxhClacygOAxVKNAAAwhmoEAIAZEaYAAAYQpgAABhCmYAmpOwBYHcIULCF1BwCrQ5iCJaTuAGB1qEYAABhDNQIAwIwIUwAAAwhTAAADCFMAAAMIUzBH+qMA1o8wBXOkPwpg/QhTMEf6owDWj54pAIAx9EwBAMyIMAUAMIAwBQAwgDAFU6DyAGBzCVMwBSoPADaXMAVToPIAYHNNFKaq6saqerKqzlfVXUccf3tV7VXV50YfPzP9UWF57ewkFy/ufwZgs5wct6CqTiTZSfLmJBeSPFxV57r78UNLf7+775zBjAAAS2uSO1M3JDnf3U9393NJ7k9yy2zHAgBYDZOEqWuSPHNg+8Jo32Fvq6pHq+ojVXXtUS9UVbdV1W5V7e7t7V3BuAAAy2VaD6D/1yTXdferkvyPJL971KLuvre7t7t7e2tra0qnhtlQdwDAJCYJU88mOXin6dRo37d195e6+xujzf+U5LXTGQ8WR90BAJOYJEw9nOR0VV1fVVcluTXJuYMLquqlBzZvTvLE9EaExVB3AMAkxv42X3dfrKo7kzyU5ESS+7r7saq6O8lud59L8nNVdXOSi0m+nOTtM5wZ5mJnR9UBAONVdy/kxNvb2727u7uQcwMAHEdVPdLd20cd04AOADCAMAUAMIAwxcZReQDANAlTbByVBwBMkzDFxlF5AMA0+W0+AIAx/DYfAMCMCFMAAAMIUwAAAwhTrA2VBwAsgjDF2lB5AMAiCFOsDZUHACyCagQAgDFUIwAAzIgwBQAwgDAFADCAMMVSU3cAwLITplhq6g4AWHbCFEtN3QEAy041AgDAGKoRAABmRJgCABhAmAIAGECYYiFUHgCwLoQpFkLlAQDrQphiIVQeALAuVCMAAIyhGgEAYEaEKQCAAYQpAIABhCmmSuUBAJtGmGKqVB4AsGmEKaZK5QEAm0Y1AgDAGKoRAABmRJgCABhAmAIAGECYYix1BwBwacIUY6k7AIBLE6YYS90BAFyaagQAgDEGVyNU1Y1V9WRVna+quy6z7m1V1VV15MkAANbN2DBVVSeS7CS5KcmZJGer6swR616c5F1JPjPtIQEAltUkd6ZuSHK+u5/u7ueS3J/kliPW/WqSX0/y11OcDwBgqU0Spq5J8syB7Qujfd9WVa9Jcm13f/RyL1RVt1XVblXt7u3tHXtYpkvlAQAMN/i3+arqRUnel+Td49Z2973dvd3d21tbW0NPzUAqDwBguEnC1LNJrj2wfWq071tenOSVST5RVX+e5HVJznkIffmpPACA4cZWI1TVySRPJXlj9kPUw0n+WXc/don1n0jyC9192d4D1QgAwKoYVI3Q3ReT3JnkoSRPJHmgux+rqrur6ubpjgoAsFpOTrKoux9M8uChfe+5xNo3DB8LAGA1+HMyAAADCFNrSOUBAMyPMLWGVB4AwPwIU2tI5QEAzM/YaoRZUY0AAKyKQdUIAABcmjAFADCAMAUAMIAwtSLUHQDAchKmVoS6AwBYTsLUilB3AADLSTUCAMAYqhEAAGZEmAIAGECYAgAYQJhaMJUHALDahKkFU3kAAKtNmFowlQcAsNpUIwAAjKEaAQBgRoQpAIABhCkAgAGEqRlQdwAAm0OYmgF1BwCwOYSpGVB3AACbQzUCAMAYqhEAAGZEmAIAGECYAgAYQJg6BpUHAMBhwtQxqDwAAA4Tpo5B5QEAcJhqBACAMVQjAADMiDAFADCAMAUAMIAwFZUHAMCVE6ai8gAAuHLCVFQeAABXTjUCAMAYqhEAAGZkojBVVTdW1ZNVdb6q7jri+D+vqv9VVZ+rqv9ZVWemPyoAwPIZG6aq6kSSnSQ3JTmT5OwRYelD3f3D3f3qJL+R5H3THhQAYBlNcmfqhiTnu/vp7n4uyf1Jbjm4oLv/74HN70mymAexAADmbJIwdU2SZw5sXxjt+w5VdUdV/Z/s35n6uemMd+V0RwEA8zC1B9C7e6e7fzDJv0ryy0etqarbqmq3qnb39vamdeoj6Y4CAOZhkjD1bJJrD2yfGu27lPuT/OhRB7r73u7e7u7tra2tiYe8ErqjAIB5mCRMPZzkdFVdX1VXJbk1ybmDC6rq9IHNtyb539Mb8crs7CQXL+5/BgCYlZPjFnT3xaq6M8lDSU4kua+7H6uqu5Psdve5JHdW1ZuSfDPJV5L81CyHBgBYFmPDVJJ094NJHjy07z0Hvn7XlOcCAFgJGtABAAYQpgAABhCmAAAGEKYAAAYQpgAABhCmAAAGEKYAAAYQpgAABhCmAAAGEKYAAAYQpgAABhCmAAAGqO5ezImr9pL8xYxPc3WSv5rxObhyrs/ycm2Wm+uz3Fyf5TXk2ry8u7eOOrCwMDUPVbXb3duLnoOjuT7Ly7VZbq7PcnN9ltesro23+QAABhCmAAAGWPcwde+iB+CyXJ/l5dosN9dnubk+y2sm12atn5kCAJi1db8zBQAwU8IUAMAAaxGmqurGqnqyqs5X1V1HHP8bVfX7o+OfqarrFjDmxprg+vzLqnq8qh6tqj+qqpcvYs5NNO7aHFj3tqrqqvLr3nM0yfWpqh8fff88VlUfmveMm2qCf9deVlUfr6rPjv5te8si5txEVXVfVX2xqv70Eserqn5rdO0erarXDD3nyoepqjqRZCfJTUnOJDlbVWcOLXtHkq90999J8ptJfn2+U26uCa/PZ5Nsd/erknwkyW/Md8rNNOG1SVW9OMm7knxmvhNutkmuT1WdTvJLSf5hd/+9JP9i3nNuogm/d345yQPd/SNJbk3yH+c75Ub7QJIbL3P8piSnRx+3JXn/0BOufJhKckOS8939dHc/l+T+JLccWnNLkt8dff2RJG+sqprjjJts7PXp7o9399dHm59OcmrOM26qSb53kuRXs/8DyF/Pczgmuj7vTLLT3V9Jku7+4pxn3FSTXJtO8n2jr78/yRfmON9G6+5PJvnyZZbckuSDve/TSV5SVS8dcs51CFPXJHnmwPaF0b4j13T3xSRfS/IDc5mOSa7PQe9I8t9mOhHfMvbajG5/X9vdH53nYCSZ7HvnFUleUVV/XFWfrqrL/TTO9Exybd6b5Ceq6kKSB5P87HxGYwLH/X9prJODxoEpqqqfSLKd5PWLnoWkql6U5H1J3r7gUbi0k9l/q+IN2b+j+8mq+uHu/uoihyJJcjbJB7r731fVP0jye1X1yu7+f4sejOlbhztTzya59sD2qdG+I9dU1cns33L90lymY5Lrk6p6U5J/neTm7v7GnGbbdOOuzYuTvDLJJ6rqz5O8Lsk5D6HPzSTfOxeSnOvub3b3nyV5Kvvhitma5Nq8I8kDSdLdn0ry3dn/I7ss3kT/Lx3HOoSph5Ocrqrrq+qq7D/od+7QmnNJfmr09Y8l+VhrK52Xsdenqn4kyT3ZD1Ke+Zify16b7v5ad1/d3dd193XZf57t5u7eXcy4G2eSf9v+MPt3pVJVV2f/bb+n5zjjpprk2nw+yRuTpKp+KPtham+uU3Ip55L85Oi3+l6X5Gvd/ZdDXnDl3+br7otVdWeSh5KcSHJfdz9WVXcn2e3uc0n+c/ZvsZ7P/kNpty5u4s0y4fX5t0m+N8l/Gf1ewOe7++aFDb0hJrw2LMiE1+ehJP+4qh5P8nySX+xud91nbMJr8+4kv1NVP5/9h9Hf7of4+aiqD2f/h4yrR8+s/UqS70qS7v7t7D/D9pYk55N8PclPDz6nawsAcOXW4W0+AICFEaYAAAYQpgAABhCmAAAGEKYAAAYQpgAABhCmAAAG+P85WEzs91IXUAAAAABJRU5ErkJggg==\n" + }, + "metadata": { + "needs_background": "light" + } + } + ], + "source": [ + "plot_predictions();" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0eFsorRHec00" + }, + "source": [ + "## 2. Build model\n", + "\n", + "Now we've got some data, let's build a model to use the blue dots to predict the green dots.\n", + "\n", + "Let's replicate a standard linear regression model using pure PyTorch." + ] + }, + { + "cell_type": "markdown", + "source": [ + "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyQAAAEkCAYAAADelfUtAAAgAElEQVR4nOzdfVxUdf7//8fSBKKf0QVz8gMGKrjggoWmqJQU8fNiXTI3zKvM1Kx0tcwsP+pHV81SV7tyy60t01ZN06DIyFKXKCpMc5UKVvgIKpOwNSaszldHaKLfHzNcCsxwkWP2vN9u3m4yM+ec93mf95x5v95X51c//vjjj4iIiIiIiHiAl6cTICIiIiIiv1wKSERERERExGMUkIiIiIiIiMcoIBEREREREY9RQCIiIiIiIh6jgERERERERDxGAYmIiIiIiHiMAhIREREREfEYBSQiIiIiIuIxCkhERERERMRjFJCIiIiIiIjHKCARERERERGPUUAiIiIiIiIeo4BEREREREQ8RgGJiIiIiIh4jAISERERERHxGAUkIiIiIiLiMQpIRERERETEYxSQiIiIiIiIxyggERERERERj1FAIiIiIiIiHqOAREREREREPEYBiYiIiIiIeIwCEhERERER8RgFJCIiIiIi4jEKSERERERExGMUkIiIiIiIiMcoIBEREREREY8xtGzz79g+NZZ5Gc3fw9St/2LB9S4+9EMZX2elsn1bKh9+lkXON2WO1/26E9F3AKN/l8iw3/Wk0xXuHfNk8jT6z28g0c3cZ3VavyPnvd1sfy+ZvQcOk1/qeNmnc0+uHzCAobEJ3Dy4J9f4NLD9N8lMiV3Eh7HL2LcukU5uHrbqnKZu4ejcqCYm2uHgqt8yal397znSH8uI225l2IDutG9qvmBlx6z+PPQeEDiLd/5xPxFu7yOL5b8Zzzrgmskb2D2/Pw1lXy3/XE33cRugkbxstCy0vYqI7r3pMzyW236XQJ9At45aZ7+xrEx/kdGB7mxV/X1y63shIiIichloYQ/JVYxe9y+O/l/9/5KmOj41dWvDn3FV6Tr52d+YENObm8YtYm3KvupgBKD0KDl7trD44UT6X5/IvOTDnPmhZWdUa58xk1mXXeZ6GwDKyE9exK3Xx3Lrw4+zaU91MAJQ9s1hMlM2sPjhRG7qH8uUtVmcaWFSLyZH+v/GvMkJ9L99EbuKmriDb3aT8p7z/0Uvsv0zd/O1tq83LOKpZm7bZOe+Iyd7D5tWLWJUXG+iJv+Nvd80dScZzFuczNctLZciIiIil6lLeMhWGfmbphE7cQ2ZpT50umUyK7fuJiurRkDz1T4+enM1C0ZG0f7cYbbPTyT2YfcrfzevyLggQDp8IIPdGxZyV5QRSvexfMIstruqfP/wHR8uS2TI/GRyzhnpM3Iu69/PIOur2mnd9/4GVs5IoI/Pd3x4qICLVK1usvoCyMP7dvPO07O4ubMPZYeTmX7Pag6ec3+fX3+QzIf4cNecWURQxqY9+5px/j74tD3Buvl/YW8Tju2O+srC0a/2kbUnmWeXjCfGD858uoY7YxOY994J91Pc1gcyFrE42f1tRERERH5JLtmA5OvkWdy6LIOytlHM2LiHfS8+yujru9C+bY0P+Ri5JvL3TF21hX2pyxjdHc68t4ghM97lZDOP69P+KkJvGM/Srf/gtand4VwGi9c3Vnku4+BTk5iy6Sh0T2RlagZJqyZxc/eraF9zhI+PkU7d+zN61iqSMvfx0eMJbg/HuhT4+HUhIuF+1qe/w8pbfODoBpanuFvJPsyOl7MgcBqjp/6e0VHAlmR2lbrcsI7xLF0Ui0/RBuataU5A00Q+RtoH92TE+IVsztzHO0sG056jbJ81jeX/dO/oA2ctZGp3+HDh464DWxEREZFfoEszIClKZvH8DMraxrJ0+wbmDLjK5SY+v0lk5fYNTO0OZR8s5In3rC1LwxVGBt47naFA2aYMchr4WFnWi8xbdxTaxrLylWWM/o0b8wyuMHJNZ/fnI1xSrujC6JnTuAY4+ME+9wK/7Ay2F0HElMFEXNGF2yYOBvbw3qdNv0ahIxey9BYfvt4wjeUZF7GP6QojEePXsGfdeEI5yrp7FrkXULW/iTlPTCaUDObN3kK+hm6JiIiI1HIJBiRl7F3/OB8CNy9YyF3uVPArte/PnJX3E0oZO1ZtIaellT+/64gbAFBAfr1zB6zs2vA38vHhrr+scXPi8mWgZxRDATKO8rXLD5ex960X+ZooRsd2B6D9DUMYCuza+I4b29dxRRdGL1rIzW3L2PTgaj5s5aFbrnSKfZSVM7rDuVSWbzns1jY+1z/IyqndIetx5m05+hOnUEREROTn5dILSEr/wcZNZRA4izmJXZq8uU/UFGb+jhZNnK5m5eTXAO3p0LaetysnagdOY/QNP9Mej+Y4ewYLQGB716tdlWXxfnIZ/O5Obgt2vub3/zHxLh/ISuXDwmYcPzDRMXTr3Bamr8q4yHNxfOgz0dFz9vXLqRx0K+j1oc/Mx5naHQ4uW8gmxSQiIiIiVS65gKTsq0PsAq4ZG9uEZWFrMnLzsASgjO0H3GvBbtDRQ6QXAVFR/Lb9hW+fObSPD2lJWn+eyrL2sQvwuSWKUBefPbNnC5vOwYhhsVRnoQ99bknEhyzWvd+8a3SNc+hW2ZZZF3foFoBfLCMTgHN72Pt/bm7TNso5dCuLxfM0dEtERESk0iUXkHxdeAiA3kFN7x2p1P7a/sQAZV8VNHtyOyczWPzHxzmIDyMm38o19XzE8l0B0LK0/tyU/d8Wpj+4hTK6M3OUq+eBWPlwzx5oO54xtxhrveMzIJGZgfD16xnNG1rn0aFbRrqEdQFOcOS4+/NgNHRLRERE5EItfDBi6zvzb0eLuclkdPHJRlwB3k3dpszKmXNQVvov0pM28NSWDE6e8yFixgb+/Lv609IqaW1IxiL6/2ZR6++3qc5ZOVMGp4sy2LXxNZ5PyeIMRoaufpEZPV1sW/gOG98Dn6kJDKwbuVzRk5vHduGpp15k+2eTWNqcIW+BiSxdtIch87cwfVUsWUti3XtgYiswXdUdOMGZc2WAu9ffOXTrg/GsW7aQTTds4a7uP2EiRURERH4GLrmA5GL6cH4s3ec38KZff+asXc2MG1yv8HW5WTfutzTwwHZ8eiaycuVcRvd0XQnPz0jmIF2YM7z+J8dHDJtEn6ceZ/sHWSy4wc2nr9dxzciFLN1zK/O2zGL5LZ+xNPYSn8vjHLr14bgNLJ63hYFbxxP6CxruJyIiIlLXJTdkq/1/O5rd80981/yd/ADlAIY2bnzYSGhkTyIiexLR3VnJHrmGrMwNLoMRnw6OoVoWSwuXGK5P7DL2NfB0+/r+7VsR27rHb3uVI08iexLhXKI4dE4yh99e5lYwwg+H2bX+MERNYkRkA58JvpWJv4OyTcl82NzH1nto6JblO8eQq2s6Nb13TEO3RERERKpdcgHJNcG9Adib1/wnW5cVHCYTuKZ390YfPuh4Ovc+dr+ZzDtvJvPOu68ypzuQsoZNbkxWvibI0fJ/yHx5PYV76tZ/cTQrw5EnbybzzturGNEW8l/Y4P7DDLNSeb4IyHqcm37zW7rX+68/D70HkEpKRguCusBE/rz0Yq66ZeVE3gmgC6ZOzemR8aHPrMeZoVW3RERERC69gMSnV2/Hwwi3pLK3WTVLK7uStwA+DO3rapJDHVf0ZOoTk7mGozy12PVKSO3DooigBROzfy78BvO/i2LhXCoPrXGnwl/G3p1bmhQY7ErZ3fwFCIBOtz3Ony/WqlulGaSkAm0HM/A3zdyHTxQzV96vVbdERETkF++SC0gcS6r6wLktPLW9GU3Hh7fwvPPZILfVP3WhUT7XP8jS8T7uDafpHsvoKBzPPPn0Ii89e5F1GjmXpVG4V+E/k8G2TWVuDjvbxwsJQEYy7zfnmSRVrmLEkscvytCtnO0vOJamvjeBPi2Y/+ETNc3xkEUN3RIREZFfsEsvIMHI0JmP0gc4uOxBlv+zCRX9c1ksn72GfHwYMXd8M58N4sPNsxxDlA4uW8X2osY+24XRM8fjQxmbHpzl4rM/c1d0566lswiljE2L/8LeRir8ZzJS2QEMHTmk0SFzDkZuHjkeyGJ7Rgsr5Z1//5MP3Sr752pmP3UU2iawYHwTe+Au4EOfaTWHbp1vlTSKiIiI/JxcggEJ0H08z6yIxYejrLtnPMs/cGOCe+G7zLt9POuOQujUhpfqdYvfYP53aSw+ZLB42buNDiXyiX2U16Z2h3MZzLtnEdv/z41q8A/fsfefP8N5Jz0nOVr0izaw+OWsBir8J3h74x5gML+7wb1r4NN/MHe1hZz1e1o89K3W0K13mjtTvh4/WMlJXsSt4zaQj5HRqxcx1K8V9ltr6Nar5JS3wj5FREREfkYuzYAEuCZxDUlzovA5d5h102KJmrya7Z8e5eSZGtXgMisnj+5j+6rJRA1+lO1HITRxDa/MiWrx8yg6JSxkaSyUfbCQ/3m7sYDIhz5zXmTlLT5wNJl5CbGMmvsqHx79jppJ5VxlWqfR//pY7nz5ixbNmWiyoneZ3v+39LxtDQebPZzJhz73LmJqIOSvXcjzWfWEJEcz2J4FJCRws7sVdp8ohiX6QFEyHzbvwe011Bi6tSW5Zbsqs3Km8DAfJq9mSlwst85PJp/ujF6TzMrBrffsmeqhW1vY9Fmr7VZERETkZ+ESfg6JDxH3byHjur8xe9YaMj/dwLxPNzT88bY9Gb1oNUsTu7fOw/Gu6MLoBQvZnvE4Hy5exa7YVQ23iF/RhdFrMwjd8CBTVu3jYMoqpqSsajytv+tJ+9ZIp5tOfvaOY4Ws0r/x8gd38kJCM5+v0rY/c5aOZ9PULayd9yrD3r2/1tC4nPdeJQcYMXhAE87Phz63JOKzaQvPv7WPqZHNeyZJlc6/589L3yH2UfeHbTX6TBqn9jdM5tklj3JzcEsSVx/n0K33xrNWU0lERETkF+aS7SGp1GnA/WzOPMRHW5cxY2T/qmdiAODXnYjB41n6dDL7/pnMytYKRip1H8/SOd3dW13qCiN9pm4g66vdJK24nxE39CS0RgDj07knMSMns/TFVLL+mczK21o5rS50GnArQ/3Ap+f93HtLyx726BM7i2cTfODoGhbXnIz9QxZvv3wCGM/IW5rWg+AzIIG72kLZpj31rq5miuxJRKTrGSmVKodutYhfdyIiB3PXktW8k36IrA1NCEbadiIisiemtm5+vmroloiIiMgvy69+/PHHHz2dCBERERER+WW65HtIRERERETk8qWAREREREREPEYBiYiIiIiIeIwCEhERERER8RgFJCIiIiIi4jEKSERERERExGMUkIiIiIiIiMcoIBEREREREY9RQCIiIiIiIh6jgERERERERDxGAYmIiIiIiHiMAhIREREREfEYBSQiIiIiIuIxCkhERERERMRjFJCIiIiIiIjHKCARERERERGPUUAiIiIiIiIeo4BEREREREQ8RgGJiIiIiIh4jAISERERERHxGAUkIiIiIiLiMQpIRERERETEYxSQiIiIiIiIxyggERERERERj1FAIiIiIiIiHqOAREREREREPEYBiYiIiIiIeIwCEhERERER8RgFJCIiIiIi4jEKSERERERExGMMLd1B6a29qv5/5euft3R3IiIiIiLiAf/Vro1HjqseEhERERER8RgFJCIiIiIi4jEKSERERERExGMUkIiIiIiIiMcoIBEREREREY9RQCIiIiIiIh6jgERERERERDxGAYmIiIiIiHiMAhIREREREfEYBSQiIiIiIuIxCkhERERERMRjLuOAxEpBWgp7T3g6HSIiIiIi0pDLOCApZO+eTApOezoddeWyY9kCNuc0b+tTO2ZyfZ/ruH7NF62bLBERERERD7iMA5JLlZ0zZ+3N2zTrKYYsgUXvfcE/Z13Xusmq63Qm6xbOZf7zGZT+tEe6RBWyc8VclqcWNn3T8v28Om8u8+etIf1UzdcWk5zXqolsVZa0J5k/byNZFTVfbUE+XBR2DmyYy/x5c3kuvaTWa0uTcj2asmrVaXT8W8nubz2dpsa0wjU3p7J83jJ2mlsvVU1izeSFeXN5IdP60x+rlc61IGUZ8+ctcH2PqMhm84K5rN5jaf7Bcja2oBxaSH9qLvM3HWr+8Rvhdj78BMypyxzf0e25tV97PoOLUJJEftEuckCSzeZ5T5J+8uIe9fLwHSmvbOTGJUsYefVFOFxxIQV28O8ejN9FONwl51Q+Bad9CQsLbvq2R3Jx/JaewXa25mvBBHdrvSS2LisFRyzQNZSQmneFluTDxVCRS+4Rx3+tZ621XgvuGuq5dNVioO/kVaxYuYqp0b7gG0rIxfgON1crXPPSggKsvj0JC2rFdDXF0XzMmAjvYfzJD9U651qC2WyFdqGEdHbx0YJcCip86d7V1OyjFeQVNL8cWvPJPQlBIT/F96sJ+dDqLBTkOe8hZ89gr/GasWswP31JEvllM1y0Ix1OZVu5P2Bh91Nz2Q2YBj/C7HgTVFgpSH+Dbem5WO1g8AvnljHjiOvq69g2L4Wlmyz0joPstHysGBk0bRHDg+D88TRe35ZGXqkdvAz4Bw3jzmmxBDgPe74gjVdTne97BzNkxgzi6rsJ13uM+USeeIN3P/4Kc6kdMGAMi2fC2HiCfGtuYyDvo1xKysHYYyRTJ8dgqlmpK7dQsH8/B4qtGAN8sULVza00cy1P7/VnULCFvVlFnLcbMA2YyPSR4bSpmb6sv7Ps04lseO4ql1ltTl3GCyfiWTAtpvk30Z7jWLFyXHO3bjlzKsv/Wkjc/85goAd+CUq/zKaYYAY2o05mPlYIXQIJOFFEibXGa13jCfNu3XS2mopCCszgHxdaq8y0JB8uihPHMVcEEtCliOLT1hqvBRMXdvFub+6xYD5mg+4hhHg6KY1o8TWvKCErpwiCBuKpYlNQUAC+vQjq9BMfqNXO1Z+4B1cR58YnLebjnCeEkGYXohaWQ3M+ZvyJC/kpbszu50OrKz9O4UlfArr4UlxipQQwOV8LG3Gp3gBFLh8X7xe7ZwJjTqbxDCaGzHmEuBo/FAUpT7LuRD+m/s8UQox2Svdv5C8vvQSzZjmCB/sZztvzOXRiFA8+dh9+NivWdsCRJP68IZ/IO+ezLMKIobyEgmxLrQpVUamR6bOXM8lgJWvrk2zbnEbEnHguaFuq9xgGig9fSb9x87k3yIihPJ8dz7zEhveCWXx7qHObIkqMM5i9ZAqGs4fY/OxWXksPdQRawPnjqbz8UgYWUziRAQYKP/4KMxDhPOz3Nhv2UxbsI6bwv6OMULyLF55fT1KPVUyIqE7eKfMxuPternWZ0Y4WHVNU6IXBiL2ErJRN7PyyCGs5YPAlIGokE27vjZ8XjmEOT6RQNfKgQyzT5ydQ3fBnZe+Ly9hxvO6Oa19Ta0EGb7/zPjnf2B3H6DuOe+sGWK7O4kgu1k69aepv3vnjGSSlppF3wobdy4B/jwTuvrsyQCxk54q1fB52n+P6gWPowqZi4mbPY8jV+SQvfYkDtur9Jf9pLskAHeOZ/ejQC8vNhSmnIM+K/7UJhNm2kvONBSJwtLKFBVZdk8bTaWH36idJDxjHA90O8erOXKwY6TvuERIjfKuO1JJ8rhnIGzpEM3Y4FFT4EtnVBLiRD0eSWPrKfkLuqlFOKwrZuXotn3eZyP/cGcyhF5exg3jG9ChiZ3ou1nqCbfupQ+x8I5XPj1vryQfXLEdysXbsxfAwG9u+/BYLkXAkF2uHcAIrM9tWyMdvpvDB4SLOOxs8hk+awsDKhokvNjJ/q4Uh0xKwpmxk7zd2DEFDeWBafFU6WppOoLpleWCdluUKKwWZb7EzPZfis/VcS+f38nxlA47jpDiwYTHJ1qHMnulIp8fLvrNMnK96IYlF85IA8I9/hEcHV94TG0tn7XSYBs9geFkqyZmFWP2GMtt57651PTDQpks0Y+8ZSZgvgIWCAhuEhuCbuZ7lDXx/XF6ugjSSUzIoOGnDTp2GMjfO1Zq5luU7nMPevCIZMzOc7O0p5HxjJ8L5vSl4czHr9ldnetio5UzqW+NnuaKErDc38XZWEecrDAQNnkjkEQt0janqyXRZNksPkfxaClknbNi9gxlyZ6/6y2G9bJjTtrLto1xKyg0Y+45nuJcj2KvqXXGW3x27s7GUg6FdIP1G38cIx8Ug57W5bC6uXYYsaU/yzCcmxsydSJSvG/lQ9zwvuOat8B09kkseIYyJgW3bizgJmJw924mV8Ugr3Euwl5CVupWdBwqrGmBr7UPkF8rzTYjWTHbvNzDojwnOyqcBv+iJ3HZ4Adv+kU3cnZHOD4by+9HR+BkAoxEjVvam7Yc+U0iMcNY8vP0J6eNfa/chfaMxeQMYiYoKYdsm542m3sTUPQaEDR1V/bZ3KFERRvaeLHH8QDmOQL9ok+P/xt707r6Vzd9YHEeoyOfdTRmcj5nBsoTKO1o2m+dtrH3YjpH07WF07COgN5Ed08j6xgIR1aksOvYxN3ZbUn8eVhSyc/XfyOoQzZB4IwUnDfh5FZG+/Q0OZBsZvGQiUV5W9q5byQ4SmL1wFiZvsJ/I4PXPvocKHIP3jDFMXxlDdeUlnNqjEIwMnLaKgc6/KoOt830TGOgMRs7nbOTp10rpN3k+y3oYsR1O4sW/b+XdiKUk9qg/+ZXMqSt5+Ssj/eKHYjxqwXAVFKdtJfnAVxiHLWeCi2kz53M28ufXSuk3+REW9zDCqQxefiqFv78fzKPDA+FkPnmnISSs+ofYfKwQfHs6flwrQklcvIrEimw2L9yIOWYGCxJqtow1FJBRHbw5W9S6dw3FWABnTp+B8pJarWwu02nNp+AU+AdfiW/fKSyILmL3mjWkv5/JoAhHhawl+cy3abz80i6IuY8Fw0IxUkT6i2s5T7ij1dVlPgD+fvgD1tPV/X2WtDf4uLw3k+6IpE1FNgVmaNPNgH/0FBbE2yne9SzPpaey94ZwR/D6bRovrMnAOHIGi+8zQakjH5I/68X0GGM9lb5qjgoLmM0W2oSEENIuH/5fKVbslJottAkb6Si7tmw2P7mR0t738fCSUIyU8PG6lex4NZXg/0kgAGdrOibwDWT4Q8uJO7iRp7fvYvfheEew5Sqd7mqgZdn+TSbpeYEMf2giIcbKfNrF3kHhxHUEjP74e0Hu6TNU3rnOf/EG7x4JZMhsZzByKZT9kFEsXjnKGegUMuiPjl7smlymszIdR5JY+spX+FWU4DtoBguG19hJnethOLufV5/NoOBbO2FdDW59f1w6ksTTr+QTedcjLI4wYrBbyUl9i8IfnPP/3DhX44AZrIhx5ltFW2yloYx9aHmtH92Q25ey4nawH1jPoiQbYbV69WzkbF3DtqPBJD6wnL5XG7B+sZXnavZkuiqbtmw2P7+VgqBRPPBYNCaDlaykZ93u4bCkvcQLaTBo8iKG9DBCcRov/NUGYZW9KzZytj7J5lP9mDpnOSFGG3lvruXV198lYvEoQoBOJn/46tvq393TmSSnWQi7fQZRvu7kgxvXvMX3EkNVL3aIsZg2fMsZa52e7da4l1RYSP/rk2R0GMX0P83A5OXcx1uZRLZkRIPIZcADAYkJU81u9BOFmAkmttbN3EBAFxNkOVo8HT8gbfCt1bhVhNkMwX1bcxxr3WM4lB5OY/dHh8g9YeG8Hejk7M51tbuCLLLPBjPkppZ2937H0XygwfkH/kQNi6ckPY3kV5w/mGmp2K6NYfDU3jhCukIKjkObMAP2Mht2b18MXWKZMKqe3TkrL8FBXRtMUemB9fwlqZDAEY/wQIwzJ8oP8fpr2QSOXMrwHo6MNIaF8N/sx2y2QI/Gc8wvaii3nE7jg5SXsFcApLHT1ouBw+4nqmejm0JFLu8mZdP2xlkMrxw33jGcEL9U0ktLgUDshYVYCGZAVVlzjhmuHLpQ2YJ14jjmivrG0NcOyOpV2coWYuTK/XD+rBV7zVY2N9LpGP/uT9xNkfh5A7TH2A7AF19oYT7byXp/F8UdYnkgobIXrT1tDPYL5480mA9Ax0BMXlBQUgIY4WQa29KtRN11h+PH+4hjnHvkzfEEORsaArqYgEJsNoAidr66C+uAGTwQbaqdD8cKISYSejgrfQ2pyCb3CISMCcVoyATbOax25/yR2x1lN+/dN8jxrnmu/oSF+LMzrYRSIKCyNT0ihrirHZ8wGts68trbzXS6yTFuv9cF4/YNAUOZek/VX/h6XwkYMVbVTkyYOkPWaWdDSHkub+/Ixn+ws1fyUin7zn1UBjoXzKlwp+xX7iPvMOe9gom8sbdjeGwVx/Uo6TOl+noYo5m0KLr6I66+P25wDIsycKX399jLweBtJGLkxKqebZfnWvmZ8sPkHQf/G/sxMMK/ng855OXlQofY6l49gOI0dn5lI2z0RPpe7fipNra9ku8rfAnrasKdslmcnkqOLZwx46ob5owGe3Ug2pjyQ+xMK8K/5vUy+mKwV88fsR/cyuavAklcnECIr2P/YT0C4MBxzCchpBP4tjMCdmzlgLednNRdmINGsqDvhVej3nxwec1b4V5S1bMdivHXNtqTj/WcpVbPdsvvJVD8/np2n41h+sxoZ49J5T7yMRNDRD0pE/ml8HwPySXOkr6GZ/bYiBo+invv6orhs2d5JsvNjcvPcR4j7Vvc7HEV3UMhvaG3vYwEXBfPKEMRBa/lYseOf9z9TI+vWTENJ3ZoOP9OT+G5J1IAMHSOZuzkUUR0qL27qspLvWPwbZh3vsTLn9iImjyPxLDqHxX7l4fIqwDeXMz8N2tvFRXgul3S2KU3cbdfSdHRjeSVgd0vnqnT3GzRPLyfAzZf+vYIrH6t/DjFp8DUx7GHvLxc6DS0ehhY+XEK6xm6YDmSi5Vggps7f6RrPCFeYOtsgqwSsmq2suW4Tmfl+PfqIRG153e0KJ/LD5J1GIw39qqaZ9XQ/JHG86GykmwFrOxN3kVJxETu7ekoMxbzcc57hRBeY5C6pbgYCMTUGTAfIqsUrJlrmZ9Ze88BfQNwS+Vcke7A+UBMHKI0q8b8kYpsPj9oo03f8OpzxY75RAl06kcnqGpND7qhugzUGj+LEI0AACAASURBVKPfGul07LWBcfs2zJ+8wduVw7Wqdj6UkKr5Rv4EdAa+djSEWFO3ktVhKLNvcl7rS6TsV55nQZ4VgoIunFPhRjodrBSdsEKPyKoW9CrO6xHSI7zBFLj6/rjD1HcYfQ+/weevrGQvgMFIWPxEJsQF1/jRbORcKxWaKcRIv2sby9BCCs3Qpk6PtPngIUoIJy6yxhGbVDYLyfqiBHrcQmRVWXJ//ojjPmNkUGSN61Wrl89O1heO1aiSlzqH91Xy6k2As+HR2MEIWLCeBv6TQlKOP0Nm19cbUH8+uLzmrfEdrerZNkGnq+lECZavj2Ov7NlujXtJ5fU4nckLC+omtB9NuZuIXI4ubkBSZuOCBW+7BBNEGgXHIaKqQd5O8QkLbYIDG6mMBhIUBLuP50Pfhn+cWqaQA58V4R/3CGNudKSkSQstGv0xkk1hMUS08G4T2G0QnxwrAuqf1H4+ZytPby0k/M759M17knWfvE/OoIlEVP0QGQiKm8KjcUCFnfMn0njhr2nsPhBLRHztXK6/lQqoKOHAa2tJPn41w2fOYFBA7eJTcrqknnknTWDLZtvTWzCHTWRu71yefiWD3Tk3MSHCdTG1fGOBOhM97dnZ5BHI8L6OlnnHj11wdZk6kkveBUMX7I5ehq4xF0xAt6Q9yTP1LrVZOYemRisb4NvBH05ls7dGK5vrdFaOfw+vrjA4V9WJdK6q06J8/sZCMRDUrUYFqc7+XeWDQ2Ul+VvMB/az42QkE+6OdM4Nca7YFRRDUFWPi5W8vBLoOZQob8BagpVwEh+bQt/69u8cOpRTUc97vSay4s7IqvkjIUbAx4gfFrIzqZ4/cvJbTlZASM3VgMq/IvsIBAyPdpQD52pMQ6pWY6pcbcw5Rt9VOt3VwPwRa+Z6XthpJ+6Pj/FAFwOQy7YF6ynoHlprdTtTp0A4WESxOZU9B/wZMqt6TPqlUfadf1YFOj0v+HFxnc7KfTh6FUyDu174A2UtwYqJwJr308ohp46juPz+uKVDJIkzI0kE7LYSspLWkLzrI/LiavSSNHKulSp7eoK7NHKsenuk7VhOWqFTMEFV16KJZbM8H8tpMEXXyMeG5jHVo/ibIiCY4Bo3mdq9fCVYS8B4Yz1DOmsyBWKigJL/WEjfsZ+2MTNqzSFtPB9wfc1b4V5S3bMNYMS/A+Rl7sVS2bN9qhXuJVgpOV3//BgRudgBia8vbSjFcgoiOtg4jy9tjNHc0mcXr76RRMh9I4noAKX7N/J2XiCxsxsLNIz0u6k3u/++lc3dH2Fsn+pJ7aY+4a00FtPR1XrumyKsFSZ8rfkcyLFAhZvPEQnqTZRfBh8n7SJq2lACvO1Yc3L5N/DfTUxJx6BusOQffDnrunomtpeQ/VkRQeMeYUyEL3SJJ+SYBWyAN2DNJfmlVOxxE0mMcsx3OXOiiDNegcReW/dHuoFWKrtjnsFuWx8mPTSKsHoy2N/PH07nkntiGEFdDNhP5/NxUgolNz5E4q93MytxCSfvfJlXp19Hfb8bpV9lUtx1PA+PjqQNwdwSchwLNsBI+VfPM/7u1+m2bBurfx94wbamziagmII8K1E9jdi/3c/md3MJGPwIgzoA2LBa4fzpEqwV4GvNZ3daNvhG1xm6kE/BETDGBF5Qhkzxj7Aivp6EV6rZygYY2vlCRS7FJ33p65w/4jKdVa1s1T/wdVfVaTSfw/7D+w8N57FvJ/LiKzO5tm2dNLbzxQj8u6AQe0QwBmsh6R9+Vc+qPQ3nQyW/jiY4/Dnb3rURMWoeVfOFnS3ShgAbVjsYsVLw/np2FgcyZFZvx03Hzx9/Pif3SytRfY0Yyi1kpWxkb5cpTI/xB69IJixf1Uhm26vmj5gAvNvj6wV5xRbaRDvnj3S8mk5e8O8juVh7hWO0WziwNYW8zkOZ7Zz7ccFqTHVb0xtNZzu+fGEsk9/oxhObn2ZYY40ODcwfMR8rBGMMIZ0NUGGn+KM0cisgPKR2Bc/P5AcUs2drNm3iH6m1UuAlUfYrFRSQh5FBXS4sNa7T6VTZq1Df0EM/f/wpoOhrK3QyYj91iOT1qdjiZjCpr79b3x8AynNJXr2eA2dNDJlVMz/tFGesZ1teCCNG30RIBwOUl1BssdGmTzRhbp6rQ2VPTzhhjUysrr9H2uAYZnakELMVTEY7pV+k8slx8I93p2z6V38nzMexVpgwVpSQ9V5GVTl0dU81tmsPFFNw3E5EVwPW42mkf1Wzd8Ufv6vAmpdL8ZBgx+/bkY/Y9m4pg2aOoup0Ohjxw0Zx2hscOBfN1OH1By8N9sy7uuYtvpfU7tkGI77twFJcBF1HOoLz1riX4Ie/H3x++CDWqGiMXnYsWW/w2v5gJk27nq/dvZeIXKYubkDSMYYhAw6xefVcdnsZCBnxCFMH+BM26hEmvb+R5NUL2GwHQ+dIhs+cUTVRuiGGnuN4cFwSm3esYNH2Gsv+uhOQlGbywlMpWPrex+KRDbUWmYibMJLC9W+wfMFWDJ1jSIyLxZScRnJmDNM7NLBZlUCGTxpJ8boUnvtTGgDGHqFuj2OuJepuFt0Qzys77mbNiLq9JP70vecR+lb+2SGWqXNqvG0Mpl+UP9tSnmTRdsDLgDGgD7fNmUhUR8dH6q5ywv6XmL8fiBjHirt6Y92fwu4TdmA/rz6xv/pznapXvjFE3cEk83pef3EB6XbHaithcSO5rYcBfhzEH2dGM3nVy3ww+nmGdbzwFP2i72N21VBwI4PufaTqPe/fjmX22DQeXPEWX/5+5oVBWcQdTBrwEq+/towDdjD4BdNvxDwmXVc5bjucuMHBZO9JYvnCFEx9biKgjHqGLgQSFGIg65O1zP/Euetxq1xOqAfqtLIBV5lw9HnUGALjKp0uW9lc5DNXccP0h+l3zwpeSR/Lmt/XKSuV38HMtSzabyQsLoGAH2wXzh9xIx9MnU1gy+ZcxDhG1Vy9qCCXggojA+OvZu/qBRw4bcfgF86Q+8ZVV/wChnFngoXXdixjURLg7U9Qn6GMjW54nH0tzmeNhIyp/O76Y+oInKzRwuoVyai7Y3h5+0aWL7A7jzGSuXc5V5VzpzXdRTqvvf1hRu+ZyWNvf8Gw6XUKST0ts+nPzCUdqlZZCukbgykvk3ULMx2rOA0LJ9BQSNamjUQ+PpGIymtiCsRENhZDPLPj6lTUL4WyXykgmBDvQ3z817l8DEAkY5ZPJMrLdTqtn6xheWpR1a4q99EmusbKYAHxjBlsYfOby5i/vbLsT2RsX/e/PwAU55JtBYxdCaj1oCUDAb36EfRlCq+u2OWYs9POREjMfTwcF1r7B7OhcyWX5GXra6xYVrkCVzAjKpcxr7uiIbDjibnswN+56hmE3TSSkCMpJD8xl7c7R5MYcyXncL9sQjhxCaEUpFZe9zsY5GWr6uHwvqrxe6rfgKEMPLiRvS8u4PMO4dwyPJDvbTWfP2Ig6vYpmP++lRf+lOZc+SqcW279A7ViisrA6LiFqLvvr30d3MgHl9e8pfcSavdsV91LisHYxRmct8a9hECG3JWAZVMKyxcmOesr0Qy+Ixo/DPg1di8R+QX41Y8//vhjS3ZQemuvqv9f+frnLU7Q5cnOeauN7w2+GH1bEANmPcX1U46x6L3nL87DEVvb8dcZP9PK4tR7a7c0uqn80xUMfDeaD5fHazWSRh1l+4SZWBft5J7mZLSbrJlrWb7HWLV0ZyVL2pM8k2ZiQs0K9WWrjE9XRLOz7yc8MfgnLJVHklj6ynEGzn7EUUkTaQW6p15KLtK9RMSF/2rXlIc0tJ7LvrpwaTDQxmhsWTACEDWH3Utg2e+u4/o1X7RO0i6WH4pIefYt+j0+sVnBCOe+YO3zRSyaqR9OV4p2PE1KvxXc9RMGI9j2k5xqIWLkHXUmHjtbpF0MU7lclGc9z9pvl/DHW37CUllRyM4399M+fqKCEWk9uqdeUi7KvUTkEqaZVT8zHUc8zz9HeDoVTXfsjac4dsdaZkf5NGNrK58+/zqBy55ipMbWNu746zx7ZAzPPVT/PJ0WO5nGM0/twmIwEjb8PsZeV2cAonPctCm+ngnJlxvrxzyzPZAnlv+BwCt+gv1XDvnCgCl6ClPjmzApW6RRuqdeUn7qe4nIz4CGbImIiIiIiIZsiYiIiIjIL48CEhERERER8RgFJCIiIiIi4jEKSERERERExGMUkIiIiIiIiMcoIBEREREREY9RQCIiIiIiIh6jgERERERERDxGAYmISGOynuL6Ptdx/ZovPJ0SERGRy5Ke1O5UkLKMdZ/Z6Dt5OYlhnk7NJSJnI/M3FRM3ex5Drnbj8+X7efVPSeRd8IaRQX9cxPAgN/ZhTmX5XzOwVv4dMY4Vd/VuQqIvoop8dqx4ib22aCY9PgoVm+axH1jPoqTcC9/oEMv0+Qm4V2yW8cInVaWGiHGrmHBd66URviPlgXjSB6exZsRVrbnjRmW/8X88ltPIByL82H5Hp4uWHhERubx56kntBo8c9ZJTgtlshXbhhHT2dFouHQV5BeDbixB3ghGAI7nkNSX4qE9QAgtWJoA1kxeeSIGQ0Gbu6CIoLaTQCm3CQgjwdFp+xvLycpsUfNQnKGERKxLAmrmW5TsgpHurJhG4ipELl5D+uyWk9H+eke5+J1oo8o7fsP0O5x/fFbHweRuDZoYy9OLFRCIiIj85BSQA+BP34CriPJ0Md53M4Lk1qRRXRDLh8YlE/CQD7yyYj9mgewghbm5hPlYIXsEEd2mFw5vzMeNPXIixFXb2E+kYzwMr4z2dip+5QgrNQFDXZgcjNZmPFULHeH6SYnP1H7jn7iW8su87Rl7EXhJXzpqLeOXNs+z7D+DtRf/4TjzYvwPw//hgfTEf+LUj7OQ53i/+ke8NXtw+JpCxPXyrt006yydnAC8I7NGB+eOuxgRgt3Lo3ZP85Qs7Zyvg1wE+TB99Db1/7bjh/GffMWZ/7sXt13zP64cq+N7gzZyHu9K/rYcyQkREfrY8EJBY2fviMnYcd/7ZaxwP9PiKbanZWOzVFWz7qUPsfCOVz49bsXsZ8O+RwN13x2CqrHyXHiL5tRSyTtiwewcz5M5e5G5IhRGLmB5jdA4DsTHif2cw0AhgIf2pJ9ltqh4CVPDmYtbtt1WlLGzUcib1rc4SR2troeMPr0jGzAwne3sKOd/YibhrFRMi3EinvYSslE3s/LIIazlg8CUgaiQTbu+Nn7uBhK2Qj99M4YPDRZy3g8HgS9i45TwQYXAvnT2tFGS+xc70XIrP2h1p6DuOe0eGU9UxV19+noSgge72UFgoyLNCj3DCGjovewlZqVvZc7CQknLA25+wYVOYFGO64KMN9s64kZ8ur0mDnGWTeMb0KGJnei5WuwHTgIlMr8qrfJKXvsSBqmITTuJjU+jrXb2XrE1z2WYZyvQEK8mvZWKxGwga/BDT4y48z3qz6cB6Fu3xJ3FAIW/vsTFo5iwCP3mSzQdtRIx7jAnXOa97QQZvv/M+Od/UvaZ2DmxYwO6OoxhoTmG37SYemBnIx09vJMsWyZg/TSTKG6goIWtH9fUwtAuk3+j7GBHmqKy6LFfeSSx9ZT8hzu8CABWF7Fy9ls+7TOR/7ozEZcfvyXzyTkNYWHjD+XHqEDvf2MXn5hLsFWDwC2f4pCkMvKCXIp/cfGjTK4QLctqaT3pSChkFFs7bcZS9m8YxNj64Ko0N52e1wG6D+GTPx5wa8Qc6ujo3l77gmQeOMvG5FuyrwMzs1+wMu6sLD3Zrw9ljxaz4+0neCe7ArZ1/5PtzcKz8RyaO78Zd7aEo3czcrRYiFwYT6VXCm1vOYulnYnv8r+H8abILcebdWT7Z+G82+nbgzwuuxuR1jmN7vmHhS18z/8FgIttA+fkfOPvd9+zrcRUv/cmfdv/vLGcVjIiISDN4ICAxMvC+VQw86xyS42OjNGQ8sx+rkZRv03hhTQbGkTNYfJ8JSjN4+akUkj/rxfQYI9iy2fz8VgqCRvHAY9GYDFaykp6t1aKel5cLnYZWt5SWH6ewTgU75PalrLidquAlLKx2dhgHzGBFjLOSWtEWW2koYx9aXp1prtKJlb3rVrKDBGYvnIXJG+wnMnj9s++hAveWFKiwkP7yWjI6juPhJb0xks+O1S+xNysXIiLdSqe9OJP0vECGPzSREKOd4l3P8lz6LvYOCieuI27kZ50gsqbKoTbO/OVkEovmJVW/33UkC6bFYASsOansLY9h6p9m4OdlJeu1J9mWmkZezLg68y8a6p1xIz9dXpPG8rqQAjO06WbAP3oKC+Ir8yqVvTeEE9cJIJTExatIdFb6k8vCCfOuuZN8Co4CV0ObgARmL4l3nOeu98mJm0hEvQeuLS8vF/wSCLsphkEHnqTgsJXYUfcTV/gkeSVnAH/O52zk6ddK6Td5Pst6GLEdTuLFv2/l3YilJIbkknsE/Hr0JK5/KQeeySfvTCxjpsZjfiaXUivgZyH9+SfJ6DCK6QtnYDJYyXtzLa/+fStBS6YQ5e1G+T/lhz9gPW0FHHlrSXuDj8t7M+mOSNpcELzVEDaKZZOjobAQC2BJWsD8GsUmyNmwAFay38/EFn0fi6f5Y7AeYvPTW9nxUS4DR9cJYk4WYrZBSN1hfhX5JD/7Ekd7TuThJZEYDXasOam8Xfg931dAGy8az88e1bvqGNQNPj1KEbRCQHIds+/5B9c/8Ba7mxWUnOOTj87j3b8Tt3dzRALtunVg0FXn+DjPyq3O4afdenbgN+0dVy2wlw/dPrJRVAKRV/1A+flfERbU3vHBNh2IrPwiFpziFfOVTFtwNSYDQFu6DfVn7GELr++z8vhNzu9S+7bcM9ifdl5A+3a0a3ZeiIjIL5lnhmx5gT0vFzP+DIqOIaLWL3ERO1/dhXXADB6IdrZzdgwnxC+V9GOFEBNJcXoqObZwxoyLxuQNYMRosINvT2eLumMYSJuw4OqW0iO55DUwBKhyDHtg3be8gPLD5B0H/xv7MTDCv0nphEIKjkObMAP2Mht2b18MXWKZMKoJeZWXxu5if+LG9MbovFrfl4O/n5+b6QRDwFCm3lP1F77eVzryzHm+rvPTyMBpqxjYWDrdmD9ivG4i06smGhvx9bGB0Yhv3Q9a8xvonXGVn+5ck0YU5FJQ4UvkzfEEGR15FdDFBBRiq1uprnBU+o0xgdQqNt8WcNQGEQPjMRkd52lsB/i2dd1b4DzHQjOEDY7B6HUc6zkwdjbRxr6f4lO+BHTxh/JDvP5aNoEjlzLcOfTGGBbCf7Mfs9kCPscxV4QzJNoIhVbOeRkxdfLFfrCIEt9AgjqC9ZOt7P4mnDHTqq95WI8AOGCh9DTQCZflio6BmLygoKQEMMLJNLalW4m66w5nkFYZvDUsx+X8ESNRd84gqupPX64sA2PbC0oN1iO5WAhmQN35I6cKMZ8Fg3cb7GV2MBgwRoys7tVxlZ893OvZapaoOWwIvY4hfdJY9F5T56acJe8EWMwnGb33ZK13ere5AvjBxfYd6H/df1jxdiG/HvprBoV14NfezuFYxXbOXuVDt1rB9q/pFmhhk+U8lQEo3r9yBCMiIiIt4LE5JIXmQujQj8i6tRDzIbJKHcNF5mfWfiugbwBQSNYXJdDjFiKrfizrtKg7h4GEhFVXaM3HCmtUsGulxBm8hNdfISo0U4iRftcGNzGdAOHEDg3n3+kpPPdECgCGztGMnTyKiA4NZk29ztvsAJRm7OJAWTiJcYHupRMb5k/e4O3K4VpViRxKiDe4lZ9ucDl/pM7woCp9Qi/M9wbnj7jIT7euScMs5uOc9wohvMZJW4qLgUBMdRc7OHEcc4UvYWG189takE8JwdxYVSluYl6ezCfvtImoYAOYc8mzBRMXhOP6eoUwKgTsBw+RVwG8uZj5b9bePCrAhOVILtZOvQnyBnPeYc4HxRPk5fzOhd5BCBbS9xVB0EhnGXAwf+24hqaaDQQNlisAE6bOkHXaCljZm7yLkoiJ3NvT3duKG/NHSg+xY1v1cK2q8wy5MD0Nzh/pFM2Qvtkk7X+JVZkABoxh8Uy4M54gb7B/2Xh+No1jNa5lnzZxM+DYv7+Dq5s+N+WWxFCm9aovKrDW81pN3kTeFsjjH1v4yzsn2ZR8km79/PnT7y+d+TEiIvLL4KGApJC8PFv9FRFrCdZ6xuVXKc/HchpM0V2rE1+nRd1eWOhoKa3auXN+Q32VQmfwEhzUtd6UmvMOc76+irardAJgIChuCo/GARV2zp9I44W/prH7QCwR8W5WdMKGMjxoJTtfXMBeDBi7RjPm0ZFE1Wkgbiid1sz1vLDTTtwfH+OBLgYgl20L1lPQPRQ/gPJvXeanJe1JntljqSdxJobMeYS4Tq7njxSkrGFbTghj5iwnqoMBTqXx3OpdGLtfOEel4dW9XOSnW9ekIVYKjlggKIYgr+rX8vJKoOdQx5yLGixHcrESTHCdenFBQWHtoYIN9vbUz15YiKVDOCGdwJpZiLVTOCFGMH9UHViUnC5ppFfBzoHPLBjDQjFhZe8JK6YeoRgp5OM8G0E3BQOFFJ0E/2tDa/TuFJGdY6VNn+haiyQ0WP4B8CegM/D1t5gP7GfHyUgm3F09byTntbls/qqezbycc8VOuZg/UpFP8vNbye4+jrmPOXoIS9PXsGqXkZALvsiNzB/BSMSoWUSMAuw2SrPe4C9Ju8g4Es+ECFf5Wdsp8zG4IZ7ABj9xFSOf+4KRLvZTtb8dMxly7F7+Oas5axR3ILJHKS8eLWFar2YGEV5t6XZTV565qYKzX5iZ+dYZPhl4FUMDDLRLK+f/zoOpqmvvPxwrgkjNWhcRkVbmmYDEWkRhQxURP3/8+ZzcL61E9TViKLeQlbKRvV2mMD3GH7zb4+sFeebjWCtMGCtKyHovo1aLutVqBWycPGWHdmA5kEpGA5XCquAlrL6ssFJ0ooGKtqt0WnNJfikVe9xEEqNMGIAzJ4o44xVI7LVuBiMV+SQ/sQnGLmdFj8YuVcPpNB8rBGMMIZ0NUGGn+KM0cisgvLKF2Y38NMU/worGFpNyzh8xRXVtoEA5ewk6hxJiNIDdSk765xQTyJBudbdooEfBnfxs9Jq048sXxjL5jW48sflphtXtMHHOHzEE2LDawYiVgvfXs7M4kCGzetc5L7tjKE/XmHrnj7TpVWOoYBNXC8vLy4WgaIKwcyCvkDbdhtYJLMDXzx9O55J7YhhBXQzYT+fzcVIKJTc+RGIPx1CyoOhgKN9P3nFfguJNYM2k8LSJ8B5GwERgJ8gpyKU0zoQfVgreT+Lj8kgm/L7md7KR8u/k19EEhz9n27s2IkbNI6JGoBxx5ypWNHKuju+eszeoPs6hVoHdQjEawH46mw/2F9Xo3auhgfkj9uIMXt2aS8iIcQzqYcSAnZJiC+fb9aafc26If2P5Wee+UHTsY24cvKQV5o8AfMHGZgcjAN70j2/HOy+W8pcuVzK9dweurCijKOc05b1MdHM5lOo/7Pvoe7r09yOwjfPDXl78+r8Afz/Gdv43G7cV021cZwIN5zm2p4TXy32Yf71mioiISOu6uAFJXhJLN+znfOXflRNZa0x8JmAYdyZYeG3HMhYlAd7+BPUZytjoyvHr4cQlhFKQmsTyhSmY+tzBIC9brRZ1vwHx9P1sC3tfXMDnHcIZFGHnPP6EVFYKnc+4MNdI2o4n5rIDf8dDADvlkrxsfY3JuJUTtYOrV+1ylU5jMP2i/NmW8iSLtgNeBowBfbhtzkSimlCbudLLxt5XFnCg6hUDxrAEpk6OwVThOp0hfWMw5WWybmEmBr9wbhkWTqChkKxNG4l8fCIRXq7z06XK+SMNjrU3ETEwnL07U1i+IJU2Xfpw24BwjAcz2f1qGhFz4qFuL8zJrcyftxU6DWX2nHhM7uSni2ty7e0PM3rPTB57+wuGTa9TCSzIpaDCyMD4q9m7egEHTtsx+IUz5L5xxDnzodaqUwCksHxeCnSMZ/ajQzE554/UHCrYtGe5VM4fCYeKfArNEDIqtHZgARii7mCSeT2vv7iAdLtjdaywuJHc1sPgHEoWzpAewLGaw7xyMft2ZUgnx/WImzCSwldTWbUw1bGiVM94pj8SS5Av4Ea5qrqynU1gy+ZcxDhGRVw4r6MxlXO3Qhp6rl/HXgwIy2TnjmXM3+lYTW1gmJEDn+3i1bRezI7HsXJejekTlq1zmb8VTIMfYXa8CUPnSPoGH+Ltvy9jtx0w+GIKiWXqQ/FVQU2j+VnTt2/xyt8HEfdeaw1puo7Zs1q4C1Mg8ycV8cqb3zLpnW/5Hvh1QBseDAH+y8W25RVYCk7zysel/McOV7b3ZuKkq+nvnFM0dOoPtHurhIUr8zlbAYFBbZn/x4AaQztFRERax2XwpPYLl/O9PNjIS1nL7nYTeWBwdUXffnA9i7aXOIdKeTB5P0tlfLoimp19P+GJwbV7LCxpT/JMmuknfK7L5cmauZble4yMmTvxgmGElxfPPKldRETkYtKT2puriWP0LxVZm+ayLae+dyIZs3wiUaWZ7P7MgvH2GrW8cgtZXxVCQCwRCkaarDzredZ+u4TVt9QdPuWcP9IjtuFnqLQCl9f85xYI2faTnGohYsyUyzsYyXqK66dshLs38k8FIyIiIq3u5x+Q/Bye6F2PqLtWVS9lWh+/XvQNO8TuHcuqVv4xtDMREBbP9NGx9UzclUZZP+aZ7YE8sfwPBF5R5z3n/BFTfENzYFqHy2v+c3EyjWee2oXFYCRs+H2Mve5yjkaAqDn88+AcT6dCRETksnUZDNkSEREREZGW8tSQrZ/bIBEREREREbmMKCARERERERGPUUAiIiIiIiIeo4BEREREREQ8RgGJiIiIiIh4jAISERERERHxGAUkIiIiIiLiMQpIRERERETEYxSQiIiIiIiIxyggERERERERj1FAIiIiIiIiHvOrH3/88UdPJ0JERERERH6Z1EMiIiIicFiejAAAIABJREFUIiIeo4BEREREREQ8RgGJiIiIiIh4jAISERERERHxGAUkIiIiIiLiMQpIRERERETEYxSQiIiIiIiIxyggERERERERj1FAIiIiIiIiHqOAREREREREPEYBiYiIiIiIeIwCEhERERER8RgFJCIiIiIi4jEKSERERERExGMUkIiIiIiIiMcoIBG5XJT+i6R1uznh6XSIiHjcGf6VvJ7dhZ5Oh4i4QwGJ/LL8cILdi/7AtWFhXHvTY3z0/7d3/8FN1fni/5932Nk4+DE77LR+uUPulzvke5Mx2k5zpUO+dqdZyxIpH3uBL1mpdoXdfmk/di/90AvFrla7EH+1wGpZdqlbsbsCQdR00BuuQriicewaRiCMQPjWNXz0Sxj5mowM2ZXLceic7x9J27TNz/4ggK/HjCNNzjnv93mf9/uc8zrn/X4nOur7Ez0ssnTgByCC59Ey6nsj407Ov2MRpZv9E8hwDs66ad3sJ3xtUstexEOztR7XhXxnZGpFDjRT1uBi/LVlEpz30LqkGKOxmLKnvYyu3mKi/PRUltJxbKqWnwz5SPN6FMT9eAf+vDbIm8t1cY670QxE8e9YyaINmc7H0m4lIMnVsQ6MRmPCf6UsWt2BV57C3BAibzpoHGjgw0A/n3hasGr9dFmMLNoRiC0wECZ4cXj56MUJnnovBiflpjCyvxGjpRnPjXgluBolepMHIwB8EyVyNZ8ZiODa0Aj/+iH9/Z9weL0VbT6zc5MKn82tRScuPzXteNQ5jNzzmJWIm0ZjKc2HroOT0PWUl++Sa3qOG1uvbzgDITxPVtN61k73k5nPx1PSbm8gEpCMi5VnvP309/fT/8k7bJkXpHFJM56LmdcE8G82Tuip+/Ug0luP8Vo9+Z9E5856YUYh2mmARgOYafD18/YqU76zllbB/dvo923BVhD7+0YtfzGVzhH8ALQ/jF32NJo8Z0eMMbodp+anw5jtW8UpOIddcFFvHHxTHFdQxbb+j9myIGPmp97ovCTLr7jB3RjX5nRCvQ6aL9bS/VQVumn5zs31TwKSidIUYHqolprLbnwnlaGPlb+4cDxUFu8aVI3jQAjw07PVR/gqeB8vi71hqYu9/vRvNo66wYzgqksIXE70sMjSSk9vK0vNRozmapxnYfA1X+ufEtJb0orn/FBOCPY6qLbE3ugUW6txvJckGLrox7mhkUVDy9XTdSRZ0BTBvd3FZ98CO6rjb4mGLwTJ9zu2nufRMuq3uuhavYhSoxGjZSU9p0aV2fLS2DbNZVQ/7R1+NTwQwbe9njLz4FupLnwRxm67NrZ+6ZjXo8NlP5TvOheRXLtlDUTwbR8sp1IWre4hcDnxe4XQKTfOrZ10vurGn2pAR9RDs3E+XWeGPsCzzohxg5ehEjnRSbGxGU+UhK5k6csfovh7HVRb49121rkJDSTPQuRAM2V1nbi2r4wfDwfeaBT/qw4aK4ePQ/12H5HBbQzVw+E0lj7pGZvGxSC+3h46t/bg+iA4phtZ9MzwsS62VuPoDQ7v92Cd3u6kOb5M6XIHnjMB3JvjZW8uo/nVxHVG7FnGupZte+t6tTnedkqp3uAhcMZNR3ybxdZmnH8Zm4MrF/y4dnTS0+sjOPqBV8SXkKdFNO4IDO1D8uMRX6ch1qZidc5JYHSyJ3roPBIr5Z6HYm24vjeS5rwBXEjYrmURjYnHebAMn+uho2GwPdfTcyKIb0dz/NiXUr3Zm7DOiCOcvh6NLpahfR9uWysTyiZzfofPg4Nd1qp3Bce1H8pZD52PVsfPNbG650pynFNSQgT2O+nc2olzv3/kmK4RXUJTnJuHjqWXVmvCsUx5nktxDrsaJfiek86tTtynEr/z02E0juwacsFFfTwAiuzvwhVUgB6q4z0BYsuO7VKSVTtOeW1KFDv/zf/D8NPw6IFmjEYHXmV4e51mI80HoiPykjq/8e2cTLwmNeNOmj7jbvMpr3kZrqlZ1fkkIn1OHKuXsnTZUho3u/CnClonKf3oWR+uHZ107nDx/qcpOgSf6mK+sRF34kNZxYfDXIyjT8nQprKp1wrBA500PzR8Hqze4CI4dO3Npq7F25u1eKi+Nu8fbp2p713GUqIhAn1unFsddKZ6S3fBheM5eObX9qFgJKdzS7rzftr7oRuYKnJztF01GOrUN75M+OzqR2q7oUh9wR//O7BdrSypU3cHLqmqqqqXAi+rdYbl6u7PVVVVw+obqwxqnSs8YrPHNxlUw6bjCZ+MWu5ou2owGNTKp95Xw1dVVb10Sb10VVVV9bjaHv/83BVVVa9eUj/aVKkaVr2hhuN5qTDUxdNW1SufH1ePh5Ls15cH1Zf3HI9tQ1XVS56NapFhnXrwUvJiGJvf7Pa76JHd6umvY4ufe61OLSppV49fVVVVPa1uv9eg1u05F/vym3PqcX/83+ol9eDaouF9v3pJPb2zTi1a2K4e/2Z424bEtL++kjTfHz2bTTm3q8eTfTcmH2H14K8r1KKnPlKvqKqqfvOZuvuRIrWovE7d2PmC2r5+uTrXkKSchrZlUJfs/Cz255X31Y0Gg2owbFTfvzJYPiuGj+OIfKUo/6PtqsFQqW48fE69clVV1Usfqe0Lx9a1oT131akGQ5Fat/N0rC59fUm9oobVgy/tVo+H4pn4+qC6scSgrnvnUtZphL0b1UrDXHX5+nb1hc6Nal150cg2M6qeqF++r25cWKTWuQaPd6xOL9/0Ubycz6m7VxlUQ8ly9YUP4+l8+rK6fKhujdmzDHUt2/a2XG0fTO/z3WqdwaAWPfiC+lF8Pz7743LV8MBudTDXYVedaigpUueW16kbO9vVdQ/MVQ0ldeobg+3t64PqupJKdePh+Da/PKhuvLdI3fjhlTTHI1ZPKp79KPbZlbD62dHP1KTN8upHarvBoLYfTfgs1XljdF4unVZ3P1KkVm46rl5JKI+KtW+opy+psfPKsxWqwVCpPu6K5+/S++rGkuH8jz4GaevR6KVddbH2ezL+/edvqHUlRcP7kjG/sTpjWLhRff/LWH4vXRrnfvh3q9sPfxb77uoV9fQfl6uGe7erpxPSGVHGCa58ulutKylS73lko/rCYB1IXD6xHac7N3/5hlo3+jqT8jw3+jwVy2NRyT2xNvjrOvWeEsOYshqxD6PTG3W+UZOtl2U7TnltGuXSO+tUw/+1W42dEa+o7//aoBoMBnWjN35cQm+oK4byOCovafKbbfrjafNpr3kZrqkZ6/wYV9TjmypVw7wV6sY9/65+9OG/qy+vXaIWlSxRXw4kWXyi6V8Nq+8/Vaka5i1X1z37wlA9Sl5+sev3avdw+77y4cbhc27aNpVdvT6+Z7v6fjC+/Sun1ZcfMKgVL56Op5b5WJ9+sVItenC7ejx+TbgSPq1+NLjvae9dRjuuvjDvHnXJg0vUisT7vlE+27lErfj96ZEfZn1uSXfez3Q/dOOSNyQTdTGI+zkHPXMasBUBRPHs6ESztoWaO2JdJ7R3/Au2cj/eoxMNYe20rLVSMA3QamPdjuKslVZ0GmCaFkuFFT4Icg7gcpTQLDNmXWw5zWwz5llJNj3TRu2D5tg2AG25FTtBwllnObv9tvzkPkwzYv/WlduwXA4SDAMoRM/rMBfFMzpdh7kk/u8zTjr222lZH9/3aVpMD7fRou2h653hbZvWJKQ9I3l/Fc33st2fJMbkowBblQ1ll5cAENjViOPrFvYd7qZtTRMtm/bSvSrVxrT88z1WAm/6CALKES/OWTp0OPEeUYAogSM+TOVmcusgYaXqXh2aaYDWgq0CvGfPpV78riZaHjLF6tIMLRoKsK2qwTwrXn4zrFgXQzCc+GQsTRoXPbSvcWF56UP2bmqhaU0b3a+1YR1ad2w9YaaVlifs+J524k94gm6usMTLWcd9C60w187PyuKl8U9mzPgJpamfqetatszYBtObfR+2crAs+xmWmbGP9EVmOBEa+fanaPD4t7BlzztsqfDRuslNFAi83oF7cQst98a3OdNGVaWC888JfaTHHI9viUbBVBT/TFOA/m598r7I01L10Rp73hiTF62Jml+1oN3RhTvhiav+nh9j0hI7r9hsgBX7snhetHdimqsQuvDXJGlmU49GmWvjvrviezb7x9jmKgQ/jx3gbPNrf7QF68xYfrUJhZTTfpTU0HCvPvbdNA2m8ipM58OEMz26HgjQ8z8dRNfuo6+rjaY1LWx5rZvaVMtne24eJZvzHID9d32xNrihm75dTbCjmZ5Tmbefnezbccpr0yjaEgvWUy58ZwHFh/dVHbpZ4DzsQwGiJ3347rJinplbTrNNPyaXNp/hmpfNNTVNnR9N+aCDlXt0PNP7Cm0PVmEpq6L2N3vZtjhIxw7P2LGKE0w/eqidxl4L3Yf3suWxJpo2dPPWE1aSM1FVZ8ZzaLBngoL/PReFdVWYp5FVm8pUr80PNmCdE8+rxoT1fhOhL8Mj3uikPNZRDz3PKzQ83oA5fk3QFJiw3KUl93s2M02+PvY9bUeLDdOcZOURxPdmENu8Ud3Nsj63pDnvZ3k/dCOayO3Zd1jsVXorAAVY1jzLu73W+Gu5IIHDENi/COPTI9eqWvZ94FsA9P84nn64WrTTc1zlLhsNmpW0btDS8ov7MM8uiN1IJnM5hPetHlyvHcR7JoIC1KYZF2Od8w8Jf2W336mZsP1Sw8onWtGur+W+uXoK4uekyKd+QuU27hxxjtJhmguOs+eAWD4Kb/tBhjQmJvKpnxBe6oudI7+Y1YSGAN7XglQ9WoU+y76iBWU2rI978F+wE/7ARc2Gveh/v5SOD/y0lGsIHNZhW6VPuf7I8h+nH2r5wZj8KoTec9HT6+JgX4DIZWBVdoPtokc8uGc0sK881Y1SrJ5YHhq5Xxq9CctlD8EwOd9wXFc0Gm4ZLM9pBVQ9UEPzigCBgf+T8IkQHK6n+NWRq+jWJpTVmONRwI/vt+J4upEuWqiqMKHL9Rww5rwRIXgihPUndzLiKM02YcZB8DwwKcdg/PVopOzzq9VO0sCZiB+X04n7gBff2ShgJXyR9OXyqRfX2SpaFqdusyNkPDfr0SdJL9vznOaWxLSqqJ3XiTsQouGu7LJHuZ7UZ5gpaMczf4ytvBWPP4L9Sy+uB9vYa+xi6SYv/sesaPo96Gy1pCzdtPmdCpmueeR8TU0tivctJ/q1b2MfEbRqsFTYYU2AIDbMo1cbd/pRfIfcFD6yD2uW5xvdPVWYN7jxXKjCXhjA21uIfU/CDXmGNpVNvY6ccLF7jxvPB77Y5DPlYf4KZGz1ZwN4sLLtn5J9mcVxTEIJBQncZUaf7OnQQJjQKQum2cl2IptzS+rzfub7oetgjNc4yRuScRkc1N7HM+URwt9oKRzVaGt3xge9J/y3ZWEe5rzRmGl6bS8109ysW1ZGcdnIvvRDFD+dy+bTcVJPzfNv8XFgb+one2mMf781mFfvZe/DGtyPL6WsOLv+tNfcvGd4d9T+9R9uwIRC9DwUFuZwjGdasJZ48R314u21Y51nwrrEjNLrJfCXAF5sWAxTtyvJKfi3LmX+pgD6B7fwVl8/e1O+5Rnr22+ioC/k9qnL4I3lewBhot/E/rQ89e6Y9vHu/0g/aLNgcTeHf2Mh8PuVzDenGLNz3ZlYPcqrC27qF6zEhZWWF9/lk8PPkOq58AiXo4QoJOtTQLbn5klxC2ggdPF6nsWnAEuFGe+fP8L7Zxf2Cgume6owX3bhPRUk8AHY5l5/A5xTXvMm6ZoaE8S/HyymseFY+EIQinQUjv5iQunHntDrC3I4k8++D3u5F09fBE54cM6pxXZH/LvxtqkEkbfqqfiFC8pb6PZ8wrvP5rqF9HK9dwme8cJcffIAORwiiB7d6Nggh3K4Mc/7EyMByYQUYH+sDe2OVrpODF5ITFiWafAcSzVV3RWUZNeE72vg6hRdjLQm7Bv20nf0Y15ZHKLj954x84grR9x0na2l7akaLHPSvEWJLT10gzUs035nYZoW07I29no/4eMeO6HNnXgiUGAwo/vAz+nEweOECBwF2x1ZPo2cBAUlVsxHvPiTPmHSorsL/J/m8rOEOiz3m3A/2YxzsRWLBnQLa7BdduJ80UegwpzibUuy8p8kig/39iC1G56hpkxPQY5P428r1MHRAJ+lrMp6TBXgOxkcmWwwgG+WGdOYq+oUmcr2liD6ZQhm6dFpCzCXm/H1+cc1DXRBWQPb3vmYfk8Lha834zo5kVwVoC/R4T12emTA/0UAf8ouCDmaYD0a6RrkN0HwUA/eoja2rKnCNFub4VyYYIYOE34CuUwBn+rcrCiT+xsyA18RDoJ5dqx/mGY6kGb6VuWbTG1jatqxbl4Vpv2tNO+xY52ngdlV1CxUcO7pwnfKhjnpE+5s8jsV0l/zcrumZqJBO4ux3bkGgnh7fZgrLegmNf3bKJwFvjOf5fBQsADbYhve1w/iPOzEZLcO3ayPu00NCXJwpxfzY1tout+ETqvhlswrDZtjwoaXwF+SfTmee5fYW1uLfnSpxxXq0BMc06U413JIdt6/Xu6HpoIEJBM1x07LL6HrsR4CAwAaLL9oQ/9SM473QigDxGZe6vMRHAC4Be0PIRSvqUq8UumNNjjkxRcdXH43rqOTkL8vvLiOxfMRp5lVyG2jFtPcqkFDkGAonv57bryQ4qKlQTtDQzB0Lr4PShb7nUkIb6+fUGIjm66j8Dbgjp/SVOGm42k3QQUYiBLY5aAj2kDtgmv41mm2nZZVQVofdxKIByVKJIDvVBTQY7Wb8W9y4DobP4Vf9OPLcAz1c23oLivYKyyx184zrNgWKrj3e7De888p5i1PVv6T5HtaNNMheDYWWCnnvbgPk/XNu2aejRqcOH7rIxqvA8EjsXEyMVqsDzegbG2l61j8luuCl46nXVjW/BTTNZoaccra29dRvhosqi/cOJ73Yl3zU0yAbnELtZ+20rwrMFQ2kVO+obqUXBT/AS/BSGL5myicMbFsmhY3YX2nA8f+YKytRgM42zuI/rIW2wS3DUy4Hl3z/Cb4wfRCCAUJRYmdaw558KFApvPYHCv2Ej8d7a7YeQqIHvOlnoo23bn5Vi1aQrEbmgFlxDLZiobj7Wsgiv8PDjo1DdTcq2UwmPAcjrdRJYTvNRe+hHU1Wi2aYIhzA4CSLP0pasdzzNhmKSjxBzSgxbrAhrLfjafcwj+nON1nzu9USH/Ny+2amokJ63I93t+34x68vighvM814ojW0rJ47I3xxNLXYFlQA7scdPbFj68S5KM/B9OupS2zYTvRQcceE1X3DOdp3G1qeAtofgihYChWZy8GOPifPlDgSjara63U/FJD5/oOfGOu3eO4dxn4jMBhDWZDioBkmg79PN+YhxPZl0Oa8/71cj80BSQgmTAN5roWai504tgTb6yz7Gx7vRZeWk6pyYjR9CPqX/XHn2oXYPvXFqLPlWE0FlPxXOyJmHZBE9vK/TSWGjGWraTzjA6LRU/hrRPLnXI5hO/39fzIZMRomk+n0sC+tZaxfS7vbmLveti+JJZ+j2Kj6UEzrse6SPbcwGzfguW9lRiNRn70kDO2TNr9zpTRKKEjndRXxKbDm/9bhYbXW+IXpQKqfvc2LTPcrLDE9sNx0sorrzVhvqa/taDBvHYn3UU+mivj0/Ytc+D6PHbC1j2whW33h3FUxqcV/IWLUKbzwx1W7LNqYk8DAdBirawCTFjNqfuCJi3/yTDNTNOuFnhxUWwayJcUbGtqML/VmjBFcRoaCy27WijsXTlUBxxHGRFYaUqa2PvHKoKb5sfKabkTzaNv07342vV9nar2xqkOlhbHfzTV5iD80F62De7XdDMtu7oxH2lmvsmI0VTK4mddY6cGHuFboiddNC+LT626wot55++oSdY3ORczq+h+swXtf6yIHSebA3/5K+xdbc7cHzsbE61H1zq/CQqWtA3VjeKKZnyGWpoqPsPxfJKBwyPoqHl+G1VfOlhUbMRoLGblm6GUP4aW9txcYKNpfRRHmRHj3Aoch3MfrOpaE59y2VTKSo+ebTsGz5dabGu3YT3RSKnJSOmKTgJ/b8EypxDt4KjSIjtbyrysNBkxWqtxJjlmU9OOTViX66ipGL5GacurqIL0E3xkkd8pke6al+M1NRPTqp10V4aGri/G4vk0n63ilV0tmJO9gZxg+pqyFvauL8QVn4rXaHXgy/TTfjNs2B9UUMprqEo4R42/TQ1tAfuGwTpbTNnjPvQ/b8L6hYPOA9lsQYN59U5eqQrhSHLtzvne5UKIQNq3szrMC/R4jows6ezLId15/3q5H5p8f6eqqprvTAhxU7kcJapo0s6Ac9MbUIhGFZiuZbLGGV/3FAXlexr4JooyEH9qKz+G9Z2kXIyiaMYxCclkpX8ZNJp4G5ymmbzB/iL/lCjRy8C1ql/x9OR8lqOLHporXFje6cZ+I0/Ucg1JQCKEEEIIIcQkCvXWs/Q/bez7nV1+qT0L0mVLCCGEEEKISaRbto3uOT0sXeu66WfImgwSkAghhBBCCDGpNJjX7qW7yM3yDd7JnTnvJiRdtoQQQgghhBB5I29IhBBCCCGEEHkjAYkQQgghhBAibyQgEUIIIYQQQuSNBCRCCCGEEEKIvJGARAghhBBCCJE3EpB8l1wM4NrhIZTvfAghhBBCCBEnAck1E8HzaBn1vZH43356KkvpODbe7Y1j/bNuWjf7CY83yetM5EAzZQ0uIpkXzcFEj8v1Itf9yMd+3wBlHfXiqFxJ15HJrWVCCCGEGCYByTUUvTjypiZ8dmI/kzPR9W9430SJXJ38zd4s5ZrrfiQuH9nfiNHSjGdS78P9dFmMLNoRSJrmpIm4aTSW0nxoEjKvtdL2op3QhhW0HpB3i0IIIcRUkIDkJhXprce42Z+n1P10GOtxXchT8mLCCu7fRr9vC7aCTEvmcqzNNPj6eXuVaeIZHHTBRb2xgxE1vaCKbf0fs2VBxsxnZ3YVz+yoJfyYA9f5ydmkEEIIIYZJQDIeER9dqxdRajRiNJayaHUX/ihw0Y9zQyOLLEaMRiPF1vqcunoof3HheKiMYqORYms1jtFPZJUQgf1OOrd24tzvTzEWJIJ7u4vPvgV2VGM0GjGOuGGL4u91UG0txmgspmydm9BAwuoDEXzbB/ehlEWrewhcTppbgr0Oqof2tRrHexE40UPnkTDgpdUa+66+N5KxbCIHmimr68SVkPbKHQGUUalGz/pw7eikc4eL9z9N6HyWdvvx7nJbXXTVlmI0Gind4GXo2XxW5TpqO4PH37KSnlOjcnnBR1dD7DgaLYto3O4jMlTGsW5KrX9y0bokdgyqdwWHt/1cDx0Ng3Wgnp4TQXw7muPHq5Tqzd6hbSlnPXQ+Wk2ZObbPpcsduP4yusTSSLffJ3pYZBmsNzke65TlPbrbYtzVKMH3nHRudeI+lfidnw6jcWSXrgsu6uMBUGR/F66gAvRQbYylH1t2bFew6BkXjuWlw/nvDSbUreFjMtT+lrTiSQw+Ztl59glo3TDZXQSFEEIIIQFJri56aF7gILRwGx8G+ukPfEj3g2a0twJKGMVYS7e3n/7+fj58QkfXinY82fRKOdPF0gc86J98h0/6+/nwRRuhNc04v4h9rfzFSb1lEXX/EQQU/Lu68CTdUAFVv7SjCXlh1V76+/vp72/BPPS9n9APa3nl8Cf0f9zNvwSacbw5eIsVxfNoBY6v7ezs66c/8A5NBU6qn/eNCQw400P94yGqXovt68c7W6gyFEBJLU2zowSx8ky8HLqXFWQum2+iRI4GUMqf4W1fP/2eFjS/r6Zz8KZyIIL36UWUPtSJLwyEPLzw+nDXn0zbj16M4P2TB+36d+nv7+fdNRa0OZXr8HZ8/4+C9am3+bi/n3fXauh8uBP/YMBx0UNzpYPwA2/xSX8//Z4tWE7Ws+J5/1AZhs9Gcb0WwPbiJ7H6s1g/tO1gREtVRx+fBD6me2GQjuWNuGfU0n34E/o/3oJpTyPbfbEtaaJhNJXP8M7RfvoDn/BKZYDW/9FDYGy2x8i43wNhghfHeazTlPfobosAzkcqaX0nBBe9tD9cxqLN/rH1LYmC+xuw3xLCSy17+2Ppt9w9XMZDznRR/ZAH/YZYXj55rQF2LKWxdzgEC5+N4g8VUvvHeNmX+WkcFXwULGmiKdjFG2eyyJwQQgghsva9fGfgRhN4vQN3eRMf369HA4AGXZkl9uVMG7UPDi+rLbdip5NwBNCm22oUz45ONGvfpuaO2ILaO/4FW3kHnqMRanRf0fM/HUTXvk3fw7GbV9bYKDRWj2MPrFTdq4vlXWvBVgHVZ88BBXDGScd+O22fWCmYBlCArcqG8pCXwBOWhKAGuBwlNMuMWRf7UzPbPPL70bIpm7k27rsr/sfsH2Ob24rn8wjcXUD0UDuNvRa6+9qwTo8t8rOiesoOZL9905qW4fKdoYGBwLjK1fKT+zDNiP1bV27D8qSHYBjMM+P1Y3Ebn9wb7y6kNVHzqxbcti7cD3djnxn72P5oC9aZAFq0CXVDf8+PMWljn1tsNvgT2JeZYrugvRPTXAXPhb8CGiipoWFoTQ2m8ipMzwUJK7GvU8p1v3M91nFjypu/Jl3O/rs+2spi/26yd7FoWTM9le/ScFcWiWQ0tm0x00rLE3ZKVzvxL2nBPC32sbXSii7WMLBUWGFHkHjLiJlmwlIZxnE0SMMd+snInBBCCCGQgCRHEYInQpjuMaWOLy6H8L7Vg+u1g3jPRFCA2oupFh4UJHAYAvsXYXx65DdVy74Pn3pxna2iZXFuN0HWOf+Q0/KRT/2E8FJf7Bz5xaymsfe3d9lo0KykdYOWll/ch3l2AZppiQvo0c8ctc64ygYgiu+Qm8JH9g0FI0ll2H7hbT8Yufw4yzW1WP2w/uTOkeU124QZB8HzQLxMtNp0EUMuSfpxOZ24D3jxnY0CVsIXh9NJKtf9Hs+xJkl5p6CYaOtiAAAgAElEQVS5JTGtKmrndeIOhLIPSMr1pK7psbZleWjkvmr0JiyXhwPJbOmLbARO3hyTHgghhBDXC+myNZkUP53L5tNxUk/N82/xcWAvtTmsXruzP97Favi/LQu1sSfUFFKY9i3LJJn3DO+OykP/4QbGDEPWmGl6bS8109ysW1ZGcVmSsRSJJlQ23xKNgr7g9snd/rUs16lwwU39gpW4sNLy4rt8cvgZrNmsl+t+53qsJ+QW0EDo4vV5068tLIRPgzKORAghhJhEEpDkpAB9iY7A0QDJbpeUI266ztbS9lQNljmjnyKnY8KyTIPnWIre/zN0mPAT+CLb7SlEv8l22WEFJVbMR7z4s3prAWhN2Dfspe/ox7yyOETH7z2xGzVFGVM+4y8bgNsonAW+M5+lHFswru3nXK6ZxOqH99jpkfn8IoAfG6Y5k5VOTPBQD96iNrasqcI0W5t9mY5nv3M41hMy8BXhIJhnx/qHaaYDaaZ2Vr7JFBjpMVWA72Rw5HrBAL5ZZkyFuWUvEgqCQc8kzd8lhBBCCCQgyZlpcRPWD1ppfj2IMgAMKIT6fAQHQHOrBg1BgqH45++58ULaG6oYDZZftKF/qRnHe6Ex22WOFXuJn452F8H4/Vf0mI/Uk/pq0M7QEAydA0C5nOXT7Nl2WlYFaX3cSSAelCiRAL5TSW45v/DiOhbP62Cqswq5DeBWLVpChCKx/VAmVDax/bEsqIFdDjr74nlRgnz05+GbzHFtP+dyzcy0uAnrOx049sfrRzSAs72D6C9rsc2YwIaT+MH0QggFCUWBgSiBQx58KDCQYcVc9zvHY52raDh+TAei+P/goFPTQM29WgaDCc9hH9EBQAnhe82FL2FdjVaLJhji3ACgJEtfi/XhBpStrXQdi6dzwUvH0y4sa36KKafAGM596sM080Z9pSaEEEJcnyQgydXMKra93obuzRWUmowYTT+i/lV/7I3E3U3sXQ/blxgxlq2kR7HR9KAZ12NdmWc+mmVn2+u18NLysdtFR83z26j60sGiYiNGYzEr3wylHSdvtm/B8t5KjEYjP3rImdXMS6DBvHYn3UU+mivjU8kuc+D6fGxAolwO4ft9PT8yGTGa5tOpNLBvrSU2dqLARtP6KI4yI8a5FTgORyZWNoCmrIW96wtxxaeRNVod+BJLYFzbz71cM5pZRfebLWj/I14/bA785a+wd7U57Tjz8ShY0sa2cj+NpUaKK5rxGWppqvgMx/OeDG8tctvvnI91jlxr4sfUVMpKj55tO5owxweX29Zuw3qikVKTkdIVnQT+3oJlTiHawdFvRXa2lHlZaTJitFbjTDIDlqakib1/rCK4aX5s2t/lTjSPvk334hzfcwwE8L2jx14WH48S8dA86ocehRBCCJG7v1NVVc13JkT2lItRFI0WbbrB3TczJUr0cvzJeI5Pt9Nu9jtarvneb+UyaDQK0agC0zSTN9h/CoRer2d+XxUfb62aWNAqhBBCiBEkIBFCiEzOu6i/34Ntfzf2WfnOjBBCCHFzkWl/hRAinS9cNC7vQbd1pwQjQgghxBSQgEQIIVKJeGhe4cb84l5qS6SjlhBCCDEVpMuWEEIIIYQQIm9kli0hhBBCCCFE3khAIoQQQgghhMgbCUiEEEIIIYQQeSMBiRBCCCGEECJvJCARQgghhBBC5I0EJEIIIYQQQoi8kYBECCGEEEIIkTcSkAghhBBCCCHyRgISIYQQQgghRN5IQCKEEEIIIYTIGwlIhBBCCCGEEHkjAYkQQgghhBAibyQgEUIIIYQQQuSNBCRCCCGEEEKIvJGARAghhBBCCJE3EpAIIYQQQggh8kYCEiGEEEIIIUTeSEAihBBCCCGEyBsJSIQQQgghhBB5IwGJEEIIIYQQIm8kIBFCCCGEEELkjQQkQgghhBBCiLyRgEQIIYQQQgiRNxKQCCGEEEIIIfJGAhIhhBD5d8FH1+pFlBqNtB6K5js3QgghriEJSIQQQuTXeRf1lSvpPBQkOt2KpUg7ZhHlWCf12/0oecgeQPSDDhxvhfKUuhBC3Nz+TlVVNd+ZEEII8V2l4Hu6Gvcdz/Bv95so0CRZ5LyLxk3Q8rwd3bSEzyM+up7swPMVgBbzQ220LNOTbBNZG1BQ0KCZNvoLBf/WapzGV9iycGzAJIQQYvzkDYkQQtyAoseceL7Idy4yieB5cinF6zyk7ISl+PB8UcO/LUsRjAwEca51YVkzKhi57MWxwEF01V729e5j3+vPYPrPahp7x/8WI3rWTceyRtzhZN9qMD/SRuFWB56L405CCCFEEhKQCCHEDSgccOGP5DsXGUSP43k9gLLfgy9VRHLKR2jhjylI8bXS56RjZg1Vc0Z+HviTA2dRLTV3x6OYaTrsP7fjfboHX079ugK41tRTvawRx+ZOes6kWVRjpmZFiI5XA7kkIIQQIgMJSIQQ4oYTwrf/BrgpjoTR/LwWK248f04ekfgP+7GaU4UjUTy9TuzLbIzsJBXA6wpBkR5d4sez9VgvO/EcySUiMWHf2s3e3m386if6jEvr7qmi8CVXjkGPEEKIdCQgEUKI691A4r+jBHY56DiRt9xkLfqpH21FDbZycL/jTdJtK4DvqBXznCQrA1zw4D5gxWwa1ZcrEsR/PnW6/i+mcPD5bDPWGbkGPUIIIdKRgEQIIaba5SDuzY0sXbaUpUuWsnR1F76E7laRA60stRZjNBqp741/caKHpUvKKDYaMT7iIvZpBM+GlbTuic025XpiaWybT3q4HntvBfs1mO/Q8eOFVjjgwTt67MX5AL4SM6YU6yufBvDOMqMf/QLlavpgIHBhKqcN1mMqB++Z4BSmIYQQ3y0SkAghxFS67Kdj2VK8d/w6Nvj6zX3s/UUUx4J6XPGn/AULn2GfZxs1ieuV1LLvzddom5f4YQG2p/ax72k7APan98W2+ZRt5BiMgQBdlcV0ZvsWRQkR6PMTzDRYOxJJPTh9jCCBsyb0Wigos2HFg/PAyDcX0ZM+Cs2pwhEIXwiCvpDbR39xIYQv63xMNg2Ff68j9Gkoh7IQQgiRjgQkQggxZRT8v2+mZ3oLDfcPhwyau5toWeyldYNr+M2GRptkutpbGNccttNux7SsDZsh86LRPgeLLMtpfd5BY2UxS5/zEhlIvmzE+z5Zvxe44MdvMqMHmGmjaiH4X3ITGNq2gt8XxVKSegrd8Bc+MOhTDnjPl9sL9BCN8m2+MyKEEDcJCUiEEGKqDATw7AlBkW7k4Gs0aG8FPvDw0ZT0tSrAusqOaXqGxS566HjdRPfRPvb17uPtvg955h9dLF7tIjQmKIny0acK2f4CR/Soj8K5g28/tFgX2OB8F67B0eADAXxnrJhnZr1Tw2bqsIxjtUkVDPNVvvMghBA3CQlIhBBiqoSDBC8DtyZ7+wHgJZDH3xKJHglgWpvw+x7TtJge3MZbDwZoXNaB98LwspFDnXgN95F5HiqACJ7/0GK9a/gT7QI7NSg4X4//JskXAfxzTVlub5Tp2rSBkWmm/HChEELcSCQgEUKIqTKjMPZm5EI4xXgDC/rxvCEYxb+5Hlc8eAgd6qTj0XrqX83cuSoY0mKePfbzgvI29m7W4VpSTNmSpSy1FrPiPQtNS7LsPPXFQdyzbJgTozCNBdvDGjjQhfMMRPxeTPNSjx9JS6tDP4uxg9sHQAHMs3XJ1ppcyca2CCGEGBcJSIQQYqoM3oR/Hmbkj39HCV8ASmxYZqXbwCWUr3NIL+rBc/lntCzX4/sgkHHmLdNDNSlnuNL8Uw3b+j7mna1b2NL7MW8/axv5S+mjDUSIxBMM/IcT/b3mUW+FNFgebsFMkM7nnRw8CSZD+gEy2pkm+DSYZD9MWJfr4EyIxGHySjCAb3oNtnkjt6tEJ3eK3q8iQdBq+f6kblUIIb67JCARQogpo8Gy6hls53tw9SW8I/mLi57DVp55viZhbIkG7SzwfvrZ0CfRIy6cpwAFriRudo4JGxD6MgpECUXNmGYCFwsxlxcQOOLBVG7OOBhcMz3DiPlpGrSz9egLMo+sj7z5OGULyli6bBErvTXUWJKsM9tO08Ma+MCB44N4ntPQzTanHKtheriNmpM9OI8NjkkJ4d7lwvJoDYlJB3csori0lJWvT9ZvkyiEvwyhM+iyHk8jhBAivb9TVVXNdyaEEOKmFg3ger4D58l4UPL3NlqeaMAy6oZcOdVD8/qXOa65nduna9D9qAnbpZU0/0lDwR16/u8N+6gtiS0bOuCg+bmDKLPuxLZuGw13x+/CBwJ0LeikcE839knoDpYt/9ZiqrcrgJ7aPftouTtFEHPRQ3NFK8qGfWxbnKFr1QUX9VYPVX3dVCWLrr7w0tHeie8rAC3mh9poWaYf8WYmsr+RynVelIXPcHhrVZIgLYLnyTq6AqCcDxC8CJqZpthvn1S2sW+VedTyAboqqolu/oSWu9NnXwghRHYkIBFCiOvZAJCuq9RoxzooflHP4ZcsnDumxXz3NXqOPxAleCJAdIYJ85zJSjOEc/l8gv/6CW3l45n/eFCArseD/PTZZAFJjr5wUr0kSJOvjWQvgYQQQuROumwJIcT1LJdgBPB/6MS8wELBKS+B6dewU9E0Lfq7LZMYjADoqPpFFa5ez8R+hPALP6GizF3YshH6s5twnV2CESGEmEQSkAghxE1Eb7ajvNdJa58J+x35zs3EaRc00HLBifvseLeg4N0VwLpwEmbeUvw4d+poeXCcs4MJIYRISrpsCSGEuL6dd9G4CVqet6ef6SuZCx5c563YU41pyZqCf2s1bvPeCXYfE0IIMZoEJEIIIa57yrFOGo9Y2fbL0dMJXxvRDzrovFhDW6aB+EIIIXImAYkQQgghhBAib2QMiRBCCCGEECJvJCARQgghhBBC5I0EJEIIIYQQQoi8kYBECCGEEEIIkTcSkAghhBBCCCHyRgISIYQQQgghRN5IQCKEEEIIIYTIGwlIhBBCCCGEEHkjAYkQQgghhBAibyQgEUIIIYQQQuSNBCRCCCGEEEKIvJGARAghhBBCCJE3EpAIIYQQQggh8kYCEiGEEEIIIUTeSEAihBBCCCGEyBsJSIQQQgghhBB5IwGJEEIIIYQQIm8kIBFCCCGEEELkjQQkQgghhBBCiLyRgEQIIYQQQgiRNxKQCCGEEEIIIfJGAhIhhBBCCCFE3khAIoQQQgghhMgbCUiEEEIIIYQQeSMBiRBCCCGEECJvvpfvDNyULgZw9YawrLKhuxnSybfvyn6K64RC6AMX3lurqLlbm+/MiBvA3765kpd0/9utt+QlXSGEmGzyhmQqnHXTutlP+GZJZxIpp3pYaTFiNJZS/Wowu5VuwP28rkU8NFvrcV3IcvmBEJ4nl1JsNFJsdeCNTmnuAIgcaKaswUUk2Zcnelhk6cA/Zan/Fd8uB97Pv52yFK4dPz2VpXQcS71E2rIWQgghrgEJSHJxwUW90Ygx5X853OR9Fw346XzYjfWP/fQHPuSVB/T5ztENxE+XxciiHYGJb+pqlGgO9TTypoPGgQY+DPTziacF67duGo2lNB+awlvYb6JErqb4biBM8OLUJX39y60uhM9miCDTlbWYXOF3eeFXj/KY81S+cyKEENcVCUhyMdNOd38//fH/9q4CVu0d+ru/vxv7zHxn8joWDhK8XIh2BjBNg2ba2EUivfUYN0/ds+9rlcbkM9Pg6+ftVaZrnvK5s16YUYh2GqDRQEEV2/o/ZsuCgmuel8nnp+OGe5CQv7rwXeXfbKS+dxIC8ML/jUKAC/8fX018a0IIcdOQgGTKRPH3Oqi2FmM0FlO2zk1oIOHrgQi+7Y0sindfWrS6h8Dl5FtS/uLCsbw09hbGXEb1096E7hWp01HOeuh8tJoyc+wNTulyB66/KINbJXTCjedYhMgxFz1be3D1BYkOjEo84qNr9SJKjUaMlkU07gigkMJABN/2+nh6pSxa3YUvntHI/i5cQQXw0mqN5WdkN5II7u0uPvsW2FEdf+OU2C1nMsozQxop9zWC59Ey6re6Er5fSc+phJKI+OhqKKPYOJi+k8Dg12nKBeJdZuo6cW1fGd92sm5R8TwM3hSlSy+Zi0F8vT10bu3B9UFwVPc3hWBi2T7kwHMewE/PVh/hqwnlVeciMrob0IkeFlla6UnYxtInPTkfn+hZH64dnXTucPH+p5k76CnRIN5XO+l81U1gMKAYCNBVYaT5QGIBKvieLqb4ad/Iunuih84jYRLrZOJNpxJ6P+F4V9N1bGQBK39x4XioLN6VrRrHgdDYTJ7qYr6xEXfiGx3Fh8NcjKNPydBGY+Xc+icXrUti5Vq9K8jIuqAQPNBJ80PDdaF6g4vg6Lp/NUrwPSedW524T2W4sc723HTRj3PD4HJGiq31dB0Z3nbKej0Qxb8jsT046Hh0JUvXuYlccFE/OkA81jGinca220HP5vg2zGXU7/ATPNIzVA6lyzvwJu5mun1KW3+H24D38bKENjBed/Gz9k08t24+t497G0IIcRNSxbgd32RQDZuOj/3iaLtqMFSqGw+fU69cVVX10kdq+0KDWucKxxe4pB5cW6RWPvW+Gr6qqurVsHrw1xVq0VMfqVfGbOy0uv1eg1q351zsz2/Oqcf957JLx79b3X74M/XSVVVVr15RT/9xuWq4d7t6WlVVVQ2rb6wyqEUlc9V7HtmovvDsOnX5PINa9Mgb6rmr8aS/PqiuK6lUNx6Ob+/Lg+rGe4vUjR+OzeXYfbqknt5ZpxYtbFePfxNfJPSGusJQp77xZY5lOqnlmSKNtPsaL6tHdqunv459fe61OrWopF09fnUwfYNa8exHsbK+ElY/O/qZeinLcgm76lSDoUit23k6tv7Xl5LkO5aH2D6nS2+ssHejWmmYqy5f366+0LlRrSsvUg0Jx+H0i5WxfbsUz98f61TDA7vVeC1TP3p2dHkdV9sNBrX96CQdn6th9f2nKlXDvOXqumdfUF/4dZ16T4lBNax6Qw2rSRxtVw2GIrWoPLb8xkfuUYsMlWr70Svx/alQDWsPDpfH1eNqe0mFuv1kkm19+YZaN6ZOZjreqqoGtquVJXXq7kD8KAdeVusMy9Xdn49OINZ+V7uHj86VDzcObyttG42Vs2HhRvX9L2PH5tKl4fwNlu/xPdvV94Px7V85rb78gEGtePH0iGNVVHJP7PjHy7Zy0/GhOhZ21SWUdQ5t6cuD6st7jqvn4l9c8mxUiwzr1IOXErabpF7H0ns5Xt/OqW88UqTe868vq//uPa1eSnY8jrarBkO7OlgDw6461XDvOvWNwbL/sF2tMBjUysfeGKrD7/+6KCHPGfYpY/0dWd7J/PVv/6X+9X/tVx2rG9XVQ//9QfX97b+Sfud464vY53/7L/Wvf/tC/feNievF//vDx8PLHPlDwndPqf/+v2KfCyHEzULekEwZK1X36mLdkrQWbBXgPXsu9tUZJx377bSst1IwDZhWgK3KhrLLy9he4QrR8zrMRfH5pabrMJckzjWVJp2SGhru1ce62kzTYCqvwnQ+TDjhQa/50X30dbXR9NgW9u7fgs3XSsc7safLgdc7cC9uoeXeeNecmTaqKhWcf07Sd33MPmkxPdxGi7aHrnfizxOngWZ8hZl+P3Mqz+Sy2VfLT+7DNCP2b125DcvlIMEwwLdEo2AqMsW7NRWgv1uPNttyAbiriZaH4uvP0GYopzTpjXbRQ/saF5aXPmTvphaa1rTR/Vob1sHvox56ntfQsr4Gkzaev4U2rCe8+OJPqTVZzcU3/uMTPdROY6+F7sN72fJYE00bunnrCWuatADsQ8u3dfWxdw30rO8hMACmBbWY9rvxDr6VOOHBOcOO9Y5s9mNY6uMdxbOjE83aFmruiJW69o5/wVbux3t09LNzE1V1ZjyHvMRalYL/PReFdVWYp5FVG7U/2oJ1JjBNizbJQTY/2IB1TvwLjQnr/SZCX4ZHvA2y/64vdvw3dNO3qwl2NNOTbBhDLm1ppo3aB83o4pVVW27FTpBwYhGMqdcR3j/gpWqZPV7fdNh/XkOkL4qu3JS8Diejt/DjwbIvs2EDrEPb1HJnkQXlixB/zXqf0tTfrJxi928OwoJmnmvfxHPtm/i3BV+xe3CsSOF8/q19E8+1N2MrHL3u7dy7btPQes89fBcAd5bE/k/4XV7Y9RW2wWUevh3Pb3ZyOofcCSHE9U6m/c2DyKd+QnipL3aO/GJWU5IbURO2X2pY+UQr2vW13DdXT0Eud/URPy6nE/cBL76zUcBK+CIQH+ui+X7CtJEFVdgfaGblyQDc/38QPBGCw/UUvzpyk7q1YzMQ+dRPqNzGnSO+0mGaC46z54DsxxtY5/xD1ssOpZ11eSZLI5LTvo5VwI/vt+J4upEuWqiqMKGbnpC3bMrlh1p+kGRMTa7pjRY94sE9o4F95Sn242wADwHclUYcI76owp5im7lKf3yi+A65KXxkH9ac0hs5Bsm0uBbLVjeBCw2Y5lixlzjw9EWpul+L/7AT/S/2YUpZvnr0OY39ChI4DIH9izA+PfKbqmXfH7O07p4qzBvceC5UYS8M4O0txL4nYfxHhjaq1Waug5ETLnbvceP5wBcb8F8e5q8MPwDQJM4Oe1cVtfM6cQdCNNw1ciLtnNvS5RDet3pwvXYQ75kIClCb2D0tp3o9NcZzfkhG/4+5jZm6fX4zz+W0BvHg4xS3L2jmZ3fGPjrtOchXRSu4dzCQufOfuZOdeN79inn/8r/nmoIQQlyXJCDJl3nP8O5Oexa/q6HBvHove3WdOB5fiuOCBsv6V+heZcp8Mb3gpr6ylejPn6HtxTb03/PQWOFJn9r3gAtRBnvgW556l1ceuAF+/SPr8kwt9b5m7jFesLibwwVdbHx6JZ3rFEwPbGHbBhtT9SsBqdLTjbr5+/abKOgLM/RXr+WVQAuWqbxxTHl8IgSioC+YYI/6aaAhRDgKzNJhXWLGcchLtFKHZ48J+5uTP6Nb7c5+WuZlseDs+7CXO/D0RbD/owfnnFr2Db6tGUcbHS3yVj0VG6LUPtVG9xN6ONTI/APp1rgFNBC6mGL2rWzbkuKnc1k1HnMbbc+/xZbZ5+g0VWdYqQDTXD2tvS5q76nFdGsI15+c6B/ZizlTehMxCeeH9O7iZ+vu44XfbOGxQwmfta/gzqy3EXvL8lXRCp6bP6o9nNzJY78a+ZGMQRFC3Eyky1YeFJRYMR/x4s926tJpWkzL2tjr/YSPe+yENnfiyWJUZfBQD96iNrasqcI0W5t0VquRooRCoDPo0FKAudyMr89PNj87UWAwo/vAz+kRg19DBI6C7Y5sbwYVot9kuWhi2jmVZ7I0ctvXlPkoa2DbOx/T72mh8PVmXCcnq1yyT2+02wp1cDTAZ6kGvN9hwT7dg//MhLKSPp9pj89tFM4C35nPUk+WkI1ImCBm9LNif+rutWM94MH7jgfnXDv3zU6xnqKM45ibsCzT4DmWbYfAAmyLbXhfP4jzsBOT3crgkc+9jY4W5OBOL+bHttB0vwmdVpM5CB74inAQzLPH3p7n0paUI266ztbS9lQNljkFWefd9PM2ar94mTqbEePc5XiKutk5OGPY9zSAAqMn15iAnM+3Y1xByaaSDHXLGuyadWq4y1ZGX/Heb3ZyuvA+/q3mrrFfF60Y7tI12CVsdNAihBA3MAlI8mG2nZZVQVofdxKIXySVSADfqWRXvRDeXj+hxBva6ToKb8uczA+mF0IoSCg+s03gkAffqIt9+OJXKPG/Q2856PzAStPi2M2BbnELtZ+20rwrEJt9a0Ahcso3lOcR7vgpTRVuOp52E1Ti6e1y0BFtoHZBtj3DNWhnaAiGYn23lctZ3qLmVJ7J08hpX8eI4j/gJRhJzK+JwhlMUrnkkN4omnk2anDi+K1vaL+CR3wM/SSlxkLtE3q61jvwno9v73II35Esf7QyG2mPjwbLghrY5aCzL368lCAf/TlT+tFYtyaAqJ+upzrR/LIG62CRzrRRtdBD5/MerIttqTsM3qpFS4hQBBhQhtpCehosv2hD/1IzjvdCsXUGFEJ9PoIp1teW2bCd6KBjj4mqe4YDgWzaaHo/QPNDCAVDseN7McDB//SBAom/HR4Nx8t2IIr/Dw46NQ3U3Juk/uXQljS3atAQJBiK7/97brwAaX/TJIK7aSXKY4fp8/XT7++j+5eW2NgOgAI95lk+PB/EZyy7GMDV6822MJLL6fyQzC1ofwihUOwpkJJsxrHwu7zwq/GP6zjt3IInfBc/SzL71p0ld8HJA7w3NPHcKXZPIC0hhLgeSUCSFxrMa3fSXeSjuTI+3ecyB67Pk1wglSihI53UV8SWm/9bhYbXW7Bk0fm5YEkb28r9NJYaKa5oxmeopaniMxzPe4aeCgc2L6XYFN/2c2Fqdm2jarA//XQzLbu6MR9pZr7JiNFUyuJnXQSTXscLqPrd27TMcLPCYsRomo/jpJVXXmvCnENHbbN9C5b3VmI0GvnRQ84sB6XnUJ6p0shpX0f7luhJF83LYlMzF6/wYt75O2pmw2SVS/bpjS4aCy27WijsXUmpyYjR9CMcRxkxeFi3bBv7VkHXQ/GppSvqcZ6ITuht0ahMpD0+mrIW9q4vxFUbT9/qwJdxeLOLxvh0s8bSlXgM2+hebU7oxqjFusBG6LwJW1mabRXYaFofxVFmxDi3AsfhLCd0nWVn2+u18NLyoXKtf9Wf+g3fDBv2BxWU8hqqEo5TNm00vQLsG7ZhPdFIqamYssd96H/ehPULB50JUx+71sTL1lTKSo+ebTtS1b8c2tLdTexdD9uXGDGWraRHsdH0oBnXY11p2m0B5v9ehe+R4oQflC1l0eoOvOcBTNT+tgVeXBSbgvpxNxqDhYI7xowEz0Fu54dkebb9awvR58owGoupeM6TpBNnEbYFX7H7V4/y2K8e5bFfbcFDwtuOwR9E/NUWPGH46tCW2HLOU6FO+FEAAAHaSURBVMAp/CchFmgMrv8oj/3m3dhvldy5Ij6QffC7nYQXzILTQMRD82T9YKoQQuTR36mqquY7EyIfIrjqyvBUvM2WhYUMvjkQN6kBhWhUgelashgjnR9KlOhl0GgzdF0aUFDQoFGiRBVS7lN0fyOlR+18ssE6gdndbnzKZdBo4sd/miarQfJT5oKb+iVubK91Yx8MzAaieDctpf7/baKvqyqH6S+uH3/75krmhabAf7t1qkapCSHEtSWD2r/rvv8DtDPG23VI3DCm3QABpybLYGmaJhZgTNeiTTkzVwTPWx5qHt7ynQ5GADTT4Xp54BDpc+MtsvFs4tu8aVruNOghomHsHGVCCCG+C6TLlhDi5nE5SnQAlGMv0xNswl6W/5twMey2mTo0H/SwfXDsDRA95eKFHSEaHrZm/zskQgghbioSkHyHaWeaKLw137kQYvIE36pnvslI6dog9t/WpvntEZEPmrI2Du+0Ex0ce2MspfrFIJbf7qOpRIJHIYT4rpIxJEIIIcQEyBgSIYSYGAlIhBBCCCGEEHkjXbaEEEIIIYQQeSMBiRBCCCGEECJvJCARQgghhBBC5I0EJEIIIYQQQoi8kYBECCGEEEIIkTcSkAghhBBCCCHy5v8HhkS7VOFvdFkAAAAASUVORK5CYII=)" + ], + "metadata": { + "id": "twWMxhZaO2cs" + } + }, + { + "cell_type": "code", + "source": [ + "# Create a Linear Regression Model Class\n", + "class LinearRegressionModel(nn.Module): # <- almost everything in PyTorch is a nn.Module (think of this as neural network lego blocks)\n", + " def __init__(self):\n", + " super().__init__()\n", + " self.weights = nn.Parameter(torch.randn(1,# start with random weights\n", + " dtype = torch.float),\n", + " requires_grad= True) # we can update this value with \n", + " self.bias = nn.Parameter(torch.randn(1, dtype= torch.float), requires_grad = True)\n", + "\n", + " # Forward defines the computation in the model\n", + " def forward(self, x: torch.Tensor) -> torch.Tensor: # <- \"x\" is the input data (e.g. training/testing features)\n", + " return self.weights * x + self.bias # linear regression formula (y = m*x + b)" + ], + "metadata": { + "id": "hZvv1DINFaBQ" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xhu5wxVO7s_q" + }, + "source": [ + "> **Resource:** We'll be using Python classes to create bits and pieces for building neural networks. If you're unfamiliar with Python class notation, I'd recommend reading [Real Python's Object Orientating programming in Python 3 guide](https://realpython.com/python3-object-oriented-programming/) a few times.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "iRRq3a0Gvvnl" + }, + "source": [ + "### PyTorch model building essentials\n", + "\n", + "PyTorch has four (give or take) essential modules you can use to create almost any kind of neural network you can imagine.\n", + "\n", + "They are [`torch.nn`](https://pytorch.org/docs/stable/nn.html), [`torch.optim`](https://pytorch.org/docs/stable/optim.html), [`torch.utils.data.Dataset`](https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset) and [`torch.utils.data.DataLoader`](https://pytorch.org/docs/stable/data.html). For now, we'll focus on the first two and get to the other two later (though you may be able to guess what they do).\n", + "\n", + "| PyTorch module | What does it do? |\n", + "| ----- | ----- |\n", + "| [`torch.nn`](https://pytorch.org/docs/stable/nn.html) | Contains all of the building blocks for computational graphs (essentially a series of computations executed in a particular way). |\n", + "| [`torch.nn.Parameter`](https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html#parameter) | Stores tensors that can be used with `nn.Module`. If `requires_grad=True` gradients (used for updating model parameters via [**gradient descent**](https://ml-cheatsheet.readthedocs.io/en/latest/gradient_descent.html)) are calculated automatically, this is often referred to as \"autograd\". | \n", + "| [`torch.nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module) | The base class for all neural network modules, all the building blocks for neural networks are subclasses. If you're building a neural network in PyTorch, your models should subclass `nn.Module`. Requires a `forward()` method be implemented. | \n", + "| [`torch.optim`](https://pytorch.org/docs/stable/optim.html) | Contains various optimization algorithms (these tell the model parameters stored in `nn.Parameter` how to best change to improve gradient descent and in turn reduce the loss). | \n", + "| `def forward()` | All `nn.Module` subclasses require a `forward()` method, this defines the computation that will take place on the data passed to the particular `nn.Module` (e.g. the linear regression formula above). |\n", + "\n", + "If the above sounds complex, think of like this, almost everything in a PyTorch neural network comes from `torch.nn`,\n", + "* `nn.Module` contains the larger building blocks (layers)\n", + "* `nn.Parameter` contains the smaller parameters like weights and biases (put these together to make `nn.Module`(s))\n", + "* `forward()` tells the larger blocks how to make calculations on inputs (tensors full of data) within `nn.Module`(s)\n", + "* `torch.optim` contains optimization methods on how to improve the parameters within `nn.Parameter` to better represent input data \n", + "\n", + "![a pytorch linear model with annotations](https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/01-pytorch-linear-model-annotated.png)\n", + "*Basic building blocks of creating a PyTorch model by subclassing `nn.Module`. For objects that subclass `nn.Module`, the `forward()` method must be defined.*\n", + "\n", + "> **Resource:** See more of these essential modules and their uses cases in the [PyTorch Cheat Sheet](https://pytorch.org/tutorials/beginner/ptcheat.html). \n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HYt5sKsgufG7" + }, + "source": [ + "\n", + "### Checking the contents of a PyTorch model\n", + "Now we've got these out of the way, let's create a model instance with the class we've made and check its parameters using [`.parameters()`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.parameters). " + ] + }, + { + "cell_type": "code", + "source": [ + "# Set manual seed since nn.parameter are randomly initialzied\n", + "torch.manual_seed(42)\n", + "\n", + "# Create an instance of the model\n", + "model_0 = LinearRegressionModel()\n", + "\n", + "# Check the nn.Parameters within the nn.module subclass we created\n", + "list(model_0.parameters())" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "DcRDs0QhUhRj", + "outputId": "058729be-9cca-44b6-fb6c-fae987babb34" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "[Parameter containing:\n", + " tensor([0.3367], requires_grad=True), Parameter containing:\n", + " tensor([0.1288], requires_grad=True)]" + ] + }, + "metadata": {}, + "execution_count": 7 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "CNOmcQdSq34e" + }, + "source": [ + "We can also get the state (what the model contains) of the model using [`.state_dict()`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.state_dict)." + ] + }, + { + "cell_type": "code", + "source": [ + "# list named parameters\n", + "model_0.state_dict()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "u68u2yXAVjww", + "outputId": "21a344c9-36dd-4efc-b0ef-b55d3752c391" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "OrderedDict([('weights', tensor([0.3367])), ('bias', tensor([0.1288]))])" + ] + }, + "metadata": {}, + "execution_count": 8 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tdTEPSwSec02" + }, + "source": [ + "Notice how the values for `weights` and `bias` from `model_0.state_dict()` come out as random float tensors?\n", + "\n", + "This is becuase we initialized them above using `torch.randn()`.\n", + "\n", + "Essentially we want to start from random parameters and get the model to update them towards parameters that fit our data best (the hardcoded `weight` and `bias` values we set when creating our straight line data).\n", + "\n", + "> **Exercise:** Try changing the `torch.manual_seed()` value two cells above, see what happens to the weights and bias values. \n", + "\n", + "Because our model starts with random values, right now it'll have poor predictive power.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BDKdLN7nuheb" + }, + "source": [ + "### Making predictions using `torch.inference_mode()` \n", + "To check this we can pass it the test data `X_test` to see how closely it predicts `y_test`.\n", + "\n", + "When we pass data to our model, it'll go through the model's `forward()` method and produce a result using the computation we've defined. \n", + "\n", + "Let's make some predictions. " + ] + }, + { + "cell_type": "code", + "source": [ + "# Make predications with model\n", + "with torch.inference_mode():\n", + " y_preds = model_0(X_test)\n", + "y_preds" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "Hyt6oiYfV7V7", + "outputId": "44d90ef6-5473-4836-b3be-f16aee7e70d7" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[0.3982],\n", + " [0.4049],\n", + " [0.4116],\n", + " [0.4184],\n", + " [0.4251],\n", + " [0.4318],\n", + " [0.4386],\n", + " [0.4453],\n", + " [0.4520],\n", + " [0.4588]])" + ] + }, + "metadata": {}, + "execution_count": 9 + } + ] + }, + { + "cell_type": "code", + "source": [ + "# Note: in older PyTorch code you might also see torch.no_grad() ---> Lets also try it\n", + "with torch.no_grad():\n", + " y_preds = model_0(X_test)\n", + "y_preds" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "ZXgCJU3ZWdMT", + "outputId": "2d64bd29-a9e7-4ee4-95d3-6c571757b636" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[0.3982],\n", + " [0.4049],\n", + " [0.4116],\n", + " [0.4184],\n", + " [0.4251],\n", + " [0.4318],\n", + " [0.4386],\n", + " [0.4453],\n", + " [0.4520],\n", + " [0.4588]])" + ] + }, + "metadata": {}, + "execution_count": 10 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "L_Bx5I1FsIS0" + }, + "source": [ + "Hmm?\n", + "\n", + "You probably noticed we used [`torch.inference_mode()`](https://pytorch.org/docs/stable/generated/torch.inference_mode.html) as a [context manager](https://realpython.com/python-with-statement/) (that's what the `with torch.inference_mode():` is) to make the predictions.\n", + "\n", + "As the name suggests, `torch.inference_mode()` is used when using a model for inference (making predictions).\n", + "\n", + "`torch.inference_mode()` turns off a bunch of things (like gradient tracking, which is necessary for training but not for inference) to make **forward-passes** (data going through the `forward()` method) faster.\n", + "\n", + "> **Note:** In older PyTorch code, you may also see `torch.no_grad()` being used for inference. While `torch.inference_mode()` and `torch.no_grad()` do similar things,\n", + "`torch.inference_mode()` is newer, potentially faster and preferred. See this [Tweet from PyTorch](https://twitter.com/PyTorch/status/1437838231505096708?s=20) for more.\n", + "\n", + "We've made some predictions, let's see what they look like. " + ] + }, + { + "cell_type": "code", + "source": [ + "# Check the predictions\n", + "print(f\"Number of testing samples: {len(X_test)}\")\n", + "print(f\"Number of predictions: {len(y_preds)}\")\n", + "print(f'Predicted values:\\n{y_preds}')" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "vNtUa8RYXk76", + "outputId": "10a8ac53-2457-4651-849e-ffda3d54a50d" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Number of testing samples: 10\n", + "Number of predictions: 10\n", + "Predicted values:\n", + "tensor([[0.3982],\n", + " [0.4049],\n", + " [0.4116],\n", + " [0.4184],\n", + " [0.4251],\n", + " [0.4318],\n", + " [0.4386],\n", + " [0.4453],\n", + " [0.4520],\n", + " [0.4588]])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "FnSwGbQEupZs" + }, + "source": [ + "Notice how there's one prediction value per testing sample.\n", + "\n", + "This is because of the kind of data we're using. For our straight line, one `X` value maps to one `y` value. \n", + "\n", + "However, machine learning models are very flexible. You could have 100 `X` values mapping to one, two, three or 10 `y` values. It all depends on what you're working on.\n", + "\n", + "Our predictions are still numbers on a page, let's visualize them with our `plot_predictions()` function we created above." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 428 + }, + "id": "pwjxLWZTec02", + "outputId": "cfe6fff8-38b7-46e2-c70b-c0708c52f0f6" + }, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAGbCAYAAADgEhWsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAAsTAAALEwEAmpwYAAAuXElEQVR4nO3de3xU9Z3/8fcnCZdwFSWg3C/iBREUIsq6ghesF0Dqz7WA1oJaRZGt7ErViqKgtbW1WlujjVaEihcU0aWAaMuCtxUhgFBDQBGogBGC26UCFUjy+f0xMU1Ckpkw95nX8/GYR3LO+c45n+SE8M73nPmMubsAAABwZDLiXQAAAEAyI0wBAACEgTAFAAAQBsIUAABAGAhTAAAAYciK14Hbtm3r3bp1i9fhAQAAQrZq1ard7p5T27a4halu3bqpoKAgXocHAAAImZn9ta5tXOYDAAAIA2EKAAAgDIQpAACAMBCmAAAAwkCYAgAACEPQV/OZ2QxJwyXtcvc+tWw3SY9JulTSfknj3H11uIX9/e9/165du3To0KFwd4UU16hRI7Vr106tWrWKdykAgDQUSmuEmZIel/SHOrZfIqlXxeNMSU9WfDxif//737Vz50517NhR2dnZCuQ14HDurn/84x/asWOHJBGoAAAxF/Qyn7u/I+l/6xkyUtIfPGC5pKPM7Lhwitq1a5c6duyoZs2aEaRQLzNTs2bN1LFjR+3atSve5QAA0lAk7pnqKGlbleXtFeuO2KFDh5SdnR1WUUgv2dnZXBIGAMRFTG9AN7MbzazAzApKSkqCjY1RVUgF/LwAAOIlEmFqh6TOVZY7Vaw7jLs/5e657p6bk1Pr29sAAAAklUiEqfmSfmABZ0na4+7FEdgvAABAwgsapszsRUkfSDrRzLab2fVmdpOZ3VQxZJGkzZI2SXpa0oSoVZuGxo0bp+HDhzfoOeeee64mTpwYpYrqN3HiRJ177rlxOTYAAPEQtDWCu48Jst0l3RKxipJUsHt2xo4dq5kzZzZ4v4899pgC3+LQzZs3T40aNWrwseJh69at6t69u1auXKnc3Nx4lwMAQIOF0mcKISgu/ueVzQULFuiGG26otq7mqxMPHToUUuBp3bp1g2s5+uijG/wcAABwZHg7mQg59thjKx9HHXVUtXXffPONjjrqKL344os6//zzlZ2drfz8fH311VcaM2aMOnXqpOzsbJ1yyil69tlnq+235mW+c889VxMmTNBdd92ltm3bql27dpo8ebLKy8urjal6ma9bt2564IEHNH78eLVq1UqdOnXSL3/5y2rH+eSTTzRkyBA1bdpUJ554ohYtWqQWLVrUO5tWVlamyZMnq02bNmrTpo0mTZqksrKyamMWL16sc845R23atNHRRx+tiy66SEVFRZXbu3fvLkk644wzZGaVlwhXrlyp73znO2rbtq1atWqlf/3Xf9UHH3wQ/EQAANLKLQtvUdb0LN2yMH4XyQhTMfSTn/xEEyZM0Pr16/Xd735X33zzjfr3768FCxaosLBQt956q8aPH68lS5bUu5/nn39eWVlZ+p//+R89/vjj+vWvf605c+bU+5xHH31Up556qlavXq077rhDt99+e2U4KS8v1+WXX66srCwtX75cM2fO1LRp03TgwIF69/mrX/1KTz/9tPLz8/XBBx+orKxMzz//fLUx+/bt06RJk7RixQotW7ZMrVu31ogRI3Tw4EFJ0ooVKyQFQldxcbHmzZsnSfr66691zTXX6N1339WKFSt02mmn6dJLL9VXX31Vb00AgPSSvypfZV6m/FX58SvC3ePyGDBggNdl/fr1dW5rqAkT3DMzAx9j5ZVXXvHAtzZgy5YtLskffvjhoM8dNWqUX3/99ZXLY8eO9WHDhlUuDxkyxM8666xqzxk6dGi15wwZMsRvueWWyuWuXbv66NGjqz3n+OOP9/vvv9/d3RcvXuyZmZm+ffv2yu3vv/++S/Jnn322zlqPO+44f+CBByqXy8rKvFevXj5kyJA6n7N3717PyMjwd999193/+b1ZuXJlnc9xdy8vL/djjz3Wn3vuuTrHRPLnBgCQHCYsmOCZ0zJ9woLo/kcvqcDryDQpPzOVny+VlQU+xlvNG6zLysr005/+VH379tUxxxyjFi1aaN68efr888/r3U/fvn2rLXfo0CHoW6nU95wNGzaoQ4cO6tjxn43rzzjjDGVk1P3jsWfPHhUXF2vQoEGV6zIyMnTmmdXflvGzzz7TVVddpZ49e6pVq1Zq3769ysvLg36Nu3bt0vjx43XCCSeodevWatmypXbt2hX0eQCA9JI3LE+lU0uVNywvbjWk/A3o48cHgtT48fGuRGrevHm15Ycffli/+tWv9Nhjj+nUU09VixYtdNdddwUNRjVvXDezavdMReo5kTB8+HB16tRJ+fn56tixo7KystS7d+/Ky3x1GTt2rHbu3KlHH31U3bp1U5MmTXTBBRcEfR4AALGW8mEqLy/wSETvvfeeRowYoWuuuUZS4JLrJ598UnkDe6ycdNJJ+uKLL/TFF1+oQ4cOkqSCgoJ6w1br1q113HHHafny5Tr//PMlBepfsWKFjjsu8D7XX331lTZs2KAnnnhC5513niRp9erVKi0trdxP48aNJemwG9ffe+89/eY3v9GwYcMkSTt37qz26kgAABJFyl/mS2QnnHCClixZovfee08bNmzQxIkTtWXLlpjXceGFF+rEE0/U2LFjtXbtWi1fvlz/+Z//qaysrHr7Z9166636xS9+oblz52rjxo2aNGlStcDTpk0btW3bVk8//bQ2bdqkt99+WzfddJOysv6Z4du1a6fs7Gy9+eab2rlzp/bs2SMp8L2ZPXu21q9fr5UrV2r06NGVwQsAgERCmIqju+++WwMHDtQll1yiwYMHq3nz5rr66qtjXkdGRoZee+01HThwQAMHDtTYsWM1ZcoUmZmaNm1a5/Nuu+02XXvttfrhD3+oM888U+Xl5dXqz8jI0Jw5c7Ru3Tr16dNHt9xyi+6//341adKkckxWVpZ+85vf6Pe//706dOigkSNHSpJmzJihvXv3asCAARo9erSuu+46devWLWrfAwBA4kiEdgcNYd7A7tqRkpub6wUFBbVuKyoq0sknnxzjilDV2rVrddppp6mgoEADBgyIdzkh4ecGAFJD1vQslXmZMi1TpVNLgz8hBsxslbvX+lYdzExBkvTaa6/prbfe0pYtW7R06VKNGzdO/fr1U//+/eNdGgAgzYwfMF6ZlqnxAxLg1WMhSPkb0BGar7/+WnfccYe2bdumNm3a6Nxzz9Wjjz4a9D0HAQCItLxheXFtddBQhClIkn7wgx/oBz/4QbzLAAAg6XCZDwAAIAyEKQAAgDAQpgAAQEwkW8uDUBGmAABATOSvyleZlyl/VQK8YW4EEaYAAEBMJFvLg1Dxaj4AABATydbyIFTMTCWxbt266eGHH47LsYcPH65x48bF5dgAACQSwlSEmFm9j3CCx3333ac+ffoctn7lypWaMGFCGFXHzrJly2Rm2r17d7xLAQAgorjMFyHFxcWVny9YsEA33HBDtXXZ2dkRP2ZOTk7E9wkAABqGmakIOfbYYysfRx111GHr3nnnHQ0YMEBNmzZV9+7dNWXKFB08eLDy+fPmzVPfvn2VnZ2to48+WkOGDNHOnTs1c+ZMTZs2TYWFhZWzXDNnzpR0+GU+M9NTTz2lK6+8Us2bN1ePHj00e/bsanV++OGH6t+/v5o2barTTz9dixYtkplp2bJldX5t+/fv17hx49SiRQu1b99eDz744GFjZs+erTPOOEMtW7ZUu3btdOWVV2rHjh2SpK1bt+q8886TFAiAVWfqFi9erHPOOUdt2rTR0UcfrYsuukhFRUUN/fYDAOIoVVsehIowFQNvvvmmrr76ak2cOFGFhYWaMWOG5s6dq7vuukuS9OWXX2r06NEaO3asioqK9M477+iaa66RJI0aNUq33XabTjzxRBUXF6u4uFijRo2q81jTp0/XyJEjtXbtWo0aNUrXXXedPv/8c0nS3r17NXz4cJ100klatWqVfvGLX+jHP/5x0PonT56sP/3pT3r11Ve1ZMkSrVmzRu+88061MQcPHtS0adO0du1aLViwQLt379aYMWMkSZ07d9arr74qSSosLFRxcbEee+wxSdK+ffs0adIkrVixQsuWLVPr1q01YsSIakETAJDYUrXlQcjcPS6PAQMGeF3Wr19f57aGmrBggmdOy/QJCyZEbJ/BvPLKKx741gacc845Pn369GpjXnvtNW/evLmXl5f7qlWrXJJv3bq11v3de++9fsoppxy2vmvXrv7LX/6yclmS33nnnZXLhw4d8uzsbH/uuefc3f13v/udt2nTxvfv31855vnnn3dJvnTp0lqP/fXXX3vjxo199uzZ1da1bt3ax44dW+f3oKioyCX5tm3b3N196dKlLslLSkrqfI67+969ez0jI8PffffdesfVJpI/NwCA0MXj/9pYk1TgdWSalJ+ZSoS0vGrVKv30pz9VixYtKh9XXXWV9u3bpy+//FL9+vXT0KFD1adPH11xxRV68sknVVJSckTH6tu3b+XnWVlZysnJ0a5duyRJGzZsUJ8+fardv3XmmWfWu7/PPvtMBw8e1KBBgyrXtWjRQqeeemq1catXr9bIkSPVtWtXtWzZUrm5uZJUOStW3/6vuuoq9ezZU61atVL79u1VXl4e9HkAgMSRNyxPpVNLU7LtQShSPkwlQoOw8vJy3Xvvvfroo48qH+vWrdOnn36qnJwcZWZm6q233tJbb72lvn376plnnlGvXr20du3aBh+rUaNG1ZbNTOXl5ZH6Umq1b98+XXTRRWrWrJmee+45rVy5UosXL5akoJfrhg8frpKSEuXn5+vDDz/UmjVrlJWVxWU+AEDSSPlX8yVCg7D+/ftrw4YNOv744+scY2YaNGiQBg0apKlTp+qUU07RnDlz1K9fPzVu3FhlZWVh13HSSSdp1qxZ+sc//lE5O7VixYp6n9OzZ081atRIy5cvV48ePSQFwtPHH3+snj17SgrMeO3evVsPPvigunfvLilwQ31VjRs3lqRqX8dXX32lDRs26Iknnqi8QX316tUqLS0N+2sFACBWUn5mKhFMnTpVL7zwgqZOnaqPP/5YGzZs0Ny5c3X77bdLkpYvX64HHnhAK1eu1Oeff6758+dr27Zt6t27t6TAq/b++te/avXq1dq9e7cOHDhwRHVcddVVyszM1A033KD169frz3/+c+Ur88ys1ue0aNFC119/ve644w796U9/UmFhoa677rpqoahLly5q0qSJHn/8cW3evFkLFy7UPffcU20/Xbt2lZlp4cKFKikp0d69e9WmTRu1bdtWTz/9tDZt2qS3335bN910k7KyUj7jAwBSCGEqBi666CItXLhQS5cu1cCBAzVw4ED9/Oc/V5cuXSRJrVu31vvvv6/hw4erV69euu2223TPPffo+9//viTpiiuu0KWXXqoLLrhAOTk5evHFF4+ojpYtW+qPf/yjCgsLdfrpp+vHP/6x7rvvPklS06ZN63zeww8/rPPOO0+XX365zjvvPPXp00eDBw+u3J6Tk6NZs2bp9ddfV+/evTVt2jQ98sgj1fbRsWNHTZs2TVOmTFH79u01ceJEZWRkaM6cOVq3bp369OmjW265Rffff7+aNGlyRF8fACBy0r3dQUNY4Ab12MvNzfWCgoJatxUVFenkk0+OcUXp6b/+6790+eWXa9euXWrbtm28ywkLPzcAEDlZ07NU5mXKtEyVTuX2CzNb5e65tW1jZirNzJo1S++++662bt2qBQsWaNKkSRoxYkTSBykAQGQlwgu4kgU3p6SZnTt36t5771VxcbGOPfZYDRs2TA899FC8ywIAJJhEeAFXsiBMpZnbb7+98sZ3AAAQPi7zAQAAhIEwBQAAEAbCFAAAaYSWB5FHmAIAII0kwnvWphrCFAAAaYSWB5HHq/kAAEgjtDyIPGamktDcuXOrvZfezJkz1aJFi6gfd9y4cRo+fHjUjwMAQDIhTEXQuHHjZGYyMzVq1Eg9evTQ5MmTtW/fvqged9SoUdq8eXPE9rds2TKZmXbv3l1t/WOPPabZs2dH7DgAAKSCkMKUmV1sZhvNbJOZ3VnL9q5mtsTM1pnZMjPrFPlSk8PQoUNVXFyszZs364EHHtATTzyhyZMnHzautLRUkXpfxOzsbLVr1y4i+6pP69atddRRR0X9OAAAJJOgYcrMMiXlSbpEUm9JY8ysd41hD0v6g7v3lTRd0s8iXWiyaNKkiY499lh17txZV111la6++mq9/vrruu+++9SnTx/NnDlTPXv2VJMmTbRv3z7t2bNHN954o9q1a6eWLVtqyJAhqvkG0H/4wx/UtWtXNWvWTMOHD9fOnTurba/tMt+iRYt05plnKjs7W8ccc4xGjBihb775RpI0e/ZsnXHGGWrZsqXatWunK6+8Ujt27JAkbd26Veedd54kKScnR2amcePGSTr8Mt+BAwc0adIktW/fXk2bNtVZZ52l9957r3L7tzNcS5Ys0ZlnnqlmzZopNzdXq1evrhyzZ88eXXPNNWrXrp2aNm2qHj166Ne//nV4JwEA0hAtD+InlJmpgZI2uftmdz8o6SVJI2uM6S3pvys+X1rL9rSVnZ2tQ4cOSZK2bNmiF154Qa+88orWrl2rJk2aaNiwYdqxY4cWLFigNWvWaPDgwTr//PNVXFwsSfrwww81btw43Xjjjfroo480YsQITZ06td5jLl68WJdddpkuvPBCrVq1SkuXLtWQIUNUXl4uSTp48KCmTZumtWvXasGCBdq9e7fGjBkjSercubNeffVVSVJhYaGKi4v12GOP1Xqc22+/XXPmzNGMGTO0Zs0anXrqqbr44osra//WT37yE/385z/X6tWrdcwxx+jqq6+unJW7++679Ze//EULFizQxo0bNWPGDHXs2PEIv9sAkL5oeRBH7l7vQ9K/Sfp9leVrJD1eY8wLkm6t+Pz/SXJJx9SyrxslFUgq6NKli9dl/fr1dW5rsAkT3DMzAx+jbOzYsT5s2LDK5Q8//NCPOeYY/973vuf33nuvZ2Vl+Zdfflm5fcmSJd68eXPfv39/tf3069fPH3roIXd3HzNmjA8dOrTa9uuvv94Dpy7g2Wef9ebNm1cu/8u//IuPGjUq5LqLiopckm/bts3d3ZcuXeqSvKSkpM6vb+/evd6oUSOfNWtW5fbS0lLv0aOHT5kypdp+Fi9eXDnmvffeq3asESNG+LXXXhtyrfWJ6M8NACSZCQsmeOa0TJ+wIPr/36UjSQVeR1aK1A3okyUNMbM1koZI2iGprJbg9pS757p7bk5OToQOHUR+vlRWFvgYA4sXL1aLFi3UtGlTDRo0SIMHD9Zvf/tbSVKnTp3Uvn37yrGrVq3S/v37lZOToxYtWlQ+Pv74Y3322WeSpKKiIg0aNKjaMWou17RmzRpdcMEFdW5fvXq1Ro4cqa5du6ply5bKzc2VJH3++echf52fffaZDh06pLPPPrtyXWZmpgYNGqT169dXG9u3b9/Kzzt06CBJ2rVrlyTp5ptv1pw5c9SvXz9NnjxZb7/9dsg1AAD+KW9YnkqnltL2IA5C6TO1Q1LnKsudKtZVcvcvFJiRkpm1kHSFu/9fhGoMz/jxgSA1PjbNyQYPHqynnnpKjRo1UocOHdSoUaPKbc2bN682try8XO3bt9e777572H5atWoVlfr27duniy66SEOHDtVzzz2ndu3aaffu3TrnnHN08ODBiByjatsGSdW+B99u+/aS4yWXXKK//vWveuONN7RkyRINGzZMV155pZ599tmI1AIAQLSFMjO1UlIvM+tuZo0ljZY0v+oAM2trZt/u6yeSZkS2zDDk5UmlpYGPMdCsWTMdf/zx6tq1a7UQUZv+/ftr586dysjI0PHHH1/t8e2r804++WQtX7682vNqLtd0+umna8mSJbVu27Bhg3bv3q0HH3xQgwcP1kknnVQ5S/Stxo0bS5LKyg6bXKzUs2dPNW7cWO+//37lurKyMn3wwQfq3bvm6xPq17ZtW11zzTWaOXOmnnnmGc2aNUsHDhxo0D4AAIiXoGHK3UslTZT0pqQiSS+7e6GZTTezyyqGnStpo5l9Iqm9pJ9Gqd6UMnToUJ199tkaOXKk3njjDW3ZskUffPCB7r333srZqh/96Ef685//rJ/97Gf69NNP9fTTT+u1116rd79TpkzRK6+8orvvvlvr169XYWGhHn30Ue3fv19dunRRkyZN9Pjjj2vz5s1auHCh7rnnnmrP79q1q8xMCxcuVElJifbu3XvYMZo3b66bb75Zd9xxhxYtWqSioiLdfPPN2rlzpyZMmBDy92Dq1Kl6/fXX9emnn6qoqEjz5s1Tjx491KRJk5D3AQBAPIV0z5S7L3L3E9y9p7v/tGLdVHefX/H5XHfvVTHmh+7OtEIIzEyLFi3S+eefrxtuuEEnnniivve972njxo2V9xadddZZeuaZZ/Tkk0+qb9++mjdvnu67775693vppZfqtdde0xtvvKHTTz9dQ4YM0dKlS5WRkaGcnBzNmjVLr7/+unr37q1p06bpkUceqfb8jh07atq0aZoyZYrat2+viRMn1nqchx56SKNGjdK1116r0047TevWrdPixYt13HHHhfw9aNKkiaZMmaJ+/frp7LPP1tdff60//vGPIT8fAFIZ7Q6Sg3mEGkc2VG5urtfsp/StoqIinXzyyTGuCMmOnxsAqSZrepbKvEyZlqnSqaXxLietmdkqd8+tbRtvJwMAQIIaP2C8Mi1T4wfE5kVUODKhvJoPAADEQd6wPFodJAFmpgAAAMJAmAIAAAhDwoapb5s6AqHg5wUAEC8JGaaaN2+uHTt26ODBg4rXqw2RHNxdBw8e1I4dOw7rMA8AiYqWB6klIVsjlJeXa/fu3dqzZ49KS3kpKOqXlZWl1q1bq23btsrISMi/DwCgGloeJJ/6WiMk5Kv5MjIy1K5du8q3VAEAIJWMHzBe+avyaXmQIhJyZgoAACCR0LQTAAAgSghTAAAAYSBMAQAAhIEwBQBAhNDyID0RpgAAiJD8Vfkq8zLlr8qPdymIIcIUAAARMn7AeGVaJi0P0gytEQAAAIKgNQIAAECUEKYAAADCQJgCAAAIA2EKAIB63HKLlJUV+AjUhjAFAEA98vOlsrLAR6A2hCkAAOoxfryUmRn4CNSG1ggAAABB0BoBAAAgSghTAAAAYSBMAQAAhIEwBQBIS7Q8QKQQpgAAaYmWB4gUwhQAIC3R8gCRQmsEAACAIGiNAAAAECWEKQAAgDAQpgAAAMJAmAIApAzaHSAeCFMAgJRBuwPEA2EKAJAyaHeAeKA1AgAAQBC0RgAAAIgSwhQAAEAYCFMAAABhCClMmdnFZrbRzDaZ2Z21bO9iZkvNbI2ZrTOzSyNfKgAgXdHyAIks6A3oZpYp6RNJF0raLmmlpDHuvr7KmKckrXH3J82st6RF7t6tvv1yAzoAIFRZWYGWB5mZUmlpvKtBOgr3BvSBkja5+2Z3PyjpJUkja4xxSa0qPm8t6YsjLRYAgJpoeYBElhXCmI6StlVZ3i7pzBpj7pP0lpn9u6TmkobWtiMzu1HSjZLUpUuXhtYKAEhTeXmBB5CIInUD+hhJM929k6RLJT1nZoft292fcvdcd8/NycmJ0KEBAADiJ5QwtUNS5yrLnSrWVXW9pJclyd0/kNRUUttIFAgAAJDIQglTKyX1MrPuZtZY0mhJ82uM+VzSBZJkZicrEKZKIlkoAABAIgoapty9VNJESW9KKpL0srsXmtl0M7usYthtkm4ws7WSXpQ0zuP1PjUAgKRBywOkAt6bDwAQN7Q8QLLgvfkAAAmJlgdIBcxMAQAABMHMFAAAQJQQpgAAAMJAmAIAAAgDYQoAEFG0O0C6IUwBACIqPz/Q7iA/P96VALFBmAIARBTtDpBuaI0AAAAQBK0RAAAAooQwBQAAEAbCFAAAQBgIUwAAAGEgTAEAQkL/KKB2hCkAQEjoHwXUjjAFAAgJ/aOA2tFnCgAAIAj6TAEAAEQJYQoAACAMhCkAAIAwEKYAIM3R8gAID2EKANIcLQ+A8BCmACDN0fIACA+tEQAAAIKgNQIAAECUEKYAAADCQJgCAAAIA2EKAFIQ7Q6A2CFMAUAKot0BEDuEKQBIQbQ7AGKH1ggAAABB0BoBAAAgSghTAAAAYSBMAQAAhIEwBQBJhJYHQOIhTAFAEqHlAZB4CFMAkERoeQAkHlojAAAABEFrBAAAgCghTAEAAISBMAUAABAGwhQAJABaHgDJK6QwZWYXm9lGM9tkZnfWsv1RM/uo4vGJmf1fxCsFgBRGywMgeQUNU2aWKSlP0iWSeksaY2a9q45x9/9w99Pc/TRJv5U0Lwq1AkDKouUBkLxCmZkaKGmTu29294OSXpI0sp7xYyS9GIniACBd5OVJpaWBjwCSSyhhqqOkbVWWt1esO4yZdZXUXdJ/17H9RjMrMLOCkpKShtYKAACQcCJ9A/poSXPdvay2je7+lLvnuntuTk5OhA8NAAAQe6GEqR2SOldZ7lSxrjajxSU+AACQRkIJUysl9TKz7mbWWIHANL/mIDM7SVIbSR9EtkQASE60OwDSQ9Aw5e6lkiZKelNSkaSX3b3QzKab2WVVho6W9JLH683+ACDB0O4ASA9ZoQxy90WSFtVYN7XG8n2RKwsAkt/48YEgRbsDILVZvCaScnNzvaCgIC7HBgAAaAgzW+XuubVt4+1kAAAAwkCYAgAACANhCgAAIAyEKQBoIFoeAKiKMAUADUTLAwBVEaYAoIHGj5cyM2l5ACCA1ggAAABB0BoBAAAgSghTAAAAYSBMAQAAhIEwBQAVaHkA4EgQpgCgAi0PABwJwhQAVKDlAYAjQWsEAACAIGiNAAAAECWEKQAAgDAQpgAAAMJAmAKQ0mh3ACDaCFMAUhrtDgBEG2EKQEqj3QGAaKM1AgAAQBC0RgAAAIgSwhQAAEAYCFMAAABhIEwBSEq0PACQKAhTAJISLQ8AJArCFICkRMsDAImC1ggAAABB0BoBAAAgSghTAAAAYSBMAQAAhIEwBSCh0PIAQLIhTAFIKLQ8AJBsCFMAEgotDwAkG1ojAAAABEFrBAAAgCghTAEAAISBMAUAABAGwhSAqKPdAYBURpgCEHW0OwCQykIKU2Z2sZltNLNNZnZnHWO+Z2brzazQzF6IbJkAkhntDgCksqCtEcwsU9Inki6UtF3SSklj3H19lTG9JL0s6Xx3/5uZtXP3XfXtl9YIAAAgWYTbGmGgpE3uvtndD0p6SdLIGmNukJTn7n+TpGBBCgAAIFWEEqY6StpWZXl7xbqqTpB0gpm9b2bLzezi2nZkZjeaWYGZFZSUlBxZxQAAAAkkUjegZ0nqJelcSWMkPW1mR9Uc5O5PuXuuu+fm5ORE6NAAAADxE0qY2iGpc5XlThXrqtouab67H3L3LQrcY9UrMiUCSFS0PACA0MLUSkm9zKy7mTWWNFrS/BpjXldgVkpm1laBy36bI1cmgEREywMACCFMuXuppImS3pRUJOlldy80s+lmdlnFsDclfWVm6yUtlfRjd/8qWkUDSAy0PACAEFojRAutEQAAQLIItzUCAAAA6kCYAgAACANhCgAAIAyEKQDV0O4AABqGMAWgGtodAEDDEKYAVEO7AwBoGFojAAAABEFrBAAAgCghTAEAAISBMAUAABAGwhSQJmh5AADRQZgC0gQtDwAgOghTQJqg5QEARAetEQAAAIKgNQIAAECUEKYAAADCQJgCAAAIA2EKSHK0PACA+CJMAUmOlgcAEF+EKSDJ0fIAAOKL1ggAAABB0BoBAAAgSghTAAAAYSBMAQAAhIEwBSQg2h0AQPIgTAEJiHYHAJA8CFNAAqLdAQAkD1ojAAAABEFrBAAAgCghTAEAAISBMAUAABAGwhQAAEAYCFNADNE/CgBSD2EKiCH6RwFA6iFMATFE/ygASD30mQIAAAiCPlMAAABRQpgCAAAIA2EKAAAgDIQpIAJoeQAA6YswBUQALQ8AIH0RpoAIoOUBAKSvkMKUmV1sZhvNbJOZ3VnL9nFmVmJmH1U8fhj5UoHElZcnlZYGPgIA0ktWsAFmlikpT9KFkrZLWmlm8919fY2hc9x9YhRqBAAASFihzEwNlLTJ3Te7+0FJL0kaGd2yAAAAkkMoYaqjpG1VlrdXrKvpCjNbZ2ZzzaxzbTsysxvNrMDMCkpKSo6gXAAAgMQSqRvQ/yipm7v3lfQnSbNqG+TuT7l7rrvn5uTkROjQQHTQ7gAAEIpQwtQOSVVnmjpVrKvk7l+5+4GKxd9LGhCZ8oD4od0BACAUoYSplZJ6mVl3M2ssabSk+VUHmNlxVRYvk1QUuRKB+KDdAQAgFEFfzefupWY2UdKbkjIlzXD3QjObLqnA3edL+pGZXSapVNL/ShoXxZqBmMjLo9UBACA4c/e4HDg3N9cLCgricmwAAICGMLNV7p5b2zY6oAMAAISBMAUAABAGwhTSDi0PAACRRJhC2qHlAQAgkghTSDu0PAAARBKv5gMAAAiCV/MBAABECWEKAAAgDIQpAACAMBCmkDJoeQAAiAfCFFIGLQ8AAPFAmELKoOUBACAeaI0AAAAQBK0RAABAakqAG2YJUwAAIHklwA2zhCkAAJC8EuCGWcIUEloCzN4CABJZXp5UWhr4GCeEKSS0BJi9BQDEWpL9JU2YQkJLgNlbAECsJdlf0oQpJLQEmL0FAMRakv0lTZgCAACxEerluyT7S5owBQAAYiPJLt+FijAFAABiI8ku34WKMIW4SLIXagAAIiHJLt+FijCFuEjRmV4ASE9p/hcyYQpxkaIzvQCQntL8L2TCFOIiRWd6ASA9pflfyIQpAABwuIZcukvzv5AJUwAA4HBpfumuIQhTAADgcGl+6a4hCFOIqDR/QQcAJL4U7UIeT+bucTlwbm6uFxQUxOXYiJ6srMCscGZm4N8gACDB8Iv6iJjZKnfPrW0bM1OIKGaFASDB8Ys64piZAgAACIKZKQAAUh03rcYNYQoAgFRAK4O4IUwBAJAKuBcqbghTCIqZYwCIE7qQJwVuQEdQvIoWAOKEX8AJgxvQERZmjgEgTvgFnBSYmQIAAAgi7JkpM7vYzDaa2SYzu7OecVeYmZtZrQcDAADiZtQUEzRMmVmmpDxJl0jqLWmMmfWuZVxLSbdK+jDSRQIAkFJoY5BSQpmZGihpk7tvdveDkl6SNLKWcfdLekjSNxGsDwCA1MO9UCkllDDVUdK2KsvbK9ZVMrP+kjq7+8L6dmRmN5pZgZkVlJSUNLhYRBazzAAQYaH+YqWNQUoJ+9V8ZpYh6RFJtwUb6+5PuXuuu+fm5OSEe2iEiVlmAIgwfrGmpVDC1A5Jnassd6pY962WkvpIWmZmWyWdJWk+N6EnPmaZASDC+MWaloK2RjCzLEmfSLpAgRC1UtJV7l5Yx/hlkia7e719D2iNAAAAkkVYrRHcvVTSRElvSiqS9LK7F5rZdDO7LLKlAgAAJJesUAa5+yJJi2qsm1rH2HPDLwsAACA58HYyAAAAYSBMpSBaHgAAEDuEqRTEK3MBAIgdwlQK4pW5AADETtDWCNFCawQAAJAswmqNAAAAgLoRpgAAAMJAmAIAAAgDYSpJ0O4AAIDERJhKErQ7AAAgMRGmkgTtDgAASEy0RgAAAAiC1ggAAABRQpgCAAAIA2EKAAAgDISpOKPlAQAAyY0wFWe0PAAAILkRpuKMlgcAACQ3WiMAAAAEQWsEAACAKCFMAQAAhIEwBQAAEAbCVBTQ7gAAgPRBmIoC2h0AAJA+CFNRQLsDAADSB60RAAAAgqA1AgAAQJQQpgAAAMJAmAIAAAgDYaoBaHkAAABqIkw1AC0PAABATYSpBqDlAQAAqInWCAAAAEHQGgEAACBKCFMAAABhIEwBAACEgTAlWh4AAIAjR5gSLQ8AAMCRI0yJlgcAAODI0RoBAAAgCFojAAAARElIYcrMLjazjWa2yczurGX7TWb2FzP7yMzeM7PekS8VAAAg8QQNU2aWKSlP0iWSeksaU0tYesHdT3X30yT9QtIjkS4UAAAgEYUyMzVQ0iZ33+zuByW9JGlk1QHu/vcqi80lxedGLAAAgBgLJUx1lLStyvL2inXVmNktZvaZAjNTP4pMeUeO3lEAACAWInYDurvnuXtPSXdIuru2MWZ2o5kVmFlBSUlJpA5dK3pHAQCAWAglTO2Q1LnKcqeKdXV5SdJ3a9vg7k+5e6675+bk5IRc5JGgdxQAAIiFUMLUSkm9zKy7mTWWNFrS/KoDzKxXlcVhkj6NXIlHJi9PKi0NfAQAAIiWrGAD3L3UzCZKelNSpqQZ7l5oZtMlFbj7fEkTzWyopEOS/iZpbDSLBgAASBRBw5QkufsiSYtqrJta5fNbI1wXAABAUqADOgAAQBgIUwAAAGEgTAEAAISBMAUAABAGwhQAAEAYCFMAAABhIEwBAACEgTAFAAAQBsIUAABAGAhTAAAAYSBMAQAAhIEwBQAAEAZz9/gc2KxE0l+jfJi2knZH+Rg4cpyfxMW5SWycn8TG+Ulc4Zybru6eU9uGuIWpWDCzAnfPjXcdqB3nJ3FxbhIb5yexcX4SV7TODZf5AAAAwkCYAgAACEOqh6mn4l0A6sX5SVycm8TG+UlsnJ/EFZVzk9L3TAEAAERbqs9MAQAARBVhCgAAIAwpEabM7GIz22hmm8zszlq2NzGzORXbPzSzbnEoM22FcH7+08zWm9k6M1tiZl3jUWc6CnZuqoy7wszczHi5dwyFcn7M7HsV/34KzeyFWNeYrkL4vdbFzJaa2ZqK322XxqPOdGRmM8xsl5l9XMd2M7PfVJy7dWbWP9xjJn2YMrNMSXmSLpHUW9IYM+tdY9j1kv7m7sdLelTSQ7GtMn2FeH7WSMp1976S5kr6RWyrTE8hnhuZWUtJt0r6MLYVprdQzo+Z9ZL0E0lnu/spkibFus50FOK/nbslvezup0saLemJ2FaZ1mZKurie7ZdI6lXxuFHSk+EeMOnDlKSBkja5+2Z3PyjpJUkja4wZKWlWxedzJV1gZhbDGtNZ0PPj7kvdfX/F4nJJnWJcY7oK5d+OJN2vwB8g38SyOIR0fm6QlOfuf5Mkd98V4xrTVSjnxiW1qvi8taQvYlhfWnP3dyT9bz1DRkr6gwcsl3SUmR0XzjFTIUx1lLStyvL2inW1jnH3Ukl7JB0Tk+oQyvmp6npJb0S1Inwr6LmpmP7u7O4LY1kYJIX2b+cESSeY2ftmttzM6vtrHJETyrm5T9L3zWy7pEWS/j02pSEEDf1/KaissMoBIsjMvi8pV9KQeNcCycwyJD0iaVycS0HdshS4VHGuAjO675jZqe7+f/EsCpKkMZJmuvuvzGyQpOfMrI+7l8e7MEReKsxM7ZDUucpyp4p1tY4xsywFply/ikl1COX8yMyGSpoi6TJ3PxCj2tJdsHPTUlIfScvMbKuksyTN5yb0mAnl3852SfPd/ZC7b5H0iQLhCtEVyrm5XtLLkuTuH0hqqsCb7CL+Qvp/qSFSIUytlNTLzLqbWWMFbvSbX2PMfEljKz7/N0n/7XQrjZWg58fMTpeUr0CQ4p6P2Kn33Lj7Hndv6+7d3L2bAvezXebuBfEpN+2E8rvtdQVmpWRmbRW47Lc5hjWmq1DOzeeSLpAkMztZgTBVEtMqUZf5kn5Q8aq+syTtcfficHaY9Jf53L3UzCZKelNSpqQZ7l5oZtMlFbj7fEnPKDDFukmBm9JGx6/i9BLi+fmlpBaSXql4XcDn7n5Z3IpOEyGeG8RJiOfnTUnfMbP1ksok/djdmXWPshDPzW2Snjaz/1DgZvRx/BEfG2b2ogJ/ZLStuGftXkmNJMndf6fAPWyXStokab+ka8M+JucWAADgyKXCZT4AAIC4IUwBAACEgTAFAAAQBsIUAABAGAhTAAAAYSBMAQAAhIEwBQAAEIb/D2AjI7jt0RDpAAAAAElFTkSuQmCC\n" + }, + "metadata": { + "needs_background": "light" + } + } + ], + "source": [ + "plot_predictions(predictions=y_preds)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "JLJWVANkhY3-", + "outputId": "1a5a7b40-bb72-4339-e65f-4a2a0705d7a5" + }, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[0.4618],\n", + " [0.4691],\n", + " [0.4764],\n", + " [0.4836],\n", + " [0.4909],\n", + " [0.4982],\n", + " [0.5054],\n", + " [0.5127],\n", + " [0.5200],\n", + " [0.5272]])" + ] + }, + "metadata": {}, + "execution_count": 13 + } + ], + "source": [ + "y_test - y_preds" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lxt8WUzdv1qS" + }, + "source": [ + "Woah! Those predictions look pretty bad...\n", + "\n", + "This make sense though when you remember our model is just using random parameter values to make predictions.\n", + "\n", + "It hasn't even looked at the blue dots to try to predict the green dots.\n", + "\n", + "Time to change that." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZZpa-fXLec03" + }, + "source": [ + "## 3. Train model\n", + "\n", + "Right now our model is making predictions using random parameters to make calculations, it's basically guessing (randomly).\n", + "\n", + "To fix that, we can update its internal parameters (I also refer to *parameters* as patterns), the `weights` and `bias` values we set randomly using `nn.Parameter()` and `torch.randn()` to be something that better represents the data.\n", + "\n", + "Instead, it's much more fun to write code to see if the model can try and figure them out itself.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aD8pnhJUyZUT" + }, + "source": [ + "### Creating a loss function and optimizer in PyTorch\n", + "\n", + "For our model to update its parameters on its own, we'll need to add a few more things to our recipe.\n", + "\n", + "And that's a **loss function** as well as an **optimizer**.\n", + "\n", + "The rolls of these are: \n", + "\n", + "| Function | What does it do? | Where does it live in PyTorch? | Common values |\n", + "| ----- | ----- | ----- | ----- |\n", + "| **Loss function** | Measures how wrong your models predictions (e.g. `y_preds`) are compared to the truth labels (e.g. `y_test`). Lower the better. | PyTorch has plenty of built-in loss functions in [`torch.nn`](https://pytorch.org/docs/stable/nn.html#loss-functions). | Mean absolute error (MAE) for regression problems ([`torch.nn.L1Loss()`](https://pytorch.org/docs/stable/generated/torch.nn.L1Loss.html)). Binary cross entropy for binary classification problems ([`torch.nn.BCELoss()`](https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html)). |\n", + "| **Optimizer** | Tells your model how to update its internal parameters to best lower the loss. | You can find various optimization function implementations in [`torch.optim`](https://pytorch.org/docs/stable/optim.html). | Stochastic gradient descent ([`torch.optim.SGD()`](https://pytorch.org/docs/stable/generated/torch.optim.SGD.html#torch.optim.SGD)). Adam optimizer ([`torch.optim.Adam()`](https://pytorch.org/docs/stable/generated/torch.optim.Adam.html#torch.optim.Adam)). | \n", + "\n", + "Let's create a loss function and an optimizer we can use to help improve our model.\n", + "\n", + "Depending on what kind of problem you're working on will depend on what loss function and what optimizer you use.\n", + "\n", + "However, there are some common values, that are known to work well such as the SGD (stochastic gradient descent) or Adam optimizer. And the MAE (mean absolute error) loss function for regression problems (predicting a number) or binary cross entropy loss function for classification problems (predicting one thing or another). \n", + "\n", + "For our problem, since we're predicting a number, let's use MAE (which is under `torch.nn.L1Loss()`) in PyTorch as our loss function. \n", + "\n", + "![what MAE loss looks like for our plot data](https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/01-mae-loss-annotated.png)\n", + "*Mean absolute error (MAE, in PyTorch: `torch.nn.L1Loss`) measures the absolute difference between two points (predictions and labels) and then takes the mean across all examples.*\n", + "\n", + "And we'll use SGD, `torch.optim.SGD(params, lr)` where:\n", + "\n", + "* `params` is the target model parameters you'd like to optimize (e.g. the `weights` and `bias` values we randomly set before).\n", + "* `lr` is the **learning rate** you'd like the optimizer to update the parameters at, higher means the optimizer will try larger updates (these can sometimes be too large and the optimizer will fail to work), lower means the optimizer will try smaller updates (these can sometimes be too small and the optimizer will take too long to find the ideal values). The learning rate is considered a **hyperparameter** (because it's set by a machine learning engineer). Common starting values for the learning rate are `0.01`, `0.001`, `0.0001`, however, these can also be adjusted over time (this is called [learning rate scheduling](https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate)). \n", + "\n", + "Woah, that's a lot, let's see it in code." + ] + }, + { + "cell_type": "code", + "source": [ + "# Create the loss function\n", + "loss_fn = nn.L1Loss() # MAE loss is same as L1Loss\n", + "\n", + "# Create the optimizer\n", + "optimizer = torch.optim.SGD(params = model_0.parameters(), # parameters of target model to optimize\n", + " lr = 0.01) # learning rate" + ], + "metadata": { + "id": "EB25cqATeFq_" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aFcKCsPcRfnA" + }, + "source": [ + "### Creating an optimization loop in PyTorch\n", + "\n", + "Woohoo! Now we've got a loss function and an optimizer, it's now time to create a **training loop** (and **testing loop**).\n", + "\n", + "The training loop involves the model going through the training data and learning the relationships between the `features` and `labels`.\n", + "\n", + "The testing loop involves going through the testing data and evaluating how good the patterns are that the model learned on the training data (the model never see's the testing data during training).\n", + "\n", + "Each of these is called a \"loop\" because we want our model to look (loop through) at each sample in each dataset.\n", + "\n", + "To create these we're going to write a Python `for` loop in the theme of the [unofficial PyTorch optimization loop song](https://twitter.com/mrdbourke/status/1450977868406673410?s=20) (there's a [video version too](https://youtu.be/Nutpusq_AFw)).\n", + "\n", + "![the unofficial pytorch optimization loop song](https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/01-pytorch-optimization-loop-song.png)\n", + "*The unoffical PyTorch optimization loops song, a fun way to remember the steps in a PyTorch training (and testing) loop.*\n", + "\n", + "There will be a fair bit of code but nothing we can't handle.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "agXn72H-sgyd" + }, + "source": [ + "\n", + "\n", + "### PyTorch training loop\n", + "For the training loop, we'll build the following steps:\n", + "\n", + "| Number | Step name | What does it do? | Code example |\n", + "| ----- | ----- | ----- | ----- |\n", + "| 1 | Forward pass | The model goes through all of the training data once, performing its `forward()` function calculations. | `model(x_train)` |\n", + "| 2 | Calculate the loss | The model's outputs (predictions) are compared to the ground truth and evaluated to see how wrong they are. | `loss = loss_fn(y_pred, y_train)` | \n", + "| 3 | Zero gradients | The optimizers gradients are set to zero (they are accumulated by default) so they can be recalculated for the specific training step. | `optimizer.zero_grad()` |\n", + "| 4 | Perform backpropagation on the loss | Computes the gradient of the loss with respect for every model parameter to be updated (each parameter with `requires_grad=True`). This is known as **backpropagation**, hence \"backwards\". | `loss.backward()` |\n", + "| 5 | Update the optimizer (**gradient descent**) | Update the parameters with `requires_grad=True` with respect to the loss gradients in order to improve them. | `optimizer.step()` |\n", + "\n", + "![pytorch training loop annotated](https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/01-pytorch-training-loop-annotated.png)\n", + "\n", + "> **Note:** The above is just one example of how the steps could be ordered or described. With experience you'll find making PyTorch training loops can be quite flexible.\n", + ">\n", + "> And on the ordering of things, the above is a good default order but you may see slightly different orders. Some rules of thumb: \n", + "> * Calculate the loss (`loss = ...`) *before* performing backpropagation on it (`loss.backward()`).\n", + "> * Zero gradients (`optimizer.zero_grad()`) *before* stepping them (`optimizer.step()`).\n", + "> * Step the optimizer (`optimizer.step()`) *after* performing backpropagation on the loss (`loss.backward()`).\n", + "\n", + "For resources to help understand what's happening behind the scenes with backpropagation and gradient descent, see the extra-curriculum section.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OXHDdlfjssDc" + }, + "source": [ + "\n", + "### PyTorch testing loop\n", + "\n", + "As for the testing loop (evaluating our model), the typical steps include:\n", + "\n", + "| Number | Step name | What does it do? | Code example |\n", + "| ----- | ----- | ----- | ----- |\n", + "| 1 | Forward pass | The model goes through all of the training data once, performing its `forward()` function calculations. | `model(x_test)` |\n", + "| 2 | Calculate the loss | The model's outputs (predictions) are compared to the ground truth and evaluated to see how wrong they are. | `loss = loss_fn(y_pred, y_test)` | \n", + "| 3 | Calulate evaluation metrics (optional) | Alongisde the loss value you may want to calculate other evaluation metrics such as accuracy on the test set. | Custom functions |\n", + "\n", + "Notice the testing loop doesn't contain performing backpropagation (`loss.backward()`) or stepping the optimizer (`optimizer.step()`), this is because no parameters in the model are being changed during testing, they've already been calculated. For testing, we're only interested in the output of the forward pass through the model.\n", + "\n", + "![pytorch annotated testing loop](https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/01-pytorch-testing-loop-annotated.png)\n", + "\n", + "Let's put all of the above together and train our model for 100 **epochs** (forward passes through the data) and we'll evaluate it every 10 epochs.\n" + ] + }, + { + "cell_type": "code", + "source": [ + "torch.manual_seed(42)\n", + "# Set the number of epochs (how many times the model will pass over the training data)\n", + "epochs = 100\n", + "\n", + "# Create empty loss list to track values\n", + "train_loss_values = []\n", + "test_loss_values = []\n", + "epoch_count = []\n", + "\n", + "for epoch in range(epochs):\n", + " ### Training\n", + "\n", + " # Putting model in training model (this is the default state of a model)\n", + " model_0.train()\n", + " # 1. Forward pass on train data using the forward() method inside\n", + " y_pred = model_0(X_train)\n", + " # 2. Calculate the loss\n", + " loss = loss_fn(y_pred, y_train)\n", + " # 3. Zero grad of the optimizer\n", + " optimizer.zero_grad()\n", + " # 4. Loss backwards\n", + " loss.backward()\n", + " # 5. Progress Optimizer\n", + " optimizer.step()\n", + "\n", + " ### Testing Loop\n", + "\n", + " # Pul model under evalution mode\n", + " model_0.eval()\n", + "\n", + " with torch.inference_mode():\n", + " # 1. Forward Pass on test data\n", + " test_pred = model_0(X_test)\n", + " # 2. Calculate loss on test data\n", + " test_loss = loss_fn(test_pred,y_test.type(torch.float)) # Predications comes in torch.float\n", + " # Printing out details of what happening\n", + " if epoch % 10 == 0:\n", + " epoch_count.append(epoch)\n", + " train_loss_values.append(loss.detach().numpy())\n", + " test_loss_values.append(test_loss.detach().numpy())\n", + " print(f'Epoch: {epoch} | MAE Train Loss :{loss} | MAE Test Loss: {test_loss}')" + ], + "metadata": { + "id": "BOdTzvIDfkt9", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "dcb22f45-5822-4160-c989-f3b6444ac0ad" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Epoch: 0 | MAE Train Loss :0.31288138031959534 | MAE Test Loss: 0.48106518387794495\n", + "Epoch: 10 | MAE Train Loss :0.1976713240146637 | MAE Test Loss: 0.3463551998138428\n", + "Epoch: 20 | MAE Train Loss :0.08908725529909134 | MAE Test Loss: 0.21729660034179688\n", + "Epoch: 30 | MAE Train Loss :0.053148526698350906 | MAE Test Loss: 0.14464017748832703\n", + "Epoch: 40 | MAE Train Loss :0.04543796554207802 | MAE Test Loss: 0.11360953003168106\n", + "Epoch: 50 | MAE Train Loss :0.04167863354086876 | MAE Test Loss: 0.09919948130846024\n", + "Epoch: 60 | MAE Train Loss :0.03818932920694351 | MAE Test Loss: 0.08886633068323135\n", + "Epoch: 70 | MAE Train Loss :0.03476089984178543 | MAE Test Loss: 0.0805937647819519\n", + "Epoch: 80 | MAE Train Loss :0.03132382780313492 | MAE Test Loss: 0.07232122868299484\n", + "Epoch: 90 | MAE Train Loss :0.02788739837706089 | MAE Test Loss: 0.06473556160926819\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1krgBqXBdYHc" + }, + "source": [ + "Oh would you look at that! Looks like our loss is going down with every epoch, let's plot it to find out." + ] + }, + { + "cell_type": "code", + "source": [ + "# Plot the loss curves\n", + "plt.plot(epoch_count, train_loss_values, label = 'Train loss')\n", + "plt.plot(epoch_count, test_loss_values, label = 'Test loss')\n", + "plt.title('Training and Test loss Curves')\n", + "plt.ylabel('Loss')\n", + "plt.xlabel('Epochs')\n", + "plt.legend();" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 295 + }, + "id": "frw6nfFxqsXm", + "outputId": "fa9084eb-2df5-4d56-e094-0579bfcbf899" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAAsTAAALEwEAmpwYAAA0aklEQVR4nO3dd3hUZdrH8e+dSSWF3hJKQLFAIAERFBcpdlCxoSgq7uq6+K6iriLYFdFdyyq66wrurmVtoCiKotgLiIUiVUABA4TeEwgh7X7/OGfCBJKQQCZnkrk/1zVXZs45M3NnGPKbp8xzRFUxxhgTviK8LsAYY4y3LAiMMSbMWRAYY0yYsyAwxpgwZ0FgjDFhzoLAGGPCnAWBOWwi8pGIDKvuY70kIpkicrrXdUBo1WLqNguCMCMiuwMuxSKyN+D20Ko8lqqeo6ovV/exocgNMv/rVCAi+QG3xx/G4z0gIq8Go9bDJSI9RORDEdkpIttF5EcR+b3XdZngsyAIM6qa4L8Aa4DzAra95j9ORCK9qzL0uEHmf91eAx4LeN2Ge13fkRKRk4EvgK+Bo4HGwA3AOYf5eL7qq84EmwWBAUBE+opIloiMEpGNwIsi0lBEPhCRLSKyw73eKuA+X4nIde71a0Rkpog84R77m4icc5jHthORb0QkR0Q+E5Fny/v0XMkaHxKRb93H+0REmgTsv0pEVovINhG5+zBfu3NFZL77SXqWiHQJ2DdKRNa5z71cRE4TkbOBu4DL3BbFgko8R4yIjBOR9e5lnIjEuPuauL+3/5P8DBGJKO/5y3mKx4GXVfVRVd2qjrmqeqn7ONeIyMwDalIROdq9/pKIPOe2KPYAt4vIxsBAEJELRWShez1CREaLyEr3tX9TRBq5+2JF5FV3+04RmS0izSv9D2KqzILABGoBNALaAtfjvD9edG+3AfYC/6zg/j2B5UAT4DHgvyIih3Hs68CPOJ9KHwCuquA5K1PjFcDvgWZANHA7gIh0BJ5zHz/Zfb5WVIGIdAVeAP7k3n8CMNX9w30scCNwoqomAmcBmao6HXgEmOS2KNIr8VR3AycBGUA60AO4x913G5AFNAWa44SMlvf8ZfwO9YCTgclV+d3LcAXwMJAIPA3sAfofsP919/pNwAVAH5zXfgfwrLtvGFAfaI3zmg7H+Xc1QWJBYAIVA/er6j5V3auq21T1bVXNVdUcnP/kfSq4/2pV/beqFgEvAy1x/jBV+lgRaQOcCNynqvmqOhOYWt4TVrLGF1X1F1XdC7yJ88cU4BLgA1X9RlX3Afe6r0FVXA9MUNUfVLXIHQfZh/NHuwiIATqKSJSqZqrqyio+vt9QYIyqblbVLcCD7A/IApzXr62qFqjqDHUWEavs8zfE+Vuw4TBr83tPVb9V1WJVzQPeAC4HEJFEYIC7DZw/7nerapb72j8AXOJ2SRbgBMDR7ms6V1Wzj7A2UwELAhNoi/sfGHA+KYrIBLfrJBv4BmhQQf/vRv8VVc11ryZU8dhkYHvANoC15RVcyRo3BlzPDagpOfCxVXUPsK285ypHW+A2twtjp4jsxPkkm6yqK4BbcP7IbRaRiSKSXMXH90sGVgfcXu1uA6dbZwXwiYisEpHR7u9T2effgROALQ+zNr8D/51eBy5yu7AuAuapqv93aAtMCXjNluIEV3PgFeBjYKLbDfaYiEQdYW2mAhYEJtCBS9HeBhwL9FTVJOBUd3t53T3VYQPQyO2u8GtdwfFHUuOGwMd2n7Nx1cplLfCwqjYIuNRT1TcAVPV1Vf0dzh8+BR5171fVZX/Xu4/h18bdhqrmqOptqtoeOB/4i38soILnL+GG7nfAxRU8/x6g5N9ERFqUcUyp30lVf8YJrHMo3S0Ezut2zgGvW6yqrnNbNQ+qakegF3AucHUFtZkjZEFgKpKI0ze70x3Iuz/YT+h+YpwDPCAi0eLMZjkvSDVOBs4Vkd+JSDQwhqr/n/g3MFxEeoojXkQGikiiiBwrIv3dT8R5bp3+rqdNQKp/ULcS3gDuEZGm7mD3fcCrUDJYfbQ7xrIL55N18SGe/0B3ANeIyEgRaew+brqITHT3LwA6iUiGiMTitDIq43XgZpyAfitg+3jgYRFp6z5XUxEZ5F7vJyKd3VZdNk5XUVW77EwVWBCYiowD4oCtwPfA9Bp63qE4g5fbgLHAJJx+97KM4zBrVNUlwJ9x/lhtwOkiyapKoao6B/gjzgD1Dpwummvc3THA39zaNuIMVt/p7vP/UdwmIvMq8VRjcQJyIbAImOduA+gAfAbsxvlk/y9V/fIQz3/g7zELZ2C3P7BKRLYDzwMfuvt/wQnKz4BfgZllPU4Z3sAZs/lCVbcGbH8aZ+znExHJwfm36+nua4ET0tk4XUZf43QXmSAROzGNCXUiMglYpqpBb5EYE46sRWBCjoicKCJHuXPNzwYGAe96XJYxdZZ9e9SEohbAOzgDt1nADar6k7clGVN3WdeQMcaEuaB2DYnI2e7X2lf45zYfsP8acZYGmO9ergtmPcYYYw4WtK4hd+rXs8AZOM372SIy1Z1bHGiSqt5Y2cdt0qSJpqamVl+hxhgTBubOnbtVVZuWtS+YYwQ9gBWqugrAnY88CDgwCKokNTWVOXPmVEN5xhgTPkRkdXn7gtk1lELpr5xnudsOdLGILBSRySJS5jdIReR6EZkjInO2bNkSjFqNMSZseT199H0gVVW7AJ/iLD52EFV9XlW7q2r3pk3LbNkYY4w5TMEMgnWUXiOmlbuthLtypP8bo/8BTghiPcYYY8oQzDGC2UAHEWmHEwBDcBaeKiEiLVXVv/Tt+ThfJzfGhKGCggKysrLIy8s79MGmXLGxsbRq1YqoqMov2Bq0IFDVQhG5EWc5WR/wgqouEZExwBxVnQqMEJHzgUJgO/vXaDHGhJmsrCwSExNJTU2l/PMZmYqoKtu2bSMrK4t27dpV+n5B/Waxqn6Iu2hVwLb7Aq7fSTmLYBljwkteXp6FwBESERo3bkxVJ9V4PVhsjDElLASO3OG8huETBBsXw6f3gy2pYYwxpYRPEGTOhG/HwfIPD3moMSb8bNu2jYyMDDIyMmjRogUpKSklt/Pz8yu875w5cxgxYkSVni81NZWtW7ce+sAaED6rj554Lcx5AT6+G44+HSJjvK7IGBNCGjduzPz58wF44IEHSEhI4Pbbby/ZX1hYSGRk2X8yu3fvTvfu3WuizKAInxaBLwrOfgR2/AY/jPe6GmNMLXDNNdcwfPhwevbsyR133MGPP/7IySefTNeuXenVqxfLly8H4KuvvuLcc88FnBD5wx/+QN++fWnfvj3PPPPMIZ/nySefJC0tjbS0NMaNGwfAnj17GDhwIOnp6aSlpTFp0iQARo8eTceOHenSpUupoDoS4dMiAKcl0OEs+PpxSL8cEpp5XZExpgwPvr+En9dnV+tjdkxO4v7zOlX5fllZWcyaNQufz0d2djYzZswgMjKSzz77jLvuuou33377oPssW7aML7/8kpycHI499lhuuOGGcuf1z507lxdffJEffvgBVaVnz5706dOHVatWkZyczLRp0wDYtWsX27ZtY8qUKSxbtgwRYefOnVX+fcoSPi0Cv7MegcK98PkYrysxxtQCgwcPxufzAc4f48GDB5OWlsatt97KkiVLyrzPwIEDiYmJoUmTJjRr1oxNmzaV+/gzZ87kwgsvJD4+noSEBC666CJmzJhB586d+fTTTxk1ahQzZsygfv361K9fn9jYWK699lreeecd6tWrVy2/Y3i1CACaHA09h8N3z8KJ10FyhtcVGWMOcDif3IMlPj6+5Pq9995Lv379mDJlCpmZmfTt27fM+8TE7B+D9Pl8FBYWVvl5jznmGObNm8eHH37IPffcw2mnncZ9993Hjz/+yOeff87kyZP55z//yRdffFHlxz5Q+LUIAE4dCfUawfTRNp3UGFNpu3btIiXFWUT5pZdeqpbH7N27N++++y65ubns2bOHKVOm0Lt3b9avX0+9evW48sorGTlyJPPmzWP37t3s2rWLAQMG8NRTT7FgwYJqqSH8WgQAcQ2g/73wwS2wZAqkXeR1RcaYWuCOO+5g2LBhjB07loEDB1bLY3br1o1rrrmGHj16AHDdddfRtWtXPv74Y0aOHElERARRUVE899xz5OTkMGjQIPLy8lBVnnzyyWqpodads7h79+5aLSemKS6CCX0gbyfcOBui4o78MY0xh23p0qUcf/zxXpdRJ5T1WorIXFUtc45reHYNAUT44Oy/wq61MOsfXldjjDGeCd8gAGjXG44/H2Y+Bdnrva7GGGM8Ed5BAHDmQ0430WcPeF2JMcZ4woKgYSr0uhEWToK1s72uxhhjapwFAcDv/gIJLWD6KCgu9roaY4ypURYEADEJcPr9sG4uLHrT62qMMaZGWRD4dRkCyd2ccxbs2+11NcaYGnYky1CDs/DcrFmzytz30ksvceONN1Z3ydXGgsAvIgLOeRR2b3RmERljwop/Ger58+czfPhwbr311pLb0dHRh7x/RUEQ6iwIArXuAZ0HO98r2LHa62qMMR6bO3cuffr04YQTTuCss85iw4YNADzzzDMlS0EPGTKEzMxMxo8fz1NPPUVGRgYzZswo9zEzMzPp378/Xbp04bTTTmPNmjUAvPXWW6SlpZGens6pp54KwJIlS+jRowcZGRl06dKFX3/9NSi/Z3guMVGR0x+EZdPg03vh0v95XY0x4emj0bBxUfU+ZovOcM7fKn24qnLTTTfx3nvv0bRpUyZNmsTdd9/NCy+8wN/+9jd+++03YmJi2LlzJw0aNGD48OEHncymLDfddBPDhg1j2LBhvPDCC4wYMYJ3332XMWPG8PHHH5OSklKyvPT48eO5+eabGTp0KPn5+RQVFR3JK1AuaxEcqH4KnHIL/Pyec3pLY0xY2rdvH4sXL+aMM84gIyODsWPHkpWVBUCXLl0YOnQor776arlnLSvPd999xxVXXAHAVVddxcyZzt+ZU045hWuuuYZ///vfJX/wTz75ZB555BEeffRRVq9eTVxccJbCsRZBWXrdBPP+56xOev3XznIUxpiaU4VP7sGiqnTq1InvvvvuoH3Tpk3jm2++4f333+fhhx9m0aIjb72MHz+eH374gWnTpnHCCScwd+5crrjiCnr27Mm0adMYMGAAEyZMoH///kf8XAeyFkFZouvBmWOcpulPr3hdjTHGAzExMWzZsqUkCAoKCliyZAnFxcWsXbuWfv368eijj7Jr1y52795NYmIiOTk5h3zcXr16MXHiRABee+01evfuDcDKlSvp2bMnY8aMoWnTpqxdu5ZVq1bRvn17RowYwaBBg1i4cGFQflcLgvJ0ugjanAyfPwR5u7yuxhhTwyIiIpg8eTKjRo0iPT2djIwMZs2aRVFREVdeeSWdO3ema9eujBgxggYNGnDeeecxZcqUQw4W/+Mf/+DFF1+kS5cuvPLKKzz99NMAjBw5ks6dO5OWlkavXr1IT0/nzTffJC0tjYyMDBYvXszVV18dlN81fJehroz1P8Hz/ZwlKM4cWzPPaUyYsmWoq48tQ12dkrtC16Hw/XjYttLraowxJigsCA6l/30QGQsf3+11JcYYExQWBIeS2BxOvQ1++QhWfO51NcbUabWtqzoUHc5raEFQGSf9HzRsBx/fBUWFXldjTJ0UGxvLtm3bLAyOgKqybds2YmNjq3Q/+x5BZUTGOIPFk4bCnBeg5/VeV2RMndOqVSuysrLYsmWL16XUarGxsbRq1apK97EgqKzjBkK7U+HLh6HzJVCvkdcVGVOnREVF0a5dO6/LCEvWNVRZInD232BfNnz1V6+rMcaYamNBUBXNO8EJv4fZ/4XNS72uxhhjqkVQg0BEzhaR5SKyQkRGV3DcxSKiIlLmlx1CSr+7nTOaTb8TbFDLGFMHBC0IRMQHPAucA3QELheRjmUclwjcDPwQrFqqVXxj6HsnrPoSfpnudTXGGHPEgtki6AGsUNVVqpoPTAQGlXHcQ8CjQF4Qa6leJ14HTY5xvmRWeOhT2BljTCgLZhCkAGsDbme520qISDegtapOC2Id1c8XBWc9AttXwo8TvK7GGGOOiGeDxSISATwJ3FaJY68XkTkiMidk5hh3OAM6nAlfPwa7Q6QmY4w5DMEMgnVA64DbrdxtfolAGvCViGQCJwFTyxowVtXnVbW7qnZv2rRpEEuuorMegYJc+OIhrysxxpjDFswgmA10EJF2IhINDAGm+neq6i5VbaKqqaqaCnwPnK+qNbTGdDVo0gF6XO+czWxDcE4YYYwxwRa0IFDVQuBG4GNgKfCmqi4RkTEicn6wnrfG9bnD+Zbx9NE2ndQYUysFdYkJVf0Q+PCAbfeVc2zfYNYSNHENne8WTPuLc8L7Thd4XZExxlSJfbO4OnQbBs06waf3QsFer6sxxpgqsSCoDr5IOPuvsHMNfPdPr6sxxpgqsSCoLu37wHHnwoynIHuD19UYY0ylWRBUpzPHQnEBfP6g15UYY0ylWRBUp0bt4OQ/w4I3IKv2zII1xoQ3C4Lq1vs2SGhu00mNMbWGBUF1i0mE0+6DrNmw6C2vqzHGmEOyIAiG9CugZQZ8ej/k7/G6GmOMqZAFQTBERMA5j0LOepg5zutqjDGmQhYEwdLmJEi7GGY943y/wBhjQpQFQTCd/iAg8GmZq2oYY0xIsCAIpgat4ZSbYckUWD3L62qMMaZMFgTBdsrNkJQCH42C4iKvqzHGmINYEARbdD04YwxsXAjzX/O6GmOMOYgFQU1Iuxha94TPx0BettfVGGNMKRYENUHEWZ10zxaY8YTX1RhjTCkWBDUl5QTIGArf/Qu2rfS6GmOMKWFBUJNOuw8iY+CTe7yuxBhjSlgQ1KTEFnDq7bD8Q/jlE6+rMcYYwIKg5p30Z2jcAT66AwryvK7GGGMsCGpcZDQMeAx2/Aaz/uF1NcYYY0HgiaP6Q8cLnBlEO1Z7XY0xJsxZEHjlrEdAfDD9Tq8rMcaEOQsCr9RPgT4jYfk0+OVjr6sxxoQxCwIv2cCxMSYEWBB4KTIaBjwOOzKd8xYYY4wHLAi8dlQ/6HQhzPi7EwjGGFPDLAhCwZkP28CxMcYzFgShoH4K9LnD/caxDRwbY2qWBUGoOOn/oMkxNnBsjKlxFgShInDg+Nunva7GGBNGwiYIioqVRVm7vC6jYu37QqeLYOaTsP03r6sxxoSJsAmCpz/7hYvHz2LF5hyvS6nYmWNt4NgYU6PCJgiuOjmV+Ggft721kMKiYq/LKV/9FOg7Cn75CJZP97oaY0wYCJsgaJoYw0MXpLFg7U4mfLPK63Iq1vMGaHKsO3C81+tqjDF1XFCDQETOFpHlIrJCREaXsX+4iCwSkfkiMlNEOgaznnO7JDOwc0vGffYLyzaG8Enk/QPHO1fbwLExJuiCFgQi4gOeBc4BOgKXl/GH/nVV7ayqGcBjwJPBqsdvzKBOJMVGcftbCygI5S6i9n2cgeMZNnBsjAmuYLYIegArVHWVquYDE4FBgQeoauDH8nhAg1gPAI0TYnj4ws4sXpfNv74M8ZPIn/UwRETC9IMaU8YYU22CGQQpwNqA21nutlJE5M8ishKnRTCirAcSketFZI6IzNmyZcsRF3Z2WgsGZSTzjy9+Zcn6EJ5SmpQMfUfDL9Nh+UdeV2OMqaM8HyxW1WdV9ShgFHBPOcc8r6rdVbV706ZNq+V5HzivEw3jo7ntzQXkF4ZwF9FJN0DT4+CjUTZwbIwJimAGwTqgdcDtVu628kwELghiPaU0jI/mrxd2ZtnGHP75xa819bRV54vaP3A8c5zX1Rhj6qBgBsFsoIOItBORaGAIMDXwABHpEHBzIFCjf5FP79ici7u14tmvVrIwa2dNPnXVtDsV0i6GmU/B9hCf+mqMqXWCFgSqWgjcCHwMLAXeVNUlIjJGRM53D7tRRJaIyHzgL8CwYNVTnvvO60iTBKeLaF9hUU0/feWdOdZpHXw0CjToY+rGmDAS1DECVf1QVY9R1aNU9WF3232qOtW9frOqdlLVDFXtp6pLgllPWerHRfHoxV34dfNuxn0Wwl1E/oHjXz+xgWNjTLXyfLA4FPQ9thlDTmzNhK9X8tOaHV6XU76ew52B4+k2cGyMqT4WBK67Bx5Pi6RYbntrAXkFIdpF5IuCAU/AzjXOeIExxlQDCwJXYmwUj12Szqote/j7J8u9Lqd87XpD2iXODKJtIf6FOGNMrWBBEOB3HZowtGcb/jPzN+Zkbve6nPLZwLExphpZEBzgrgHHk9IgjtvfWsDe/BDtIkpqCX3vhBWfOuc5NsaYI2BBcID4mEgevySdzG25PDp9mdfllK/nn6Dp8fDRaMjP9boaY0wtVqkgEJF4EYlwrx8jIueLSFRwS/POyUc15ppeqbw0K5PvV23zupyy+aJg4BOwywaOjTFHprItgm+AWBFJAT4BrgJeClZRoeCOs48ltXE9Rk5ewJ59hV6XU7bU30HnwfDtOBs4NsYctsoGgahqLnAR8C9VHQx0Cl5Z3qsXHcnjg9PJ2rGXv3601OtyynfGQ+CLsYFjY8xhq3QQiMjJwFBgmrvNF5ySQseJqY249pR2vPr9Gmb+utXrcsqW1BL62cCxMebwVTYIbgHuBKa46wW1B74MWlUh5PazjqV903hGvb2QnLwCr8spW4/roVlHGzg2xhyWSgWBqn6tquer6qPuoPFWVS3zJDJ1TWyUjycGp7Nh114e+TBEu4j83zjetQZmBv1sn8aYOqays4ZeF5EkEYkHFgM/i8jI4JYWOrq1acj1px7FGz+u5avlm70up2ypp0DnS52T3dvAsTGmCirbNdTRPb/wBcBHQDucmUNh45bTO9ChWQKj317Err0h2kV0pn/g+A4bODbGVFplgyDK/d7ABcBUVS2gBk40H0pio3z8/dJ0tuzex0Mf/Ox1OWVLbAH97oIVn8GyaYc+3hhjqHwQTAAygXjgGxFpC2QHq6hQ1aVVA/6v71FMnpvF50s3eV1O2fwDx9Nt4NgYUzmVHSx+RlVTVHWAOlYD/YJcW0i6qX8HjmuRyOh3FrEzN9/rcg7mi3QHjtfCjL97XY0xphao7GBxfRF5UkTmuJe/47QOwk50ZAR/vzSdHXvyeWBqjZ9QrXJST4Eul8GsZ2DrCq+rMcaEuMp2Db0A5ACXupds4MVgFRXqOiXX56b+HXh3/nqmL97odTllO+MhiIy1gWNjzCFVNgiOUtX7VXWVe3kQaB/MwkLd//U7ik7JSdzz7iK27wnBLqLE5s7A8crPYdkHXldjjAlhlQ2CvSLyO/8NETkFCOuT5kb5nC6iXXsLuPe9xV6XU7YT/wjNOsH0OyF/j9fVGGNCVGWDYDjwrIhkikgm8E/gT0GrqpY4rkUSt5x+DNMWbuCDheu9Ludgvkh3qWobODbGlK+ys4YWqGo60AXooqpdgf5BrayW+NOp7UlvVZ97313Mlpx9XpdzsLa9oMsQ+NYGjo0xZavSGcpUNdv9hjHAX4JQT60T6YvgicHp7Mkv4p53F6GhODB7xhiIioOPRtrAsTHmIEdyqkqptipquQ7NE7ntjGP4eMkmpi4IwS6ikoHjL2Dp+15XY4wJMUcSBPbRMsB1vdvTrU0D7ntvCZuz87wu52A2cGyMKUeFQSAiOSKSXcYlB0iuoRprBV+E8MTgdPIKirhrSgh2EfkHjrOz4JsnvK7GGBNCKgwCVU1U1aQyLomqGllTRdYW7ZsmcMfZx/HZ0s28PW+d1+UcrG0vSL8cZv0Dtv7qdTXGmBBxJF1Dpgy/75VKj9RGPPj+EjbsCsGvWvgHjj+0gWNjjMOCoJpFRAiPD+5CYZEy+u0Q7CJKaAb97oZVX8KSKV5XY4wJARYEQdC2cTx3DjiOr3/ZwqTZa70u52AnXgct0+HdG+DXT72uxhjjMQuCILmyZ1tObt+YsdOWkrUjxM4L4IuEK9+BJsfAG0Ng8TteV2SM8ZAFQZBERAiPXdIFVWXU2wtDr4sovglc8wG0OhEm/wHmvux1RcYYj1gQBFHrRvW4e2BHvl2xjdd+WON1OQeLre+0DI4+Hd4f4cwmMsaEHQuCILu8R2t6d2jCIx8uZc22EOsiAoiuB0Neh04Xwif3wBdjbTaRMWEmqEEgImeLyHIRWSEio8vY/xcR+VlEForI5+65kOsUEeHRi7vgE2Hk5AUUF4fgH9nIaLj4v9DtavjmcedkNsXFXldljKkhQQsCEfEBzwLnAB2By0Wk4wGH/QR0V9UuwGTgsWDV46XkBnHce15HfvhtO//7LtPrcsoW4YPznoGTb4Qfn3dmFBUVel2VMaYGBLNF0ANY4Z7RLB+YCAwKPEBVv1RVf3/J90CrINbjqcEntKLfsU352/RlZG4N0bV+RODMsdD/Hlg4Ed4aBgUhuG6SMaZaBTMIUoDASfRZ7rbyXAt8VNYOEbleROaIyJwtW7ZUY4k1R0T460VdiPZFcNtbCygsCtGuFxE4dSSc87hzisvXB8O+3V5XZYwJopAYLBaRK4HuwONl7VfV51W1u6p2b9q0ac0WV41a1I/loQvSmLt6B89+udLrcirW83q4cAJkfgv/GwS5272uyBgTJMEMgnVA64DbrdxtpYjI6cDdwPmqGoKn+KpegzJSuCAjmWe++JW5q3d4XU7F0ofApf+DjQvhpXMhZ5PXFRljgiCYQTAb6CAi7UQkGhgCTA08QES6AhNwQmBzEGsJKWMuSKNl/VhumfQTOXkFXpdTsePPhaFvwY5MeOEs2LHa64qMMdUsaEGgqoXAjcDHwFLgTVVdIiJjROR897DHgQTgLRGZLyJTy3m4OiUpNoqnh2Swfmce97+3xOtyDq19X7j6Pdi7HV44G7Ys97oiY0w1kpBb+uAQunfvrnPmzPG6jGox7rNfGPfZrzw9JINBGRWNo4eIjYvhlQtBi5xvJCdneF2RMaaSRGSuqnYva19IDBaHqxv7Hc0JbRtyz5TFrN0egt86PlCLNPjDdIiKh5fPg9WzvK7IGFMNLAg8FOmLYNxlGQDcOml+6E4pDdT4KCcMEls4rYNfPvG6ImPMEbIg8FjrRvV46II05qzewb++CvEppX71U+D3H0HTY2Hi5bD4ba8rMsYcAQuCEHBBV2dK6dOf14IppX7xTWDY+9CqB0y+Fua+5HVFxpjDZEEQImrVlFK/2Ppw5dvuMtY3w7fPeF2RMeYwWBCEiKTYKMZdlsG6HXu5f2otmFLqV7KM9UXw6b3w+UO2jLUxtYwFQQjpntqIm/p34J1565i6YL3X5VReZDRc/B/oNgxmPAEfjrRlrI2pRSwIQsxN/Z0ppXdPWRR65zquSIQPznsaeo2A2f+Gd4fbMtbG1BIWBCHGP6VUtRZNKfUTgTPGwGn3wcJJ8ObVtoy1MbWABUEIcqaUdmJ2Zi2aUuonAr1vgwFPwPJp7jLWOV5XZYypgAVBiLqwaysGuVNK562pJVNKA/X4I1z4vLuM9QW2jLUxIcyCIIQ95J9SOnF+7ZlSGij9MrjsFXcZ64GQs9HriowxZbAgCGH+KaVZO3Jr15TSQMcNdJexXu2sXGrLWBsTciwIQlytnVIaqH1fGDYV9u6wZayNCUEWBLXATf2PplubBrVvSmmgVt2d9Ym0yAmD9T95XZExxmVBUAtE+iJ4ekjXkimlRcW19Ju7zTs6K5fGJMBL5zkDycYYz1kQ1BKlppR+ucLrcg5fo/bwh48hKRlevciWsTYmBFgQ1CL+KaXjauuUUr+kZHcZ6+OcZaw/e9AGkY3xkAVBLfPQBWm0SKrFU0r94hs7y1gfNxC+HQdPp8OrF8PS96GoFv9extRCFgS1jP/E97V6SqlfbBJc+j+4eSH0GQWbfoZJV8JTac4qptZKMKZGWBDUQt1TG3FjbZ9SGqhBa+h3J9yyCC6fCMkZMPNJayUYU0NEa9na8d27d9c5c+Z4XYbnCouKuXTCd/y6eTcf3dybVg3reV1S9dq5Fn56Feb9D3LWQ0IL6HoldLsaGrb1ujpjah0Rmauq3cvaZy2CWipwSulfJi2ovVNKy3NgK6FlurUSjAkSC4JazD+l9MfM7bV7SmlFfJFw7Dkw9E13LOEOG0swpppZENRyF2SkcH56HZhSWhkNWkO/u5xWwpA3nFbCjL9bK8GYI2RjBHVAdl4B54ybgS9C+PDm3iTERHpdUs3ZuRZ+egXmvbJ/LKHbVdD1KhtLMCaAjRHUcaWmlL5Xy6eUVlVZrYRvnghoJXxgrQRjDsGCoI7wTyl9e14W79eFKaVV5YuE4wY4Ywm3LHLHEpbApKHOWMIXY20swZhyWNdQHVJYVMzgCd+xoq5OKa2qokL49ROY+yL8+qmz7ejT4ITfwzFnO+FhTJiwrqEwEemL4OnL6vCU0qoqaSW8VUYroZPTSti5xusqjfGcBUEd06ZxPcYMcqaUPvdVHZ1SejhKxhIWu2MJXZyxhHFd4NVLnBlHdl5lE6asbVwHXdg1ha+Wb+Gpz37llKOb0LVNQ69LCh3+VsJxA5zWwLxXnFlHk6509ie1ghad3Uua87NBKkTYZyZTd9kYQR21a28BA54O0ymlVVVUCKtnwvr5sGkxbFwEW38BLXb2RydA87T9wdCiMzTrCFFxnpZtTFVUNEZgQVCHzc7czmUTvuPCrq34+6XpXpdTuxTshc1LnVDwh8PGxZCf4+yXCGjcoXQ4NO8Mic29rduYclQUBEH9mCgiZwNPAz7gP6r6twP2nwqMA7oAQ1R1cjDrCTcnpjbixn5H88wXK+h7bFPOS0/2uqTaIyoOUro5F7/iYti5unQ4rJ0Ni9/ef0x8s4Bw6OK0JBofbTOUTEgL2rtTRHzAs8AZQBYwW0SmqurPAYetAa4Bbg9WHeFuxGkdmLFiK3dNWUS3tg1JaWDdGYctIgIatXMuHc/fv33vDmc20sZF+y/fPwdF+c7+yFhodvz+VkOLztC8k3M+BmNCQDA/pvQAVqjqKgARmQgMAkqCQFUz3X3FQawjrPmnlA54Zga3TpzPG9efhC9CvC6rbolrCKm/cy5+RQXOOENgOCz9wFlW269haulwaNEZ6rcCsX8fU7OCGQQpwNqA21lAz8N5IBG5HrgeoE2bNkdeWZjxTyn9y5sLeO6rFdzYv4PXJdV9vijnU3/zTpA+xNmmCjkbSoeDPyBwx+pi6kOjVGjYzgmKwEv9Vs7jGlPNakXHpao+DzwPzmCxx+XUShd2TeFLm1LqLRFISnYux5y1f/u+3bD5ZycUNv8MOzKdMYhl06A4YJ0k8Tlh0DDV6Z46MCji7N/UHJ5gBsE6oHXA7VbuNuMBEWHsBWnMW72DWybNZ9oIm1IaMmISoHUP5xKouMhpQezI3H/Z/pvzc+kHkLu19PGx9QOCoZ21JkylBfMvwWygg4i0wwmAIcAVQXw+cwj146IYNySDyyZ8xwNTl/DEYJtSGtIi3BZA/Valxx/89uU4C+mVBIUbEpuWwPKP9g9WQ+nWRFktCmtNhLWgBYGqForIjcDHONNHX1DVJSIyBpijqlNF5ERgCtAQOE9EHlTVTsGqyZSeUpqWnMSVJ7Ul0mffmq2VYhLdqappB+8rqzXhb1Esm3aI1kQqNGjrdGEltoSkFKjX2L5dXYfZF8rCUGFRMVf+9we+X7Wd1o3i+GPv9gw+oTVx0T6vSzM15aDWROb+FsXONaVbEwARUW4oJENSS0hMPuB6S2d/ZEzN/y6mUuybxeYgxcXKp0s3Mf7rlfy0ZieN4qO5plcqV5/clgb1or0uz3ipuAh2b4LsDc5Z37I3QPY6p4WRvd655GyAgtyD71uviRMKSSkBwZG8/3piS6f1YVNka5wFgSmXqvLjb9uZ8M0qvli2mXrRPi47sTXX9W5vXz4z5VOFvF1uKPjDoozrudsOvm9U/P4WRFJK2S2MhGbOGImpNhYEplKWbczm+a9XMdU9w9n56clc36c9x7Wwb8Caw1S47+CWxIHXczZAcWHp+4kPElu4YVFBC8MW/qs0CwJTJet27uW/M35j4uw15OYX0e/Ypvypz1H0bNcIsSa9qW7Fxc7gdfa6gO6o9Qd0Ta3fv+BfoLiG+8cokpIDrgcER1xD64rCgsAcpp25+bzy3WpempXJtj35ZLRuwPA+R3Fmx+ZE2DIVpqbtyyl/vMJ/fc8WSr6l7RcZW3ZrIjA4ElrU+YUBLQjMEckrKOKtOWt5fsYq1m7fS/sm8Vx/ansu7JZCTKT145oQUlQAORsPGK/wB0fA9QNnRUmEs3JsqdZEwPXEls4S4zFJtbZ1YUFgqkVhUTEfLd7I+K9XsmR9Nk0TY/jDKe0YelIbkmLtW6umllB1Tksa2LLI2RDQNeVez9t18H0j45xASGhRwc8WENco5L53YUFgqpWq8u2KbYz/eiUzV2wlMSaSK05qw7WntKNZUqzX5RlTPfL3uK2LdZCzyQmI3ZucbYE/92UffN+IKGfmU2KLikMjvmmNdUlZEJigWbxuF+O/XsmHizYQGRHBhV1TuL5Pe45qmuB1acbUjPxc2L3RCYuKfpY1lVYinO9elNWqSGhe+ucRflnPgsAE3epte/jPjN94c85a8ouKObNjc/7U5yi62SqnxjgK82HP5oDWRTmhsWfz/vNlB4prCGc+DF2HHtbTWxCYGrN19z5enpXJ/75bza69BfRo14jhfdrT79hmNvXUmMooLoI9W8sOirSLoW2vw3pYCwJT4/bsK2Ti7LX8d8Yq1u/K49jmifypT3vOS08myha5M6bGWRAYzxQUFfP+gvVM+HoVyzflkFw/lmt7t2fIia2Jt/MhGFNjLAiM51SVL5dvZvzXq/jxt+3Uj4vi6pPbMqxXKk0SbMVKY4LNgsCElHlrdjD+q5V8unQT0b4IjmuZRPPEGJolxdA8MZZmSTE0S4qlWWIMzRJjaRwfbd9kNuYIVRQE1jY3Na5bm4Y8f3V3Vmzezavfr2bllt2s3pbLj5nb2ZlbcNDxkRFCk4QYmifF0DQxluZJTkA0T3LCo5kbHo3jY/BZYBhTZRYExjNHN0vggfNLn5Aur6CILTn72Jyzj83Zec7PnDw2ZTvbsnbkMm/NDrbvyT/o8XwRQpOE6JKQOCg0SgIj2s7KZkwACwITUmKjfLRuVI/WjepVeFx+YTFbdu9jU3Yem7P3saUkLJyf63bm8dOanWwrIzAiBBq7LYyS0EiIITE2iviYSOJjfNSLdn7GR0eWbIuPiaRelM9CxNQ5FgSmVoqOjCClQdwhT56TX1jM1t37WxibcvaxJXt/aGzclcfCrF1s27OPyg6XxURGkBATSb1SQRFJfLQTIAkxPurFRDrHRPvcfc7x/m3OT+eY2KgI+46F8ZQFganToiMjSG4QR/IhAqOoWMnNLyQ3v4jd+wrJ3ef+zC9kT34Re/YVupcicvML3X3u9vxCsvcWsGHnXmdbvnNsQVHlkkUEN1B8JYFRL8r96YZL6Z/7WyxxUU7QxEW79w3YbwFjKsuCwBic8YXE2CgSY6NoXk2PmV9YXBIUZQbMPidkcvcVsjsgYPa6YbJjTz5ZO4pKbufmF5FfWMbSA+UQgXpRTuvk4CBxgiPObbE4geIjLtrfstl/fJw/eAKu26B83WJBYEyQREdGEB0ZTcP46Gp7zMKiYnILish1gyM3v6ikFbLXbaH4t+3f72+9FLG3oJCcvEI2Z+/bf5/8QvIKKh8w4HSPHRgucQHhUlYLJvCYkutuCPm70GIirRXjBQsCY2qRSF8ESb6Iaj//Q1GxsrfAaZ0EBkvpQNl/PbCV4lwvYm9+IRuz80rty80voqi48t9VOrAV4+/6CrweF+07oKVTduj4r/uDyZY2KZ8FgTEGX4SQ4A5wVydVJb+ouMJQyXW7x3IL3FBxWy7OeIxzffc+pxWTW1AYcExRlWqJ9kW4LZL9XVxlhoc7O8zfYjkohA5o+cRF+Wp9K8aCwBgTNCJCTKSPmEgfDSqeEVxlxcVKXqEbCge1UCrfotmyex+5+bklXWt7C4oqPdDv/I6UdG8d2AVW2W6ywNZLvDtZIC7aR3RkzbRiLAiMMbVSRIS4f1Sr/89YfqHbinFbJnv9IRIwPrO3wB9C/u60/df9obN1975St/cWFFV6mjJAlE/ckHFmkd16+jGcl55c7b+vBYExxhzAGeiPoD7VOxajquQVFJdqsZTbeinVXeZcb1iv+iYeBLIgMMaYGiIixLndQqHEhtGNMSbMWRAYY0yYsyAwxpgwZ0FgjDFhzoLAGGPCnAWBMcaEOQsCY4wJcxYExhgT5kSr8n3nECAiW4DVh3n3JsDWaiyntrPXozR7Pfaz16K0uvB6tFXVpmXtqHVBcCREZI6qdve6jlBhr0dp9nrsZ69FaXX99bCuIWOMCXMWBMYYE+bCLQie97qAEGOvR2n2euxnr0Vpdfr1CKsxAmOMMQcLtxaBMcaYA1gQGGNMmAubIBCRs0VkuYisEJHRXtdTk0SktYh8KSI/i8gSEbnZ3d5IRD4VkV/dnw29rrUmiYhPRH4SkQ/c2+1E5Af3PTJJRIJzOqgQJCINRGSyiCwTkaUicnK4vj9E5Fb3/8liEXlDRGLr+nsjLIJARHzAs8A5QEfgchHp6G1VNaoQuE1VOwInAX92f//RwOeq2gH43L0dTm4GlgbcfhR4SlWPBnYA13pSlTeeBqar6nFAOs7rEnbvDxFJAUYA3VU1DfABQ6jj742wCAKgB7BCVVepaj4wERjkcU01RlU3qOo893oOzn/yFJzX4GX3sJeBCzwp0AMi0goYCPzHvS1Af2Cye0jYvB4iUh84FfgvgKrmq+pOwvf9EQnEiUgkUA/YQB1/b4RLEKQAawNuZ7nbwo6IpAJdgR+A5qq6wd21EWjuVV0eGAfcARS7txsDO1W10L0dTu+RdsAW4EW3q+w/IhJPGL4/VHUd8ASwBicAdgFzqePvjXAJAgOISALwNnCLqmYH7lNnHnFYzCUWkXOBzao61+taQkQk0A14TlW7Ans4oBsoXN4f7jjIIJxwTAbigbM9LaoGhEsQrANaB9xu5W4LGyIShRMCr6nqO+7mTSLS0t3fEtjsVX017BTgfBHJxOkm7I/TR97A7Q6A8HqPZAFZqvqDe3syTjCE4/vjdOA3Vd2iqgXAOzjvlzr93giXIJgNdHBH/qNxBn+melxTjXH7v/8LLFXVJwN2TQWGudeHAe/VdG1eUNU7VbWVqqbivBe+UNWhwJfAJe5h4fR6bATWisix7qbTgJ8Jz/fHGuAkEann/r/xvxZ1+r0RNt8sFpEBOP3CPuAFVX3Y24pqjoj8DpgBLGJ/n/hdOOMEbwJtcJb2vlRVt3tSpEdEpC9wu6qeKyLtcVoIjYCfgCtVdZ+H5dUYEcnAGTiPBlYBv8f5oBh27w8ReRC4DGe23U/AdThjAnX2vRE2QWCMMaZs4dI1ZIwxphwWBMYYE+YsCIwxJsxZEBhjTJizIDDGmDBnQWCMS0SKRGR+wKXaFlkTkVQRWVxdj2dMdYo89CHGhI29qprhdRHG1DRrERhzCCKSKSKPicgiEflRRI52t6eKyBcislBEPheRNu725iIyRUQWuJde7kP5ROTf7lr3n4hInHv8CPdcEQtFZKJHv6YJYxYExuwXd0DX0GUB+3apamfgnzjfUAf4B/CyqnYBXgOecbc/A3ytquk4a/Yscbd3AJ5V1U7ATuBid/tooKv7OMOD86sZUz77ZrExLhHZraoJZWzPBPqr6ip38b6NqtpYRLYCLVW1wN2+QVWbiMgWoFXgEgTu8t+fuid5QURGAVGqOlZEpgO7gXeBd1V1d5B/VWNKsRaBMZWj5VyvisC1aYrYP0Y3EOcMet2A2QGrXBpTIywIjKmcywJ+fuden4WzeinAUJyF/cA5reMNUHJe5PrlPaiIRACtVfVLYBRQHzioVWJMMNknD2P2ixOR+QG3p6uqfwppQxFZiPOp/nJ32004Z/UaiXOGr9+7228GnheRa3E++d+Ac7arsviAV92wEOAZ9zSRxtQYGyMw5hDcMYLuqrrV61qMCQbrGjLGmDBnLQJjjAlz1iIwxpgwZ0FgjDFhzoLAGGPCnAWBMcaEOQsCY4wJc/8PuAR83PIccPwAAAAASUVORK5CYII=\n" + }, + "metadata": { + "needs_background": "light" + } + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lmqQE8Kpec04" + }, + "source": [ + "Nice! The **loss curves** show the loss going down over time. Remember, loss is the measure of how *wrong* your model is, so the lower the better.\n", + "\n", + "But why did the loss go down?\n", + "\n", + "Well, thanks to our loss function and optimizer, the model's internal parameters (`weights` and `bias`) were updated to better reflect the underlying patterns in the data.\n", + "\n", + "Let's inspect our model's [`.state_dict()`](https://pytorch.org/tutorials/recipes/recipes/what_is_state_dict.html) to see see how close our model gets to the original values we set for weights and bias.\n", + "\n" + ] + }, + { + "cell_type": "code", + "source": [ + "# Find our model learned parameters\n", + "print('The model learned the following values for weights and bias:')\n", + "print(model_0.state_dict())\n", + "print(\"\\n And the original values for weights and bias are:\")\n", + "print(f\"weights: {weight}, bias: {bias}\")" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "1HTl2FpJ9Pn9", + "outputId": "f59c7501-4f4b-45a1-de75-3cc0de68d82b" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "The model learned the following values for weights and bias:\n", + "OrderedDict([('weights', tensor([0.5784])), ('bias', tensor([0.3513]))])\n", + "\n", + " And the original values for weights and bias are:\n", + "weights: 0.7, bias: 0.3\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BZyBa9rMelBv" + }, + "source": [ + "Wow! How cool is that?\n", + "\n", + "Our model got very close to calculate the exact original values for `weight` and `bias` (and it would probably get even closer if we trained it for longer).\n", + "\n", + "> **Exercise:** Try changing the `epochs` value above to 200, what happens to the loss curves and the weights and bias parameter values of the model?\n", + "\n", + "It'd likely never guess them *perfectly* (especially when using more complicated datasets) but that's okay, often you can do very cool things with a close approximation.\n", + "\n", + "This is the whole idea of machine learning and deep learning, **there are some ideal values that describe our data** and rather than figuring them out by hand, **we can train a model to figure them out programmatically**." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c-VBDFd2ec05" + }, + "source": [ + "## 4. Making predictions with a trained PyTorch model (inference)\n", + "\n", + "Once you've trained a model, you'll likely want to make predictions with it.\n", + "\n", + "We've already seen a glimpse of this in the training and testing code above, the steps to do it outside of the training/testing loop are similar.\n", + "\n", + "There are three things to remember when making predictions (also called performing inference) with a PyTorch model:\n", + "\n", + "1. Set the model in evaluation mode (`model.eval()`).\n", + "2. Make the predictions using the inference mode context manager (`with torch.inference_mode(): ...`).\n", + "3. All predictions should be made with objects on the same device (e.g. data and model on GPU only or data and model on CPU only).\n", + "\n", + "The first two items make sure all helpful calculations and settings PyTorch uses behind the scenes during training but aren't necessary for inference are turned off (this results in faster computation). And the third ensures that you won't run into cross-device errors." + ] + }, + { + "cell_type": "code", + "source": [ + "# 1. Set the model in evaluation mode\n", + "model_0.eval()\n", + "\n", + "# 2. Setup the inference mode context manager\n", + "with torch.inference_mode():\n", + " # 3. Make sure the calculations are done with the model and data on the same device\n", + " # in our case, we haven't setup device-agnostic code yet so our data and model are\n", + " # on the CPU by default.\n", + " # model_0.to(device)\n", + " # X_test = X_test.to(device)\n", + " y_preds = model_0(X_test)\n", + "y_preds" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "zTIpQLrh_5Vy", + "outputId": "7c89a29b-399a-46a9-d358-413d700688ab" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[0.8141],\n", + " [0.8256],\n", + " [0.8372],\n", + " [0.8488],\n", + " [0.8603],\n", + " [0.8719],\n", + " [0.8835],\n", + " [0.8950],\n", + " [0.9066],\n", + " [0.9182]])" + ] + }, + "metadata": {}, + "execution_count": 18 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Cn21JvzmjbBO" + }, + "source": [ + "Nice! We've made some predictions with our trained model, now how do they look?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 428 + }, + "id": "b_kBqpCfec05", + "outputId": "d8817fa6-0d27-402c-b86d-d9ee5e957785" + }, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAGbCAYAAADgEhWsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAAsTAAALEwEAmpwYAAAuWUlEQVR4nO3de3hV9Zn3/89NwiEcRQko54N4QASFiDKO4AHHAyDjz8cCWgtqFQWeyjNStaIoaJ3aWq0do4NWhIoHFNGhgGjLgKcRIYAwhoAiUAERgtOhAhVIcv/+2DFNYpK9w9rn/X5dV65krfXda93JCvBhre+6t7m7AAAAcHQaJLoAAACAVEaYAgAACIAwBQAAEABhCgAAIADCFAAAQADZiTpwmzZtvGvXrok6PAAAQMRWr169191za9qWsDDVtWtXFRQUJOrwAAAAETOzP9e2jdt8AAAAARCmAAAAAiBMAQAABECYAgAACIAwBQAAEEDYp/nMbKakYZL2uHvvGrabpMclXS7poKSx7r4maGF//etftWfPHh05ciTorpDmGjZsqLZt26ply5aJLgUAkIEiaY0wS9ITkn5fy/bLJPUs/zhb0lPln4/aX//6V+3evVsdOnRQTk6OQnkN+D5319/+9jft3LlTkghUAIC4C3ubz93flfQ/dQwZIen3HrJC0jFmdkKQovbs2aMOHTqoadOmBCnUyczUtGlTdejQQXv27El0OQCADBSNOVMdJG2vtLyjfN1RO3LkiHJycgIVhcySk5PDLWEAQELEdQK6md1sZgVmVlBcXBxubJyqQjrg9wUAkCjRCFM7JXWqtNyxfN33uPvT7p7n7nm5uTW+vQ0AAEBKiUaYWiDpRxZyjqR97r4rCvsFAABIemHDlJm9JOlDSSeb2Q4zu9HMbjGzW8qHLJa0RdJmSc9IGh+zajPQ2LFjNWzYsHq95vzzz9fEiRNjVFHdJk6cqPPPPz8hxwYAIBHCtkZw99FhtrukCVGrKEWFm7MzZswYzZo1q977ffzxxxX6EUdu/vz5atiwYb2PlQjbtm1Tt27dtGrVKuXl5SW6HAAA6i2SPlOIwK5df7+zuXDhQt10001V1lV/OvHIkSMRBZ5WrVrVu5Zjjz223q8BAABHh7eTiZLjjz++4uOYY46psu7bb7/VMccco5deekkXXnihcnJyNGPGDH399dcaPXq0OnbsqJycHJ122ml67rnnquy3+m2+888/X+PHj9fdd9+tNm3aqG3btpo8ebLKysqqjKl8m69r16568MEHNW7cOLVs2VIdO3bUr371qyrH+fTTTzV48GA1adJEJ598shYvXqzmzZvXeTWttLRUkydPVuvWrdW6dWtNmjRJpaWlVcYsWbJE5513nlq3bq1jjz1Wl1xyiYqKiiq2d+vWTZJ01llnycwqbhGuWrVK//RP/6Q2bdqoZcuW+sd//Ed9+OGH4U8EACCjTFg0QdnTszVhUeJukhGm4uhnP/uZxo8frw0bNuif//mf9e2336pfv35auHChCgsLddttt2ncuHFaunRpnft54YUXlJ2drf/6r//SE088od/85jeaO3duna957LHHdPrpp2vNmjW68847dccdd1SEk7KyMl155ZXKzs7WihUrNGvWLE2bNk2HDh2qc5+//vWv9cwzz2jGjBn68MMPVVpaqhdeeKHKmAMHDmjSpElauXKlli9frlatWmn48OE6fPiwJGnlypWSQqFr165dmj9/viTpm2++0XXXXaf33ntPK1eu1BlnnKHLL79cX3/9dZ01AQAyy4zVM1TqpZqxekbiinD3hHz079/fa7Nhw4Zat9XX+PHuWVmhz/Hy6quveuhHG7J161aX5I888kjY144cOdJvvPHGiuUxY8b40KFDK5YHDx7s55xzTpXXDBkypMprBg8e7BMmTKhY7tKli48aNarKa0488UR/4IEH3N19yZIlnpWV5Tt27KjY/sEHH7gkf+6552qt9YQTTvAHH3ywYrm0tNR79uzpgwcPrvU1+/fv9wYNGvh7773n7n//2axatarW17i7l5WV+fHHH+/PP/98rWOi+XsDAEgN4xeO96xpWT5+YWz/oZdU4LVkmrS/MjVjhlRaGvqcaNUnWJeWlurnP/+5+vTpo+OOO07NmzfX/Pnz9cUXX9S5nz59+lRZbt++fdi3UqnrNRs3blT79u3VocPfG9efddZZatCg9l+Pffv2adeuXRo4cGDFugYNGujss6u+LePnn3+ua665Rj169FDLli3Vrl07lZWVhf0e9+zZo3Hjxumkk05Sq1at1KJFC+3Zsyfs6wAAmSV/aL5KppYof2h+wmpI+wno48aFgtS4cYmuRGrWrFmV5UceeUS//vWv9fjjj+v0009X8+bNdffdd4cNRtUnrptZlTlT0XpNNAwbNkwdO3bUjBkz1KFDB2VnZ6tXr14Vt/lqM2bMGO3evVuPPfaYunbtqsaNG+uiiy4K+zoAAOIt7cNUfn7oIxm9//77Gj58uK677jpJoVuun376acUE9ng55ZRT9OWXX+rLL79U+/btJUkFBQV1hq1WrVrphBNO0IoVK3ThhRdKCtW/cuVKnXBC6H2uv/76a23cuFFPPvmkLrjgAknSmjVrVFJSUrGfRo0aSdL3Jq6///77+u1vf6uhQ4dKknbv3l3l6UgAAJJF2t/mS2YnnXSSli5dqvfff18bN27UxIkTtXXr1rjXcfHFF+vkk0/WmDFjtG7dOq1YsUL/8i//ouzs7Dr7Z91222365S9/qXnz5mnTpk2aNGlSlcDTunVrtWnTRs8884w2b96sd955R7fccouys/+e4du2baucnBy99dZb2r17t/bt2ycp9LOZM2eONmzYoFWrVmnUqFEVwQsAgGRCmEqge+65RwMGDNBll12mQYMGqVmzZrr22mvjXkeDBg30+uuv69ChQxowYIDGjBmjKVOmyMzUpEmTWl93++236/rrr9ePf/xjnX322SorK6tSf4MGDTR37lytX79evXv31oQJE/TAAw+ocePGFWOys7P129/+Vr/73e/Uvn17jRgxQpI0c+ZM7d+/X/3799eoUaN0ww03qGvXrjH7GQAAkkcytDuoD/N6dteOlry8PC8oKKhxW1FRkU499dQ4V4TK1q1bpzPOOEMFBQXq379/osuJCL83AJAesqdnq9RLlWVZKplaEv4FcWBmq929xrfq4MoUJEmvv/663n77bW3dulXLli3T2LFj1bdvX/Xr1y/RpQEAMsy4/uOUZVka1z8Jnh6LQNpPQEdkvvnmG915553avn27WrdurfPPP1+PPfZY2PccBAAg2vKH5ie01UF9EaYgSfrRj36kH/3oR4kuAwCAlMNtPgAAgAAIUwAAAAEQpgAAQFykWsuDSBGmAABAXMxYPUOlXqoZq5PgDXOjiDAFAADiItVaHkSKp/kAAEBcpFrLg0hxZSqFde3aVY888khCjj1s2DCNHTs2IccGACCZEKaixMzq/AgSPO6//3717t37e+tXrVql8ePHB6g6fpYvXy4z0969exNdCgAAUcVtvijZtWtXxdcLFy7UTTfdVGVdTk5O1I+Zm5sb9X0CAID64cpUlBx//PEVH8ccc8z31r377rvq37+/mjRpom7dumnKlCk6fPhwxevnz5+vPn36KCcnR8cee6wGDx6s3bt3a9asWZo2bZoKCwsrrnLNmjVL0vdv85mZnn76aV199dVq1qyZunfvrjlz5lSp86OPPlK/fv3UpEkTnXnmmVq8eLHMTMuXL6/1ezt48KDGjh2r5s2bq127dnrooYe+N2bOnDk666yz1KJFC7Vt21ZXX321du7cKUnatm2bLrjgAkmhAFj5St2SJUt03nnnqXXr1jr22GN1ySWXqKioqL4/fgBAAqVry4NIEabi4K233tK1116riRMnqrCwUDNnztS8efN09913S5K++uorjRo1SmPGjFFRUZHeffddXXfddZKkkSNH6vbbb9fJJ5+sXbt2adeuXRo5cmStx5o+fbpGjBihdevWaeTIkbrhhhv0xRdfSJL279+vYcOG6ZRTTtHq1av1y1/+Uj/96U/D1j958mT98Y9/1GuvvaalS5dq7dq1evfdd6uMOXz4sKZNm6Z169Zp4cKF2rt3r0aPHi1J6tSpk1577TVJUmFhoXbt2qXHH39cknTgwAFNmjRJK1eu1PLly9WqVSsNHz68StAEACS3dG15EDF3T8hH//79vTYbNmyodVt9jV843rOmZfn4heOjts9wXn31VQ/9aEPOO+88nz59epUxr7/+ujdr1szLysp89erVLsm3bdtW4/7uu+8+P+200763vkuXLv6rX/2qYlmS33XXXRXLR44c8ZycHH/++efd3f3f//3fvXXr1n7w4MGKMS+88IJL8mXLltV47G+++cYbNWrkc+bMqbKuVatWPmbMmFp/BkVFRS7Jt2/f7u7uy5Ytc0leXFxc62vc3ffv3+8NGjTw9957r85xNYnm7w0AIHKJ+Lc23iQVeC2ZJu2vTCVDWl69erV+/vOfq3nz5hUf11xzjQ4cOKCvvvpKffv21ZAhQ9S7d29dddVVeuqpp1RcXHxUx+rTp0/F19nZ2crNzdWePXskSRs3blTv3r2rzN86++yz69zf559/rsOHD2vgwIEV65o3b67TTz+9yrg1a9ZoxIgR6tKli1q0aKG8vDxJqrgqVtf+r7nmGvXo0UMtW7ZUu3btVFZWFvZ1AIDkkT80XyVTS9Ky7UEk0j5MJUODsLKyMt133336+OOPKz7Wr1+vzz77TLm5ucrKytLbb7+tt99+W3369NGzzz6rnj17at26dfU+VsOGDassm5nKysqi9a3U6MCBA7rkkkvUtGlTPf/881q1apWWLFkiSWFv1w0bNkzFxcWaMWOGPvroI61du1bZ2dnc5gMApIy0f5ovGRqE9evXTxs3btSJJ55Y6xgz08CBAzVw4EBNnTpVp512mubOnau+ffuqUaNGKi0tDVzHKaecotmzZ+tvf/tbxdWplStX1vmaHj16qGHDhlqxYoW6d+8uKRSePvnkE/Xo0UNS6IrX3r179dBDD6lbt26SQhPqK2vUqJEkVfk+vv76a23cuFFPPvlkxQT1NWvWqKSkJPD3CgBAvKT9lalkMHXqVL344ouaOnWqPvnkE23cuFHz5s3THXfcIUlasWKFHnzwQa1atUpffPGFFixYoO3bt6tXr16SQk/t/fnPf9aaNWu0d+9eHTp06KjquOaaa5SVlaWbbrpJGzZs0J/+9KeKJ/PMrMbXNG/eXDfeeKPuvPNO/fGPf1RhYaFuuOGGKqGoc+fOaty4sZ544glt2bJFixYt0r333ltlP126dJGZadGiRSouLtb+/fvVunVrtWnTRs8884w2b96sd955R7fccouys9M+4wMA0ghhKg4uueQSLVq0SMuWLdOAAQM0YMAA/eIXv1Dnzp0lSa1atdIHH3ygYcOGqWfPnrr99tt177336oc//KEk6aqrrtLll1+uiy66SLm5uXrppZeOqo4WLVroD3/4gwoLC3XmmWfqpz/9qe6//35JUpMmTWp93SOPPKILLrhAV155pS644AL17t1bgwYNqtiem5ur2bNn64033lCvXr00bdo0Pfroo1X20aFDB02bNk1TpkxRu3btNHHiRDVo0EBz587V+vXr1bt3b02YMEEPPPCAGjdufFTfHwAgejK93UF9WGiCevzl5eV5QUFBjduKiop06qmnxrmizPQf//EfuvLKK7Vnzx61adMm0eUEwu8NAERP9vRslXqpsixLJVOZfmFmq909r6ZtXJnKMLNnz9Z7772nbdu2aeHChZo0aZKGDx+e8kEKABBdyfAAV6pgckqG2b17t+677z7t2rVLxx9/vIYOHaqHH3440WUBAJJMMjzAlSoIUxnmjjvuqJj4DgAAguM2HwAAQACEKQAAkLomTJCys0OfE4QwBQBABkm7lgczZkilpaHPCUKYAgAggyTDe9ZG1bhxUlZW6HOCEKYAAMggadfyID9fKikJfU4QnuYDACCD0PIg+rgylYLmzZtX5b30Zs2apebNm8f8uGPHjtWwYcNifhwAAFIJYSqKxo4dKzOTmalhw4bq3r27Jk+erAMHDsT0uCNHjtSWLVuitr/ly5fLzLR3794q6x9//HHNmTMnascBAKBGSfCEXn1EFKbM7FIz22Rmm83srhq2dzGzpWa23syWm1nH6JeaGoYMGaJdu3Zpy5YtevDBB/Xkk09q8uTJ3xtXUlKiaL0vYk5Ojtq2bRuVfdWlVatWOuaYY2J+HABAhkuCJ/TqI2yYMrMsSfmSLpPUS9JoM+tVbdgjkn7v7n0kTZf0r9EuNFU0btxYxx9/vDp16qRrrrlG1157rd544w3df//96t27t2bNmqUePXqocePGOnDggPbt26ebb75Zbdu2VYsWLTR48GBVfwPo3//+9+rSpYuaNm2qYcOGaffu3VW213Sbb/HixTr77LOVk5Oj4447TsOHD9e3334rSZozZ47OOusstWjRQm3bttXVV1+tnTt3SpK2bdumCy64QJKUm5srM9PYsWMlff8236FDhzRp0iS1a9dOTZo00TnnnKP333+/Yvt3V7iWLl2qs88+W02bNlVeXp7WrFlTMWbfvn267rrr1LZtWzVp0kTdu3fXb37zm2AnAQAyUFq1PEiCJ/TqI5IrUwMkbXb3Le5+WNLLkkZUG9NL0n+Wf72shu0ZKycnR0eOHJEkbd26VS+++KJeffVVrVu3To0bN9bQoUO1c+dOLVy4UGvXrtWgQYN04YUXateuXZKkjz76SGPHjtXNN9+sjz/+WMOHD9fUqVPrPOaSJUt0xRVX6OKLL9bq1au1bNkyDR48WGVlZZKkw4cPa9q0aVq3bp0WLlyovXv3avTo0ZKkTp066bXXXpMkFRYWateuXXr88cdrPM4dd9yhuXPnaubMmVq7dq1OP/10XXrppRW1f+dnP/uZfvGLX2jNmjU67rjjdO2111Zclbvnnnv03//931q4cKE2bdqkmTNnqkOHDkf50waAzJVWLQ+S4Am9enH3Oj8k/R9Jv6u0fJ2kJ6qNeVHSbeVf/3+SXNJxNezrZkkFkgo6d+7stdmwYUOt2+pt/Hj3rKzQ5xgbM2aMDx06tGL5o48+8uOOO85/8IMf+H333efZ2dn+1VdfVWxfunSpN2vWzA8ePFhlP3379vWHH37Y3d1Hjx7tQ4YMqbL9xhtv9NCpC3nuuee8WbNmFcv/8A//4CNHjoy47qKiIpfk27dvd3f3ZcuWuSQvLi6u9fvbv3+/N2zY0GfPnl2xvaSkxLt37+5Tpkypsp8lS5ZUjHn//ferHGv48OF+/fXXR1xrXaL6ewMAKWb8wvGeNS3Lxy+M/b93mUhSgdeSlaI1AX2ypMFmtlbSYEk7JZXWENyedvc8d8/Lzc2N0qHDiPN91yVLlqh58+Zq0qSJBg4cqEGDBunf/u3fJEkdO3ZUu3btKsauXr1aBw8eVG5urpo3b17x8cknn+jzzz+XJBUVFWngwIFVjlF9ubq1a9fqoosuqnX7mjVrNGLECHXp0kUtWrRQXl6eJOmLL76I+Pv8/PPPdeTIEZ177rkV67KysjRw4EBt2LChytg+ffpUfN2+fXtJ0p49eyRJt956q+bOnau+fftq8uTJeueddyKuAQDwd/lD81UytSS52x6k2MTySEXSZ2qnpE6VljuWr6vg7l8qdEVKZtZc0lXu/r9RqjGYceNCQSpO910HDRqkp59+Wg0bNlT79u3VsGHDim3NmjWrMrasrEzt2rXTe++99739tGzZMib1HThwQJdccomGDBmi559/Xm3bttXevXt13nnn6fDhw1E5RuW2DZKq/Ay+2/bdLcfLLrtMf/7zn/Xmm29q6dKlGjp0qK6++mo999xzUakFAJBEKl/gSJVbeBGI5MrUKkk9zaybmTWSNErSgsoDzKyNmX23r59JmhndMgOI833Xpk2b6sQTT1SXLl2qhIia9OvXT7t371aDBg104oknVvn47um8U089VStWrKjyuurL1Z155plaunRpjds2btyovXv36qGHHtKgQYN0yimnVFwl+k6jRo0kSaWl37u4WKFHjx5q1KiRPvjgg4p1paWl+vDDD9WrV/XnE+rWpk0bXXfddZo1a5aeffZZzZ49W4cOHarXPgAAKSDFJpZHKmyYcvcSSRMlvSWpSNIr7l5oZtPN7IryYedL2mRmn0pqJ+nnMao3rQwZMkTnnnuuRowYoTfffFNbt27Vhx9+qPvuu6/iatVPfvIT/elPf9K//uu/6rPPPtMzzzyj119/vc79TpkyRa+++qruuecebdiwQYWFhXrsscd08OBBde7cWY0bN9YTTzyhLVu2aNGiRbr33nurvL5Lly4yMy1atEjFxcXav3//947RrFkz3Xrrrbrzzju1ePFiFRUV6dZbb9Xu3bs1fvz4iH8GU6dO1RtvvKHPPvtMRUVFmj9/vrp3767GjRtHvA8AQIpItYnlEYpozpS7L3b3k9y9h7v/vHzdVHdfUP71PHfvWT7mx+7OZYUImJkWL16sCy+8UDfddJNOPvlk/eAHP9CmTZsq5hadc845evbZZ/XUU0+pT58+mj9/vu6///4693v55Zfr9ddf15tvvqkzzzxTgwcP1rJly9SgQQPl5uZq9uzZeuONN9SrVy9NmzZNjz76aJXXd+jQQdOmTdOUKVPUrl07TZw4scbjPPzwwxo5cqSuv/56nXHGGVq/fr2WLFmiE044IeKfQePGjTVlyhT17dtX5557rr755hv94Q9/iPj1AJDO0qrdQRozj1LjyPrKy8vz6v2UvlNUVKRTTz01zhUh1fF7AyDdZE/PVqmXKsuyVDK1JNHlZDQzW+3ueTVt4+1kAABIUuP6j1OWZWlc/ySfY5SmT+lFiitTSBv83gBAgmRnh57Sy8oKzYlKQ1yZAgAAsZOmT+lFKpI+UwAAALXLz0+7J/TqI2mvTH3X1BGIBL8vAIBEScow1axZM+3cuVOHDx9WouZ0ITW4uw4fPqydO3d+r8M8ACQrWh6kl6ScgF5WVqa9e/dq3759KknTiWyInuzsbLVq1Upt2rRRgwZJ+f8DAKgiJVoeTJjw97djy+BbeN+pawJ6Us6ZatCggdq2bVvxlioAAKSTcf3HacbqGcnd8iBN30cvFvhvPAAAcZY/NF8lU0uUPzSJQ0qGP6FXH0l5mw8AACCZ0GcKAAAgRghTAAAAARCmAACIkpRoeZDh76MXC8yZAgAgSlKi5UEGvI9eLDBnCgCAOBjXf5yyLCu5Wx7wlF7UcWUKAAAgDK5MAQAAxAhhCgAAIADCFAAA6YCn9BKGMAUAQB1SJqNUfi89xBVhCgCAOqRMRuEpvYQhTAEAUIeUySj5+aG+UflJ/ObJaYrWCAAAAGHQGgEAACBGCFMAACSrlJn9ntkIUwAAJKuUmf2e2QhTAICMlBIXfVJm9ntmYwI6ACAjZWeHLvpkZYUeggPqwgR0AACq4aIPooUwBQDISAlty5QS9xgRKcIUAADxxsTytEKYAgAg3rjHmFaYgA4AABAGE9ABABmBqUhIBMIUACBtMBUJiUCYAgCkjYRPReLSWEZizhQAANFCJ9C0xZwpAADiIeGXxpAIXJkCAAAIgytTAAAAMRJRmDKzS81sk5ltNrO7atje2cyWmdlaM1tvZpdHv1QAQKZiXjeSWdjbfGaWJelTSRdL2iFplaTR7r6h0pinJa1196fMrJekxe7eta79cpsPABAp5nUj0YLe5hsgabO7b3H3w5JeljSi2hiX1LL861aSvjzaYgEAqI553Uhm2RGM6SBpe6XlHZLOrjbmfklvm9n/ldRM0pCadmRmN0u6WZI6d+5c31oBABkqPz/0ASSjaE1AHy1plrt3lHS5pOfN7Hv7dven3T3P3fNyc3OjdGgAAIDEiSRM7ZTUqdJyx/J1ld0o6RVJcvcPJTWR1CYaBQIAACSzSMLUKkk9zaybmTWSNErSgmpjvpB0kSSZ2akKhaniaBYKAACQjMKGKXcvkTRR0luSiiS94u6FZjbdzK4oH3a7pJvMbJ2klySN9UR1AwUApAxaHiAd0AEdAJAwtDxAqqADOgAgKdHyAOmAK1MAAABhcGUKAAAgRghTAAAAARCmAAAAAiBMAQCiinYHyDSEKQBAVM2YEWp3MGNGoisB4oMwBQCIKtodINPQGgEAACAMWiMAAADECGEKAAAgAMIUAABAAIQpAACAAAhTAICI0D8KqBlhCgAQEfpHATUjTAEAIkL/KKBm9JkCAAAIgz5TAAAAMUKYAgAACIAwBQAAEABhCgAyHC0PgGAIUwCQ4Wh5AARDmAKADEfLAyAYWiMAAACEQWsEAACAGCFMAQAABECYAgAACIAwBQBpiHYHQPwQpgAgDdHuAIgfwhQApCHaHQDxQ2sEAACAMGiNAAAAECOEKQAAgAAIUwAAAAEQpgAghdDyAEg+hCkASCG0PACSD2EKAFIILQ+A5ENrBAAAgDBojQAAABAjhCkAAIAACFMAAAABEKYAIAnQ8gBIXRGFKTO71Mw2mdlmM7urhu2PmdnH5R+fmtn/Rr1SAEhjtDwAUlfYMGVmWZLyJV0mqZek0WbWq/IYd/9/7n6Gu58h6d8kzY9BrQCQtmh5AKSuSK5MDZC02d23uPthSS9LGlHH+NGSXopGcQCQKfLzpZKS0GcAqSWSMNVB0vZKyzvK132PmXWR1E3Sf9ay/WYzKzCzguLi4vrWCgAAkHSiPQF9lKR57l5a00Z3f9rd89w9Lzc3N8qHBgAAiL9IwtROSZ0qLXcsX1eTUeIWHwAAyCCRhKlVknqaWTcza6RQYFpQfZCZnSKptaQPo1siAKQm2h0AmSFsmHL3EkkTJb0lqUjSK+5eaGbTzeyKSkNHSXrZE/VmfwCQZGh3AGSG7EgGuftiSYurrZtabfn+6JUFAKlv3LhQkKLdAZDeLFEXkvLy8rygoCAhxwYAAKgPM1vt7nk1bePtZAAAAAIgTAEAAARAmAIAAAiAMAUA9UTLAwCVEaYAoJ5oeQCgMsIUANTTuHFSVhYtDwCE0BoBAAAgDFojAAAAxAhhCgAAIADCFAAAQACEKQAoR8sDAEeDMAUA5Wh5AOBoEKYAoBwtDwAcDVojAAAAhEFrBAAAgBghTAEAAARAmAIAAAiAMAUgrdHuAECsEaYApDXaHQCINcIUgLRGuwMAsUZrBAAAgDBojQAAABAjhCkAAIAACFMAAAABEKYApCRaHgBIFoQpACmJlgcAkgVhCkBKouUBgGRBawQAAIAwaI0AAAAQI4QpAACAAAhTAAAAARCmACQVWh4ASDWEKQBJhZYHAFINYQpAUqHlAYBUQ2sEAACAMGiNAAAAECOEKQAAgAAIUwAAAAEQpgDEHO0OAKQzwhSAmKPdAYB0FlGYMrNLzWyTmW02s7tqGfMDM9tgZoVm9mJ0ywSQymh3ACCdhW2NYGZZkj6VdLGkHZJWSRrt7hsqjekp6RVJF7r7X8ysrbvvqWu/tEYAAACpImhrhAGSNrv7Fnc/LOllSSOqjblJUr67/0WSwgUpAACAdBFJmOogaXul5R3l6yo7SdJJZvaBma0ws0tr2pGZ3WxmBWZWUFxcfHQVAwAAJJFoTUDPltRT0vmSRkt6xsyOqT7I3Z929zx3z8vNzY3SoQEAABInkjC1U1KnSssdy9dVtkPSAnc/4u5bFZpj1TM6JQJIVrQ8AIDIwtQqST3NrJuZNZI0StKCamPeUOiqlMysjUK3/bZEr0wAyYiWBwAQQZhy9xJJEyW9JalI0ivuXmhm083sivJhb0n62sw2SFom6afu/nWsigaQHGh5AAARtEaIFVojAACAVBG0NQIAAABqQZgCAAAIgDAFAAAQAGEKQBW0OwCA+iFMAaiCdgcAUD+EKQBV0O4AAOqH1ggAAABh0BoBAAAgRghTAAAAARCmAAAAAiBMARmClgcAEBuEKSBD0PIAAGKDMAVkCFoeAEBs0BoBAAAgDFojAAAAxAhhCgAAIADCFAAAQACEKSDF0fIAABKLMAWkOFoeAEBiEaaAFEfLAwBILFojAAAAhEFrBAAAgBghTAEAAARAmAIAAAiAMAUkIdodAEDqIEwBSYh2BwCQOghTQBKi3QEApA5aIwAAAIRBawQAAIAYIUwBAAAEQJgCAAAIgDAFAAAQAGEKiCP6RwFA+iFMAXFE/ygASD+EKSCO6B8FAOmHPlMAAABh0GcKAAAgRghTAAAAARCmAAAAAiBMAVFAywMAyFyEKSAKaHkAAJmLMAVEAS0PACBzRRSmzOxSM9tkZpvN7K4ato81s2Iz+7j848fRLxVIXvn5UklJ6DMAILNkhxtgZlmS8iVdLGmHpFVmtsDdN1QbOtfdJ8agRgAAgKQVyZWpAZI2u/sWdz8s6WVJI2JbFgAAQGqIJEx1kLS90vKO8nXVXWVm681snpl1qmlHZnazmRWYWUFxcfFRlAsAAJBcojUB/Q+Surp7H0l/lDS7pkHu/rS757l7Xm5ubpQODcQG7Q4AAJGIJEztlFT5SlPH8nUV3P1rdz9Uvvg7Sf2jUx6QOLQ7AABEIpIwtUpSTzPrZmaNJI2StKDyADM7odLiFZKKolcikBi0OwAARCLs03zuXmJmEyW9JSlL0kx3LzSz6ZIK3H2BpJ+Y2RWSSiT9j6SxMawZiIv8fFodAADCM3dPyIHz8vK8oKAgIccGAACoDzNb7e55NW2jAzoAAEAAhCkAAIAACFPIOLQ8AABEE2EKGYeWBwCAaCJMIePQ8gAAEE08zQcAABAGT/MBAADECGEKAAAgAMIUAABAAIQppA1aHgAAEoEwhbRBywMAQCIQppA2aHkAAEgEWiMAAACEQWsEAACAGCFMAQAABECYAgAACIAwhaRGuwMAQLIjTCGp0e4AAJDsCFNIarQ7AAAkO1ojAAAAhEFrBAAAgBghTAEAAARAmAIAAAiAMIWEoOUBACBdEKaQELQ8AACkC8IUEoKWBwCAdEFrBAAAgDBojQAAABAjhCkAAIAACFMAAAABEKYQVbQ8AABkGsIUooqWBwCATEOYQlTR8gAAkGlojQAAABAGrREAAABihDAFAAAQAGEKAAAgAMIUwqLdAQAAtSNMISzaHQAAUDvCFMKi3QEAALWjNQIAAEAYgVsjmNmlZrbJzDab2V11jLvKzNzMajwYAABAugkbpswsS1K+pMsk9ZI02sx61TCuhaTbJH0U7SIBAACSVSRXpgZI2uzuW9z9sKSXJY2oYdwDkh6W9G0U6wMAAEhqkYSpDpK2V1reUb6ugpn1k9TJ3RfVtSMzu9nMCsysoLi4uN7FIrpoeQAAQHCBn+YzswaSHpV0e7ix7v60u+e5e15ubm7QQyMgWh4AABBcJGFqp6ROlZY7lq/7TgtJvSUtN7Ntks6RtIBJ6MmPlgcAAAQXtjWCmWVL+lTSRQqFqFWSrnH3wlrGL5c02d3r7HtAawQAAJAqArVGcPcSSRMlvSWpSNIr7l5oZtPN7IrolgoAAJBasiMZ5O6LJS2utm5qLWPPD14WAABAauDtZAAAAAIgTKUhWh4AABA/hKk0RMsDAADihzCVhmh5AABA/IRtjRArtEYAAACpIlBrBAAAANSOMAUAABAAYQoAACAAwlSKoN0BAADJiTCVImh3AABAciJMpQjaHQAAkJxojQAAABAGrREAAABihDAFAAAQAGEKAAAgAMJUgtHyAACA1EaYSjBaHgAAkNoIUwlGywMAAFIbrREAAADCoDUCAABAjBCmAAAAAiBMAQAABECYigHaHQAAkDkIUzFAuwMAADIHYSoGaHcAAEDmoDUCAABAGLRGAAAAiBHCFAAAQACEKQAAgAAIU/VAywMAAFAdYaoeaHkAAACqI0zVAy0PAABAdbRGAAAACIPWCAAAADFCmAIAAAiAMAUAABAAYUq0PAAAAEePMCVaHgAAgKNHmBItDwAAwNGjNQIAAEAYtEYAAACIkYjClJldamabzGyzmd1Vw/ZbzOy/zexjM3vfzHpFv1QAAIDkEzZMmVmWpHxJl0nqJWl0DWHpRXc/3d3PkPRLSY9Gu1AAAIBkFMmVqQGSNrv7Fnc/LOllSSMqD3D3v1ZabCYpMROxAAAA4iySMNVB0vZKyzvK11VhZhPM7HOFrkz9JDrlHT16RwEAgHiI2gR0d8939x6S7pR0T01jzOxmMysws4Li4uJoHbpG9I4CAADxEEmY2impU6XljuXravOypH+uaYO7P+3uee6el5ubG3GRR4PeUQAAIB4iCVOrJPU0s25m1kjSKEkLKg8ws56VFodK+ix6JR6d/HyppCT0GQAAIFayww1w9xIzmyjpLUlZkma6e6GZTZdU4O4LJE00syGSjkj6i6QxsSwaAAAgWYQNU5Lk7oslLa62bmqlr2+Lcl0AAAApgQ7oAAAAARCmAAAAAiBMAQAABECYAgAACIAwBQAAEABhCgAAIADCFAAAQACEKQAAgAAIUwAAAAEQpgAAAAIgTAEAAARAmAIAAAjA3D0xBzYrlvTnGB+mjaS9MT4Gjh7nJ3lxbpIb5ye5cX6SV5Bz08Xdc2vakLAwFQ9mVuDueYmuAzXj/CQvzk1y4/wkN85P8orVueE2HwAAQACEKQAAgADSPUw9negCUCfOT/Li3CQ3zk9y4/wkr5icm7SeMwUAABBr6X5lCgAAIKYIUwAAAAGkRZgys0vNbJOZbTazu2rY3tjM5pZv/8jMuiagzIwVwfn5FzPbYGbrzWypmXVJRJ2ZKNy5qTTuKjNzM+Nx7ziK5PyY2Q/K//wUmtmL8a4xU0Xw91pnM1tmZmvL/267PBF1ZiIzm2lme8zsk1q2m5n9tvzcrTezfkGPmfJhysyyJOVLukxSL0mjzaxXtWE3SvqLu58o6TFJD8e3yswV4flZKynP3ftImifpl/GtMjNFeG5kZi0k3Sbpo/hWmNkiOT9m1lPSzySd6+6nSZoU7zozUYR/du6R9Iq7nylplKQn41tlRpsl6dI6tl8mqWf5x82Sngp6wJQPU5IGSNrs7lvc/bCklyWNqDZmhKTZ5V/Pk3SRmVkca8xkYc+Puy9z94PliyskdYxzjZkqkj87kvSAQv8B+TaexSGi83OTpHx3/4skufueONeYqSI5Ny6pZfnXrSR9Gcf6Mpq7vyvpf+oYMkLS7z1khaRjzOyEIMdMhzDVQdL2Sss7ytfVOMbdSyTtk3RcXKpDJOenshslvRnTivCdsOem/PJ3J3dfFM/CICmyPzsnSTrJzD4wsxVmVtf/xhE9kZyb+yX90Mx2SFos6f/GpzREoL7/LoWVHagcIIrM7IeS8iQNTnQtkMysgaRHJY1NcCmoXbZCtyrOV+iK7rtmdrq7/28ii4IkabSkWe7+azMbKOl5M+vt7mWJLgzRlw5XpnZK6lRpuWP5uhrHmFm2Qpdcv45LdYjk/MjMhkiaIukKdz8Up9oyXbhz00JSb0nLzWybpHMkLWASetxE8mdnh6QF7n7E3bdK+lShcIXYiuTc3CjpFUly9w8lNVHoTXaReBH9u1Qf6RCmVknqaWbdzKyRQhP9FlQbs0DSmPKv/4+k/3S6lcZL2PNjZmdKmqFQkGLOR/zUeW7cfZ+7t3H3ru7eVaH5bFe4e0Fiys04kfzd9oZCV6VkZm0Uuu23JY41ZqpIzs0Xki6SJDM7VaEwVRzXKlGbBZJ+VP5U3zmS9rn7riA7TPnbfO5eYmYTJb0lKUvSTHcvNLPpkgrcfYGkZxW6xLpZoUlpoxJXcWaJ8Pz8SlJzSa+WPxfwhbtfkbCiM0SE5wYJEuH5eUvSP5nZBkmlkn7q7lx1j7EIz83tkp4xs/+n0GT0sfwnPj7M7CWF/pPRpnzO2n2SGkqSu/+7QnPYLpe0WdJBSdcHPibnFgAA4Oilw20+AACAhCFMAQAABECYAgAACIAwBQAAEABhCgAAIADCFAAAQACEKQAAgAD+fy9PTOtPUDc5AAAAAElFTkSuQmCC\n" + }, + "metadata": { + "needs_background": "light" + } + } + ], + "source": [ + "plot_predictions(predictions=y_preds)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fEHGrjLgji6E" + }, + "source": [ + "Woohoo! Those red dots are looking far closer than they were before!\n", + "\n", + "Let's get onto saving an reloading a model in PyTorch." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8NRng9aEec05" + }, + "source": [ + "## 5. Saving and loading a PyTorch model\n", + "\n", + "If you've trained a PyTorch model, chances are you'll want to save it and export it somewhere.\n", + "\n", + "As in, you might train it on Google Colab or your local machine with a GPU but you'd like to now export it to some sort of application where others can use it. \n", + "\n", + "Or maybe you'd like to save your progress on a model and come back and load it back later.\n", + "\n", + "For saving and loading models in PyTorch, there are three main methods you should be aware of (all of below have been taken from the [PyTorch saving and loading models guide](https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-model-for-inference)):\n", + "\n", + "| PyTorch method | What does it do? | \n", + "| ----- | ----- |\n", + "| [`torch.save`](https://pytorch.org/docs/stable/torch.html?highlight=save#torch.save) | Saves a serialzed object to disk using Python's [`pickle`](https://docs.python.org/3/library/pickle.html) utility. Models, tensors and various other Python objects like dictionaries can be saved using `torch.save`. | \n", + "| [`torch.load`](https://pytorch.org/docs/stable/torch.html?highlight=torch%20load#torch.load) | Uses `pickle`'s unpickling features to deserialize and load pickled Python object files (like models, tensors or dictionaries) into memory. You can also set which device to load the object to (CPU, GPU etc). |\n", + "| [`torch.nn.Module.load_state_dict`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=load_state_dict#torch.nn.Module.load_state_dict)| Loads a model's parameter dictionary (`model.state_dict()`) using a saved `state_dict()` object. | \n", + "\n", + "> **Note:** As stated in [Python's `pickle` documentation](https://docs.python.org/3/library/pickle.html), the `pickle` module **is not secure**. That means you should only ever unpickle (load) data you trust. That goes for loading PyTorch models as well. Only ever use saved PyTorch models from sources you trust.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SdAGcH2aec05" + }, + "source": [ + "### Saving a PyTorch model's `state_dict()`\n", + "\n", + "The [recommended way](https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-model-for-inference) for saving and loading a model for inference (making predictions) is by saving and loading a model's `state_dict()`.\n", + "\n", + "Let's see how we can do that in a few steps:\n", + "\n", + "1. We'll create a directory for saving models to called `models` using Python's `pathlib` module.\n", + "2. We'll create a file path to save the model to.\n", + "3. We'll call `torch.save(obj, f)` where `obj` is the target model's `state_dict()` and `f` is the filename of where to save the model.\n", + "\n", + "> **Note:** It's common convention for PyTorch saved models or objects to end with `.pt` or `.pth`, like `saved_model_01.pth`.\n" + ] + }, + { + "cell_type": "code", + "source": [ + "pip install pathlib" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "r_vfgjVqCMVY", + "outputId": "7de4b315-cb90-4e78-85e0-ca5358cf8671" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n", + "Requirement already satisfied: pathlib in /usr/local/lib/python3.9/dist-packages (1.0.1)\n" + ] + } + ] + }, + { + "cell_type": "code", + "source": [ + "from pathlib import Path\n", + "\n", + "# 1. Create models directory\n", + "MODEL_PATH = Path('models')\n", + "MODEL_PATH.mkdir(parents = True, exist_ok = True)\n", + "\n", + "# 2. Create model save path\n", + "MODEL_NAME = '01_pytorch_workflow_model_0.pth'\n", + "MODEL_SAVE_PATH = MODEL_PATH / MODEL_NAME\n", + "\n", + "# 3. Save the model state dict\n", + "print(f\"Saving model to: {MODEL_SAVE_PATH}\")\n", + "torch.save(obj = model_0.state_dict(), # only saving the state_dict() only saves the models learned parameters\n", + " f=MODEL_SAVE_PATH)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "uVA1USqp4Tvm", + "outputId": "fc4fd94b-3a83-43c4-93ad-63814fd86831" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Saving model to: models/01_pytorch_workflow_model_0.pth\n" + ] + } + ] + }, + { + "cell_type": "code", + "source": [ + "# Check the saved file path\n", + "!ls -l models/01_pytorch_workflow_model_0.pth" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "d9i9Q6rlCt94", + "outputId": "bc3e9365-8b1f-4e43-e79d-09aaea7429a7" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "-rw-r--r-- 1 root root 1207 Mar 25 10:35 models/01_pytorch_workflow_model_0.pth\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "jFQpRoH5ec06" + }, + "source": [ + "### Loading a saved PyTorch model's `state_dict()`\n", + "\n", + "Since we've now got a saved model `state_dict()` at `models/01_pytorch_workflow_model_0.pth` we can now load it in using `torch.nn.Module.load_state_dict(torch.load(f))` where `f` is the filepath of our saved model `state_dict()`.\n", + "\n", + "Why call `torch.load()` inside `torch.nn.Module.load_state_dict()`? \n", + "\n", + "Because we only saved the model's `state_dict()` which is a dictionary of learned parameters and not the *entire* model, we first have to load the `state_dict()` with `torch.load()` and then pass that `state_dict()` to a new instance of our model (which is a subclass of `nn.Module`).\n", + "\n", + "Why not save the entire model?\n", + "\n", + "[Saving the entire model](https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-load-entire-model) rather than just the `state_dict()` is more intuitive, however, to quote the PyTorch documentation (italics mine):\n", + "\n", + "> The disadvantage of this approach *(saving the whole model)* is that the serialized data is bound to the specific classes and the exact directory structure used when the model is saved...\n", + ">\n", + "> Because of this, your code can break in various ways when used in other projects or after refactors.\n", + "\n", + "So instead, we're using the flexible method of saving and loading just the `state_dict()`, which again is basically a dictionary of model parameters.\n", + "\n", + "Let's test it out by created another instance of `LinearRegressionModel()`, which is a subclass of `torch.nn.Module` and will hence have the in-built method `load_state_dit()`." + ] + }, + { + "cell_type": "code", + "source": [ + "# Instantiate a new instance of our model (this will be instantiated with random weights)\n", + "loaded_model_0 = LinearRegressionModel()\n", + "\n", + "# Load the state_dict of our saved model (this will update the new instance of our model with trained weights)\n", + "loaded_model_0.load_state_dict(torch.load(f=MODEL_SAVE_PATH))" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "jZ79T9KWFy9C", + "outputId": "7a23a13d-3f2a-49e9-c817-c759d867b4e0" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ] + }, + "metadata": {}, + "execution_count": 22 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vK8PRtY7Qgpz" + }, + "source": [ + "Excellent! It looks like things matched up.\n", + "\n", + "Now to test our loaded model, let's perform inference with it (make predictions) on the test data.\n", + "\n", + "Remember the rules for performing inference with PyTorch models?\n", + "\n", + "If not, here's a refresher:\n", + "\n", + "
\n", + " PyTorch inference rules\n", + "
    \n", + "
  1. Set the model in evaluation mode (model.eval()).
    2. \n", + "
  2. Make the predictions using the inference mode context manager (with torch.inference_mode(): ...).
    3. \n", + "
  3. All predictions should be made with objects on the same device (e.g. data and model on GPU only or data and model on CPU only).
    4. \n", + "
\n", + "
\n", + "\n" + ] + }, + { + "cell_type": "code", + "source": [ + "# 1. Put the loaded model into evalution mode\n", + "loaded_model_0.eval()\n", + "\n", + "# 2. Use the inference mode context manger to make predictions\n", + "with torch.inference_mode():\n", + " loaded_model_preds = loaded_model_0(X_test) # perform a forward pass on the test data with the loaded model\n" + ], + "metadata": { + "id": "Stw1IdaCGQLw" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "e81XpN8WSSqn" + }, + "source": [ + "Now we've made some predictions with the loaded model, let's see if they're the same as the previous predictions." + ] + }, + { + "cell_type": "code", + "source": [ + "# Compare previous model predications with loaded model predictions (these should be the same)\n", + "y_preds == loaded_model_preds" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "bEXGm8IBG3Es", + "outputId": "7d3aff13-d4df-4afe-c6cd-d4cdb03bc9bb" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[True],\n", + " [True],\n", + " [True],\n", + " [True],\n", + " [True],\n", + " [True],\n", + " [True],\n", + " [True],\n", + " [True],\n", + " [True]])" + ] + }, + "metadata": {}, + "execution_count": 24 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9Y4ZcxxfNcVu" + }, + "source": [ + "Nice! \n", + "\n", + "It looks like the loaded model predictions are the same as the previous model predictions (predictions made prior to saving). This indicates our model is saving and loading as expected.\n", + "\n", + "> **Note:** There are more methods to save and load PyTorch models but I'll leave these for extra-curriculum and further reading. See the [PyTorch guide for saving and loading models](https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-and-loading-models) for more. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "FeAITvLXec06" + }, + "source": [ + "## 6. Putting it all together \n", + "\n", + "We've covered a fair bit of ground so far. \n", + "\n", + "But once you've had some practice, you'll be performing the above steps like dancing down the street.\n", + "\n", + "Speaking of practice, let's put everything we've done so far together. \n", + "\n", + "Except this time we'll make our code device agnostic (so if there's a GPU available, it'll use it and if not, it will default to the CPU). \n", + "\n", + "There'll be far less commentary in this section than above since what we're going to go through has already been covered.\n", + "\n", + "We'll start by importing the standard libraries we need.\n", + "\n", + "> **Note:** If you're using Google Colab, to setup a GPU, go to Runtime -> Change runtime type -> Hardware acceleration -> GPU. If you do this, it will reset the Colab runtime and you will lose saved variables." + ] + }, + { + "cell_type": "code", + "source": [ + "# Import Pytorch and matplotlib\n", + "import torch\n", + "from torch import nn \n", + "import matplotlib.pyplot as plt\n", + "\n", + "# Check PyTorch Version\n", + "torch.__version__" + ], + "metadata": { + "id": "Val2gvSIHPgo", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 35 + }, + "outputId": "c46438a0-60f1-49fb-cc70-5a0ec1736f62" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'1.13.1+cu116'" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + } + }, + "metadata": {}, + "execution_count": 25 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bT-krbNMIw0d" + }, + "source": [ + "Now let's start making our code device agnostic by setting `device=\"cuda\"` if it's available, otherwise it'll default to `device=\"cpu\"`.\n", + "\n" + ] + }, + { + "cell_type": "code", + "source": [ + "# Setup device agnostic code\n", + "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n", + "print(f\"Using device: {device}\")" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "m9C5SRsWV_zV", + "outputId": "923bfedd-63a4-47d5-8488-1bca545605c0" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Using device: cuda\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "G1t0Ek0GJq6T" + }, + "source": [ + "If you've got access to a GPU, the above should've printed out:\n", + "\n", + "```\n", + "Using device: cuda\n", + "```\n", + "Otherwise, you'll be using a CPU for the following computations. This is fine for our small dataset but it will take longer for larger datasets." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DmilLp3Vec07" + }, + "source": [ + "### 6.1 Data\n", + "\n", + "Let's create some data just like before.\n", + "\n", + "First, we'll hard-code some `weight` and `bias` values.\n", + "\n", + "Then we'll make a range of numbers between 0 and 1, these will be our `X` values.\n", + "\n", + "Finally, we'll use the `X` values, as well as the `weight` and `bias` values to create `y` using the linear regression formula (`y = weight * X + bias`)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "fJqgDWUfec07", + "outputId": "1bf1a19d-971e-4ede-c1f1-3b61dda2bfc7" + }, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "(tensor([[0.0000],\n", + " [0.0200],\n", + " [0.0400],\n", + " [0.0600],\n", + " [0.0800],\n", + " [0.1000],\n", + " [0.1200],\n", + " [0.1400],\n", + " [0.1600],\n", + " [0.1800]]), tensor([[0.3000],\n", + " [0.3140],\n", + " [0.3280],\n", + " [0.3420],\n", + " [0.3560],\n", + " [0.3700],\n", + " [0.3840],\n", + " [0.3980],\n", + " [0.4120],\n", + " [0.4260]]))" + ] + }, + "metadata": {}, + "execution_count": 27 + } + ], + "source": [ + "# Create weight and bias\n", + "weight = 0.7\n", + "bias = 0.3\n", + "\n", + "# Create range values\n", + "start = 0\n", + "end = 1\n", + "step = 0.02\n", + "\n", + "# Create X and y (features and labels)\n", + "X = torch.arange(start, end, step).unsqueeze(dim=1) # without unsqueeze, errors will happen later on (shapes within linear layers)\n", + "y = weight * X + bias \n", + "X[:10], y[:10]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Oaar6rDGLGaQ" + }, + "source": [ + "Wonderful!\n", + "\n", + "Now we've got some data, let's split it into training and test sets.\n", + "\n", + "We'll use an 80/20 split with 80% training data and 20% testing data." + ] + }, + { + "cell_type": "code", + "source": [ + "# Split data \n", + "train_split = int(0.8* len(X))\n", + "X_train, y_train = X[:train_split], y[:train_split]\n", + "X_test, y_test = X[train_split:], y[train_split:]\n", + "\n", + "len(X_train), len(y_train), len(X_test), len(y_test)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "vJairLsEWesu", + "outputId": "9f1b0f56-a830-4196-8c9f-64a1df39ec3b" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "(40, 40, 10, 10)" + ] + }, + "metadata": {}, + "execution_count": 28 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "INW8-McyLeFE" + }, + "source": [ + "Excellent, let's visualize them to make sure they look okay." + ] + }, + { + "cell_type": "code", + "source": [ + "plot_predictions(X_train, y_train,X_test,y_test)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 428 + }, + "id": "noueFAQ8bCFg", + "outputId": "84ff2926-d0d0-43d7-8f32-25a871932901" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAGbCAYAAADgEhWsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAAsTAAALEwEAmpwYAAAXjElEQVR4nO3df4hl93ke8Of1btRA4sQlmhajlS2Rrmm2jnHsQXXpHza1XSQbpLQOQVtC4uBYKkiJmzqhCg2OUQglaetAytSR0grHAVtRTQhbrKKW2MY02EYj7KiRhNStklgrBzzxr/5hYnnVt3/MtRmPZvfe2XN/388Hhplzzpd7XjjM7jPnnnmmujsAAFyZFy16AACAVSZMAQAMIEwBAAwgTAEADCBMAQAMcHJRJ7766qv7uuuuW9TpAQAm9sgjj/xVd28ddWxhYeq6667L7u7uok4PADCxqvqLSx3zNh8AwADCFADAAMIUAMAAwhQAwADCFADAAGPDVFXdV1VfrKo/vcTxqqrfqqrzVfVoVb1m+mMCACynSe5MfSDJjZc5flOS06OP25K8f/hYAACrYWyY6u5PJvnyZZbckuSDve/TSV5SVS+d1oAAAMtsGs9MXZPkmQPbF0b7AADW3lwfQK+q26pqt6p29/b25nlqAICZmEaYejbJtQe2T432vUB339vd2929vbV15J+3AQBYKdMIU+eS/OTot/pel+Rr3f2XU3hdAIClN/YPHVfVh5O8IcnVVXUhya8k+a4k6e7fTvJgkrckOZ/k60l+elbDAgAsm7FhqrvPjjneSe6Y2kQAACtEAzoAwADCFACwsu746B05effJ3PHRxb1JJkwBACvrnkfuyfP9fO555J6FzSBMAQAr6/bX3p4TdSK3v/b2hc1Q+8+Pz9/29nbv7u4u5NwAAMdRVY909/ZRx9yZAgAYQJgCABhAmAIAGECYAgCWyjLUHRyHMAUALJVlqDs4DmEKAFgqy1B3cByqEQAAxlCNAAAwI8IUAMAAwhQAwADCFAAwF6tWeTApYQoAmItVqzyYlDAFAMzFqlUeTEo1AgDAGKoRAABmRJgCABhAmAIAGECYAgAGWdfKg0kJUwDAIOtaeTApYQoAGGRdKw8mpRoBAGAM1QgAADMiTAEADCBMAQAMIEwBAC+w6XUHxyFMAQAvsOl1B8chTAEAL7DpdQfHoRoBAGAM1QgAADMiTAEADCBMAQAMIEwBwAZReTB9whQAbBCVB9MnTAHABlF5MH2qEQAAxlCNAAAwI8IUAMAAE4Wpqrqxqp6sqvNVddcRx19eVX9UVY9W1Seq6tT0RwUAWD5jw1RVnUiyk+SmJGeSnK2qM4eW/bskH+zuVyW5O8m/mfagAMClqTxYnEnuTN2Q5Hx3P93dzyW5P8kth9acSfKx0dcfP+I4ADBDKg8WZ5IwdU2SZw5sXxjtO+hPkvzT0df/JMmLq+oHDr9QVd1WVbtVtbu3t3cl8wIAR1B5sDjTegD9F5K8vqo+m+T1SZ5N8vzhRd19b3dvd/f21tbWlE4NAOy8dScX33MxO2/dWfQoG+fkBGueTXLtge1To33f1t1fyOjOVFV9b5K3dfdXpzQjAMDSmuTO1MNJTlfV9VV1VZJbk5w7uKCqrq6qb73WLyW5b7pjAgAsp7FhqrsvJrkzyUNJnkjyQHc/VlV3V9XNo2VvSPJkVT2V5G8n+bUZzQsAsFQmemaqux/s7ld09w9296+N9r2nu8+Nvv5Id58erfmZ7v7GLIcGgE2g7mA1aEAHgCWl7mA1CFMAsKTUHayG6u6FnHh7e7t3d3cXcm4AgOOoqke6e/uoY+5MAQAMIEwBAAwgTAEADCBMAcCcqTxYL8IUAMyZyoP1IkwBwJypPFgvqhEAAMZQjQAAMCPCFADAAMIUAMAAwhQATInKg80kTAHAlKg82EzCFABMicqDzaQaAQBgDNUIAAAzIkwBAAwgTAEADCBMAcBl3HFHcvLk/mc4ijAFAJdxzz3J88/vf4ajCFMAcBm3356cOLH/GY6iGgEAYAzVCAAAMyJMAQAMIEwBAAwgTAGwkVQeMC3CFAAbSeUB0yJMAbCRVB4wLaoRAADGUI0AADAjwhQAwADCFADAAMIUAGtD3QGLIEwBsDbUHbAIwhQAa0PdAYugGgEAYAzVCAAAMyJMAQAMIEwBAAwwUZiqqhur6smqOl9Vdx1x/GVV9fGq+mxVPVpVb5n+qABsKpUHLLOxD6BX1YkkTyV5c5ILSR5Ocra7Hz+w5t4kn+3u91fVmSQPdvd1l3tdD6ADMKmTJ/crD06cSC5eXPQ0bKKhD6DfkOR8dz/d3c8luT/JLYfWdJLvG339/Um+cKXDAsBhKg9YZicnWHNNkmcObF9I8vcPrXlvkv9eVT+b5HuSvOmoF6qq25LcliQve9nLjjsrABtqZ2f/A5bRtB5AP5vkA919KslbkvxeVb3gtbv73u7e7u7tra2tKZ0aAGBxJglTzya59sD2qdG+g96R5IEk6e5PJfnuJFdPY0AAgGU2SZh6OMnpqrq+qq5KcmuSc4fWfD7JG5Okqn4o+2Fqb5qDAgAso7FhqrsvJrkzyUNJnkjyQHc/VlV3V9XNo2XvTvLOqvqTJB9O8vZe1N+pAWBlqDxgHfjbfAAsjMoDVoW/zQfAUlJ5wDpwZwoAYAx3pgAAZkSYAgAYQJgCABhAmAJgqtQdsGmEKQCm6p579usO7rln0ZPAfAhTAEyVugM2jWoEAIAxVCMAAMyIMAUAMIAwBQAwgDAFADCAMAXARPRHwdGEKQAmoj8KjiZMATAR/VFwND1TAABj6JkCAJgRYQoAYABhCgBgAGEKYMOpPIBhhCmADafyAIYRpgA2nMoDGEY1AgDAGKoRAABmRJgCABhAmAIAGECYAlhD6g5gfoQpgDWk7gDmR5gCWEPqDmB+VCMAAIyhGgEAYEaEKQCAAYQpAIABhCmAFaLyAJaPMAWwQlQewPIRpgBWiMoDWD6qEQAAxlCNAAAwI8IUAMAAwhQAwADCFMASUHkAq2uiMFVVN1bVk1V1vqruOuL4b1bV50YfT1XVV6c+KcAaU3kAq2tsmKqqE0l2ktyU5EySs1V15uCa7v757n51d786yX9I8gczmBVgbak8gNU1yZ2pG5Kc7+6nu/u5JPcnueUy688m+fA0hgPYFDs7ycWL+5+B1TJJmLomyTMHti+M9r1AVb08yfVJPnaJ47dV1W5V7e7t7R13VgCApTPtB9BvTfKR7n7+qIPdfW93b3f39tbW1pRPDQAwf5OEqWeTXHtg+9Ro31Fujbf4AIANMkmYejjJ6aq6vqquyn5gOnd4UVX93SR/M8mnpjsiwGpSdwCbYWyY6u6LSe5M8lCSJ5I80N2PVdXdVXXzgaW3Jrm/F/XH/gCWjLoD2AwnJ1nU3Q8mefDQvvcc2n7v9MYCWH23374fpNQdwHqrRd1I2t7e7t3d3YWcGwDgOKrqke7ePuqYPycDADCAMAUAMIAwBQAwgDAFcEwqD4CDhCmAY1J5ABwkTAEc0+23JydOqDwA9qlGAAAYQzUCAMCMCFMAAAMIUwAAAwhTACMqD4ArIUwBjKg8AK6EMAUwovIAuBKqEQAAxlCNAAAwI8IUAMAAwhQAwADCFLDW1B0AsyZMAWtN3QEwa8IUsNbUHQCzphoBAGAM1QgAADMiTAEADCBMAQAMIEwBK0nlAbAshClgJak8AJaFMAWsJJUHwLJQjQAAMIZqBACAGRGmAAAGEKYAAAYQpoClovIAWDXCFLBUVB4Aq0aYApaKygNg1ahGAAAYQzUCAMCMCFMAAAMIUwAAAwhTwMypOwDWmTAFzJy6A2CdTRSmqurGqnqyqs5X1V2XWPPjVfV4VT1WVR+a7pjAKlN3AKyzsdUIVXUiyVNJ3pzkQpKHk5zt7scPrDmd5IEk/6i7v1JVf6u7v3i511WNAACsiqHVCDckOd/dT3f3c0nuT3LLoTXvTLLT3V9JknFBCgBgXUwSpq5J8syB7QujfQe9IskrquqPq+rTVXXjUS9UVbdV1W5V7e7t7V3ZxAAAS2RaD6CfTHI6yRuSnE3yO1X1ksOLuvve7t7u7u2tra0pnRoAYHEmCVPPJrn2wPap0b6DLiQ5193f7O4/y/4zVqenMyKwrFQeAEwWph5Ocrqqrq+qq5LcmuTcoTV/mP27Uqmqq7P/tt/T0xsTWEYqDwAmCFPdfTHJnUkeSvJEkge6+7Gquruqbh4teyjJl6rq8SQfT/KL3f2lWQ0NLAeVBwATVCPMimoEAGBVDK1GAADgEoQpAIABhCkAgAGEKeA7qDsAOB5hCvgO6g4AjkeYAr6DugOA41GNAAAwhmoEAIAZEaYAAAYQpgAABhCmYEOoPACYDWEKNoTKA4DZEKZgQ6g8AJgN1QgAAGOoRgAAmBFhCgBgAGEKAGAAYQpWnMoDgMUSpmDFqTwAWCxhClacygOAxVKNAAAwhmoEAIAZEaYAAAYQpgAABhCmYAmpOwBYHcIULCF1BwCrQ5iCJaTuAGB1qEYAABhDNQIAwIwIUwAAAwhTAAADCFMAAAMIUzBH+qMA1o8wBXOkPwpg/QhTMEf6owDWj54pAIAx9EwBAMyIMAUAMIAwBQAwgDAFU6DyAGBzCVMwBSoPADaXMAVToPIAYHNNFKaq6saqerKqzlfVXUccf3tV7VXV50YfPzP9UWF57ewkFy/ufwZgs5wct6CqTiTZSfLmJBeSPFxV57r78UNLf7+775zBjAAAS2uSO1M3JDnf3U9393NJ7k9yy2zHAgBYDZOEqWuSPHNg+8Jo32Fvq6pHq+ojVXXtUS9UVbdV1W5V7e7t7V3BuAAAy2VaD6D/1yTXdferkvyPJL971KLuvre7t7t7e2tra0qnhtlQdwDAJCYJU88mOXin6dRo37d195e6+xujzf+U5LXTGQ8WR90BAJOYJEw9nOR0VV1fVVcluTXJuYMLquqlBzZvTvLE9EaExVB3AMAkxv42X3dfrKo7kzyU5ESS+7r7saq6O8lud59L8nNVdXOSi0m+nOTtM5wZ5mJnR9UBAONVdy/kxNvb2727u7uQcwMAHEdVPdLd20cd04AOADCAMAUAMIAwxcZReQDANAlTbByVBwBMkzDFxlF5AMA0+W0+AIAx/DYfAMCMCFMAAAMIUwAAAwhTrA2VBwAsgjDF2lB5AMAiCFOsDZUHACyCagQAgDFUIwAAzIgwBQAwgDAFADCAMMVSU3cAwLITplhq6g4AWHbCFEtN3QEAy041AgDAGKoRAABmRJgCABhAmAIAGECYYiFUHgCwLoQpFkLlAQDrQphiIVQeALAuVCMAAIyhGgEAYEaEKQCAAYQpAIABhCmmSuUBAJtGmGKqVB4AsGmEKaZK5QEAm0Y1AgDAGKoRAABmRJgCABhAmAIAGECYYix1BwBwacIUY6k7AIBLE6YYS90BAFyaagQAgDEGVyNU1Y1V9WRVna+quy6z7m1V1VV15MkAANbN2DBVVSeS7CS5KcmZJGer6swR616c5F1JPjPtIQEAltUkd6ZuSHK+u5/u7ueS3J/kliPW/WqSX0/y11OcDwBgqU0Spq5J8syB7Qujfd9WVa9Jcm13f/RyL1RVt1XVblXt7u3tHXtYpkvlAQAMN/i3+arqRUnel+Td49Z2973dvd3d21tbW0NPzUAqDwBguEnC1LNJrj2wfWq071tenOSVST5RVX+e5HVJznkIffmpPACA4cZWI1TVySRPJXlj9kPUw0n+WXc/don1n0jyC9192d4D1QgAwKoYVI3Q3ReT3JnkoSRPJHmgux+rqrur6ubpjgoAsFpOTrKoux9M8uChfe+5xNo3DB8LAGA1+HMyAAADCFNrSOUBAMyPMLWGVB4AwPwIU2tI5QEAzM/YaoRZUY0AAKyKQdUIAABcmjAFADCAMAUAMIAwtSLUHQDAchKmVoS6AwBYTsLUilB3AADLSTUCAMAYqhEAAGZEmAIAGECYAgAYQJhaMJUHALDahKkFU3kAAKtNmFowlQcAsNpUIwAAjKEaAQBgRoQpAIABhCkAgAGEqRlQdwAAm0OYmgF1BwCwOYSpGVB3AACbQzUCAMAYqhEAAGZEmAIAGECYAgAYQJg6BpUHAMBhwtQxqDwAAA4Tpo5B5QEAcJhqBACAMVQjAADMiDAFADCAMAUAMIAwFZUHAMCVE6ai8gAAuHLCVFQeAABXTjUCAMAYqhEAAGZkojBVVTdW1ZNVdb6q7jri+D+vqv9VVZ+rqv9ZVWemPyoAwPIZG6aq6kSSnSQ3JTmT5OwRYelD3f3D3f3qJL+R5H3THhQAYBlNcmfqhiTnu/vp7n4uyf1Jbjm4oLv/74HN70mymAexAADmbJIwdU2SZw5sXxjt+w5VdUdV/Z/s35n6uemMd+V0RwEA8zC1B9C7e6e7fzDJv0ryy0etqarbqmq3qnb39vamdeoj6Y4CAOZhkjD1bJJrD2yfGu27lPuT/OhRB7r73u7e7u7tra2tiYe8ErqjAIB5mCRMPZzkdFVdX1VXJbk1ybmDC6rq9IHNtyb539Mb8crs7CQXL+5/BgCYlZPjFnT3xaq6M8lDSU4kua+7H6uqu5Psdve5JHdW1ZuSfDPJV5L81CyHBgBYFmPDVJJ094NJHjy07z0Hvn7XlOcCAFgJGtABAAYQpgAABhCmAAAGEKYAAAYQpgAABhCmAAAGEKYAAAYQpgAABhCmAAAGEKYAAAYQpgAABhCmAAAGqO5ezImr9pL8xYxPc3WSv5rxObhyrs/ycm2Wm+uz3Fyf5TXk2ry8u7eOOrCwMDUPVbXb3duLnoOjuT7Ly7VZbq7PcnN9ltesro23+QAABhCmAAAGWPcwde+iB+CyXJ/l5dosN9dnubk+y2sm12atn5kCAJi1db8zBQAwU8IUAMAAaxGmqurGqnqyqs5X1V1HHP8bVfX7o+OfqarrFjDmxprg+vzLqnq8qh6tqj+qqpcvYs5NNO7aHFj3tqrqqvLr3nM0yfWpqh8fff88VlUfmveMm2qCf9deVlUfr6rPjv5te8si5txEVXVfVX2xqv70Eserqn5rdO0erarXDD3nyoepqjqRZCfJTUnOJDlbVWcOLXtHkq90999J8ptJfn2+U26uCa/PZ5Nsd/erknwkyW/Md8rNNOG1SVW9OMm7knxmvhNutkmuT1WdTvJLSf5hd/+9JP9i3nNuogm/d345yQPd/SNJbk3yH+c75Ub7QJIbL3P8piSnRx+3JXn/0BOufJhKckOS8939dHc/l+T+JLccWnNLkt8dff2RJG+sqprjjJts7PXp7o9399dHm59OcmrOM26qSb53kuRXs/8DyF/Pczgmuj7vTLLT3V9Jku7+4pxn3FSTXJtO8n2jr78/yRfmON9G6+5PJvnyZZbckuSDve/TSV5SVS8dcs51CFPXJHnmwPaF0b4j13T3xSRfS/IDc5mOSa7PQe9I8t9mOhHfMvbajG5/X9vdH53nYCSZ7HvnFUleUVV/XFWfrqrL/TTO9Exybd6b5Ceq6kKSB5P87HxGYwLH/X9prJODxoEpqqqfSLKd5PWLnoWkql6U5H1J3r7gUbi0k9l/q+IN2b+j+8mq+uHu/uoihyJJcjbJB7r731fVP0jye1X1yu7+f4sejOlbhztTzya59sD2qdG+I9dU1cns33L90lymY5Lrk6p6U5J/neTm7v7GnGbbdOOuzYuTvDLJJ6rqz5O8Lsk5D6HPzSTfOxeSnOvub3b3nyV5Kvvhitma5Nq8I8kDSdLdn0ry3dn/I7ss3kT/Lx3HOoSph5Ocrqrrq+qq7D/od+7QmnNJfmr09Y8l+VhrK52Xsdenqn4kyT3ZD1Ke+Zify16b7v5ad1/d3dd193XZf57t5u7eXcy4G2eSf9v+MPt3pVJVV2f/bb+n5zjjpprk2nw+yRuTpKp+KPtham+uU3Ip55L85Oi3+l6X5Gvd/ZdDXnDl3+br7otVdWeSh5KcSHJfdz9WVXcn2e3uc0n+c/ZvsZ7P/kNpty5u4s0y4fX5t0m+N8l/Gf1ewOe7++aFDb0hJrw2LMiE1+ehJP+4qh5P8nySX+xud91nbMJr8+4kv1NVP5/9h9Hf7of4+aiqD2f/h4yrR8+s/UqS70qS7v7t7D/D9pYk55N8PclPDz6nawsAcOXW4W0+AICFEaYAAAYQpgAABhCmAAAGEKYAAAYQpgAABhCmAAAG+P85WEzs91IXUAAAAABJRU5ErkJggg==\n" + }, + "metadata": { + "needs_background": "light" + } + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "X0ycBrxIec07" + }, + "source": [ + "### 6.2 Building a PyTorch linear model\n", + "\n", + "We've got some data, now it's time to make a model.\n", + "\n", + "We'll create the same style of model as before except this time, instead of defining the weight and bias parameters of our model manually using `nn.Parameter()`, we'll use [`nn.Linear(in_features, out_features)`](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) to do it for us.\n", + "\n", + "Where `in_features` is the number of dimensions your input data has and `out_features` is the number of dimensions you'd like it to be output to.\n", + "\n", + "In our case, both of these are `1` since our data has `1` input feature (`X`) per label (`y`).\n", + "\n", + "![comparison of nn.Parameter Linear Regression model and nn.Linear Linear Regression model](https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/01-pytorch-linear-regression-model-with-nn-Parameter-and-nn-Linear-compared.png)\n", + "*Creating a linear regression model using `nn.Parameter` versus using `nn.Linear`. There are plenty more examples of where the `torch.nn` module has pre-built computations, including many popular and useful neural network layers.*\n" + ] + }, + { + "cell_type": "code", + "source": [ + "# Subclass nn.Module to make our model\n", + "class LinearRegressionModelV2(nn.Module):\n", + " def __init__(self):\n", + " super().__init__()\n", + " # Use nn.Linear () for creating the model parameters\n", + " self.linear_layer = nn.Linear(in_features = 1,\n", + " out_features = 1)\n", + " \n", + " # Define the forward computation \n", + " def forward(self, x: torch.Tensor) -> torch.Tensor:\n", + " return self.linear_layer(x)\n", + "torch.manual_seed(42)\n", + "model_1 = LinearRegressionModelV2()\n", + "model_1, model_1.state_dict()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "czBIFC_vbodn", + "outputId": "13cd0528-5fcb-455f-e26d-920a70f6a585" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "(LinearRegressionModelV2(\n", + " (linear_layer): Linear(in_features=1, out_features=1, bias=True)\n", + " ),\n", + " OrderedDict([('linear_layer.weight', tensor([[0.7645]])),\n", + " ('linear_layer.bias', tensor([0.8300]))]))" + ] + }, + "metadata": {}, + "execution_count": 52 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4vLN2pPXNXUs" + }, + "source": [ + "Notice the outputs of `model_1.state_dict()`, the `nn.Linear()` layer created a random `weight` and `bias` parameter for us.\n", + "\n", + "Now let's put our model on the GPU (if it's available).\n", + "\n", + "We can change the device our PyTorch objects are on using `.to(device)`.\n", + "\n", + "First let's check the model's current device." + ] + }, + { + "cell_type": "code", + "source": [ + "# Check model device\n", + "next(model_1.parameters()).device" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "Tldo2A6idi9y", + "outputId": "830bb2c1-8422-4ed2-9392-95e6b88ef5bf" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "device(type='cpu')" + ] + }, + "metadata": {}, + "execution_count": 53 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZqalUGW5N93K" + }, + "source": [ + "Wonderful, looks like the model's on the CPU by default.\n", + "\n", + "Let's change it to be on the GPU (if it's available)." + ] + }, + { + "cell_type": "code", + "source": [ + "# Set model to GPU if it's availalble, otherwise it'll default to CPU\n", + "model_1.to(device) # the device variable was set above to be \"cuda\" if available or \"cpu\" if not\n", + "next(model_1.parameters()).device" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "lg9rJAWTer1q", + "outputId": "fbeeca64-0fd3-407a-aa57-ff932550755c" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "device(type='cuda', index=0)" + ] + }, + "metadata": {}, + "execution_count": 54 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qHs0bL5_Oc1k" + }, + "source": [ + "Nice! Because of our device agnostic code, the above cell will work regardless of whether a GPU is available or not.\n", + "\n", + "If you do have access to a CUDA-enabled GPU, you should see an output of something like:\n", + "\n", + "```\n", + "device(type='cuda', index=0)\n", + "```" + ] + }, + { + "cell_type": "code", + "source": [ + "next(model_1.parameters()).device" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "61_sQEHKkQrS", + "outputId": "76e18009-b255-4b8b-a568-40c7aa6b5095" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "device(type='cuda', index=0)" + ] + }, + "metadata": {}, + "execution_count": 55 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "jwTeP_vkec08" + }, + "source": [ + "### 6.3 Training" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vPFOV3wUec09" + }, + "source": [ + "Time to build a training and testing loop.\n", + "\n", + "First we'll need a loss function and an optimizer.\n", + "\n", + "Let's use the same functions we used earlier, `nn.L1Loss()` and `torch.optim.SGD()`.\n", + "\n", + "We'll have to pass the new model's parameters (`model.parameters()`) to the optimizer for it to adjust them during training. \n", + "\n", + "The learning rate of `0.1` worked well before too so let's use that again.\n", + "\n", + "\n" + ] + }, + { + "cell_type": "code", + "source": [ + "# Create loss function\n", + "loss_fn = nn.L1Loss()\n", + "\n", + "# Create Optimizer\n", + "optimizer = torch.optim.SGD(params = model_1.parameters(),\n", + " lr = 0.01)" + ], + "metadata": { + "id": "zFOL0zyYfF6p" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "NxuBdoWRP2nU" + }, + "source": [ + "Beautiful, loss function and optimizer ready, now let's train and evaluate our model using a training and testing loop.\n", + "\n", + "The only different thing we'll be doing in this step compared to the previous training loop is putting the data on the target `device`.\n", + "\n", + "If you need a reminder of the PyTorch training loop steps, see below.\n", + "\n", + "\n", + "
\n", + " PyTorch training loop steps\n", + "
    \n", + "
  1. Forward pass - The model goes through all of the training data once, performing its\n", + " forward() function\n", + " calculations (model(x_train)).\n", + "
    2. \n", + "
  2. Calculate the loss - The model's outputs (predictions) are compared to the ground truth and evaluated\n", + " to see how\n", + " wrong they are (loss = loss_fn(y_pred, y_train).
    3. \n", + "
  3. Zero gradients - The optimizers gradients are set to zero (they are accumulated by default) so they\n", + " can be\n", + " recalculated for the specific training step (optimizer.zero_grad()).
    4. \n", + "
  4. Perform backpropagation on the loss - Computes the gradient of the loss with respect for every model\n", + " parameter to\n", + " be updated (each parameter\n", + " with requires_grad=True). This is known as backpropagation, hence \"backwards\"\n", + " (loss.backward()).
    5. \n", + "
  5. Step the optimizer (gradient descent) - Update the parameters with requires_grad=True\n", + " with respect to the loss\n", + " gradients in order to improve them (optimizer.step()).
    6. \n", + "
\n", + "
" + ] + }, + { + "cell_type": "code", + "source": [ + "torch.manual_seed(42)\n", + "\n", + "# Set the number of epochs\n", + "epochs = 100\n", + "\n", + "# Put data on the available device\n", + "# Without this, error will happen (not all model/data on device)\n", + "X_train = X_train.to(device)\n", + "X_test = X_test.to(device)\n", + "y_train = y_train.to(device)\n", + "y_test = y_test.to(device)\n", + "\n", + "for epoch in range(epochs):\n", + " ### Training\n", + " model_1.train()\n", + "\n", + " # 1. Forward Pass\n", + " y_pred = model_1(X_train)\n", + " #y_pred = model_1(X_train)\n", + "\n", + " #2. Calculate loss\n", + " loss = loss_fn(y_pred,y_train)\n", + "\n", + " # 3. Zero Grad Optimizer\n", + " optimizer.zero_grad()\n", + "\n", + " # 4. Loss Backward\n", + " loss.backward()\n", + "\n", + " # 5. Step the Optimizer \n", + " optimizer.step()\n", + "\n", + " ### Testing\n", + " model_1.eval() # put the model in evaluation mode for testing (inference)\n", + "\n", + " with torch.inference_mode():\n", + "\n", + " # 1. Forward pass\n", + " test_pred = model_1(X_test)\n", + "\n", + " # 2. Calculate the loss\n", + " test_loss = loss_fn(test_pred, y_test)\n", + "\n", + " if epoch % 100 == 0:\n", + " print(f\"Epoch: {epoch} | Train loss: {loss} | Test loss: {test_loss}\")\n" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "sjWZBSIDf4KQ", + "outputId": "74e46140-51a7-440f-b52c-96818936f2e7" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Epoch: 0 | Train loss: 0.5551779866218567 | Test loss: 0.5739762187004089\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "nt-b2Y131flk" + }, + "source": [ + "> **Note:** Due to the random nature of machine learning, you will likely get slightly different results (different loss and prediction values) depending on whether your model was trained on CPU or GPU. This is true even if you use the same random seed on either device. If the difference is large, you may want to look for errors, however, if it is small (ideally it is), you can ignore it.\n", + "\n", + "Nice! That loss looks pretty low.\n", + "\n", + "Let's check the parameters our model has learned and compare them to the original parameters we hard-coded." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "TP_tFn5rec09", + "outputId": "53b6c53a-1bab-4f13-e09a-c9473200af39" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The model learned the following values for weights and bias:\n", + "OrderedDict([('linear_layer.weight', tensor([[0.6968]], device='cuda:0')),\n", + " ('linear_layer.bias', tensor([0.3025], device='cuda:0'))])\n", + "\n", + "And the original values for weights and bias are:\n", + "weights: 0.7, bias: 0.3\n" + ] + } + ], + "source": [ + "# Find our model's learned parameters\n", + "from pprint import pprint # pprint = pretty print, see: https://docs.python.org/3/library/pprint.html \n", + "print(\"The model learned the following values for weights and bias:\")\n", + "pprint(model_1.state_dict())\n", + "print(\"\\nAnd the original values for weights and bias are:\")\n", + "print(f\"weights: {weight}, bias: {bias}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rDZo0vEU1_-1" + }, + "source": [ + "Ho ho! Now that's pretty darn close to a perfect model.\n", + "\n", + "Remember though, in practice, it's rare that you'll know the perfect parameters ahead of time.\n", + "\n", + "And if you knew the parameters your model had to learn ahead of time, what would be the fun of machine learning?\n", + "\n", + "Plus, in many real-world machine learning problems, the number of parameters can well exceed tens of millions.\n", + "\n", + "I don't know about you but I'd rather write code for a computer to figure those out rather than doing it by hand." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mBR1qvqhec09" + }, + "source": [ + "### 6.4 Making predictions\n", + "\n", + "Now we've got a trained model, let's turn on it's evaluation mode and make some predictions." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "ksqG5N5Iec09", + "outputId": "a0d4a51f-e1d9-4038-fd8a-0bbf4386f36a" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "tensor([[0.8600],\n", + " [0.8739],\n", + " [0.8878],\n", + " [0.9018],\n", + " [0.9157],\n", + " [0.9296],\n", + " [0.9436],\n", + " [0.9575],\n", + " [0.9714],\n", + " [0.9854]], device='cuda:0')" + ] + }, + "execution_count": 36, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Turn model into evaluation mode\n", + "model_1.eval()\n", + "\n", + "# Make predictions on the test data\n", + "with torch.inference_mode():\n", + " y_preds = model_1(X_test)\n", + "y_preds" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "NtOoVnbi2ysL" + }, + "source": [ + "If you're making predictions with data on the GPU, you might notice the output of the above has `device='cuda:0'` towards the end. That means the data is on CUDA device 0 (the first GPU your system has access to due to zero-indexing), if you end up using multiple GPUs in the future, this number may be higher. \n", + "\n", + "Now let's plot our model's predictions.\n", + "\n", + "> **Note:** Many data science libraries such as pandas, matplotlib and NumPy aren't capable of using data that is stored on GPU. So you might run into some issues when trying to use a function from one of these libraries with tensor data not stored on the CPU. To fix this, you can call [`.cpu()`](https://pytorch.org/docs/stable/generated/torch.Tensor.cpu.html) on your target tensor to return a copy of your target tensor on the CPU." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 428 + }, + "id": "Z4dmfr2bec09", + "outputId": "dd68d5a7-1733-4385-c1cb-7d7b44085813" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAGbCAYAAADgEhWsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAtPElEQVR4nO3de3RU9bn/8c+ThEsg3JQAQhAQUMGIChFFy0XFg3JZ1FbLrQpqNfyQc3QdL1A9Be9XrLWVVtQq3qpWBWuRopYfCFqRBBQEAh7EC2CEYH8LBauQ5Pn9MTFNQpKZsGcyk8z7tdasyd77u/d+MjuBT/be84y5uwAAAHB4UuJdAAAAQENGmAIAAAiAMAUAABAAYQoAACAAwhQAAEAAafHacfv27b179+7x2j0AAEDE1qxZs8fdM6tbFrcw1b17d+Xn58dr9wAAABEzs89qWsZlPgAAgAAIUwAAAAEQpgAAAAIgTAEAAARAmAIAAAgg7Lv5zOxxSaMl7Xb37GqWm6QHJY2U9K2kKe6+NmhhX3/9tXbv3q2DBw8G3RSSQMuWLZWVlaWUFP4+AADUr0haI8yX9JCkp2pYfr6k3mWP0yT9oez5sH399dfatWuXunTpovT0dIXyGlC90tJS7dy5U3v27FGHDh3iXQ4AIMmE/TPe3VdI+mctQ8ZKespDVklqa2ZHBSlq9+7d6tKli1q0aEGQQlgpKSnq2LGj9u7dG+9SAABJKBrXRLpI2l5hekfZvMN28OBBpaenByoKyaVJkyYqLi6OdxkAgCQUjTBV3akjr3ag2ZVmlm9m+UVFRbVvlDNSqAN+XgAA8RKNMLVDUtcK01mSvqhuoLs/4u457p6TmVntx9sAAAA0KNEIU69KusRCTpe0190Lo7BdAACAhBc2TJnZc5LelXScme0ws8vNbKqZTS0bsljSNklbJT0qaVrMqk1CU6ZM0ejRo+u0zrBhwzR9+vQYVVS76dOna9iwYXHZNwAA8RC2NYK7Twiz3CVdFbWKGqhw9+xMnjxZ8+fPr/N2H3zwQYVe4sgtWLBATZo0qfO+4uHTTz9Vjx49lJeXp5ycnHiXAwBAnUXSZwoRKCz895XNRYsW6Yorrqg0r+q7Ew8ePBhR4GnTpk2dazniiCPqvA4AADg8tIuOkk6dOpU/2rZtW2ned999p7Zt2+q5557T2WefrfT0dM2bN09fffWVJkyYoKysLKWnp+uEE07QE088UWm7VS/zDRs2TNOmTdONN96o9u3bq0OHDrruuutUWlpaaUzFy3zdu3fX7bffrtzcXLVu3VpZWVm67777Ku3no48+0tChQ9W8eXMdd9xxWrx4sTIyMmo9m1ZSUqLrrrtO7dq1U7t27XTNNdeopKSk0pglS5Zo8ODBateunY444giNGDFCBQUF5ct79OghSTr11FNlZuWXCPPy8vQf//Efat++vVq3bq0f/ehHevfdd8MfCABAUnlr9IkqTjG9NfrEuNVAmKpHv/zlLzVt2jRt2rRJP/7xj/Xdd9+pf//+WrRokTZu3Kirr75aubm5Wrp0aa3befbZZ5WWlqZ//OMfeuihh/Sb3/xGL7zwQq3rPPDAAzrxxBO1du1azZgxQzfccEN5OCktLdUFF1ygtLQ0rVq1SvPnz9ctt9yi77//vtZt3n///Xr00Uc1b948vfvuuyopKdGzzz5bacz+/ft1zTXXaPXq1Vq+fLnatGmjMWPG6MCBA5Kk1atXSwqFrsLCQi1YsECS9M033+jiiy/WypUrtXr1ap188skaOXKk9uzZU2tNAIDkcubiDUrz0HPcuHtcHgMGDPCabNq0qcZldTVtmntqaui5vrz44oseemlDPvnkE5fkc+bMCbvuuHHj/PLLLy+fnjx5so8aNap8eujQoX766adXWmf48OGV1hk6dKhfddVV5dPdunXz8ePHV1qnV69eftttt7m7+5IlSzw1NdV37NhRvvydd95xSf7EE0/UWOtRRx3lt99+e/l0SUmJ9+7d24cOHVrjOvv27fOUlBRfuXKlu//7tcnLy6txHXf30tJS79Spkz/99NM1jonmzw0AoGFYPirbD5p8+ajsmO5HUr7XkGka/ZmpefOkkpLQc7xVvcG6pKREd9xxh/r166cjjzxSGRkZWrBggT7//PNat9OvX79K0507d9bu3bsPe53Nmzerc+fO6tLl343rTz311Fo/NHjv3r0qLCzUoEGDyuelpKTotNMqfyzjxx9/rIkTJ6pnz55q3bq1OnbsqNLS0rDf4+7du5Wbm6tjjz1Wbdq0UatWrbR79+6w6wEAksvQRR8qrdQ1dNGHcauh0d+AnpsbClK5ufGuRGrZsmWl6Tlz5uj+++/Xgw8+qBNPPFEZGRm68cYbwwajqjeum1mle6bquo67x6yD+JgxY9SlSxfNmzdPXbp0UVpamvr27Vt+ma8mkydP1q5du/TAAw+oe/fuatasmc4555yw6wEAUN8afZiaOzf0SERvv/22xowZo4svvlhSKNR89NFH5Tew15c+ffpo586d+uKLL9S5c2dJUn5+fq0BrU2bNjrqqKO0atUqnX322ZJC9a9evVpHHRX6nOuvvvpKBQUFmjt3rs466yxJ0tq1ayt9hl7Tpk0l6ZAb199++2399re/1ahRoyRJu3btqvTuSAAAEkWjv8yXyI499lgtXbpUb7/9tjZv3qzp06frk08+qfc6zj33XB133HGaPHmy1q1bp1WrVum///u/lZaWVusZq6uvvlr33nuvXnrpJW3ZskXXXHNNpcDTrl07tW/fXo8++qi2bt2qt956S1OnTlVa2r8zfIcOHZSenq7XX39du3bt0t69eyWFXptnnnlGmzZtUl5ensaPH18evAAASCSEqTj6n//5Hw0cOFDnn3++hgwZopYtW2rSpEn1XkdKSooWLlyo77//XgMHDtTkyZN10003yczUvHnzGte79tprdemll+oXv/iFTjvtNJWWllaqPyUlRS+88ILWr1+v7OxsXXXVVbrtttvUrFmz8jFpaWn67W9/q8cee0ydO3fW2LFjJUmPP/649u3bpwEDBmj8+PG67LLL1L1795i9BgCAxJEI7Q7qwryO3bWjJScnx/Pz86tdVlBQoD59+tRzRaho3bp1Ovnkk5Wfn68BAwbEu5yI8HMDAI1DcYopzaVik9JK45NTqjKzNe5e7Ud1cGYKkqSFCxfqjTfe0CeffKJly5ZpypQpOumkk9S/f/94lwYASDLvjMxWsYWeG4JGfwM6IvPNN99oxowZ2r59u9q1a6dhw4bpgQceiNm7/AAAqMkPbQ6GxrmOSBGmIEm65JJLdMkll8S7DAAAGhwu8wEAAARAmAIAAAiAMAUAAOpFQ2t5ECnCFAAAqBdnLt6gNA89NyaEKQAAUC8aWsuDSPFuPgAAUC8aWsuDSHFmqgHr3r275syZE5d9jx49WlOmTInLvgEASCSEqSgxs1ofQYLHzTffrOzsQ0+J5uXladq0aQGqrj/Lly+XmWnPnj3xLgUAgKjiMl+UFBYWln+9aNEiXXHFFZXmpaenR32fmZmZUd8mAACoG85MRUmnTp3KH23btj1k3ooVKzRgwAA1b95cPXr00E033aQDBw6Ur79gwQL169dP6enpOuKIIzR06FDt2rVL8+fP1y233KKNGzeWn+WaP3++pEMv85mZHnnkEV100UVq2bKljjnmGD3zzDOV6nzvvffUv39/NW/eXKeccooWL14sM9Py5ctr/N6+/fZbTZkyRRkZGerYsaPuvPPOQ8Y888wzOvXUU9WqVSt16NBBF110kXbu3ClJ+vTTT3XWWWdJCgXAimfqlixZosGDB6tdu3Y64ogjNGLECBUUFNT15QcAxFFjbXkQKcJUPXj99dc1adIkTZ8+XRs3btTjjz+ul156STfeeKMk6csvv9T48eM1efJkFRQUaMWKFbr44oslSePGjdO1116r4447ToWFhSosLNS4ceNq3Nett96qsWPHat26dRo3bpwuu+wyffbZZ5Kkffv2afTo0Tr++OO1Zs0a3Xvvvbr++uvD1n/dddfpzTff1Msvv6ylS5fq/fff14oVKyqNOXDggG655RatW7dOixYt0p49ezRhwgRJUteuXfXyyy9LkjZu3KjCwkI9+OCDkqT9+/frmmuu0erVq7V8+XK1adNGY8aMqRQ0AQCJrbG2PIiYu8flMWDAAK/Jpk2balxWV9MWTfPUW1J92qJpUdtmOC+++KKHXtqQwYMH+6233lppzMKFC71ly5ZeWlrqa9ascUn+6aefVru92bNn+wknnHDI/G7duvl9991XPi3JZ86cWT598OBBT09P96efftrd3R9++GFv166df/vtt+Vjnn32WZfky5Ytq3bf33zzjTdt2tSfeeaZSvPatGnjkydPrvE1KCgocEm+fft2d3dftmyZS/KioqIa13F337dvn6ekpPjKlStrHVedaP7cAAAit3xUth80+fJR2fEuJWYk5XsNmabRn5mat2aeSrxE89bMi1sNa9as0R133KGMjIzyx8SJE7V//359+eWXOumkkzR8+HBlZ2frpz/9qf7whz+oqKjosPbVr1+/8q/T0tKUmZmp3bt3S5I2b96s7OzsSvdvnXbaabVu7+OPP9aBAwc0aNCg8nkZGRk68cTKp3LXrl2rsWPHqlu3bmrVqpVycnIkSZ9//nnY7U+cOFE9e/ZU69at1bFjR5WWloZdDwCQOIYu+lBppV7e+iDZNPowlTsgV6mWqtwBuXGrobS0VLNnz9YHH3xQ/li/fr3+93//V5mZmUpNTdUbb7yhN954Q/369dMf//hH9e7dW+vWravzvpo0aVJp2sxUWloqKXQW0szqtL1QGK/d/v37NWLECLVo0UJPP/208vLytGTJEkkKe7luzJgxKioq0rx58/Tee+/p/fffV1paGpf5AAANRqN/N9/cUXM1d9TcuNbQv39/bd68Wb169apxjJlp0KBBGjRokGbNmqUTTjhBL7zwgk466SQ1bdpUJSUlgevo06ePnnrqKf3rX/8qPzu1evXqWtfp1auXmjRpolWrVumYY46RFApPGzZsUM+ePSWFznjt2bNHd955p3r06CEpdEN9RU2bNpWkSt/HV199pYKCAs2dO7f8BvW1a9equLg48PcKAEB9afRnphLBrFmz9Kc//UmzZs3Shg0btHnzZr300ku64YYbJEmrVq3S7bffrry8PH3++ed69dVXtX37dvXt21dS6F17n332mdauXas9e/bo+++/P6w6Jk2apNTUVF1xxRXatGmT/v73v5e/M6+mM1YZGRm6/PLLNWPGDL355pvauHGjLrvsskqh6Oijj1azZs300EMPadu2bXrttdf0q1/9qtJ2unXrJjPTa6+9pqKiIu3bt0/t2rVT+/bt9eijj2rr1q166623NHXqVKWlNfqMDwBoRAhT9WDEiBF67bXXtGzZMg0cOFADBw7U3XffraOPPlqS1KZNG73zzjsaPXq0evfurWuvvVa/+tWv9POf/1yS9NOf/lQjR47UOeeco8zMTD333HOHVUdGRob++te/auPGjTrllFN0/fXX6+abb5YkNW/evMb15syZo7POOksXXHCBzjrrLGVnZ2vIkCHlyzMzM/Xkk0/qlVdeUd++fXXLLbfo17/+daVtdOnSRbfccotuuukmdezYUdOnT1dKSopeeOEFrV+/XtnZ2brqqqt02223qVmzZof1/QEAoifZ2x3UhUVyT0ws5OTkeH5+frXLCgoK1KdPn3quKDn95S9/0QUXXKDdu3erffv28S4nEH5uACB6ilNMaS4Vm5RWGp+skEjMbI2751S3jDNTSebJJ5/UypUr9emnn2rRokW65pprNGbMmAYfpAAA0fXOyGwVW+gZtePmlCSza9cuzZ49W4WFherUqZNGjRqle+65J95lAQASzA9tDobGuY6GgDCVZG644YbyG98BAEBwXOYDAAAIgDAFAAAQAGEKAIAkQsuD6CNMAQCQRM5cvEFpHnpGdBCmAABIIrQ8iD7ezQcAQBKh5UH0cWaqAXrppZcqfZbe/PnzlZGREWiby5cvl5lpz549QcsDACCpEKaiaMqUKTIzmZmaNGmiY445Rtddd532798f0/2OGzdO27Zti3h89+7dNWfOnErzzjjjDBUWFurII4+MdnkAADRqEYUpMzvPzLaY2VYzm1nN8nZmttDM1pvZajNL2guxw4cPV2FhobZt26bbb79dv//973XdddcdMq64uFjR+lzE9PR0dejQIdA2mjZtqk6dOlU64wUAAMILG6bMLFXSXEnnS+oraYKZ9a0y7EZJH7h7P0mXSHow2oU2FM2aNVOnTp3UtWtXTZw4UZMmTdIrr7yim2++WdnZ2Zo/f7569uypZs2aaf/+/dq7d6+uvPJKdejQQa1atdLQoUNV9QOgn3rqKXXr1k0tWrTQ6NGjtWvXrkrLq7vM99prr+m0005Tenq6jjzySI0ZM0bfffedhg0bps8++0zXX399+Vk0qfrLfAsWLNCJJ56oZs2aqWvXrrrjjjsqBcDu3bvr9ttvV25urlq3bq2srCzdd999leqYN2+ejj32WDVv3lyZmZkaMWKEiouLo/JaAwD+jZYH8RPJmamBkra6+zZ3PyDpeUljq4zpK2mpJLn7ZkndzaxjVCttoNLT03Xw4EFJ0ieffKI//elPevHFF7Vu3To1a9ZMo0aN0s6dO7Vo0SK9//77GjJkiM4++2wVFhZKkt577z1NmTJFV155pT744AONGTNGs2bNqnWfS5Ys0dixY3XuuedqzZo1WrZsmYYOHarS0lItWLBAWVlZmjVrlgoLC8v3U9WaNWt00UUX6Sc/+Yk+/PBD3X333brrrrv00EMPVRr3wAMP6MQTT9TatWs1Y8YM3XDDDXr33XclSfn5+brqqqs0e/ZsbdmyRX//+9913nnnBX1JAQDVoOVBHLl7rQ9JF0p6rML0xZIeqjLmTkm/Lvt6oKRiSQOq2daVkvIl5R999NFek02bNtW4rM6mTXNPTQ09x9jkyZN91KhR5dPvvfeeH3nkkf6zn/3MZ8+e7Wlpaf7ll1+WL1+6dKm3bNnSv/3220rbOemkk/yee+5xd/cJEyb48OHDKy2//PLLPXToQp544glv2bJl+fQZZ5zh48aNq7HObt26+X333Vdp3rJly1ySFxUVubv7xIkT/ayzzqo0Zvbs2d6lS5dK2xk/fnylMb169fLbbrvN3d1ffvllb926tX/99dc11hJNUf25AYAGZvmobD9o8uWjsuNdSqMkKd9ryEqRnJmq7iaaqjf73C2pnZl9IOk/Jb1fFqiqBrdH3D3H3XMyMzMj2HUUzJsnlZSEnuvBkiVLlJGRoebNm2vQoEEaMmSIfve730mSsrKy1LHjv0/YrVmzRt9++60yMzOVkZFR/tiwYYM+/vhjSVJBQYEGDRpUaR9Vp6t6//33dc455wT6PgoKCnTmmWdWmvejH/1IO3fu1Ndff10+r1+/fpXGdO7cWbt375YknXvuuerWrZt69OihSZMm6cknn9Q333wTqC4AQPWGLvpQaaVe3voA9SeSPlM7JHWtMJ0l6YuKA9z9a0mXSpKFbsL5pOwRf7m5oSCVm1svuxsyZIgeeeQRNWnSRJ07d1aTJk3Kl7Vs2bLS2NLSUnXs2FErV648ZDutW7eWpKjdpF5X7l7jzegV51f8/n5YVlpaKklq1aqV1q5dqxUrVujNN9/UXXfdpRtvvFF5eXnq3Llz7IoHAKAeRXJmKk9SbzPrYWZNJY2X9GrFAWbWtmyZJP1C0oqygBV/c+dKxcWh53rQokUL9erVS926dTskaFTVv39/7dq1SykpKerVq1elxw/vzuvbt69WrVpVab2q01WdcsopWrp0aY3LmzZtqpKSklq30bdvX7399tuV5r399tvKyspSq1atal23orS0NJ199tm66667tH79eu3fv1+LFi2KeH0AABJd2DNT7l5sZtMlvS4pVdLj7r7RzKaWLX9YUh9JT5lZiaRNki6PYc2NxvDhw3XmmWdq7Nixuvfee3X88cfryy+/1JIlSzR8+HANHjxY//Vf/6UzzjhDd911ly688EItX75cCxcurHW7N910k8aMGaNevXpp4sSJcne98cYbys3NVYsWLdS9e3etXLlSP//5z9WsWTO1b9/+kG1ce+21OvXUU3XzzTdr4sSJysvL0/33368777wz4u9v0aJF+vjjjzVkyBAdccQRWrZsmb755hv16dOnzq8VAACJKqI+U+6+2N2Pdfee7n5H2byHy4KU3P1dd+/t7se7+0/c/f/FsujGwsy0ePFinX322briiit03HHH6Wc/+5m2bNlSfhns9NNP1x//+Ef94Q9/UL9+/bRgwQLdfPPNtW535MiRWrhwof72t7/plFNO0dChQ7Vs2TKlpIQO96233qrt27erZ8+equnetf79++vFF1/Uyy+/rOzsbM2cOVMzZ87U9OnTI/7+2rZtq1deeUXDhw/X8ccfrzlz5uixxx7T4MGDI94GACQz2h00DBave3JycnK8aj+lHxQUFHD2AnXGzw2AxqY4xZTmUrFJaaXx+f8aIWa2xt1zqlvGx8kAAJCg3hmZrWILPSNxRfJuPgAAEAc/tDkYGuc6UDvOTAEAAARAmAIAAAggYcPUD40fgUjE640UAAAkZJhq2bKldu7cqQMHDvCfJMJyd3311Vdq3rx5vEsBgIjQ8qBxScjWCKWlpdqzZ4/27t2r4uJDPuIPOETz5s2VlZUVtus8ACQCWh40PLW1RkjId/OlpKSoQ4cO5R+pAgBAY/LOyGyduXiD3hmZzTv1GoGEDFMAADRmtDxoXBLynikAAICGgjAFAAAQAGEKAAAgAMIUAABRQsuD5ESYAgAgSs5cvEFpHnpG8iBMAQAQJe+MzFaxhZ6RPGiNAABAlNDyIDlxZgoAACAAwhQAAEAAhCkAAIAACFMAANTiqquktLTQM1AdwhQAALWYN08qKQk9A9UhTAEAUIvcXCk1NfQMVMfcPS47zsnJ8fz8/LjsGwAAoC7MbI2751S3jDNTAAAAARCmAAAAAiBMAQAABECYAgAkJVoeIFoIUwCApETLA0QLYQoAkJRoeYBooTUCAABAGLRGAAAAiBHCFAAAQACEKQAAgAAIUwCARoWWB6hvhCkAQKNCywPUN8IUAKBRoeUB6hutEQAAAMKgNQIAAECMEKYAAAACIEwBAAAEEFGYMrPzzGyLmW01s5nVLG9jZn81s3VmttHMLo1+qQCAZEW7AySysDegm1mqpI8knStph6Q8SRPcfVOFMTdKauPuM8wsU9IWSZ3c/UBN2+UGdABApNLSQu0OUlOl4uJ4V4NkFPQG9IGStrr7trJw9LyksVXGuKRWZmaSMiT9UxI/7gCAqKDdARJZJGGqi6TtFaZ3lM2r6CFJfSR9IelDSVe7e2nVDZnZlWaWb2b5RUVFh1kyACDZzJ0bOiM1d268KwEOFUmYsmrmVb02OELSB5I6SzpZ0kNm1vqQldwfcfccd8/JzMysY6kAAACJJ5IwtUNS1wrTWQqdgaroUkkLPGSrpE8kHR+dEgEAABJXJGEqT1JvM+thZk0ljZf0apUxn0s6R5LMrKOk4yRti2ahAAAAiShsmHL3YknTJb0uqUDSn919o5lNNbOpZcNuk3SGmX0oaamkGe6+J1ZFAwAaB1oeoDHgs/kAAHFDywM0FHw2HwAgIdHyAI0BZ6YAAADC4MwUAABAjBCmAAAAAiBMAQAABECYAgBEHS0PkEwIUwCAqJs3L9TyYN68eFcCxB5hCgAQdbQ8QDKhNQIAAEAYtEYAAACIEcIUAABAAIQpAACAAAhTAAAAARCmAAARoXcUUD3CFAAgIvSOAqpHmAIARITeUUD16DMFAAAQBn2mAAAAYoQwBQAAEABhCgAAIADCFAAkOVoeAMEQpgAgydHyAAiGMAUASY6WB0AwtEYAAAAIg9YIAAAAMUKYAgAACIAwBQAAEABhCgAaIdodAPWHMAUAjRDtDoD6Q5gCgEaIdgdA/aE1AgAAQBi0RgAAAIgRwhQAAEAAhCkAAIAACFMA0IDQ8gBIPIQpAGhAaHkAJB7CFAA0ILQ8ABIPrREAAADCoDUCAABAjBCmAAAAAiBMAQAABECYAoAEQMsDoOGKKEyZ2XlmtsXMtprZzGqWX29mH5Q9NphZiZkdEf1yAaBxouUB0HCFDVNmlipprqTzJfWVNMHM+lYc4+73ufvJ7n6ypF9Kesvd/xmDegGgUaLlAdBwRXJmaqCkre6+zd0PSHpe0thaxk+Q9Fw0igOAZDF3rlRcHHoG0LBEEqa6SNpeYXpH2bxDmFkLSedJermG5VeaWb6Z5RcVFdW1VgAAgIQTSZiyaubV1OlzjKR3arrE5+6PuHuOu+dkZmZGWiMAAEDCiiRM7ZDUtcJ0lqQvahg7XlziAwAASSSSMJUnqbeZ9TCzpgoFplerDjKzNpKGSvpLdEsEgIaJdgdAcggbpty9WNJ0Sa9LKpD0Z3ffaGZTzWxqhaEXSHrD3ffHplQAaFhodwAkh7RIBrn7YkmLq8x7uMr0fEnzo1UYADR0ubmhIEW7A6BxM/ea7iWPrZycHM/Pz4/LvgEAAOrCzNa4e051y/g4GQAAgAAIUwAAAAEQpgAAAAIgTAFAHdHyAEBFhCkAqCNaHgCoiDAFAHWUmyulptLyAEAIrREAAADCoDUCAABAjBCmAAAAAiBMAQAABECYAoAytDwAcDgIUwBQhpYHAA4HYQoAytDyAMDhoDUCAABAGLRGAAAAiBHCFAAAQACEKQAAgAAIUwAaNdodAIg1whSARo12BwBijTAFoFGj3QGAWKM1AgAAQBi0RgAAAIgRwhQAAEAAhCkAAIAACFMAGiRaHgBIFIQpAA0SLQ8AJArCFIAGiZYHABIFrREAAADCoDUCAABAjBCmAAAAAiBMAQAABECYApBQaHkAoKEhTAFIKLQ8ANDQEKYAJBRaHgBoaGiNAAAAEAatEQAAAGKEMAUAABAAYQoAACAAwhSAmKPdAYDGjDAFIOZodwCgMYsoTJnZeWa2xcy2mtnMGsYMM7MPzGyjmb0V3TIBNGS0OwDQmIVtjWBmqZI+knSupB2S8iRNcPdNFca0lfQPSee5++dm1sHdd9e2XVojAACAhiJoa4SBkra6+zZ3PyDpeUljq4yZKGmBu38uSeGCFAAAQGMRSZjqIml7hekdZfMqOlZSOzNbbmZrzOyS6jZkZleaWb6Z5RcVFR1exQAAAAkkkjBl1cyrem0wTdIASaMkjZD0KzM79pCV3B9x9xx3z8nMzKxzsQAAAIkmkjC1Q1LXCtNZkr6oZswSd9/v7nskrZB0UnRKBJCoaHkAAJGFqTxJvc2sh5k1lTRe0qtVxvxF0mAzSzOzFpJOk1QQ3VIBJBpaHgBABGHK3YslTZf0ukIB6c/uvtHMpprZ1LIxBZKWSFovabWkx9x9Q+zKBpAIaHkAABG0RogVWiMAAICGImhrBAAAANSAMAUAABAAYQoAACAAwhSAQ9DyAAAiR5gCcAhaHgBA5AhTAA5BywMAiBytEQAAAMKgNQIAAECMEKYAAAACIEwBAAAEQJgCkgTtDgAgNghTQJKg3QEAxAZhCkgStDsAgNigNQIAAEAYtEYAAACIEcIUAABAAIQpAACAAAhTQANHywMAiC/CFNDA0fIAAOKLMAU0cLQ8AID4ojUCAABAGLRGAAAAiBHCFAAAQACEKQAAgAAIU0CCouUBADQMhCkgQdHyAAAaBsIUkKBoeQAADQOtEQAAAMKgNQIAAECMEKYAAAACIEwBAAAEQJgCAAAIgDAF1CN6RwFA40OYAuoRvaMAoPEhTAH1iN5RAND40GcKAAAgDPpMAQAAxAhhCgAAIADCFAAAQACEKSAKaHkAAMmLMAVEAS0PACB5EaaAKKDlAQAkr4jClJmdZ2ZbzGyrmc2sZvkwM9trZh+UPWZFv1Qgcc2dKxUXh54BAMklLdwAM0uVNFfSuZJ2SMozs1fdfVOVoSvdfXQMagQAAEhYkZyZGihpq7tvc/cDkp6XNDa2ZQEAADQMkYSpLpK2V5jeUTavqkFmts7M/mZmJ1S3ITO70szyzSy/qKjoMMoFAABILJGEKatmXtXPoFkrqZu7nyTpd5JeqW5D7v6Iu+e4e05mZmadCgXqG+0OAACRiCRM7ZDUtcJ0lqQvKg5w96/dfV/Z14slNTGz9lGrEogD2h0AACIRSZjKk9TbzHqYWVNJ4yW9WnGAmXUyMyv7emDZdr+KdrFAfaLdAQAgEmHfzefuxWY2XdLrklIlPe7uG81satnyhyVdKOn/mFmxpH9JGu/uVS8FAg3K3Lm0OgAAhGfxyjw5OTmen58fl30DAADUhZmtcfec6pbRAR0AACAAwhQAAEAAhCkkHVoeAACiiTCFpEPLAwBANBGmkHRoeQAAiCbezQcAABAG7+YDAACIEcIUAABAAIQpAACAAAhTaDRoeQAAiAfCFBoNWh4AAOKBMIVGg5YHAIB4oDUCAABAGLRGAAAAiBHCFAAAQACEKQAAgAAIU0hotDsAACQ6whQSGu0OAACJjjCFhEa7AwBAoqM1AgAAQBi0RgAAAIgRwhQAAEAAhCkAAIAACFOIC1oeAAAaC8IU4oKWBwCAxoIwhbig5QEAoLGgNQIAAEAYtEYAAACIEcIUAABAAIQpAACAAAhTiCpaHgAAkg1hClFFywMAQLIhTCGqaHkAAEg2tEYAAAAIg9YIAAAAMUKYAgAACIAwBQAAEABhCmHR7gAAgJoRphAW7Q4AAKgZYQph0e4AAICa0RoBAAAgjMCtEczsPDPbYmZbzWxmLeNONbMSM7vwcIsFAABoSMKGKTNLlTRX0vmS+kqaYGZ9axh3j6TXo10kAABAoorkzNRASVvdfZu7H5D0vKSx1Yz7T0kvS9odxfoAAAASWiRhqouk7RWmd5TNK2dmXSRdIOnh2jZkZleaWb6Z5RcVFdW1VkQZLQ8AAAgukjBl1cyretf6byTNcPeS2jbk7o+4e46752RmZkZYImKFlgcAAAQXSZjaIalrheksSV9UGZMj6Xkz+1TShZJ+b2Y/jkaBiB1aHgAAEFzY1ghmlibpI0nnSNopKU/SRHffWMP4+ZIWuftLtW2X1ggAAKChqK01Qlq4ld292MymK/QuvVRJj7v7RjObWra81vukAAAAGrOwYUqS3H2xpMVV5lUbotx9SvCyAAAAGgY+TgYAACAAwlQjRMsDAADqD2GqEaLlAQAA9Ycw1QjR8gAAgPoTtjVCrNAaAQAANBS1tUbgzBQAAEAAhCkAAIAACFMAAAABEKYaCNodAACQmAhTDQTtDgAASEyEqQaCdgcAACQmWiMAAACEQWsEAACAGCFMAQAABECYAgAACIAwFWe0PAAAoGEjTMUZLQ8AAGjYCFNxRssDAAAaNlojAAAAhEFrBAAAgBghTAEAAARAmAIAAAiAMBUjtDwAACA5EKZihJYHAAAkB8JUjNDyAACA5EBrBAAAgDBojQAAABAjhCkAAIAACFMAAAABEKbqgHYHAACgKsJUHdDuAAAAVEWYqgPaHQAAgKpojQAAABAGrREAAABihDAFAAAQAGEKAAAgAMKUaHkAAAAOH2FKtDwAAACHjzAlWh4AAIDDR2sEAACAMGiNAAAAECMRhSkzO8/MtpjZVjObWc3ysWa23sw+MLN8M/tR9EsFAABIPGnhBphZqqS5ks6VtENSnpm96u6bKgxbKulVd3cz6yfpz5KOj0XBAAAAiSSSM1MDJW11923ufkDS85LGVhzg7vv83zdftZQUnxuxAAAA6lkkYaqLpO0VpneUzavEzC4ws82SXpN0WXTKC4b+UQAAINYiCVNWzbxDzjy5+0J3P17SjyXdVu2GzK4su6cqv6ioqE6FHg76RwEAgFiLJEztkNS1wnSWpC9qGuzuKyT1NLP21Sx7xN1z3D0nMzOzzsXWFf2jAABArEUSpvIk9TazHmbWVNJ4Sa9WHGBmvczMyr7uL6mppK+iXWxdzZ0rFReHngEAAGIh7Lv53L3YzKZLel1SqqTH3X2jmU0tW/6wpJ9KusTMDkr6l6RxHq9uoAAAAPWIDugAAABh0AEdAAAgRghTAAAAARCmAAAAAiBMAQAABECYAgAACIAwBQAAEABhCgAAIADCFAAAQACEKQAAgAAIUwAAAAEQpgAAAAIgTAEAAAQQtw86NrMiSZ/Vw67aS9pTD/tB3XFsEhvHJ3FxbBIbxydxBTk23dw9s7oFcQtT9cXM8mv6lGfEF8cmsXF8EhfHJrFxfBJXrI4Nl/kAAAACIEwBAAAEkAxh6pF4F4AacWwSG8cncXFsEhvHJ3HF5Ng0+numAAAAYikZzkwBAADEDGEKAAAggEYRpszsPDPbYmZbzWxmNcvNzH5btny9mfWPR53JKoLjM6nsuKw3s3+Y2UnxqDMZhTs2FcadamYlZnZhfdaX7CI5PmY2zMw+MLONZvZWfdeYrCL4d62Nmf3VzNaVHZtL41FnMjKzx81st5ltqGF59DOBuzfoh6RUSR9LOkZSU0nrJPWtMmakpL9JMkmnS3ov3nUnyyPC43OGpHZlX5/P8UmcY1Nh3P+VtFjShfGuO1keEf7utJW0SdLRZdMd4l13MjwiPDY3Srqn7OtMSf+U1DTetSfDQ9IQSf0lbahhedQzQWM4MzVQ0lZ33+buByQ9L2lslTFjJT3lIasktTWzo+q70CQV9vi4+z/c/f+VTa6SlFXPNSarSH53JOk/Jb0saXd9FoeIjs9ESQvc/XNJcneOUf2I5Ni4pFZmZpIyFApTxfVbZnJy9xUKvd41iXomaAxhqouk7RWmd5TNq+sYxEZdX/vLFfqLAbEX9tiYWRdJF0h6uB7rQkgkvzvHSmpnZsvNbI2ZXVJv1SW3SI7NQ5L6SPpC0oeSrnb30vopD2FEPROkBSonMVg186r2e4hkDGIj4tfezM5SKEz9KKYV4QeRHJvfSJrh7iWhP7BRjyI5PmmSBkg6R1K6pHfNbJW7fxTr4pJcJMdmhKQPJJ0tqaekN81spbt/HePaEF7UM0FjCFM7JHWtMJ2l0F8CdR2D2IjotTezfpIek3S+u39VT7Ulu0iOTY6k58uCVHtJI82s2N1fqZcKk1uk/7btcff9kvab2QpJJ0kiTMVWJMfmUkl3e+gmna1m9omk4yWtrp8SUYuoZ4LGcJkvT1JvM+thZk0ljZf0apUxr0q6pOwO/tMl7XX3wvouNEmFPT5mdrSkBZIu5i/qehX22Lh7D3fv7u7dJb0kaRpBqt5E8m/bXyQNNrM0M2sh6TRJBfVcZzKK5Nh8rtAZQ5lZR0nHSdpWr1WiJlHPBA3+zJS7F5vZdEmvK/QOi8fdfaOZTS1b/rBC70IaKWmrpG8V+osB9SDC4zNL0pGSfl92BqTY+cT1mIvw2CBOIjk+7l5gZkskrZdUKukxd6/27eCIngh/d26TNN/MPlTostIMd98Tt6KTiJk9J2mYpPZmtkPSbElNpNhlAj5OBgAAIIDGcJkPAAAgbghTAAAAARCmAAAAAiBMAQAABECYAgAACIAwBQAAEABhCgAAIID/D3PFxscZJbEwAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "# plot_predictions(predictions=y_preds) # -> won't work... data not on CPU\n", + "\n", + "# Put data on the CPU and plot it\n", + "plot_predictions(predictions=y_preds.cpu())" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DxZa-5-Tec0-" + }, + "source": [ + "Woah! Look at those red dots, they line up almost perfectly with the green dots. I guess the extra epochs helped.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "K8jCHl1gec0-" + }, + "source": [ + "### 6.5 Saving and loading a model\n", + "\n", + "We're happy with our models predictions, so let's save it to file so it can be used later.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "DcQo4JqL7eSU", + "outputId": "e43ada0c-c074-4b50-9207-fa01581b1d5f" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Saving model to: models/01_pytorch_workflow_model_1.pth\n" + ] + } + ], + "source": [ + "from pathlib import Path\n", + "\n", + "# 1. Create models directory \n", + "MODEL_PATH = Path(\"models\")\n", + "MODEL_PATH.mkdir(parents=True, exist_ok=True)\n", + "\n", + "# 2. Create model save path \n", + "MODEL_NAME = \"01_pytorch_workflow_model_1.pth\"\n", + "MODEL_SAVE_PATH = MODEL_PATH / MODEL_NAME\n", + "\n", + "# 3. Save the model state dict \n", + "print(f\"Saving model to: {MODEL_SAVE_PATH}\")\n", + "torch.save(obj=model_1.state_dict(), # only saving the state_dict() only saves the models learned parameters\n", + " f=MODEL_SAVE_PATH) " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lk0rvpwV7slc" + }, + "source": [ + "And just to make sure everything worked well, let's load it back in.\n", + "\n", + "We'll:\n", + "* Create a new instance of the `LinearRegressionModelV2()` class\n", + "* Load in the model state dict using `torch.nn.Module.load_state_dict()`\n", + "* Send the new instance of the model to the target device (to ensure our code is device-agnostic)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "jMnVHzf1ec0-", + "outputId": "76f10046-cd42-4b39-a372-aa95227828e8" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loaded model:\n", + "LinearRegressionModelV2(\n", + " (linear_layer): Linear(in_features=1, out_features=1, bias=True)\n", + ")\n", + "Model on device:\n", + "cuda:0\n" + ] + } + ], + "source": [ + "# Instantiate a fresh instance of LinearRegressionModelV2\n", + "loaded_model_1 = LinearRegressionModelV2()\n", + "\n", + "# Load model state dict \n", + "loaded_model_1.load_state_dict(torch.load(MODEL_SAVE_PATH))\n", + "\n", + "# Put model to target device (if your data is on GPU, model will have to be on GPU to make predictions)\n", + "loaded_model_1.to(device)\n", + "\n", + "print(f\"Loaded model:\\n{loaded_model_1}\")\n", + "print(f\"Model on device:\\n{next(loaded_model_1.parameters()).device}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Hv6EMEx99LV2" + }, + "source": [ + "Now we can evaluate the loaded model to see if its predictions line up with the predictions made prior to saving." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "fYODT7ONec0_", + "outputId": "c8184cd1-595a-43e4-8155-89dcecc4d0b0" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "tensor([[True],\n", + " [True],\n", + " [True],\n", + " [True],\n", + " [True],\n", + " [True],\n", + " [True],\n", + " [True],\n", + " [True],\n", + " [True]], device='cuda:0')" + ] + }, + "execution_count": 40, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Evaluate loaded model\n", + "loaded_model_1.eval()\n", + "with torch.inference_mode():\n", + " loaded_model_1_preds = loaded_model_1(X_test)\n", + "y_preds == loaded_model_1_preds" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7M_kcRC89YrZ" + }, + "source": [ + "Everything adds up! Nice!\n" + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "provenance": [], + "include_colab_link": true + }, + "interpreter": { + "hash": "3fbe1355223f7b2ffc113ba3ade6a2b520cadace5d5ec3e828c83ce02eb221bf" + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.4" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} \ No newline at end of file