10-SequenceModels/EVA_P2S3.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "EVA_P2S3.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/satyajitghana/TSAI-DeepVision-EVA4.0-Phase-2/blob/master/10-SequenceModels/EVA_P2S3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jofyc9OC4Qcf"
      },
      "source": [
        "#Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ahBVnrNc3E0U"
      },
      "source": [
        "import numpy as np\n",
        "%matplotlib inline\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import gdown\n",
        "from IPython import display\n",
        "plt.style.use('seaborn-white')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "crQSAaIz4SkA"
      },
      "source": [
        "# Read and process data. \n",
        "\n",
        "Download the file from this URL: https://drive.google.com/file/d/1UWWIi-sz9g0x3LFvkIZjvK1r2ZaCqgGS/view?usp=sharing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JC5gqjf6E4nf",
        "outputId": "9a9f78dc-7946-4c00-bc67-77890be3d602",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 108
        }
      },
      "source": [
        "url = 'https://drive.google.com/uc?id=1UWWIi-sz9g0x3LFvkIZjvK1r2ZaCqgGS'\n",
        "output = 'text.txt'\n",
        "gdown.download(url, output, quiet=False)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1UWWIi-sz9g0x3LFvkIZjvK1r2ZaCqgGS\n",
            "To: /content/text.txt\n",
            "100%|██████████| 10.3k/10.3k [00:00<00:00, 2.14MB/s]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'text.txt'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rgOGxPDP3Wpp",
        "outputId": "6875d515-bb85-4f77-f92d-477184f3bc53",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 108
        }
      },
      "source": [
        "data = open('text.txt', 'r').read()\n",
        "data"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'A coronavirus is a kind of common virus that causes an infection in your nose, sinuses, or upper throat. Most coronaviruses are not dangerous.\\n\\nSome types of coronaviruses are serious, though. About 858 people have died from Middle East respiratory syndrome (MERS), which first appeared in 2012 in Saudi Arabia and then in other countries in the Middle East, Africa, Asia, and Europe. In April 2014, the first American was hospitalized for MERS in Indiana and another case was reported in Florida. Both had just returned from Saudi Arabia. In May 2015, there was an outbreak of MERS in Korea, which was the largest outbreak outside of the Arabian Peninsula. In 2003, 774 people died from a severe acute respiratory syndrome (SARS) outbreak. As of 2015, there were no further reports of cases of SARS. \\n\\nBut in early 2020, following a December 2019 outbreak in China, the World Health Organization identified a new type, 2019 novel coronavirus (2019-nCoV).\\n\\nOften a coronavirus causes upper respiratory infection symptoms like a stuffy nose, cough, and sore throat. You can treat them with rest and over-the-counter medication. The coronavirus can also cause middle ear infections in children.\\n\\nWhat Is a Coronavirus?\\nCoronaviruses were first identified in the 1960s, but we don\\'t know where they come from. They get their name from their crown-like shape. Sometimes, but not often, a coronavirus can infect both animals and humans.\\n\\nMost coronaviruses spread the same way other cold-causing viruses do: through infected people coughing and sneezing, by touching an infected person\\'s hands or face, or by touching things such as doorknobs that infected people have touched.\\n\\nAlmost everyone gets a coronavirus infection at least once in their life, most likely as a young child. In the United States, coronaviruses are more common in the fall and winter, but anyone can come down with a coronavirus infection at any time.\\n\\nCommon Symptoms of Coronavirus\\nThe symptoms of most coronaviruses are similar to any other upper respiratory infection, including runny nose, coughing, sore throat, and sometimes a fever. In most cases, you won\\'t know whether you have a coronavirus or a different cold-causing virus, such as rhinovirus.\\n\\nYou could get lab tests, including nose and throat cultures and blood work, to find out whether your cold was caused by a coronavirus, but there\\'s no reason to. The test results wouldn\\'t change how you treat your symptoms, which typically go away in a few days.\\n\\nBut if a coronavirus infection spreads to the lower respiratory tract (your windpipe and your lungs), it can cause pneumonia, especially in older people, people with heart disease, or people with weakened immune systems.\\n\\nWhat to Do About Coronavirus\\nThere is no vaccine for coronavirus. To help prevent a coronavirus infection, do the same things you do to avoid the common cold:\\n\\nWash your hands thoroughly with soap and warm water or with an alcohol-based hand sanitizer.\\nKeep your hands and fingers away from your eyes, nose, and mouth.\\nAvoid close contact with people who are infected.\\nYou treat a coronavirus infection the same way you treat a cold:\\n\\nGet plenty of rest.\\nDrink fluids.\\nTake over-the-counter medicine for a sore throat and fever. But don\\'t give aspirin to children or teens younger than 19; use ibuprofen or acetaminophen instead.\\nA humidifier or steamy shower can also help ease a sore and scratchy throat.\\n\\nEven when a coronavirus causes MERS or SARS in other countries, the kind of coronavirus infection common in the U.S. isn\\'t a serious threat for an otherwise healthy adult. If you get sick, treat your symptoms and contact a doctor if they get worse or don\\'t go away.\\n\\nNews about the coronavirus outbreak that started in Wuhan, China, is changing rapidly. The respiratory infection, which is closely related to SARS and MERS, has been spreading across China, and cases have been diagnosed in several other countries, including the United States. We’ll provide the latest updates on cases, deaths, travel restrictions, and more here.\\n\\nWhat is the latest news?\\nOn Wednesday, CDC medical officers and others met a group of about 210 U.S. citizens evacuated from China. Their plane landed at March Air Reserve Base in California, where the evacuees will be monitored for coronavirus symptoms for several days. Anyone showing signs of the disease will be taken to the hospital.\\n\\nSome airlines, including British Airways and United Airlines, are stopping or reducing flights to China as the number of cases continues to rise rapidly. Worldwide there are now more than 6,000 cases and 132 deaths, according to the European Centre for Disease Prevention and Control. While the majority of cases center in China, it has been found in the U.S. and these countries: Thailand, Hong Kong, Taiwan, Australia, Macau, Singapore, Japan, South Korea, Malaysia, France, Canada, Vietnam, Nepal, Cambodia, Germany and United Arab Emirates. There have been no deaths outside China.\\n\\nIn the U.S., the CDC is reporting five confirmed cases from these states: Washington, Illinois, California and Arizona. On Monday, Jan. 27, Nancy Messonnier, MD, Director of the agency\\'s National Center for Immunization and Respiratory Diseases, said it had 165 persons under investigation for coronavirus from 36 states. In addition to the 5 confirmed positive, 68 have tested negative. They are prioritizing the testing based on a person’s risk.\\n\\nMessonnier said they had posted the blueprints for their diagnostic test on a public server and were working \"as fast as we can\" to get test kits out to states. \\n\\nRight now, all the testing for the new coronavirus is taking place at the CDC\\'s headquarters in Atlanta.\\n\\nShe said the CDC was mulling a change to its travel screening for the infection, but did not give further details.\\n\\nHow many people have been diagnosed with the virus, and how many have died?\\nAccording to European CDC, the majority of the confirmed cases – 5,994 – are in China. Another 78 cases are confirmed outside of China in 16 countries. Countries with the most confirmed cases include Thailand with 14 and Taiwan and South Korea with 8 each. All reported deaths have been in China, and include 16 healthcare workers.\\n\\nWhen did the outbreak start?\\nChina first reported the outbreak in Wuhan on Dec. 30, 2019.\\n\\nWhat are public officials doing to contain the virus?\\nOn Tuesday, the CDC urged travelers to avoid non-essential travel to China.\\n\\nChinese officials have shut down all public transportation to 10 cities, affecting 35 million people. The first was Wuhan, which has a population of about 11 million. In Wuhan, that includes buses, subways, trains, and the airport.\\n\\nAll passengers flying into the United States from Wuhan will be routed through one of five airports and screened for fever and other signs of the virus. “Patients of concern��� will be referred to a facility and given a test to find out if they have the virus.\\n\\nWhat do we know about cases in the United States?\\nIn all U.S. cases so far, patients had recently traveled to Wuhan.\\n\\nCalifornia has two patients, one in Los Angeles County and one in Orange County. The patient from Orange County is a man in his 50s. He is in a local hospital in isolation and is in good condition, according to the Orange County Health Care Agency. Los Angeles County officials did not provide additional details about the patient there.\\n\\nArizona\\'s Department of Health Services said its patient is a Maricopa County resident and member of the Arizona State University community who did not live in student housing. The patient is not severely ill and is being kept in isolation. \\n\\nAnother case involves a woman in her 60s from Chicago. The Chicago Department of Public Health reported that she had visited Wuhan, China in December and returned to Chicago earlier this month. She is hospitalized in stable condition. \\n\\nThe first U.S. patient is a man in his 30s from Washington state. He had traveled from Wuhan and entered the country before the screening was in place. He started having symptoms and contacted his doctor. He is in good condition and is in isolation at Providence Regional Medical Center.\\n\\nIs travel to China safe?\\nWuhan is closed to travelers.\\n\\nThe CDC is advising that travelers avoid non-essential travel to China. Travelers who do go should:\\n\\nAvoid contact with sick people.\\nAvoid animals, animal markets, and products that come from animals.\\nWash their hands often with soap and water, or use an alcohol-based sanitizer if that’s not available.\\nSeek medical care right away for a fever, cough, or difficulty breathing. Tell a health care professional about any travel.\\nWhat are the symptoms, and how is the virus diagnosed?\\nChina created a test for the virus and shared that information with other countries. The CDC has developed its own test.\\n\\nSymptoms include a fever, coughing, and shortness of breath. They may appear 2 to 14 days after you’re exposed to the virus.\\n\\nWhat is the source of the virus, and how is it spread?\\nHealth officials are not sure of the source of the virus yet or how easily it can spread. Coronaviruses are found in many different animals, including camels, cattle, cats, and bats. One research paper also suggested snakes as a possible source. The new virus may be linked to a seafood and live animal market in Wuhan that has since been closed\\n\\nThe virus can spread from person to person. Health officials are seeing this happen most often where people are close together and in health care settings. To date, 16 health care workers have been infected.\\n\\nThe CDC believes that severe acute respiratory syndrome (SARS) and Middle East respiratory syndrome (MERS), two other types of coronavirus, are spread through droplets when someone coughs or sneezes.\\n\\nIs there a vaccine?\\nThere is no vaccine, but the National Institutes of Health is working on one and hopes to begin testing in several months. That testing would be for safety. If it’s safe, there would be testing to see how well it works.\\n\\nHow is it treated?\\nThere is no specific treatment for the virus. Patients are generally given supportive care for their symptoms, such a fluids and pain relievers. Hospitalized patients may need support with breathing.'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZeXXMLRb4kXb"
      },
      "source": [
        "Process data and calculate indices"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E5TKeiOp4jtl",
        "outputId": "a47fa8e1-477c-42f4-9a75-b9e888e0ee8d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "chars = list(set(data))\n",
        "data_size, X_size = len(data), len(chars)\n",
        "print(\"Corona Virus article has %d characters, %d unique characters\" %(data_size, X_size))\n",
        "char_to_idx = {ch:i for i,ch in enumerate(chars)}\n",
        "idx_to_char = {i:ch for i,ch in enumerate(chars)}"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Corona Virus article has 10223 characters, 75 unique characters\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4C53MB135LRY"
      },
      "source": [
        "# Constants and Hyperparameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dfj21ORa49Ps"
      },
      "source": [
        "Hidden_Layer_size = 10 #size of the hidden layer\n",
        "Time_steps = 10 # Number of time steps (length of the sequence) used for training\n",
        "learning_rate = 1e-1 # Learning Rate\n",
        "weight_sd = 0.1 #Standard deviation of weights for initialization\n",
        "z_size = Hidden_Layer_size + X_size #Size of concatenation(H, X) vector"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OdmJf4Du5uhb"
      },
      "source": [
        "# Activation Functions and Derivatives"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "seGHei_D5FGk"
      },
      "source": [
        "def sigmoid(x): # sigmoid function\n",
        "    z = 1 / (1 + np.exp(-x))\n",
        "    return z\n",
        "\n",
        "def dsigmoid(y): # derivative of sigmoid function\n",
        "    dz = y * (1 - y)\n",
        "    return dz\n",
        "\n",
        "def tanh(x): # tanh function\n",
        "    z = np.tanh(x)\n",
        "    return z\n",
        "\n",
        "def dtanh(y): # derivative of tanh\n",
        "    dz = 1 - y * y\n",
        "    return dz"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6JsxZVC8J2Ek"
      },
      "source": [
        "#### Sigmoid\n",
        "\n",
        "\\begin{align}\n",
        "\\sigma(x) &= \\frac{1}{1 + e^{-x}}\\\\\n",
        "\\frac{d\\sigma(x)}{dx} &= \\sigma(x) \\cdot (1 - \\sigma(x))\n",
        "\\end{align}\n",
        "\n",
        "#### Tanh\n",
        "\n",
        "\\begin{align}\n",
        "\\frac{d\\text{tanh}(x)}{dx} &= 1 - \\text{tanh}^2(x)\n",
        "\\end{align}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OXpD9BCGHUPJ"
      },
      "source": [
        "# Quiz Question 1\n",
        "\n",
        "What is the value of sigmoid(0) calculated from  your code? (Answer up to 1 decimal point, e.g. 4.2 and NOT 4.29999999, no rounding off)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nt60ZTotHVRA",
        "outputId": "211e59dd-1244-4c2b-91d4-2c8582a85c2a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "sigmoid(0)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.5"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 111
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GKBqcNqyHcbp"
      },
      "source": [
        "# Quiz Question 2\n",
        "\n",
        "What is the value of dsigmoid(sigmoid(0)) calculated from your code?? (Answer up to 2 decimal point, e.g. 4.29 and NOT 4.29999999, no rounding off). "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zg8bvJYYHfwx",
        "outputId": "a5f88139-dc39-403c-bc64-ad58901e8552",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "dsigmoid(sigmoid(0))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.25"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 112
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7UAfxFA8HdUg"
      },
      "source": [
        "# Quiz Question 3\n",
        "\n",
        "What is the value of tanh(dsigmoid(sigmoid(0))) calculated from your code?? (Answer up to 5 decimal point, e.g. 4.29999 and NOT 4.29999999, no rounding off)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-GA_BOsnHnub",
        "outputId": "433dd60e-7e6c-41da-f1f7-e5fa2009f821",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "tanh(dsigmoid(sigmoid(0)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.24491866240370913"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 113
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KeCvVH1v6Me-"
      },
      "source": [
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "# Quiz Question 4\n",
        "\n",
        "What is the value of dtanh(tanh(dsigmoid(sigmoid(0)))) calculated from your code?? (Answer up to 5 decimal point, e.g. 4.29999 and NOT 4.29999999, no rounding off)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "89-fjLhgFmnV",
        "outputId": "c4ec8c6c-c516-4f99-84cc-6f15b1ca633b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "dtanh(tanh(dsigmoid(sigmoid(0))))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.940014848806378"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 114
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EeSVipDu8iKE"
      },
      "source": [
        "# Parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ICbWNemE6LGV"
      },
      "source": [
        "class Param:\n",
        "    def __init__(self, name, value):\n",
        "      self.name = name\n",
        "      self.v = value # parameter value\n",
        "      self.d = np.zeros_like(value) # derivative\n",
        "      self.m = np.zeros_like(value) # momentum for Adagrad"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j83pZNPE8212"
      },
      "source": [
        "We use random weights with normal distribution (0, weight_sd) for  tanh  activation function and (0.5, weight_sd) for  `sigmoid`  activation function.\n",
        "\n",
        "Biases are initialized to zeros."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "swHwLXOI9E7V"
      },
      "source": [
        "# LSTM \n",
        "You are making this network, please note f, i, c and o (also \"v\") in the image below:\n",
        "![alt text](http://blog.varunajayasiri.com/ml/lstm.svg)\n",
        "\n",
        "Please note that we are concatenating the old_hidden_vector and new_input."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A0DBzNY-90s5"
      },
      "source": [
        "# Quiz Question 4\n",
        "\n",
        "In the class definition below, what should be size_a, size_b, and size_c? ONLY use the variables defined above."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SFuHhqVq6Wge"
      },
      "source": [
        "size_a = Hidden_Layer_size\n",
        "size_b = z_size\n",
        "size_c = X_size\n",
        "\n",
        "class Parameters:\n",
        "    def __init__(self):\n",
        "        self.W_f = Param('W_f', np.random.randn(size_a, size_b) * weight_sd + 0.5)\n",
        "        self.b_f = Param('b_f', np.zeros((size_a, 1)))\n",
        "\n",
        "        self.W_i = Param('W_i', np.random.randn(size_a, size_b) * weight_sd + 0.5)\n",
        "        self.b_i = Param('b_i', np.zeros((size_a, 1)))\n",
        "\n",
        "        self.W_C = Param('W_C', np.random.randn(size_a, size_b) * weight_sd)\n",
        "        self.b_C = Param('b_C', np.zeros((size_a, 1)))\n",
        "\n",
        "        self.W_o = Param('W_o', np.random.randn(size_a, size_b) * weight_sd + 0.5)\n",
        "        self.b_o = Param('b_o', np.zeros((size_a, 1)))\n",
        "\n",
        "        #For final layer to predict the next character\n",
        "        self.W_v = Param('W_v', np.random.randn(X_size, size_a) * weight_sd)\n",
        "        self.b_v = Param('b_v', np.zeros((size_c, 1)))\n",
        "        \n",
        "    def all(self):\n",
        "        return [self.W_f, self.W_i, self.W_C, self.W_o, self.W_v,\n",
        "               self.b_f, self.b_i, self.b_C, self.b_o, self.b_v]\n",
        "        \n",
        "parameters = Parameters()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RzmfGLZt_xVs"
      },
      "source": [
        "Look at these operations which we'll be writing:\n",
        "\n",
        "**Concatenation of h and x:**\n",
        "\n",
        "$z\\:=\\:\\left[h_{t-1},\\:x\\right]$\n",
        "\n",
        "$f_t=\\sigma\\left(W_f\\cdot z\\:+\\:b_f\\:\\right)$\n",
        "\n",
        "$i_i=\\sigma\\left(W_i\\cdot z\\:+\\:b_i\\right)$\n",
        "\n",
        "$\\overline{C_t}=\\tanh\\left(W_C\\cdot z\\:+\\:b_C\\right)$\n",
        "\n",
        "$C_t=f_t\\ast C_{t-1}+i_t\\ast \\overline{C}_t$\n",
        "\n",
        "$o_t=\\sigma\\left(W_o\\cdot z\\:+\\:b_i\\right)$\n",
        "\n",
        "$h_t=o_t\\ast\\tanh\\left(C_t\\right)$\n",
        "\n",
        "**Logits:**\n",
        "\n",
        "$v_t=W_v\\cdot h_t+b_v$\n",
        "\n",
        "**Softmax:**\n",
        "\n",
        "$\\hat{y}=softmax\\left(v_t\\right)$\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-bUkseNnDott"
      },
      "source": [
        "def forward(x, h_prev, C_prev, p = parameters):\n",
        "    assert x.shape == (X_size, 1)\n",
        "    assert h_prev.shape == (Hidden_Layer_size, 1)\n",
        "    assert C_prev.shape == (Hidden_Layer_size, 1)\n",
        "    \n",
        "    z = np.row_stack((h_prev, x))\n",
        "    f = sigmoid(np.dot(p.W_f.v, z) + p.b_f.v)\n",
        "    i = sigmoid(np.dot(p.W_i.v, z) + p.b_i.v)\n",
        "    C_bar = tanh(np.dot(p.W_C.v, z) + p.b_C.v)\n",
        "\n",
        "    C = f * C_prev + i * C_bar\n",
        "    o = sigmoid(np.dot(p.W_o.v, z) + p.b_o.v)\n",
        "    h = o * tanh(C)\n",
        "\n",
        "    v = np.dot(p.W_v.v, h) + p.b_v.v\n",
        "    y = np.exp(v) / np.sum(np.exp(v)) # softmax\n",
        "\n",
        "    return z, f, i, C_bar, C, o, h, v, y"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jZrDhZIjFpdI"
      },
      "source": [
        "You must finish the function above before you can attempt the questions below. \n",
        "\n",
        "# Quiz Question 5\n",
        "\n",
        "What is the output of 'print(len(forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)), parameters)))'?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xWcFcDsfJHog",
        "outputId": "819256cd-ff19-44c1-baed-7141469e8b3d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "print(len(forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)), parameters)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "9\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XV-YVl_GGiX8"
      },
      "source": [
        "# Quiz Question 6. \n",
        "\n",
        "Assuming you have fixed the forward function, run this command: \n",
        "z, f, i, C_bar, C, o, h, v, y = forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)))\n",
        "\n",
        "Now, find these values:\n",
        "\n",
        "\n",
        "1.   print(z.shape)\n",
        "2.   print(np.sum(z))\n",
        "3.   print(np.sum(f))\n",
        "\n",
        "Copy and paste exact values you get in the logs into the quiz.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1GvKVWmTDt3H"
      },
      "source": [
        "z, f, i, C_bar, C, o, h, v, y = forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EVy696fNJMse",
        "outputId": "9767ad1a-2413-45bd-fe2f-3af76866c6f4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "z.shape, np.sum(z), np.sum(f)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((85, 1), 0.0, 5.0)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NeSvhkqwILsG"
      },
      "source": [
        "# Backpropagation\n",
        "\n",
        "Here we are defining the backpropagation. It's too complicated, here is the whole code. (Please note that this would work only if your earlier code is perfect)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zIa1jUZiGPmF"
      },
      "source": [
        "def backward(target, dh_next, dC_next, C_prev,\n",
        "             z, f, i, C_bar, C, o, h, v, y,\n",
        "             p = parameters):\n",
        "    \n",
        "    assert z.shape == (X_size + Hidden_Layer_size, 1)\n",
        "    assert v.shape == (X_size, 1)\n",
        "    assert y.shape == (X_size, 1)\n",
        "    \n",
        "    for param in [dh_next, dC_next, C_prev, f, i, C_bar, C, o, h]:\n",
        "        assert param.shape == (Hidden_Layer_size, 1)\n",
        "        \n",
        "    dv = np.copy(y)\n",
        "    dv[target] -= 1\n",
        "\n",
        "    p.W_v.d += np.dot(dv, h.T)\n",
        "    p.b_v.d += dv\n",
        "\n",
        "    dh = np.dot(p.W_v.v.T, dv)        \n",
        "    dh += dh_next\n",
        "    do = dh * tanh(C)\n",
        "    do = dsigmoid(o) * do\n",
        "    p.W_o.d += np.dot(do, z.T)\n",
        "    p.b_o.d += do\n",
        "\n",
        "    dC = np.copy(dC_next)\n",
        "    dC += dh * o * dtanh(tanh(C))\n",
        "    dC_bar = dC * i\n",
        "    dC_bar = dtanh(C_bar) * dC_bar\n",
        "    p.W_C.d += np.dot(dC_bar, z.T)\n",
        "    p.b_C.d += dC_bar\n",
        "\n",
        "    di = dC * C_bar\n",
        "    di = dsigmoid(i) * di\n",
        "    p.W_i.d += np.dot(di, z.T)\n",
        "    p.b_i.d += di\n",
        "\n",
        "    df = dC * C_prev\n",
        "    df = dsigmoid(f) * df\n",
        "    p.W_f.d += np.dot(df, z.T)\n",
        "    p.b_f.d += df\n",
        "\n",
        "    dz = (np.dot(p.W_f.v.T, df)\n",
        "         + np.dot(p.W_i.v.T, di)\n",
        "         + np.dot(p.W_C.v.T, dC_bar)\n",
        "         + np.dot(p.W_o.v.T, do))\n",
        "    dh_prev = dz[:Hidden_Layer_size, :]\n",
        "    dC_prev = f * dC\n",
        "    \n",
        "    return dh_prev, dC_prev"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tnc7WpRkIU5S"
      },
      "source": [
        "# Forward and Backward Combined Pass\n",
        "\n",
        "Let's first clear the gradients before each backward pass"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OJWoC3U1ITf8"
      },
      "source": [
        "def clear_gradients(params = parameters):\n",
        "    for p in params.all():\n",
        "        p.d.fill(0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7XN93UnjIgmA"
      },
      "source": [
        "Clip gradients to mitigate exploding gradients"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0LTsublxIfFl"
      },
      "source": [
        "def clip_gradients(params = parameters):\n",
        "    for p in params.all():\n",
        "        np.clip(p.d, -1, 1, out=p.d)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T7XUpDTWIl_Y"
      },
      "source": [
        "Calculate and store the values in forward pass. Accumulate gradients in backward pass and clip gradients to avoid exploding gradients.\n",
        "\n",
        "input, target are list of integers, with character indexes.\n",
        "h_prev is the array of initial h at  h−1  (size H x 1)\n",
        "C_prev is the array of initial C at  C−1  (size H x 1)\n",
        "Returns loss, final  hT  and  CT"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CQNxjTuZIia_"
      },
      "source": [
        "def forward_backward(inputs, targets, h_prev, C_prev):\n",
        "    global paramters\n",
        "    \n",
        "    # To store the values for each time step\n",
        "    x_s, z_s, f_s, i_s,  = {}, {}, {}, {}\n",
        "    C_bar_s, C_s, o_s, h_s = {}, {}, {}, {}\n",
        "    v_s, y_s =  {}, {}\n",
        "    \n",
        "    # Values at t - 1\n",
        "    h_s[-1] = np.copy(h_prev)\n",
        "    C_s[-1] = np.copy(C_prev)\n",
        "    \n",
        "    loss = 0\n",
        "    # Loop through time steps\n",
        "    assert len(inputs) == Time_steps\n",
        "    for t in range(len(inputs)):\n",
        "        x_s[t] = np.zeros((X_size, 1))\n",
        "        x_s[t][inputs[t]] = 1 # Input character\n",
        "        \n",
        "        (z_s[t], f_s[t], i_s[t],\n",
        "        C_bar_s[t], C_s[t], o_s[t], h_s[t],\n",
        "        v_s[t], y_s[t]) = \\\n",
        "            forward(x_s[t], h_s[t - 1], C_s[t - 1]) # Forward pass\n",
        "            \n",
        "        loss += -np.log(y_s[t][targets[t], 0]) # Loss for at t\n",
        "        \n",
        "    clear_gradients()\n",
        "\n",
        "    dh_next = np.zeros_like(h_s[0]) #dh from the next character\n",
        "    dC_next = np.zeros_like(C_s[0]) #dh from the next character\n",
        "\n",
        "    for t in reversed(range(len(inputs))):\n",
        "        # Backward pass\n",
        "        dh_next, dC_next = \\\n",
        "            backward(target = targets[t], dh_next = dh_next,\n",
        "                     dC_next = dC_next, C_prev = C_s[t-1],\n",
        "                     z = z_s[t], f = f_s[t], i = i_s[t], C_bar = C_bar_s[t],\n",
        "                     C = C_s[t], o = o_s[t], h = h_s[t], v = v_s[t],\n",
        "                     y = y_s[t])\n",
        "\n",
        "    clip_gradients()\n",
        "        \n",
        "    return loss, h_s[len(inputs) - 1], C_s[len(inputs) - 1]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tcy5u_vRItkV"
      },
      "source": [
        "# Sample the next character"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p8SrtJiwIsSm"
      },
      "source": [
        "def sample(h_prev, C_prev, first_char_idx, sentence_length):\n",
        "    x = np.zeros((X_size, 1))\n",
        "    x[first_char_idx] = 1\n",
        "\n",
        "    h = h_prev\n",
        "    C = C_prev\n",
        "\n",
        "    indexes = []\n",
        "    \n",
        "    for t in range(sentence_length):\n",
        "        _, _, _, _, C, _, h, _, p = forward(x, h, C)\n",
        "        idx = np.random.choice(range(X_size), p=p.ravel())\n",
        "        x = np.zeros((X_size, 1))\n",
        "        x[idx] = 1\n",
        "        indexes.append(idx)\n",
        "\n",
        "    return indexes"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SiWFaWLNIx_L"
      },
      "source": [
        "# Training (Adagrad)\n",
        "\n",
        "Update the graph and display a sample output\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ENQYU-7AIw0t"
      },
      "source": [
        "def update_status(inputs, h_prev, C_prev):\n",
        "    #initialized later\n",
        "    global plot_iter, plot_loss\n",
        "    global smooth_loss\n",
        "    \n",
        "    # Get predictions for 200 letters with current model\n",
        "\n",
        "    sample_idx = sample(h_prev, C_prev, inputs[0], 200)\n",
        "    txt = ''.join(idx_to_char[idx] for idx in sample_idx)\n",
        "\n",
        "    # Clear and plot\n",
        "    plt.plot(plot_iter, plot_loss)\n",
        "    display.clear_output(wait=True)\n",
        "    plt.show()\n",
        "\n",
        "    #Print prediction and loss\n",
        "    print(\"----\\n %s \\n----\" % (txt, ))\n",
        "    print(\"iter %d, loss %f\" % (iteration, smooth_loss))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ACXcASJuI73a"
      },
      "source": [
        "# Update Parameters\n",
        "\n",
        "\\begin{align}\n",
        "\\theta_i &= \\theta_i - \\eta\\frac{d\\theta_i}{\\sum dw_{\\tau}^2} \\\\\n",
        "d\\theta_i &= \\frac{\\partial L}{\\partial \\theta_i}\n",
        "\\end{align}"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bR08TvcjI4Pf"
      },
      "source": [
        "def update_paramters(params = parameters):\n",
        "    for p in params.all():\n",
        "        p.m += p.d * p.d # Calculate sum of gradients\n",
        "        #print(learning_rate * dparam)\n",
        "        p.v += -(learning_rate * p.d / np.sqrt(p.m + 1e-8))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "La9vyJ6RJLFK"
      },
      "source": [
        "To delay the keyboard interrupt to prevent the training from stopping in the middle of an iteration\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZVDHbMb7JNGT"
      },
      "source": [
        "# Exponential average of loss\n",
        "# Initialize to a error of a random model\n",
        "smooth_loss = -np.log(1.0 / X_size) * Time_steps\n",
        "\n",
        "iteration, pointer = 0, 0\n",
        "\n",
        "# For the graph\n",
        "plot_iter = np.zeros((0))\n",
        "plot_loss = np.zeros((0))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HF6vS0VWJqsS"
      },
      "source": [
        "# Training Loop"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OQyNSL0iJOxH",
        "outputId": "129a73bd-df7d-41ba-e36e-0887f73a721e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 353
        }
      },
      "source": [
        "iter = 1000\n",
        "while iter > 0:\n",
        "  # Reset\n",
        "  if pointer + Time_steps >= len(data) or iteration == 0:\n",
        "      g_h_prev = np.zeros((Hidden_Layer_size, 1))\n",
        "      g_C_prev = np.zeros((Hidden_Layer_size, 1))\n",
        "      pointer = 0\n",
        "\n",
        "\n",
        "  inputs = ([char_to_idx[ch] \n",
        "              for ch in data[pointer: pointer + Time_steps]])\n",
        "  targets = ([char_to_idx[ch] \n",
        "              for ch in data[pointer + 1: pointer + Time_steps + 1]])\n",
        "\n",
        "  loss, g_h_prev, g_C_prev = \\\n",
        "      forward_backward(inputs, targets, g_h_prev, g_C_prev)\n",
        "  smooth_loss = smooth_loss * 0.999 + loss * 0.001\n",
        "\n",
        "  # Print every hundred steps\n",
        "  if iteration % 100 == 0:\n",
        "      update_status(inputs, g_h_prev, g_C_prev)\n",
        "\n",
        "  update_paramters()\n",
        "\n",
        "  plot_iter = np.append(plot_iter, [iteration])\n",
        "  plot_loss = np.append(plot_loss, [loss])\n",
        "\n",
        "  pointer += Time_steps\n",
        "  iteration += 1\n",
        "  iter = iter -1"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWsAAAD1CAYAAACWXdT/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO2deWAURfr3vz2ThCRc4QggEMCDIwqCiq6oIAFEUVc8VsGI7OG5LiyHiOiyqOu7gnj8UFnFA1FkV9Gwy+rqAoLERQ1BBFFQDEGRcIVAIAnknun3j5nqqe6u6mMyk0yY5/NPMn1UV1d3P/XUU089j6KqqgqCIAgipvE0dQUIgiAIe0hYEwRBNANIWBMEQTQDSFgTBEE0A0hYEwRBNAMSolFodXU1tm/fjvT0dHi93mhcgiAI4pTD5/OhpKQE/fv3R3Jysm5fVIT19u3bcdttt0WjaIIgiFOev//97xg8eLBuW1SEdXp6unbBLl26ROMSBEEQpxyHDh3CbbfdpslQnqgIa2b66NKlC7p37x6NSxAEQZyyiMzHNMFIEATRDCBhTRAE0QwgYU0QBNEMIGFNEATRDCBhTRAE0QwgYU0QBNEMIGFNEM2Mkooa9Jr1IdZ9X9zUVSEaERLWBNHM2H6gDACwNO/nJq4J0ZiQsCYIgmgGkLAmCIJoBpCwJgiCaAbYxgbJz8/HlClT0Lt3bwBAnz59cPLkSezYsQNpaWkAgDvuuAPDhw+PakUJgiDiGUeBnC666CI8//zz2u9Zs2Zh+vTpyMrKilrFCIIgiBBkBiEIgmgGOBLWhYWFuPfee3Hrrbfi888/BwAsW7YMEydOxLRp01BaWhrVShIEQcQ7tmaQXr16YdKkSRgzZgyKioowceJEPP744+jYsSMyMzPxyiuvYOHChZgzZ05j1JcgCCIusdWsO3fujKuvvhqKoqBHjx7o2LEjevXqhczMTADAiBEjUFBQEPWKEgRBxDO2wvr999/H4sWLAQAlJSU4evQo5s2bh6KiIgABbxHmKUIQBEFEB1szyIgRIzBjxgysW7cOdXV1ePTRR9GiRQtMnToVKSkpSE1Nxdy5cxujrgRBEHGLrbBu1aoVFi1aZNq+YsWKqFSIIAhnqE1dAaJRIdc9giCIZgAJa4IgiGYACWuCIIhmAAlrgmimKE1dAaJRIWFNEATRDCBhTRAE0QwgYU0QBNEMIGFNEATRDCBhTRAE0QwgYU0QBNEMIGFNEATRDCBhTRAE0QwgYU0QBNEMIGFNEATRDCBhTRDNFAqRGl+QsCYIgmgG2CYfyM/Px5QpU7TUXX369MGdd96JmTNnwufzIT09HU899RSSkpKiXlmCIIh4xVZYA8BFF12E559/Xvv90EMPITs7G2PGjMGzzz6LnJwcZGdnR62SBEEQ8U5YZpD8/HyMHDkSAJCVlYW8vLyIVoogCILQ40izLiwsxL333ouysjJMmjQJVVVVmtmjQ4cOKCkpiWolCYIg4h1bYd2rVy9MmjQJY8aMQVFRESZOnAifz6ftV1WakyYIgog2tmaQzp074+qrr4aiKOjRowc6duyIsrIyVFdXAwCKi4vRqVOnqFeUIAginrEV1u+//z4WL14MACgpKcHRo0dx4403YvXq1QCANWvWYOjQodGtJUEQRJxjawYZMWIEZsyYgXXr1qGurg6PPvooMjMz8eCDD2L58uXo2rUrrr/++saoK0EQRNxiK6xbtWqFRYsWmbYvWbIkKhUiCIIgzNAKRoIgiGYACWuCaKYoTV0BolGJG2Ht96v4zzcH4POTqyFBEM2PuBHW731VhEn/2Iq38vY0dVXijr1HK7H/eFVTV4MgmjWOVjCeCpRU1AT+nqhp4prEH8OeWg8A2DPvmiauyakFjRHji7jRrAmCIJozcSOsaVU8QRDNmbgR1gRBEM0ZEtYEQRDNgLgR1go5pRIE0YxpFsL6ppe+wKPv72jqahAEQTQZzUJYf/XzMbzxxZ6mrgZBEEST0SyENUEQRLxDwpogCKIZQMKaIAiiGeBouXl1dTWuvfZa3Hfffdi0aRN27NiBtLQ0AMAdd9yB4cOHR7OOBEEQcY8jYf3SSy+hbdu22u/p06cjKysrapWKBrSCkSCI5oytGWT37t0oLCw8ZbRnhaIAEwTRDLEV1k8++SRmzZql27Zs2TJMnDgR06ZNQ2lpadQqRxAEQQSwFNYrV67EoEGDkJGRoW0bO3YsZsyYgaVLlyIzMxMLFy6MeiUJwojfr8Ifp4kkaGwYn1gK69zcXKxbtw633HIL3nvvPbz44otQVRWZmZkAApnPCwoKGqWiPHuOnERRaWWjX5eIHc5+ZBVGPJPb1NUgiEbDcoJxwYIF2v8vvPACunXrhrfffhsZGRnIyMhAfn4+evfuHfVKGhn+dC4ACmYfz1TX+bHnKHXYRPzgOlPMbbfdhqlTpyIlJQWpqamYO3duNOpFEARBcDgW1pMnT9b+X7FiRVQqQxAEQYiJuxWMqk3mugdzvsGUd7Y2Um0Iwj3xOa1KxI2wdhrPevnmIvz76wPRrQxBEIRLYk5Y+/0qaup9TV0NqLTkkSCIGCLmhPVLn+7G2IWfC/e9/OlulFXVhVWuW9l7vDK86xAEQUQD194g0aaiuh67S05AVVUoBtvF3P/uxPcHyxtUvtPl5j7SrAmCiCFiTrNuk5KAOp+K6jq/cH95dX2j1MNPwpogiBgi5oR125REAEB5tdgM0Wi2ZJLVBEHEEDEnrNskB4W1xDbdWDKUZDVBELFE7AnroGYtm0hsNMWapDURq9C7GXOcqKnHgeNVUb1GzAnrVi0Cc54napzbpo+eqMFLubsjaiKR2axVVcXXRccjdh2CIJo/173wGS6Z90lUrxFzwrplCy8AoLJW7GstEqEz3tuGJ1ftxNYIClGZ2F+a9zOu/9vnyP3hcMSuRRBusFuFSzQ+Px45GfVrxJ6wTgpo1iddaNbMQySS8Y1lWnpBcQUAoOhYdIc8BEEQPDEnrFOTbDRrgRBlJgujX3ZDIJs1EavQuxmfxJywbhm0WZ+sda5ZM4XaYyGr3b7f9EEQBBFLxJywbpHggUcBKmtkmrVoW2Cjx4Fm7VT5JrsgEauQIhGfxJywVhQFLZMSXGrWzAxif2y9X8Xf1heiSmJmYdh+EPTFEATRiDiKDVJdXY1rr70W9913H4YMGYKZM2fC5/MhPT0dTz31FJKSkiJaqRaJXlTXybxBzEJS1cwg9tL6n1v2obi8BuXVdXhoTKb0OFpuTsQq9GbGJ44065deeglt27YFADz//PPIzs7GP/7xD/Ts2RM5OTkRr1SiV0G9z/kryWzWVrKa7WITlzIzC8PJ1XvN+hAP/fMbB0cSBEE0DFthvXv3bhQWFmL48OEAgPz8fIwcORIAkJWVhby8vIhXKsGrwCdxw/ML4ju5sVk7xali/famoohdk3CPm8VTpwoUaz0+sRXWTz75JGbNmqX9rqqq0sweHTp0QElJScQrleDxoC4orI0vpih0qRubtVP465ZV1mHB2oKI+nETkeGhf37b1FUgiEbB0ma9cuVKDBo0CBkZGcL90erhEzwKfEEV2ngJkcB0I0OdynO+yEfe346VXx/AgG5tnV+IaBT2H6ts6io0OqQyxCeWwjo3NxdFRUXIzc3FoUOHkJSUhNTUVFRXVyM5ORnFxcXo1KlTxCvl9SioC9qsjS+mSLNmnYbIRGI61mEd+MswO3edz8EFiEaFBBcRL1gK6wULFmj/v/DCC+jWrRu2bt2K1atXY+zYsVizZg2GDh0a+UpxNmuj9s5r1kWllchon6oJVje+0XYmE5k3CDtv274yx9ciokc8mm/j8Z6JMPysJ0+ejJUrVyI7OxvHjx/H9ddfH/FKJXg8mhZrNHHUcxuGzl8fPIYJ9tBx3x0oR9/Z/8Whsuqw6mD3QeR8tS+sconIEp9yKz7vOt5xnINx8uTJ2v9LliyJSmUYAZs1M4MYJhgtbNa8gH3ziz2oqfdj/Q+HcetFPdwvN+fOiOTEZbxTePgEerRPRVJChNZjkZpJxAkxt4IRCJhBmJ+18VsUC2uzYGf/hytnSQZYU1vvx7KNP0tdLEUcOVGDUc9+ij+v3K7bPvntrfhfgbVX0Za9xzDvvztN27ftK8P5j38sXUR1KkLvZnwSm8La40G9ZLZQPMEY+MvLjVBwJ7G4Nm698K9rMfZvn5vKdEpJRY27E5o5r274EbNXbsd7m537mVcEQ9lu/OmobvsH2w5g4uubLM+98cUvsOjT3cJ9pSdrsS8OvUKI+CI2hbVX0WzTTlz3VM1mzWnW7F8L1frIiRqc8dCH+HJPKUoqarCNS17gZrJy00+luPCva/HhNwcdn9NY1Pv8qKmPvNZ5vLIWgDyxsQjRo6AFHu6hFotPYlNYe0JmEKNXRr2FzZrfZWcGUQFs3nMMfhV4bcOP5v0uvogdBwKeIZsMGmMscOurG9F39qqmroYUt+uMSLg3LfU+P2rryYW1KYhRYe3hJhj1iBfFsG1mzdpqCTrbJXKfdhPIiV0hFsXIl3uORbX8cGSnqjNXuSvgVJfVm/eU2noaNWUbjHluA/rM/m/TVSCOcewN0ph4vQrq/H4UHq7Ahl1HdPusNGv+JWYamExWKwgJcpG2JvoeZB+JopUj3n8qot2zq3PM29y2mfzwU8Nl51eLArF2fnVB9yauiZhdh080dRXilpgU1olB170xz23QVjIyxJoYM5kYt1i73bHMMqIy3QgRVk4kExYcLKvCz0crcfEZHSJWZiRpiGjk28m9Zq028OrNH0qMEZ/EpBnE6/Gg3qeaBDVg52cdnhlEZDd1ZRsNFuTU/rru+2IcO1lreczoZ/+H8a9sdF6HJsJVMwmErFvNWt7GJMCIU5uYFNaJXkXuumdhs9a77kk+Xm5zbb0a/Gu+Fn+2SMhAtz94DnfSruIKPL9ul+nYsso63PHmZty5dLNlmRVxEvrTtWZNQjmuzG1EiJgU1l6PPPmASLNik45fFx3XFleEzCB6Qctvrw3OLFYKFlRI7dMCwR3S3kMn3fxyHp79uACVhvRkdcFO6KcjJ8UXaC40wPSjm1towLkEEU/EpLBO9Ho0QWpEpHGzD/jJVTtDiyuC22Q6cWVtvaZRVwnyPboxg2jmFK5qNXXi+oe08OYtdUSd1uGKapz3lzXYeahcfI7gYZA3iHuoCWKXaH7XMSmsWycn4GRNPRI85q9bZAYRTzmKvUHYz3c37wsJa4FmLbRjS+ob0qvtJyrD8aKIZfj7W/f9YRyrrMOSz/Y4Pz/YpzU06zzlhZBTWVuP//u4gEL8NgLRVCZiUli3TUmEXxW76YkmHa28OZgGGAqjGqI2uLJPZHJxM7z3CFz3FM1MIKa5a4gi4WrVafHozSDiY3v/6SNTDBFALpTjKcGxW+1twdpdeG7dLqygSJHNmpgV1m6wEtZWbnXM1CLqFNw5ELN6iOvGL/fWEiWcgsJF66Bc3JqW7Niwvc6n4q2NP5uOlwkqJ4kn4hU2byIzLRKRI5pf9SkirEXbAht/DE7kqQKhwLR0K3dAwH6IbqVRPrumAH1nr8LJoHeHdkQzl9Ui27s2inFRTih/pjM7CGnWzqmq9eGtvD3Cdz+e8flVvLu5CPVR6LyiabO2XRRTVVWFWbNm4ejRo6ipqcF9992H1atXY8eOHUhLSwMA3HHHHVr280jgVliLpAPb9NTqH3Bu97bab/7Qmjp5ui53WWfkNo/lXwai0pVX16Fli4RmJ1R8fhU/Hz2JM9Jb6bazW3538z5k9euEc7q2DXmIyLwmLTpVx7goO9bw+VU8kLMN9ww7E327tG5weXaC4clVO/HGF3vQqXWLBl/rVOKdL/fiT//ajvKqOtw59IyIlt2kmvX69evRv39/LFu2DAsWLMC8efMAANOnT8dbb72Ft956K6KCGoDrwPRiM0hoW+HhE9rXzB/LciuKYiGHt4JRXjfNZi6wnTcWdT4/Xv3fj66i8D23bhdGPPMpdpeIlxnvLa3EnW8GfMbtbNbC7a5ltWyCsemldVllnWVc7cLDJ/DPLfsx6R9bGnQdp7d6LBgZkb3nlEUjQOmJQLuw9okkTTrBePXVV+Ouu+4CABw8eBCdO3eOXm2CJHgiIawNv4N/eZPHSS0RrrtJSyOhlZC8SUB/Xe2vKi/fCT8dOYlX/ieO62zHP/L34q8ffY/XNvxkeVzOV/u0yaj8HwORBA+Xy+N1pyR6AViPMAC+wwodILNZy4hlM8jAv6zB9VxMdCOso7FaVUs0HnaL3cIhmou2HEvF8ePHY8aMGXj44YcBAMuWLcPEiRMxbdo0lJaWRrRSXoHLnhVO5gdFQlLkXy073wqjx4lVQaJ8kVYYh7q3vrIRT3y001UcacaJoN38BLc68vbF+fjLB9/pjpvx3jbc/942XT2Nj4R/0dlIyC76oNXIwynSCcYYENYAsPNQhXSf36WbogynAiFGmiTmiGazxITr3jvvvIOXXnoJDzzwAMaOHYsZM2Zg6dKlyMzMxMKFCyNaqQRvw3s8Y5wQ9oLzmrU2PBQW4PxaIoUy5E/NhLPeDBPuB3ciAsvQ+dbdsOsIXv9crmmzzDweg7TmBQ6715A3iMQMot07t81ZlW2Pbw5+1k21VF6LQNkkV284tfV+zF+1U5ukjxTNbYBjK6y3b9+OgwcDGVAyMzPh8/nQp08fZGZmAgBGjBiBgoKCiFbKrWYtwhiBT5T6S7QYJnQOZ9Kw8wbRAjnxHYT+wzTarK2ECy+QZYc11nvG7skyIBb7a+dbLirfz7xBzNus6iMrxw2fFpRg+vKvXZ8XLppXRgOkRFllHaYt3+boWLvnEW3e2vgzzpmzqsEeEss3F+HF3N14/hNzrJ1wiKb226Sa9ebNm/H6668DAI4cOYLKykrMmTMHRUUBL4f8/Hz07t07opXyRqDL02lvqioUjlaatcxv12oxiHD+LLhNS6bg4GHe+GLI7ikTTo31ATIhaNV/sjaxNAdJthsXLwHiPJuhEyT1DKNBfv36Jvxz6/5GX/rvRBf57oB4yf4Xu0Px3e2Evnn1rP11I8mfV27HyVqf8LnnfLUPvWZ9iCMn7HOXal5b9bE/fIrm6MnWdW/8+PH405/+hOzsbFRXV2POnDlITU3F1KlTkZKSgtTUVMydOzeilYqEZm3SbAWNaDWsEpufxQ8ipMHw2rje59hnMINYPdOC4pDnhRvBZ0c4QkmWeJj/5TGaQeQ1CNaD32I+2ipjumxXQwRuTb0fycFJ0mgS0qzN++p8fiR6Q7rT1c9vwCu3X4DR53TRHdcQc4+qAmt2HMIVZ3dukHbv+rqCbW9v2gsgMGHesZW1a6FVuzWEhhb32a4j6NO5Feq4hxLNft9WWCcnJ+OZZ54xbV+xYkVUKgRExmZtQtCIVXU+nN6xpTACnqqqUFUVJVzPL/tQRMvNQ+cEhbQhTZnTCTGpG5yqYnfJCbyV9zPmXHu2yaZsBXvpnSwK8GmatXGGMfTbeGm5zdq8TWtT3gxi0TaRjA2S6FVQ51NRXedrHGHN4tUYxMQH2w5g8ttbse7+y3XbC0tOYLShDMtRhw1vb9qLHQfKMf+mc3HLhRlhl+MWv6rCa7hnppDJomvyhLxoIlOfSGm/Exbn47S2ybhuUNeIlGdHTK5gjIxmbfgtOKay1oeuacnC8/0qsOjTH3HRX9ehqLQqWKb1Q2ZLyx/65zea3Vkzg5gmGMO7D75+dy/djDe+2KOt0nRbVrWDxKeazdrqTdE0a+sVjEKbteAGrQSvvD3cf4AtEgICuloSITFcjlfW4uui46btmpui4fX+7/bAnND3B8WmD56GjCCKy6sBAAfLqsMuIxxEVfYK5nlkhNot0qq1s/K+O1CO3yzZJFyfcLCsWtf5RtNQE5PC2o2fdUmF2ObF956qKn7Ja+v9SEkUDy7e37YfT67aCSD0ksterAVrC7TrrNp+CG9vKjIdE7JZ6//aITvswPEqS99nczmciSb4clVZecPYXF9vBjFsszHd6J8N0zZDWJtBZJq1+8+EuRwaF7KoqooFawtw4HiV6zIBYPwrG4X+1n7BvQauh+B2RbhdVIYbQu6XegF5uNw6pG2kEGmyTCGzetba+Vr7RKg+LpvwoX99i9wfSnTzCHYeT9EgJoW1G836wr+uFW7nXwIVqvAB1fn8SE4UN8FH3x7S/mcftew5MBuzCvnLx7ZrKcjERZmQfZzXvvAZl03G+QvPY7XajuHTOhf5MWabteRFdhBCFrD27IikDb8FE9YGjamg+AQWrN2F+/4e3kpDma+1X6Jay3zZxWW4rw9rd/ZdMYGyNhjS9o3P97gv1M31RZo1E9YOHlwo3HFkNeuGlMZXm69WHGrWDX8oRn9kUSPWOpxYYsLaTglQVVUcwQ9mbxDHi2IE13AKL4xFZ1m5LjJki3h0ftbaX2feIPx+kfyy+oAjaQZhz9U4wmDCwW7k8eWeUrwTnCgTYXxW9RLPmlAwK/s6N8Rmzcwfje2TLqoy+8Z9TmzWEvNR2PWJwHn8c+CrFROLYhqTSNisy6tCwvqJj3YKG7HerzqKQ5LoYcLa+kmoqlyzDuWJdPc0G7Iyr9+fV3F1M+u1otyTRkJ+4fqz+SG7UaN2o/0KbdZWmnUEJxg1zdpgszaaCyqq67D8y71Ytf2gZhIDgJsX5WHWP7+Vlm+sk18yWcsOc6I5RmKYbVyY1RAhuONAGXrN+hCFh+UrN0XPzONGs9Z8/cOspASn9y06jP/OdeXEm7BuqGY959/bsbe0UrdN9pE7uZZ2rp1mDUlsbADM8SIaaaycHCMK+VpjI6xVVdWC3VhdIrQoKHheA80g4WjWPr+Ksso6LPn8J0uBlvPVPny7rwwjnsnVzFdGMwh7I9iz+vPK7Xhwxbe4d9kW3Pqq84zzxmfN3g2zzVq83UhZZR227jVPXLpFe05a9cL/3v7zTWBydNX2Q9Jj2PXqfH7sOxb4LrUJxuDOLXuP4ePvii3PDzeWx/b9ZVj3vbhsN+hcTvnvSTfBSDZrVyzN+9m0Tfb9OrkWe1nsBK1fVeGTuMO5WRTDE6kJC5Hd3i7NU3l1PY5XBmKQWN27FnXQxr7Nth+uqMGsFd+g3ufnJt24RTFhTDCqqooHV3yDxz74Dlv2HpOeP+O9bfjlws/wY0nIg8aYL1MxuGIeORGKzrbvmHnSceEnu4SeAsa6sk7IqNGFhvlGW7b+/OzXNuKdL4uk+2WYPKMMG1Zs2YfPdh2BHaLrJQV9w2stzBnsvEff34HLnlyPYydrQ657wWd944tf4K6lmy3rH65YuPaFz3DHm1zZwQLX7zzsaOWrSAOXKRRxZwZRFCUiphCeIoOmzeAXIsgImTCsj1NVuWbtNoATX2YkUFVu2BvcZmcG4RcNGT9Uvc1abzZwUuV3vizCVz8fE9ojrSbRZGX7VaA0OAqodbnSzdg5sFdPZEsWvZZPrynAm1/sMdfVUA1mnzV5fViUzbNDsqrRDmNrGDuR2no/JizOty1H9G4zM6JVx8/O+rSgBEBgPol9305Gmtv2BUcTLu01X+4pRa9ZH1qUW4Y38/a4KDFU16ZYWRyTwhqIjN2aZ93Ow2FfR+RyJjwOcmHNthsfsl1saScvc2WtD71mfYi/rS8M1cVwnqgYXliLNAx+wYJpgpH7n2kZ7Hv9+LtiDJ3/iak88ySlIrw/azOIRWcoEPxOqDf0DuwKIv9e2VC8otq8GtbYCWj31YAJRjfIWpE9J7eCRWQ2Y5p1nUXHrxp2qWrI/OhkUcwnwW/XbfO8IehAAf197zeMlOp9fsxe+S32c26bouv6JTbruHPdAyLjERKp66iONWtVPsFoWMEIAP/+ej/6zl6FHyWB/Y3HyygK2gF57c40uSV4iXhtSCQgeVuu9UIVc0dUVFplenGNdRBNegLWZhArbxDjyMEponoBoc6Zf0VkAtVJTHSf1GYdLFviZ11d53Ocmfxv6wsx96PvLY8Jd9K6RuA9lBhcbWytWZuv52aCUTvHZW/mJMaQ8eobfyzFso17MWvFN5bn6SYYLcqLJDErrEUab2pS+EuCkyTmDicLcPYcDQhDu17TyhvEJ9CsP/o2MDnDxwIxYhIkgmMm/WOraZ85NgpH8CXmE6iK6q3XrOX3blzww7DrMN75ski4UMR6ublkuyq3/dph1O60OQoWf5rbJxMYPoHtxnj/sqX7srC57Fe/P6/CuJfzJLXX89TqH/Dy/360PEZ7Ti6FtmjFa2ICs1nLhbXokzBOMDrB7chDpojJJgoB8yhLhq7ajRRnJWaFtaih09zmZuSQvUxu4pDYvdsqLPys1UCskRPccJlV6d5lX1kVGlb9jNUIxDrRb+OHtaJ68y+ucbfeL1p8jD94z7Iy/sVFvOMFbLgrGNket9+OyVxh6Hw8OjOImO8ES8WNnZemWUsKMcoJ/vQtEfAC0a6jmfXcIdKstQlGi3mC0DMObZNp1larRt0OtsMxpbL3S3Su/vuSaNZRVK1jVlh7BRpvNCKFGTuFEf06SY+1Gz5W1fqEGhYQ0CBe/t+PmPj6Jm2bE/tWOP7DXxQeQZ/Z/9Vtq/OpeGtjwEtG0bZZa9b80N6o9fHPQmQGYb/1Woz5GqImCM8MYu2Pe7yyVjo/YOyojBOlin42VcjnhUeFdeKRCetofuAiwjWDiGKoJBk06/3Hq5D7g35+SHQ1bVGMoZGs0qK5/f5lwtpq7ol9EnYmFJmfddy57gEhWxhPNEYbxgeaYrGi0a9av+jF5TWWmvV/vjlg2maHu4cfOPa9YP5EnqV5e1B6MuAt8dXPx/B/HxfoJhjFZpDQfquqGpfS8+eoguN4RM1laQaxsFmz+zNKVJ9fxaC/fIz73xUH7TfbrPXb+VfEjd1UprErUHC4ohrPrPkBfr/aoIVPjjAUz4Su28uKOrtEwwTjNc9vwG+WfKk7xmzKU6WxQQ5LYv2IKKuqw5R3tuIJiY0+HM1aM1Vx54o6Cekzi0fNOkVgn/YoCn5zSa+IXseoWVuZRUSmBJ7DFdXSJcpPr/7BtM2J1uzmg2LHiuzzvLfCZ4VH8Ny6XTphLbLV8Zo1eznLq+twuEIftU1mszZ+iFaCWbE5zni8kYrqem1uwclAp3AAACAASURBVAiLZrfue7FHkNFmzepdHAyUpU9hJq+bEVN7cOaAB977Bi98UojNPx+Tujw2REv719Z9OBg0KRjLWbHF3JkDgUU3ebsDI4T1PxzGhl0luv0ibxB2i0yzZn75+oP0P/1qqNNzEsiJYewoBz62Bv/++gBekdjoXS140+oWNIMIHjR/pMy0Hc1uN2aFdasW5mh4igI8et05uGfYGRG7ToJBsFlNOKqqtSCp8/HanZ49Ryux1yBMnJlBnD9+dmQLQXAq0YvLm0FEL1+d36xZj3g6Fxf9dZ2wjsbJIqPNWnS/2v0pgY7gsQ92OIoGaKSsKiQkjLfKIjN2aSsOh2vuVEK/135XrOu03OhqOw6U6565toJRUbS4LT6/atLkI8G05duw+Wf54iDA/DzuePNL3PrqRlTV+vDbJV/i9sWbdPt1sWYMk6JGn33+XTDelV9VNaXITljf+kpoxSiTn7X1fjxkscyfITKlBuou/p/VDdCHBBY9c31sEN4kaFutsLEV1lVVVZgyZQomTJiAm2++GevXr8fBgwdx++23Izs7G1OmTEFtrVhANQSRsGY9a5bBrvzEDQPCvo5RiInML4zAg7R+GkdPyNvCeKbxRf3b+kJ8ZfjA3GnWgYNFwalEQ0I7zbpeoFmz1Xy6RSyq/q+23Q9dMl6hZs1d9vm1u7Dk8z1450t5cCSZQONNNsZhKxOSspl+s8069P+dSzdrfr5A4B38seQENu8pldaR8ds3vsSwp9aHyhXOC/g1LwvR/nBySzrFWDKbJOXbKfvVjdi8pxTPflygm0Q1PnMVqhYqGNB39Ka5DL+qfWd2C7PyfgzNBbBX+NOCEi3TjBVOnAfe+GKPFu9l2cafNc+qiup6vLe5SKfQSCcYG8lmbZspZv369ejfvz/uuusu7N+/H7/73e9w/vnnIzs7G2PGjMGzzz6LnJwcZGdnR7RiLUWadfDvxWd0wJM3DcCDKwK9a0pS+AMEo2ZtZedSYS88+UA/RowlGxPsPiUwlbgS1sG/LQTBqUSadY0bm7XFdf1+VbhSLLfgMJ74aGfoOMHNPPrBDgCBtmEfhtUHLGsPXuDyd5r1dK52XVmEN9HEqAxFUTDimU+l+60QmXz4CWdjp6GqeqEXbVi9+Gp8sfsotu//EuWGRT8BxUDRhRhYsDaU0LZOsKAqtNIVSA4mfiiucJ4IwW1sEPkEo56FnxTi8ev74/H/fKdt27DrCDbsOgKfXxWavvyS961JNeurr74ad911FwDg4MGD6Ny5M/Lz8zFy5EgAQFZWFvLynPmAukFmBmGc1amV9r+bZAVGjELMzl5qN1Q9UCZ3PTK+8Py1ZMNBk++txeXZPpGwFqX94oe1Qm8QfigrmYQD5BOlPxvMPqK2Kzxs9jEXLTDRrivZLnIrAwI5/lg9pKtLjX7WFi9BOJPcrDw7lzlR+zhZ4WeH6PGI5l/YT2a3Nm4XbZO5bPMrGtl9sSBOPn8ogfXB4y6EdbDtnc4buk28LZo85s1rMjfUmItnPX78eMyYMQMPP/wwqqqqkJSUBADo0KEDSkpKbM52T8sW5qE8P7zlY3pYmS7sMPa+Mtc7AFi5db/tpKCbFFG8UJD5gftV4ImPvreMccBgL5Mo7KtIs+azuwuFdb3cps2/uLKOxpg+yumLbLUEXzrByMcxkZwrX7AUiAbXa9aH+Ojbg5bPWJaZSNRBMljccF22HsEra6yfCvtgW04Q3Y7Pr2KrIPUYYOP3z8rkzB/8XwaftVxVA7F5+IBo7PgDYaQYs5LBe46cxP3vbkOdz2/6tiuq63R1N5bnxnlE77pndmONBo6F9TvvvIOXXnoJDzzwgO2kUSTo26WNaRvfmLxAstOsf3fp6dJ9RkFvFdipoPiEyabcEHjBIxv6q6oqne02HRv8y3IL8ogmW/jkAyKt8/73Qq5uflXVuUhZTdIwjhkmW63eFUVRtJe+0mKCUVYEv9goFDRLf7BM8NX7VWzeE3iuAWHt/p1unSxfsHWyNlA3u8k00TOwGmVs2HUE3+4rs62b6H5W7yjGB9sOCI4WIJpr0CaVg4cYjrnZsOKSNw/y/velJ+WuerJwBVbmkPvf24YVW/ZhW9Fxk4Iy4NE1wtyYDDdJp/k25eOINKkZZPv27Th4MLAsOjMzEz6fDy1btkR1daDxi4uL0amTfCFJuAzu2c60jX9IvFD12mjWVj2xUYh1b5dqWZaTgP1O4b9NWWxp/hgnk011Pj82CSa/RO8hbwb51Utf4LbXNkqvUefTdxr8UTIhZJq4c9h0TFiLRkwygc+7JsqG5lahAJgm2LFVi7CEdZLFO1hZE7gfLTAUxAJHNgFphczHmEd0O7sskgU4RVVVfLs/0FkY24x34fOrqmlCmh1derJW+Ew3/ngUpz/0kW6b1jwW3zNflkj4fieIXMiOkvnQC58Vd51/5NtPdkYCW2G9efNmvP766wCAI0eOoLKyEpdccglWr14NAFizZg2GDh0a8YqJ4oDwbSmL9WFkxug+ltMSxt7XowBZfdOlx5dXCfxIw4R/4HJ3tdAx2w+UWafiUoH/+7gAHwYDwvOItDZegz1Z68PnhUc1LdCIMV8jX3d5bF/nE3eia4lGCJt/PiZM8HrCEM71RE29qV5WKdeOBkcBHVslhbVqNNHCDMLalN3/pwUlOi8Haf1U1dZmHe50jRsBI3puqgq8valIi2z35R75iDNweug7u+etzVoPUudTTXM5QMDPW1QPVVWF77d2reBfmYKWlOCRemyIJiRl74Ib//BIYfuox48fj9LSUmRnZ+Puu+/GnDlzMHnyZKxcuRLZ2dk4fvw4rr/++ohXTLSSkH9neDNIh5ZJuGvo6Rg3OEN3/LO3DMSkEb0tNWuje4+ihD6aB67sazq+okYszMKB/wiM2UrOPq1N8JjQthte/MKyPBXQBdXnEb1cTnIwyo7VuTFJXlxjWFqrF5x/RqzjEtmB5/13J65asMG0nRfWy78sQv9HVmOXIUCW1QiA3YPX4wnLXc7KfMbux85kKIyMaDMccRuJjuFmpaCoObYWHdMWG9mhQv98i8trdGWWCRbSiCdFA6afHMEK3cB+lcukowjbO9GrmMw6zPwmakudx5Zku129I4Wt615ycjKeeeYZ0/YlS5ZEpUIM0QpG/oPkvUW8HgV/uuZsvPzpbtfXMfamh8trNG2GFxZpqYni1VkNgH+wRjttKBN1aJtdb65yiw2MCIW1QJuXyQYrzdrpJJhVZDYFwIrgR1gZ1ESd5Mdk8GaQtcEUTj8U64WJbATg86vakNmJx48IK2F9slZvBpEhjs/iTFh/sO0AJr+91aaW4SFqj+xX83HbL3o4Pt/4VvLarei5iNqCTzMngp+sVJRQnA+eJK9HOgEtMhX6/KrQ7CJ7lnEZG0S0sIM3QfB+2GyC0fhis47SKgAM/5FdN7Arfn1JL21RAL+vjcUEUrjw1TKaV9xk0uCRLbEVLQgRaday6xmFNT/xZSWEeSz9pxEatRhjVzy7xux/buRETaj92PM2Ts5Z+Wh7tXP8YQ1xrW3WejOIDJE3iK0ZJHjZaAlqQN5uTltJpiUzxOFlzSdZPRafX98hKJIyHvvgO10GJB6RGYQ3TTlRnJp0grGpEGkqMhMEa2RjHriQM36AqaN6S88FAkvZM9qnah85r9klc0u4rdy0wsWotTOh62S2n6HCvMiHIbLXVtX6TLZ/mfZpFOysY7zi7M62iXcZVkKd76zYcYfKq7F17zE8/0mh7DQN3mWSPVHR8FoEb/ao96lh2axl7Q6E2s6uXNGiGLuOMFwziAiZoHE6JyHjUFm1SWF6MTc0Cha9m6KirUY9dT6/IYaL2AxyqLwafzfY6zWlTlCun+sEfjpyUlNaZKayaFqyY1ZYu0ET1jKhEWxtUQeQ4FE07YT9Zb0mL8gUKJh9TaZ2TiTgPzRjTBF2TzNtMlbwqKqFZi3Q0KrqfLpOCJBrf39brzcxMT/Wgd3bOtYmrFI/yQSZnZ3eir868JQAAsKCCaQ6nz8sd1QrX/8PvzmI/xWU2JZr1qztJxi/O1gu9HCIJNIY4g69eyYszrec5BfdoyxCo6wJq+t8Jq8Np1loVDUgfEVKB1/Gw//6Fn/613atLuKy4tAM4oYEG2HNHqJICUnweDTByHr/WWP64bS2yejTubV2XNe0UBAgkYkmHPjqGG1xbpIiMFS4s1lX1/mQmqSftnD6Adb5/FDgzq7s1FzSUNx+Lj5/KMt6rc/vKtUUw8pmvW7nYUx8fZNrMwigX/Iv4mBZNa5+3jzhGg4yJV1W7e8FXjkyrO5dLJjFNmupn31NvSGJs7OolkAgPsj4VzYKhbVRg96yN+D1Ih1tOLtkWMS0sH587DmW+5nmy4St3WSMqH0TvIqm4TKl9NKzOiLvoZG6VZSDe7VHm2CmmpGZzv3KLzmzg6PjzJq1+0cT0KzF5xltzkDAputUsxadqyhi9zoZkfRRt0IW+VDGiRof9hwJeNHU1PvxzJoCmzPMOBlthTPB2FgdHODe3vqNCxOd1XslMoOI2kpV5cLwRE29rv6K4m6+Z9OeUqFWbHwmaamJwfqRzVrH7UN6We5v11LfcDJhbem651FMmjWDN1MkehX86vzuWPmHSzEys7NlvXTmE6tvmNvJL89l9XKLanHeO18WCbe3StZr1jLXPyMBG6Hiyn5vnFOIFdZ+X4zVOwIeJJv3lOKnI87agMdJoHu3mvWu4hMRiQ0SC1j1OWt2HDJtE0YgtLBZn6iuF0b3c8PgXu1N277YfVT3DbdPTbIs+3BFNYY/tR4/H3X/DtkR08Lajj9knQUAmsYrN4MEEPWcCR6PJpSND5sX1gkeDzweBYMy0izDoAJ6bchqAojfc8gQIyGcLBdQ7VdzGmndQu/lMmFxvqPzwjKDNJJm3RDKq8Lzo3cy0Wdcfm/EOLRe810xXgrDHTUWsfKwESX4FQlln6pikaQ9TtTohbWqOjeD8Nc8r0eabtt3B8u1JBRASNbI7mfv0UrsOVppCmIWCZq1sJ44pBf2zLtGc6urCQpJo6CzXG7uVTDzqsDiF+NCHP483hZs1IKtOC8jTbqPL98Y0MZtxDAgaLN2KeRlAfntiCUzSEOy3hvh3br4pjy/h/w5As4613c3ixdzMETaWiRj0TAuOt2sQUYbtxNvIlm4q/iEdORnNIOE4y/v84u/Hz6iI/ss390sHqkyk05YypYNzVpYG2HeBrKZefbszukaChKV6FE0oW+cJOJjC/CNf3kf+XJ0I1NG9dH+PyO9JXp1CMUe4d8lYzQ3txoyw2242B7trWOhyAho1oorzdpNBDk+BK4dDcl6b+QEt9yef+bn92gn3M5wEwRIhsh2G4VvHteee5pwu5sVrW5xO2krErQnLFYPG80gftW9GaTer9o/WzXQ8ayVpIgL5e0kYW3JxKCNe8G48wAAQ4KTe8wbRAXwzaOjseL3l2jnWPWA/C6+xx2YkYa7g6nFBlpozsbyR2V21nmSfF10HBef0R43ntdNcO0wNGvVfbxlUShaJ9T5/PAo7nzO3WjWg3u2Q2+HAvu6Qeb2CxdeRvDP7sEx/bT/RbHWwxkJGXGaULihOI2rE0ncLjQSLr236Oxr6v26tlJV9/7yAc3a3Db7julj1FutK9AS7kahkz2lhPVlvTtiz7xrcFX/Ltgz7xqc1jYFADQ71IDubdEmOVEnMK00UV5gGr0zbr6gOwBnWna/Lq3RpU0yHryqn8kmlpqUIMyZGA4q3GsTKUkJuH5QV9fXqqtXoSjuNGs3ng0pSV7HQ8nTwjTl2OHVTTB7kBmM19I6WSCsI/B1NlZwICs3w2hhzHhuh6gprIR1bb1fJ+ADCQ4io1mbrmVRD18UzSC2sUFOBUZmdkb+wyPRuY35o7byZ+aVJaNppXfn1tgz7xr8Pf9nbduca89Gt3YpaJeahFu4eL6rpg7T/n/0unOwu+QkNv0UCGPqUSL38fj8qutg7qmJXlzV/zSs/NphbOMgdf7ABKObxA+uhHWi1zJMAE80PgxRuexXIHa1XtuKxLA3HP/ucHDTwTYFqqoKlQ6rkVlNvQ/rucBhftV9qIZ6n9/RnI9VPZgpKxJmMSOx/dQALJpwPkb0a3i8bJGgBuzMIGKbNQ8fDOl3l52OK8/pYjmB0yLBq9Ouy6rqhMI63EzX77sUuqkuNFierXuPA4o7G7kbM0hyotdxxj0n9Z/GzR04xbiEnN1qa5EZJAJfUu5OsR000jSFZu2GylofPvzWHAbV6v3519b9+H8f8skxVMcLvBg+h5q1EzNIJMxiRmL7qQG4qv9peP03F0atfKsXV++6J278K8/p4vqadfUhQSwT1uHidhFFwNwQ3rUUuFtp6UZYexTnq8GcfBjheIyYNevAb943/bKzOuLzWSMiot2Hk+IqHKIR2yaSlElixltlzdlt8BIJR7OurvOhhc3qZBXW73HIZh2HwjraWH1j/D7ZisIMiTfF3BsH4OXbLxDuq/WFtHFVDT+H5Hv3DsGSMDsydm+pSQlhv1gej+JqssqNN4gsEI+sHnakhjGRauwE2IfIu3i2SUlAt7SUqHyc0cJoBlmYfV5Eyh3mwkvKCpmwdqOI+FXVkVlpLDdfc6KmXjgfYapHUFh3EYzWQ2YQpzV1jiOb9fz58/HVV1+hvr4e99xzDz755BPs2LEDaWmB4fwdd9yB4cOHR752UWTF7y/Bmu8OWdpF+cAw3dJSXJV/60XyWL+1hszPQs3agZxqmZQQ9qQUOy01yWuZoNaKgGYdvUUxxm/tgp7thH7HXk8gHZeV/3vLJPGrflGv9sI0aIFy9e8G09T4CWr2jkTLbh4NjCEGRIk+wiHztNb4X0HDk2fLQpi6eX/4PI9W9OSUrfJqZ8KarU68a9gZePw/3+n2sXCvTTLBuHHjRuzatQvLly/HsWPHcMMNN+Diiy/G9OnTkZWVFfEKNRYX9GyHCwR5HnVw7X12V3MC33DhXzpVDf/Berml8uGSkuRFeVV4ZSiK4moRjhPN6MbzuuGfW/dDUcyB3GX36lEUfDFrBH48ckKYRQYQJ7MAApqxDOP1WMfIm1SMw96kBE/Mr9Ts0LKF7jc/Kshon4Ki0irjKY5ok5yIOy87Ha999lOD6ifLIO9mZKY6NIPwylptvV84H8Hzr6378a+t+wFA6FrKqtgkNusLL7wQzz33HACgTZs2qKqqgs8XPef5mCL4rNs46G3dwNvefKo4DKaTjBMeRW+qCUdupyZ5LROQWlF6staVvd1J4J+OrQOCRIFict+SdQyeoAuh0STD22Zl2qOiKCZ3SoZJWAc/fl7wM4HAjn1gtDkVXCRhLqMNoUOrJN1vXgj27tQao8+2jn0jo1WLBNx9+RkNqhsA/P7vW4Tb3Qhrv6o6GnUan7HIh16GyKuGadZOPZncYPuleb1epKYGhgo5OTkYNmwYvF4vli1bhokTJ2LatGkoLRUPI08VIhUSlcHPJvv8qjCLi5Nl3B6PotOKxgtML3Z2xNTEhAbFdQwnlKsVTGtNSfSYNCOZZs22G+3GzC8akE+qKQCW/u4iy3IZzJ2Mfx+Mwjoa7ne82aJXx5YNLs8okIz3Ge4oslWL8Oc/nGA1wWiExb6205SNr1QrFxmhRMI6Jpabr127Fjk5OZgzZw7Gjh2LGTNmYOnSpcjMzMTChQsjXrFYoE1KAqZf0Qdv331xRMvlNQRVFb+E/FA7o32Ktkydt517Fb2wnvCLnqZyLuhhbepJSfIKZXWfzs5WDyZGeCbl3svPxLRRfZD9i54mm6OVZg2YV2/yE7eyWf46n186OpgxWu/uVy+YYDSaQaKxsGUUF+UxEkLAqPUN79sJdw09HWmpiZg6qnfYQ/iWURbWbmCatVXWecDcFk5s1gzR5Lq/qV33NmzYgEWLFuHVV19F69atMWTIEGRmBrKmjBgxAgUF7uP/NgcURcEfR/bGmenO41Q4gU806ldVYYB5Xlivmz4cuQ9kYdsjo/Hx9NACG4+it1mLPmRe8/1i1gjt/77BxApJCR7hRExrhxqGSLP+9RBzp+GEuTcOQHKiF1NG9UZSglmzlk1mhjL96OvCt4fMa+VwRY1UAF5xtt4tk1UnhdN0WT/Livf7Vdx7+Zla+0YCvu5u5gjuG36mdN/3f7lK+58lnP56zmic2z0t7AUdAc3a+hiRSZF1qpF0Kfz4u2IcKKuyDadgfGd4TfyN31p7Wonqy0ZW0fAGsS2yoqIC8+fPx8svv6x5f0yePBlFRYGoU/n5+ejd25zbMBqE6+IWbebfdC6G9u7o+PjR53TBh3+8DEBQWAu0MT6DCxtutU1J1G33ePQvm6gz59usK6eVv333xfjnfYEYKSL7uFMNUSSsJ1wcnrA2aiNONetQPHL9dl5jlq3aO1xRY6Gx639rrnu8zdqgSflUFbPG9MNNF0QuXgnfxm4064vPkCe+kE248tcQxayxolVygq2ttqvBq+r9SZdi0YQLbOtkxeQRZ5m25Xy1D9/sKzOFAOYZf2GGwAwS+r7SUpNghUh5iKafta3O/9FHH+HYsWOYOnWqtu3GG2/E1KlTkZKSgtTUVMydOzfiFTOy8aGRMevMf8uFGbjlwgxX5zBB4vOLM4876Zi8HkXXg4s0ZJl/ePuWSWjfMkl6Hi+snxs/CFPe+VpYjsgM0rtza2x7ZDQGPrbGovZmjBqdKb64TKhKkkc40axLKmqkAsa4XTODcB2mFmWNZaMXfKzd0lKw/7gzDwvmDcPDdzoiYe31KMLO1aMouPSsDvi88KjwWjed3x2DMtqatrMrGCci7WjVwmurWS/MPh+vf/4T/hFMWntu9zTk/xioXzRWVsrMGv83biBuOK87Xtugj6XN2/PtTBl8J39ejzRs3XtccxZoEte9cePGYdy4cabtN9xwQ8QrY0W4cZdjFfagVVVF93bmhTVX9T8NT9ukl/Ioiu6FErkqORH6Ih06q18nfLs/4L0xdlA3PPL+DlMG9g4tk6QCtG0YYUuN36pjzdqQli10fKjAxISGfzysfXnBb1xezGQmE/S3XpSBx67rjz6z/+voGiKBZSusFQU+wVP0KEA7C+3wmVsGCreza4hGfDec101zXTPixGbdsVUS7r+ijyasAX0nnOBRhNe1wmoUKDPnMSXG2CHzmrWdwOVHPJOyzsIdb26mFYynIuwD9Ksq7hp6Bpb89kKc2z2k5ZzVqRXuv6IPXps4WFqGR1F0L5tYs3YgrLkT+3Vpjc2zR2HqyN5Y+YdL8d69QwAAnz84wnTeHUNP1/2efkUfbJgZvu+98QU3rmC08rMG9IuYAL1wj0RYUPYh8h2gUbPWbJZK6LpuAieJOpXhfUMePSJtT2ofVQKueG5h72a9TzWNZs/v2Q7vT7pUeJ4TbxCPRzF5V7G2UlUg+xfyxWQyrIS1zO2WvRvGV6oNJ9ztPJ14ZcD4/Cn5wCkEexH8wUUxWX07aYKRMXlkb4yy8Hk1LooRpSVy4q3Bv+oeRUHHVi20FGYXBvPStRS4QBmF45AzO0iX3zvB+KEbv0GZCyW7RZNmzX1sIq+A09omuxIOzMSR4PXgV0F/Z1ZHr8QM4tY3RKRZn5neCmemt9Rdh0c2XPcoCv6QdSYW/3owfpp7teM6sHar9/tNwjrBI1+I1TIpwTaeuuj8kMIhWc1rg5UmLjODyNw9+fu1k7d8p82eQX1Te4MQkYf1yrzpwmmKLF4r0CWxUIF/3afXepzEyu7OTfpYaRPGxRLGl7mhy5Zly7s7tkrCI788G+1SJUNazXXPnWad99BIPHHDAMs6/WXsOXjwqkDiAaY1JXoUrS2YVnd20Kf7nG6B0RG7tNtgQqJ6ej2KNmoSCcqTteJFah5FQYLXg5GZnV0t0mAdfG29auogvR5FKlCNfv/Csr0ekzlL69jU8OzWIm8qhkjJAOSatZt24icY2Wm+oM1aaQpvECI6sF7ZbbIAINT7KwY/a+bV8fpvBuNPV2fi98PPxNUD9CmclvzmQqzl3P+AwITggnGDAFjb2ubeqBdsxkNlH4ZTjB8O01JevO0C/PbS05GWIra/yrxBsrjQuuFOXk0c0gu/D7rA+TjNOjRBHNg2rE86Pn1gOK4b2DVYF70N+5oB4lRaRjq1Scafrz0bI7m6exRF09DdDK/DHYkzU0y932/q7L2KdYgDu2uKNGv+J/sufiVZqfnBpMtM26w0a9lojK114O3liyacr9WxZ4dU29gifKfDRpmaGYQ061MH1iuLVrwNsXC5AkIr84zvA3tnR/TrjLuGnYEHr+qnCRVm98zq1wlnCeyY3doFtGurD1EWMpTR0MS1xo6C5dTkXRdFMMFoPH8sl+4rEjZEpiUneBXtXiu5nI09O4RWF4a0xcA5t10sN7fwPs9nn9YGd1x2OhZz0RQ9nlA5bu4jXHnBRn31PhXJhtFeglccD4aZWew0a8Uwz8KfoyLUqcomk0U2fVG4Bu14rwff/eVK0/by6nrdtccNzsBV/QMd6g//bww+uX+45X2wshmaZk2ZYk492MtoVAq+nnOFrb/pa78ejK1Fx9EmORFlnIfGgG5mNywA2PbIaFsThZOXzOj5Yfwuw/WTZRivXRd0aWQjCZZTU3Ze5D8PPZpm7VGQHJyIkiVx1cwgwRG6VZIGvh1FKcp4zdpqUcwFPdthwbhBGDp/fbDc8FqEabd1Pj8uOr09dh0+oe3zehShfzG7VjiXZOeoXARKvpzsX/TQvEdEZiIrzTrRKzbblAfDsDIlhu9MnQpa/jD2bz15g5x6JHk96No2GXMNNtO01CRb23VaahKy+gaGyUwx794uRfqStU1JtPVIGNCtLXp1SNXssyLshnapDbRZGzsDpjExYZ3RPhV75l2DZw0uZ16BZv3xNL2pJxIwmZDg8WgTVydr5PbiwDmsEwzty2ivXxjCN2sbwejBqyhcOfLn2C41ERntEsx5fgAAEIdJREFUU9G5TQtdHdyiadZ+FY/88hx8MOkyrRNJ8FhHWgyng9Br1uz8UDn8vIJI8PosUsKI6pvoVXBFcM7htLYp2DPvGpzb3RzMy879VHSvNcEM8VFQrElYNxUej4IvHhqJmyIQRQ0If8jLaNkiAbkPZFmGjTXbGg0TehFe1MC0FGNHY7wu+8kmdVq1SEBvwXLvM9IbFgTpF8F0ba2TEzQXL5lmzWQNE/B8nd/87UW46fzu3KGhfSJXM49HCWnoFhPALEAYC4Ea7iuRwGnWSQkeDOjelpvg9ER8iM9PMLL/ZQvgREpHnYVm7fV6TEJ111+v1pmsZHRqk4xP7r/c9jgg5G2UH8ytSq57hAknoVQjhUxI3jf8TKEWcudlp0vLOq9HGh677hzdNllmGONIw9gx2f1mWNkh5904AOMGZ6BfF7lf8sLs85Fz7xBktE/VLZ4QYbRZ8x/vGemtdAtS+O9a1OF5lJAJxspfnMXRbqigYNorH3CMlen1uItP4gS+OBbzXDYSFN2/z8pmHSzcLs6HjDMcxgUymhmbJEQqEdt0b5eKq87pgoW3nh/1a8myfc+8qh+2PTLadPzsa89Gp9YtTNuBgE/ury/ppdsm88AyeiQYOw1NMPr19XLD+It64MlfnavLRG8kvXULDA76nbO2kHl5aIGdHEwM2n3YXo+iTURbCcpag4dDuEmX+UUxfB0Cfz2OR1DMu8IOhevYaur08xSmugnDklqYQbQJ9k5YNXUoXrVYZNYQGuoJ5QSaYGzmeD0KFklyPUYao5wYILDzGZEJDNGqM9mxRm1KKqyDo4xoaDUidjx2pUWcbP0EsqWwtrmOR1E0F0+rkJ9M0IXr481gWq0oYbSdzZrntLbOUuFpKxgBLcVckteDxb8ebNJsReET7CYYGf26tEG/LpHL+MTTUE8oJ5BmTTiGCcEz0lvi6zlX2KdFA/CLoBuiMeC9SFjLzSBGYW2sV+Bv25RE3DK4e9hDXre0bJEg1TIVTgAB1pOzdn2LR1G4BTnyT5a5XzJhahyppEtGOUbO7dYW91x+Bv5v/KBQHTTN2nkquYEZafjPZLNftBEP11jM7t4iMbCY53RDsgWhGcSvSoWllRdOJGkMYU2aNeGKt++6GL07t7INH8l45uaBmDKyN+5/d5sWGAoI+ZcnJ3pQHdQIZQqSUVM2/ea2z/+VODiRkf89kOXouHAxeoMwYSdyoXRkBtEW5MiPffrmgbprG80DH08bZgrGJay7R8FDYzL1dVDca9YA0F/iTqq7HucNwu6vlSS0qait6nx+rJ46THNZ5Il0JiMZqZKEzJGENGvCFUPO7ICOrZxpaEBgBVmfzq3x6sTBOk8HNnTNf2gULgp6WRi1bZnmbo4t4bg6Gj06hB/DxAnGCUYm4Kx80UVxmQNlhVa68q5rHVu10KUuY5O8vHcFT1pqUthpwbRYGh7zopaGwhc3eURvTMo6y7SCcf5N55ri3ux8/CoM65OOx67rj4z2qcLJ4UhPhspojOz2pFkTjUKXtsn4es5ofL77CG5fvEkTPm1TE9G5TcCH12hjXXbHL1BWZdYE3cRzSPQqrnL3RYqewc7g/GBaNWbDli1O2jPvGmlZCm8G4TTFzbNHAQB6zfpQd7yWDzKCKcaYhhqVxR6eUMfWqkUCZlxpTjosihmfnOgV5s/88I+X4anVPyD3h5KopRmblCXuWKMJCWui0fB4FC2+Mi9I+BCZPClJXqEmanIhtLjmF7NGoqLafugfafp3a4vcGcM1oV0dnDjj72frn69wHLuZWTScxDiJRvJeY47Jq87pglU7DgGApbsjYJ28IlB24K/T2j43fhDSBaM7RTKiiAaiDuWduy/G+Fc2Ru2ajswg8+fPx7hx43DTTTdhzZo1OHjwIG6//XZkZ2djypQpqK2tjVoFiVMLkdbHhrzn2yT3ZZj9quXiOr11C8e+spGmV8eWWt1YVh5+eN+uZZLjST+fam+zZkwd1RudWrdw3J5OSDA8N9bkM6/qi5zfX2J5Lh+jRYTMbGNV3iVnmdPoRdsQkf/wSLROTpA+M6OZJtLYatYbN27Erl27sHz5chw7dgw33HADhgwZguzsbIwZMwbPPvsscnJykJ2dHdWKEqcGCQKtb2jvdEszgBGz615k6hZNOrZqge/+cmXYYWS1xAcOvBvO69EOm/40KqzryPBIhHXP9i1Nnj4izkhvKZ3rCDmDNEwlViQjtEjRuU0yvvzTKKl5KRIJLqywLf3CCy/Ec889BwBo06YNqqqqkJ+fj5EjRwIAsrKykJeXF9VKEqcOxo8+HOyWn8cqqUn2CWXtEPlZb5iZhc8ejK53SyjwWFBYQ+/bbscn9w/Hu/cMEe6L1PO7M5i5qAeXACPSK3yTE726BTATh/TUYr1E27/ftkv0er1ITQ3cfE5ODoYNG4bPPvsMSUmBYV2HDh1QUlIS1UoSpw5epeHC+qJe7TH9ij4Y1icdGwpK0L2ds8UXQCAeMu9C2NwQeTc0JDuPU564YQCeWvMDBvcKmlYaqMX269Jai6vi1gwi44bzuuOG8yITa8cpfxnbv9Gu5XiCce3atcjJycHrr7+O0aNDS4tlCxkIQkQkPBU8HgV/HNkbADAow52dcED3thjQ3d73N1aJRgZwJ/Tq2BJ/yzYvHw/3KfLL+t1OMDqheYy13OHoyW/YsAGLFi3Cq6++itatWyM1NRXV1dUAgOLiYnTq1MmmBIIIwOyWdxqS7RJmXps4GL+9tJduG+vsZDFXGgsmDGXKWrc056MdRYm8tL71okB86rNPa74dsxFbzbqiogLz58/HG2+8gbS0gBZzySWXYPXq1Rg7dizWrFmDoUOHRr2ixKlBSpLX1WRiPDPq7M5awuRJWWdh4fpCAIGYJI2xCMMKK/vs5tmjXE2keiI0wcgz+pwuTfKePXPzQJzVKTreR7bC+qOPPsKxY8cwdepUbdu8efMwe/ZsLF++HF27dsX1118flcoRBBFgxpV9Nd/exojw1hDcrHAFms8EsRMiFZ9ehO1THzduHMaNG2favmTJkqhUiCCI2CdkBml4WZGaYDzVodggBEG4JlK+0fqyCCtIWBME4ZpIGi6MQa8IMbFt/CKIJmDBuEFN5iLXXIhkHI5ouO6dipCwJggD159nHcuCIJt1U0DCmiCIsImEfFUU4J7Lz8DV/cX5LIkAJKwJgnCPttw8EhOM5sw0hBkS1gRBuCYUyOnU5+NpwxzHHY8mJKwJgnDNKbSOxZbena2TKzQWNOVNEIRrNFnd9Apn3EDCmiAI10RyUQzhDBLWBEGEDbnbNR4krAmCcE08TTDGCiSsCYJwzeV90wEA/bueOvGiYx3yBiEIwjVXDzgN3z46Gq2TE5u6KnEDadYEQYQFCerGxZGwLigowKhRo7Bs2TIAwKxZs/DLX/4St99+O26//Xbk5uZGs44EQRBxj60ZpLKyEo8//jiGDNGnkZ8+fTqysrKiVjGCIAgihK1mnZSUhFdffZWS4hIEQTQhtsI6ISEBycnJpu3Lli3DxIkTMW3aNJSWlkalcgRBEESAsCYYx44dixkzZmDp0qXIzMzEwoULI10vgiAIgiMsYT1kyBBkZgZCGo4YMQIFBQURrRRBEAShJyw/68mTJ2PmzJnIyMhAfn4+evfurdvv8/kAAIcOHWp4DQmCIOIEJjOZDOWxFdbbt2/Hk08+if379yMhIQGrV6/GhAkTMHXqVKSkpCA1NRVz587VnVNSUgIAuO222yJRf4IgiLiipKQEPXv21G1T1CikFK6ursb27duRnp4Or9cb6eIJgiBOSXw+H0pKStC/f3+TY0dUhDVBEAQRWWi5OUEQRDMg5gI5PfHEE9i2bRsURcHDDz+Mc889t6mr1CjMnz8fX331Ferr63HPPfdgwIABmDlzJnw+H9LT0/HUU08hKSkJ77//Pt588014PB7ccsstuPnmm5u66lGhuroa1157Le677z4MGTIkrtvi/fffx2uvvYaEhAT88Y9/RN++feO2PU6ePIkHH3wQZWVlqKurwx/+8Aekp6fj0UcfBQD07dsXjz32GADgtddew6pVq6AoCiZNmoTLL7+8CWseAdQYIj8/X7377rtVVVXVwsJC9ZZbbmniGjUOeXl56p133qmqqqqWlpaql19+uTpr1iz1o48+UlVVVZ955hn173//u3ry5El19OjRanl5uVpVVaVec8016rFjx5qy6lHj2WefVW+88UZ1xYoVcd0WpaWl6ujRo9WKigq1uLhYnT17dly3x1tvvaU+/fTTqqqq6qFDh9Qrr7xSnTBhgrpt2zZVVVV1+vTpam5urrp37171hhtuUGtqatSjR4+qV155pVpfX9+UVW8wMWUGycvLw6hRowAAZ555JsrKynDixIkmrlX0ufDCC/Hcc88BANq0aYOqqirk5+dj5MiRAICsrCzk5eVh27ZtGDBgAFq3bo3k5GScf/752LJlS1NWPSrs3r0bhYWFGD58OADEdVvk5eVhyJAhaNWqFTp16oTHH388rtujXbt2OH78OACgvLwcaWlp2L9/vzYCZ+2Rn5+PoUOHIikpCe3bt0e3bt1QWFjYlFVvMDElrI8cOYJ27dppv9u3b6+5AZ7KeL1epKamAgBycnIwbNgwVFVVISkpCQDQoUMHlJSU4MiRI2jfvr123qnaPk8++SRmzZql/Y7ntti3bx+qq6tx7733Ijs7G3l5eXHdHtdccw0OHDiAK664AhMmTMDMmTPRpk0bbf+p3B4xZ7PmUePMUWXt2rXIycnB66+/jtGjR2vbZe1wKrbPypUrMWjQIGRkZAj3x1NbMI4fP46FCxfiwIEDmDhxou5e4609/v3vf6Nr165YvHgxdu7ciT/84Q9o3bq1tv9Ubo+YEtadOnXCkSNHtN+HDx9Genp6E9ao8diwYQMWLVqE1157Da1bt0Zqaiqqq6uRnJyM4uJidOrUSdg+gwYNasJaR57c3FwUFRUhNzcXhw4dQlJSUty2BRDQFM877zwkJCSgR48eaNmyJbxeb9y2x5YtW3DZZZcBAPr164eamhrU19dr+/n2+Omnn0zbmzMxZQa59NJLsXr1agDAjh070KlTJ7Rq1aqJaxV9KioqMH/+fLz88stIS0sDAFxyySVaW6xZswZDhw7FwIED8e2336K8vBwnT57Eli1bMHjw4KasesRZsGABVqxYgXfffRc333wz7rvvvrhtCwC47LLLsHHjRvj9fhw7dgyVlZVx3R49e/bEtm3bAAD79+9Hy5YtceaZZ2Lz5s0AQu1x8cUXIzc3F7W1tSguLsbhw4dx1llnNWXVG0zMLYp5+umnsXnzZiiKgkceeQT9+vVr6ipFneXLl+OFF17A6aefrm2bN28eZs+ejZqaGnTt2hVz585FYmIiVq1ahcWLF0NRFEyYMAHXXXddE9Y8urzwwgvo1q0bLrvsMjz44INx2xbvvPMOcnJyAAC///3vMWDAgLhtj5MnT+Lhhx/G0aNHUV9fjylTpiA9PR1z5syB3+/HwIED8dBDDwEA3nrrLXzwwQdQFAVTp041JVBpbsScsCYIgiDMxJQZhCAIghBDwpogCKIZQMKaIAiiGUDCmiAIohlAwpogCKIZQMKaIAiiGUDCmiAIohlAwpogCKIZ8P8B1R2veR5/xo0AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "----\n",
            " caeve. iosirne y rpalthas. oid mted liur nnof avisave int Sed te irnd lrocovarrehclerk cte C,oces rertiwe atokt set ufro ssthrortirlt regbis Dfulh n 7adav vids f topg Thiatsvin nd if te hol niuso ce\n",
            "S \n",
            "----\n",
            "iter 900, loss 33.844870\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2AKpa1BGOItQ"
      },
      "source": [
        "# Quiz Question 7. \n",
        "\n",
        "Run the above code for 50000 iterations making sure that you have 100 hidden layers and time_steps is 40. What is the loss value you're seeing?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z6RUQUTFV_TP"
      },
      "source": [
        "Hidden_Layer_size = 100 #size of the hidden layer\n",
        "Time_steps = 40 # Number of time steps (length of the sequence) used for training\n",
        "learning_rate = 1e-1 # Learning Rate\n",
        "weight_sd = 0.1 #Standard deviation of weights for initialization\n",
        "z_size = Hidden_Layer_size + X_size #Size of concatenation(H, X) vector"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tYJ9XnfxV_Tg"
      },
      "source": [
        "class Param:\n",
        "    def __init__(self, name, value):\n",
        "      self.name = name\n",
        "      self.v = value # parameter value\n",
        "      self.d = np.zeros_like(value) # derivative\n",
        "      self.m = np.zeros_like(value) # momentum for Adagrad"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3Y9Hac-0V_Tj"
      },
      "source": [
        "size_a = Hidden_Layer_size\n",
        "size_b = z_size\n",
        "size_c = X_size\n",
        "\n",
        "class Parameters:\n",
        "    def __init__(self):\n",
        "        self.W_f = Param('W_f', np.random.randn(size_a, size_b) * weight_sd + 0.5)\n",
        "        self.b_f = Param('b_f', np.zeros((size_a, 1)))\n",
        "\n",
        "        self.W_i = Param('W_i', np.random.randn(size_a, size_b) * weight_sd + 0.5)\n",
        "        self.b_i = Param('b_i', np.zeros((size_a, 1)))\n",
        "\n",
        "        self.W_C = Param('W_C', np.random.randn(size_a, size_b) * weight_sd)\n",
        "        self.b_C = Param('b_C', np.zeros((size_a, 1)))\n",
        "\n",
        "        self.W_o = Param('W_o', np.random.randn(size_a, size_b) * weight_sd + 0.5)\n",
        "        self.b_o = Param('b_o', np.zeros((size_a, 1)))\n",
        "\n",
        "        #For final layer to predict the next character\n",
        "        self.W_v = Param('W_v', np.random.randn(X_size, size_a) * weight_sd)\n",
        "        self.b_v = Param('b_v', np.zeros((size_c, 1)))\n",
        "        \n",
        "    def all(self):\n",
        "        return [self.W_f, self.W_i, self.W_C, self.W_o, self.W_v,\n",
        "               self.b_f, self.b_i, self.b_C, self.b_o, self.b_v]\n",
        "        \n",
        "parameters = Parameters()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AlmYXzwMV_Tm"
      },
      "source": [
        "def forward(x, h_prev, C_prev, p = parameters):\n",
        "    assert x.shape == (X_size, 1)\n",
        "    assert h_prev.shape == (Hidden_Layer_size, 1)\n",
        "    assert C_prev.shape == (Hidden_Layer_size, 1)\n",
        "    \n",
        "    z = np.row_stack((h_prev, x))\n",
        "    f = sigmoid(np.dot(p.W_f.v, z) + p.b_f.v)\n",
        "    i = sigmoid(np.dot(p.W_i.v, z) + p.b_i.v)\n",
        "    C_bar = tanh(np.dot(p.W_C.v, z) + p.b_C.v)\n",
        "\n",
        "    C = f * C_prev + i * C_bar\n",
        "    o = sigmoid(np.dot(p.W_o.v, z) + p.b_o.v)\n",
        "    h = o * tanh(C)\n",
        "\n",
        "    v = np.dot(p.W_v.v, h) + p.b_v.v\n",
        "    y = np.exp(v) / np.sum(np.exp(v)) # softmax\n",
        "\n",
        "    return z, f, i, C_bar, C, o, h, v, y"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DRJR79lQV_Tq"
      },
      "source": [
        "z, f, i, C_bar, C, o, h, v, y = forward(np.zeros((X_size, 1)), np.zeros((Hidden_Layer_size, 1)), np.zeros((Hidden_Layer_size, 1)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O2FFMmzPV_Ts",
        "outputId": "aac58152-1a85-43f7-b64f-adcee94c1aa4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "z.shape, np.sum(z), np.sum(f)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((175, 1), 0.0, 50.0)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 101
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L3PFzyN0V_Tu"
      },
      "source": [
        "def backward(target, dh_next, dC_next, C_prev,\n",
        "             z, f, i, C_bar, C, o, h, v, y,\n",
        "             p = parameters):\n",
        "    \n",
        "    assert z.shape == (X_size + Hidden_Layer_size, 1)\n",
        "    assert v.shape == (X_size, 1)\n",
        "    assert y.shape == (X_size, 1)\n",
        "    \n",
        "    for param in [dh_next, dC_next, C_prev, f, i, C_bar, C, o, h]:\n",
        "        assert param.shape == (Hidden_Layer_size, 1)\n",
        "        \n",
        "    dv = np.copy(y)\n",
        "    dv[target] -= 1\n",
        "\n",
        "    p.W_v.d += np.dot(dv, h.T)\n",
        "    p.b_v.d += dv\n",
        "\n",
        "    dh = np.dot(p.W_v.v.T, dv)        \n",
        "    dh += dh_next\n",
        "    do = dh * tanh(C)\n",
        "    do = dsigmoid(o) * do\n",
        "    p.W_o.d += np.dot(do, z.T)\n",
        "    p.b_o.d += do\n",
        "\n",
        "    dC = np.copy(dC_next)\n",
        "    dC += dh * o * dtanh(tanh(C))\n",
        "    dC_bar = dC * i\n",
        "    dC_bar = dtanh(C_bar) * dC_bar\n",
        "    p.W_C.d += np.dot(dC_bar, z.T)\n",
        "    p.b_C.d += dC_bar\n",
        "\n",
        "    di = dC * C_bar\n",
        "    di = dsigmoid(i) * di\n",
        "    p.W_i.d += np.dot(di, z.T)\n",
        "    p.b_i.d += di\n",
        "\n",
        "    df = dC * C_prev\n",
        "    df = dsigmoid(f) * df\n",
        "    p.W_f.d += np.dot(df, z.T)\n",
        "    p.b_f.d += df\n",
        "\n",
        "    dz = (np.dot(p.W_f.v.T, df)\n",
        "         + np.dot(p.W_i.v.T, di)\n",
        "         + np.dot(p.W_C.v.T, dC_bar)\n",
        "         + np.dot(p.W_o.v.T, do))\n",
        "    dh_prev = dz[:Hidden_Layer_size, :]\n",
        "    dC_prev = f * dC\n",
        "    \n",
        "    return dh_prev, dC_prev"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8MDrEtXFV_Tw"
      },
      "source": [
        "def clear_gradients(params = parameters):\n",
        "    for p in params.all():\n",
        "        p.d.fill(0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KjvRuUavV_Tz"
      },
      "source": [
        "def clip_gradients(params = parameters):\n",
        "    for p in params.all():\n",
        "        np.clip(p.d, -1, 1, out=p.d)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "97gJZDpuV_T1"
      },
      "source": [
        "def forward_backward(inputs, targets, h_prev, C_prev):\n",
        "    global paramters\n",
        "    \n",
        "    # To store the values for each time step\n",
        "    x_s, z_s, f_s, i_s,  = {}, {}, {}, {}\n",
        "    C_bar_s, C_s, o_s, h_s = {}, {}, {}, {}\n",
        "    v_s, y_s =  {}, {}\n",
        "    \n",
        "    # Values at t - 1\n",
        "    h_s[-1] = np.copy(h_prev)\n",
        "    C_s[-1] = np.copy(C_prev)\n",
        "    \n",
        "    loss = 0\n",
        "    # Loop through time steps\n",
        "    assert len(inputs) == Time_steps\n",
        "    for t in range(len(inputs)):\n",
        "        x_s[t] = np.zeros((X_size, 1))\n",
        "        x_s[t][inputs[t]] = 1 # Input character\n",
        "        \n",
        "        (z_s[t], f_s[t], i_s[t],\n",
        "        C_bar_s[t], C_s[t], o_s[t], h_s[t],\n",
        "        v_s[t], y_s[t]) = \\\n",
        "            forward(x_s[t], h_s[t - 1], C_s[t - 1]) # Forward pass\n",
        "            \n",
        "        loss += -np.log(y_s[t][targets[t], 0]) # Loss for at t\n",
        "        \n",
        "    clear_gradients()\n",
        "\n",
        "    dh_next = np.zeros_like(h_s[0]) #dh from the next character\n",
        "    dC_next = np.zeros_like(C_s[0]) #dh from the next character\n",
        "\n",
        "    for t in reversed(range(len(inputs))):\n",
        "        # Backward pass\n",
        "        dh_next, dC_next = \\\n",
        "            backward(target = targets[t], dh_next = dh_next,\n",
        "                     dC_next = dC_next, C_prev = C_s[t-1],\n",
        "                     z = z_s[t], f = f_s[t], i = i_s[t], C_bar = C_bar_s[t],\n",
        "                     C = C_s[t], o = o_s[t], h = h_s[t], v = v_s[t],\n",
        "                     y = y_s[t])\n",
        "\n",
        "    clip_gradients()\n",
        "        \n",
        "    return loss, h_s[len(inputs) - 1], C_s[len(inputs) - 1]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6LGME4NDV_T3"
      },
      "source": [
        "def sample(h_prev, C_prev, first_char_idx, sentence_length):\n",
        "    x = np.zeros((X_size, 1))\n",
        "    x[first_char_idx] = 1\n",
        "\n",
        "    h = h_prev\n",
        "    C = C_prev\n",
        "\n",
        "    indexes = []\n",
        "    \n",
        "    for t in range(sentence_length):\n",
        "        _, _, _, _, C, _, h, _, p = forward(x, h, C)\n",
        "        idx = np.random.choice(range(X_size), p=p.ravel())\n",
        "        x = np.zeros((X_size, 1))\n",
        "        x[idx] = 1\n",
        "        indexes.append(idx)\n",
        "\n",
        "    return indexes"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dnxsLVJcV_T5"
      },
      "source": [
        "def update_status(inputs, h_prev, C_prev):\n",
        "    #initialized later\n",
        "    global plot_iter, plot_loss\n",
        "    global smooth_loss\n",
        "    \n",
        "    # Get predictions for 200 letters with current model\n",
        "\n",
        "    sample_idx = sample(h_prev, C_prev, inputs[0], 200)\n",
        "    txt = ''.join(idx_to_char[idx] for idx in sample_idx)\n",
        "\n",
        "    # Clear and plot\n",
        "    plt.plot(plot_iter, plot_loss)\n",
        "    display.clear_output(wait=True)\n",
        "    plt.show()\n",
        "\n",
        "    #Print prediction and loss\n",
        "    print(\"----\\n %s \\n----\" % (txt, ))\n",
        "    print(\"iter %d, loss %f\" % (iteration, smooth_loss))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MdTB6FReV_T7"
      },
      "source": [
        "def update_paramters(params = parameters):\n",
        "    for p in params.all():\n",
        "        p.m += p.d * p.d # Calculate sum of gradients\n",
        "        #print(learning_rate * dparam)\n",
        "        p.v += -(learning_rate * p.d / np.sqrt(p.m + 1e-8))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9fCk3IsGV_T-"
      },
      "source": [
        "# Exponential average of loss\n",
        "# Initialize to a error of a random model\n",
        "smooth_loss = -np.log(1.0 / X_size) * Time_steps\n",
        "\n",
        "iteration, pointer = 0, 0\n",
        "\n",
        "# For the graph\n",
        "plot_iter = np.zeros((0))\n",
        "plot_loss = np.zeros((0))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c3l9CR0EV_UA",
        "outputId": "b661507d-8d52-4e76-a0a9-8aeabee76b7a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 353
        }
      },
      "source": [
        "iter = 50000\n",
        "while iter > 0:\n",
        "  # Reset\n",
        "  if pointer + Time_steps >= len(data) or iteration == 0:\n",
        "      g_h_prev = np.zeros((Hidden_Layer_size, 1))\n",
        "      g_C_prev = np.zeros((Hidden_Layer_size, 1))\n",
        "      pointer = 0\n",
        "\n",
        "\n",
        "  inputs = ([char_to_idx[ch] \n",
        "              for ch in data[pointer: pointer + Time_steps]])\n",
        "  targets = ([char_to_idx[ch] \n",
        "              for ch in data[pointer + 1: pointer + Time_steps + 1]])\n",
        "\n",
        "  loss, g_h_prev, g_C_prev = \\\n",
        "      forward_backward(inputs, targets, g_h_prev, g_C_prev)\n",
        "  smooth_loss = smooth_loss * 0.999 + loss * 0.001\n",
        "\n",
        "  # Print every hundred steps\n",
        "  if iteration % 100 == 0:\n",
        "      update_status(inputs, g_h_prev, g_C_prev)\n",
        "\n",
        "  update_paramters()\n",
        "\n",
        "  plot_iter = np.append(plot_iter, [iteration])\n",
        "  plot_loss = np.append(plot_loss, [loss])\n",
        "\n",
        "  pointer += Time_steps\n",
        "  iteration += 1\n",
        "  iter = iter -1"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXEAAAD1CAYAAACm0cXeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dfUBUZd7/8fcwMAwoiiCj+ZD2oMkGoaSVmhWiLeq2WamZmr/KuuvW/GlZRua67d1uPpW/nty1vLVcq42NHm7u1cLNsjVDWqUITFM0FQlhBlQQGB6G8/uDHCU1yAbwwOf113jNmXO+l9aH4bqucy6LYRgGIiJiSn4tXYCIiJw7hbiIiIkpxEVETEwhLiJiYgpxERET82/Oi7ndbrKzs4mIiMBqtTbnpUVETMnj8eB0OomKisJut5/2frOGeHZ2NpMnT27OS4qItApvvPEGAwcOPK29WUM8IiLCW0zXrl2b89IiIqZ0+PBhJk+e7M3PH2vWED8xhNK1a1d69OjRnJcWETG1sw1Ba2JTRMTEFOIiIiamEBcRMTGFuIiIiSnERURMTCEuImJipgnxkcs+5YG121u6DBGR80qzrhP/JfYUHmdP4fGWLkNE5Lximm/iIiJyOoW4iIiJKcRFRExMIS4iYmKNCnG3282IESN49913yc/P584772TSpEnMmjWLqqoqAFJSUrjtttsYP348b7/9dpMWLSIidRoV4n/5y1/o2LEjAC+88AKTJk3izTffpFevXiQnJ1NeXs7y5ct57bXXWLt2LWvWrOHo0aNNWriIiDQixPfu3UtOTg433HADAOnp6cTHxwMQFxdHWloamZmZREdHExISgt1uJzY2loyMjCYtXEREGhHiixcvJjEx0fvniooKbDYbAOHh4TidTlwuF2FhYd5jwsLCcDqdTVCuiIic6idD/P3336d///707NnzjO8bhvGz2kVExLd+8o7NTZs2kZuby6ZNmzh8+DA2m43g4GDcbjd2u52CggIcDgcOhwOXy+X9XGFhIf3792/y4kVE2rqfDPHnnnvO+/rFF1+ke/fufPnll6SmpnLzzTezYcMGhg0bRkxMDPPnz6ekpASr1UpGRgbz5s1r8uJFRNq6n/3slJkzZ/LYY4+RlJREt27dGDt2LAEBAcyZM4dp06ZhsViYMWMGISEhTVGviIicotEhPnPmTO/rV1999bT3ExISSEhI8E1VIiLSKLpjU0TExBTiIiImphAXETExhbiIiIkpxEVETEwhLiJiYgpxERETU4iLiJiYQlxExMQU4iIiJqYQFxExMYW4iIiJKcRFRExMIS4iYmIKcRERE1OIi4iYWIObQlRUVJCYmEhRURGVlZVMnz6d1NRUduzYQWhoKADTpk3jhhtuICUlhTVr1uDn58eECRMYP358k3dARKQtazDEP/nkE6KiorjvvvvIy8vjnnvuYcCAATz88MPExcV5jysvL2f58uUkJycTEBDAuHHjGDlypDfoRUTE9xoM8dGjR3tf5+fn06VLlzMel5mZSXR0tHdvzdjYWDIyMhg+fLiPShURkR9r9Jj4xIkTeeSRR7y72L/++utMnTqVhx56iOLiYlwuF2FhYd7jw8LCcDqdvq9YRES8Gr1R8ltvvcXOnTt59NFHmTdvHqGhoURGRvLKK6/w0ksvMWDAgHrHG4bh82JFRKS+Br+JZ2dnk5+fD0BkZCQej4e+ffsSGRkJwPDhw9m9ezcOhwOXy+X9XGFhIQ6Ho4nKFhERaESIb9u2jdWrVwPgcrkoLy9nwYIF5ObmApCenk6fPn2IiYkhKyuLkpISysrKyMjIYODAgU1bvYhIG9fgcMrEiRN54oknmDRpEm63mwULFhAcHMzs2bMJCgoiODiYhQsXYrfbmTNnDtOmTcNisTBjxgzvJKeIiDSNBkPcbrfz7LPPntb+zjvvnNaWkJBAQkKCbyoTEZEG6Y5NERETU4iLiJiYQlxExMQU4iIiJqYQFxExMYW4iIiJKcRFRExMIS4iYmIKcRERE1OIi4iYmEJcRMTEFOIiIiamEBcRMTGFuIiIiSnERURMrMHniVdUVJCYmEhRURGVlZVMnz6dfv36MXfuXDweDxERESxduhSbzUZKSgpr1qzBz8+PCRMmMH78+Obog4hIm9VgiH/yySdERUVx3333kZeXxz333ENsbCyTJk1i1KhRLFu2jOTkZMaOHcvy5ctJTk4mICCAcePGMXLkSEJDQ31a8Lqv8xlzxQU+PaeIiFk1OJwyevRo7rvvPgDy8/Pp0qUL6enpxMfHAxAXF0daWhqZmZlER0cTEhKC3W4nNjaWjIwMnxf8p3Xf+PycIiJm1eA38RMmTpzI4cOHWbFiBXfffTc2mw2A8PBwnE4nLpeLsLAw7/FhYWE4nU7fVywiIl6NDvG33nqLnTt38uijj2IYhrf91NenOlu7iIj4ToPDKdnZ2eTn5wMQGRmJx+OhXbt2uN1uAAoKCnA4HDgcDlwul/dzhYWFOByOJipbRESgESG+bds2Vq9eDYDL5aK8vJwhQ4aQmpoKwIYNGxg2bBgxMTFkZWVRUlJCWVkZGRkZDBw40OcFWywWn59TRMSsGhxOmThxIk888QSTJk3C7XazYMECoqKieOyxx0hKSqJbt26MHTuWgIAA5syZw7Rp07BYLMyYMYOQkJDm6IOISJvVYIjb7XaeffbZ09pfffXV09oSEhJISEjwTWUiItIg3bEpImJipgtxDYmLiJxkuhAXEZGTTBfi+iYuInKS6UJcREROUoiLiJiY6ULcgsZTREROMF2Ii4jISaYLcU1sioicZLoQFxGRk0wX4voiLiJykulCXERETjJdiO8vKm/pEkREzhumC3ERETlJIS4iYmKN2mNzyZIlbN++nZqaGu6//34+/vhjduzYQWhoKADTpk3jhhtuICUlhTVr1uDn58eECRMYP358kxYvItLWNRjiW7duZc+ePSQlJXHkyBFuueUWrrnmGh5++GHi4uK8x5WXl7N8+XKSk5MJCAhg3LhxjBw50hv0IiLiew2G+KBBg7jiiisA6NChAxUVFXg8ntOOy8zMJDo62rslW2xsLBkZGQwfPtzHJYuIyAkNjolbrVaCg4MBSE5O5rrrrsNqtfL6668zdepUHnroIYqLi3G5XISFhXk/FxYWhtPpbLrKRUSkcWPiAB999BHJycmsXr2a7OxsQkNDiYyM5JVXXuGll15iwIAB9Y43DMPnxYqISH2NWp2yefNmVqxYwcqVKwkJCWHw4MFERkYCMHz4cHbv3o3D4cDlcnk/U1hYiMPhaJqqRUQEaESIl5aWsmTJEl5++WXvJOXMmTPJzc0FID09nT59+hATE0NWVhYlJSWUlZWRkZHBwIEDm7Z6EZE2rsHhlPXr13PkyBFmz57tbbv11luZPXs2QUFBBAcHs3DhQux2O3PmzGHatGlYLBZmzJjhneQUEZGm0WCI33777dx+++2ntd9yyy2ntSUkJJCQkOCbykREpEG6Y1NExMQU4iIiJqYQFxExMYW4iIiJKcRFRExMIS4iYmIKcRERE1OIi4iYmClDPOG5f3G8sqalyxARaXGmDPFdh0vZfuBIS5chItLiTBniIiJSRyEuImJipg1xbTohImLiEBcREYW4iIipNWqPzSVLlrB9+3Zqamq4//77iY6OZu7cuXg8HiIiIli6dCk2m42UlBTWrFmDn58fEyZMYPz48U1dv4hIm9ZgiG/dupU9e/aQlJTEkSNHuOWWWxg8eDCTJk1i1KhRLFu2jOTkZMaOHcvy5ctJTk4mICCAcePGMXLkSO+Wbr6mEXERkUYMpwwaNIjnn38egA4dOlBRUUF6ejrx8fEAxMXFkZaWRmZmJtHR0YSEhGC324mNjSUjI6NpqxcRaeMaDHGr1UpwcDAAycnJXHfddVRUVGCz2QAIDw/H6XTicrkICwvzfi4sLAyn09lEZYuICPyMic2PPvqI5ORkFixYUK/9bEv9mnwJoMZTREQaF+KbN29mxYoVrFy5kpCQEIKDg3G73QAUFBTgcDhwOBy4XC7vZwoLC3E4HE1TNWAoxUVEGg7x0tJSlixZwssvv+ydpBwyZAipqakAbNiwgWHDhhETE0NWVhYlJSWUlZWRkZHBwIEDm7Z6EZE2rsHVKevXr+fIkSPMnj3b27Zo0SLmz59PUlIS3bp1Y+zYsQQEBDBnzhymTZuGxWJhxowZhISENFnhj779NX+6JZqEqK5Ndg0RkfOdxWjG+9cPHTpEfHw8GzdupEePHj/rs70T152xff+iMb4oTUTkvNRQbuqOTRERE1OIi4iYmEJcRMTEFOIiIiamEBcRMTGFuIiIiSnERURMTCEuImJipg/xUnd1S5cgItJiTB/i0U9uIOnfB1u6DBGRFmH6EAd47J2sli5BRKRFtIoQFxFpqxTiIiImphAXETGxVhPivRPXsfqz71q6DBGRZtVqQhzgpU9yWroEEZFm1agQ3717NyNGjOD1118HIDExkZtuuok777yTO++8k02bNgGQkpLCbbfdxvjx43n77bebrOizKS6ravZrioi0pAa3ZysvL+epp55i8ODB9doffvhh4uLi6h23fPlykpOTCQgIYNy4cYwcOdK7L6eIiPheg9/EbTYbK1eubHDn+szMTKKjowkJCcFutxMbG0tGRobPChURkdM1GOL+/v7Y7fbT2l9//XWmTp3KQw89RHFxMS6Xi7CwMO/7YWFhOJ1O31YrIiL1NDicciY333wzoaGhREZG8sorr/DSSy8xYMCAesc04/7L9dTWGvj5WVrk2iIize2cVqcMHjyYyMhIAIYPH87u3btxOBy4XC7vMYWFhQ0OwTSFi+etx1PbMj9ARESa2zmF+MyZM8nNzQUgPT2dPn36EBMTQ1ZWFiUlJZSVlZGRkcHAgQN9Wmxj1dTWtsh1RUSaW4PDKdnZ2SxevJi8vDz8/f1JTU1lypQpzJ49m6CgIIKDg1m4cCF2u505c+Ywbdo0LBYLM2bMICQkpDn6cJqj5dU4QvywWDSsIiKtW4MhHhUVxdq1a09r//Wvf31aW0JCAgkJCb6p7Be4+umN/OmWKCZf3aulSxERaVKt6o7NU23e7Wr4IBERk2u1If7hjsMtXYKISJNrtSEO8Pi72ixCRFq3Vh3if/tC27aJSOvWqkMcYPVn33HoSHlLlyEi0iRafYj/1z++4a5X/93SZYiINIlWH+IAOYXH+Z+v8lq6DBERn2sTIQ4w662vqNXt+CLSyrSZEAd498s89hSUtnQZIiI+Y5oQH9S70y8+xyNvZzLy//3LB9WIiJwfTBPiL9wxoOGDGmnFp3t9di4RkZZkmhC/oGOQz8616INd/PfmfT47n4hISzFNiPvaH9ftpHfiOrbuK+LLg0eoqPK02EYWIiLn6px29mlNJr6y1fvaERLIBaFBTLqqJwN7hxHo70ePTsEtWJ2IyE8zVYi/fOeV5BaX88d1O5vk/IWllRSWVpKZe9TbFujvx4jILoS1szHlml4EBVi5MFzBLiLnh0aF+O7du5k+fTp33XUXU6ZMIT8/n7lz5+LxeIiIiGDp0qXYbDZSUlJYs2YNfn5+TJgwgfHjx/u02F9f3hWAe4ddzH/97zes3bqfak/TDoFU1tSyLisfgLVbD3jb5yZcxkXh7YjpGUrXDnbt6ykiLaLBEC8vL+epp55i8ODB3rYXXniBSZMmMWrUKJYtW0ZycjJjx45l+fLlJCcnExAQwLhx4xg5ciShoaFNUviCm37Fgpt+xcuf7mXhB7ua5Bo/ZcmH33pfB9usXNDRzn3DLubqi8NpH+hPREhgs9ckIm1PgxObNpuNlStX1tv0OD09nfj4eADi4uJIS0sjMzOT6OhoQkJCsNvtxMbGkpGR0XSV/+D+6y/hyl6/fA35L1Fe5WGvs4zEd7OIe2YTg/70ETPezOCN9AOs/+FbvIhIU2jwm7i/vz/+/vUPq6iowGazARAeHo7T6cTlchEWFuY9JiwsDKfT6eNyz+yd/xxC78R1zXKtxlr3dT7rvj4Z4N1Dg7j+sghui+2OYUB7uz8Xd26PgUGgv7UFKxURM/vFE5tnW5an5Xr15R2t4M30g7yZfvozzq++KIxrL+2M63gl4wf2xBESiAF06WBv/kJFxFTOKcSDg4Nxu93Y7XYKCgpwOBw4HA5crpP7WhYWFtK/f3+fFdqQ96YP4ZY/f95s1/Ol9O+KSf+uGIA1aScnT23+fsT06EiXDnYmX90LPws4Otjp2anuxid/a5td5i8iPzinEB8yZAipqancfPPNbNiwgWHDhhETE8P8+fMpKSnBarWSkZHBvHnzfF3vWQ24sGXHxZtCVU0t/95/BIB/fH362PrchMvoFdaOzENHub5vBN1Cg/DU1tLBHoDVz0J4e02uirR2DYZ4dnY2ixcvJi8vD39/f1JTU3nmmWdITEwkKSmJbt26MXbsWAICApgzZw7Tpk3DYrEwY8YMQkJCmqMPbdapK2Re+deZHyPQzmalrMoDwF8mx5IQ1ZXfp+xgyCWdKXFXc1XvMEKDAwgNttX7XG5xOeVVHi7rqn9DkfOZxWjGwetDhw4RHx/Pxo0b6dGjh8/Pf75NbppJTM9QnrzpVwTZrPwt/aB3WOfbPyZgs/phsdStgzcMgypPrSZjRZpJQ7lpqjs2G5L0H9dw+ym30UvjZeYePeOcwmXzP6Rz+0D8/SzMGtGHz3JcrPs6n+3zR1BrQPL2Qzxw/cVYLBa+c5VxUed2P3mdGk8tfhaLbo4S8ZFWFeJXXxze0iW0Sq7jlQA8/m6Wt+3KP37kfe0srcTfauGVf+3j/usvZtilERytqGJYnwisfha+zj3KpV3aU+MxGLLoY+L7OVh11yCyDh3DHuBHcKA/3UNPPqWyqqaWKk8tldUeOgQFEKAJXJGzalUhDnBd3wj+tbt51qdLndVbvvO+fvnTfbz86U8/5nfjrkLmvZd12nLLRbdG0zMsmGX/3M32A3UTuuOu7MEz42Oo9tSy4/sSZv4tg3X/dxjtbP68vS2X6B4dcVd7uLJX2JkuVU92Xt0PjUsdGueX1qPVhXiPTr577rg0nTOtl0885Zv+CcnbD+Fngb9vO+RtG7nsUyprajlaXu1te+3uQfToFMSdq74gNNjGyqlX8nlOERMG9SSn8DhPpuzgs5y6JbAzh1/Kry/viqfWoFtoEJ3b103q/nHdTm4Z0J2o7h2pqPKwz3Wc+e9n8/q0q2kX2Or+V5FWolVNbAJ8nuNi0n+nN8m5xXzuGXpRvd8UzmTe6H5MuroXUb9PBWDxbdE89k79HyiDLw5n8jUXcn3fCGoN6BgUcNbzGYZBRbWHYNu5Bf8b6QcIDbIx5ooLvG2Hj7np0iHQO8EsbUebmtgEGHJpZ779YwKXzf+wpUuR80BDAQ7w9Ppd7C447v3zjwMcIG1fEWn7irx/3r9oDAv+J5u/ph2gX9cQZo/oQ0JUXej+7Ytc5r2XxfMT+9OvawdqDYPICzo0uuYn3ssGYMwVYwDY6zxO/LOfkjiqHw9cf0mjzyNtQ6sLcUDL3+RnS95+qOGDTvGH/93BX39YhrnrcCkPvF73sLeHR/blpY9zAJj11lfe41+8YwA3xXQD4MPsfLbkFBEREkj+MTd/++Ige/406qwTuIeOVADw9rZcYi/sxFUXNTz+35QOFpWzdMO3PDs+Bpv/yZrd1R4efPNLnhgT2eAqpTP5zlXG/qIy4i5zNHyweLXKEAf46OHrGbHs05YuQ1qpV7fsP2P7sn/uPmP7/2Z+j9XPwvQ3zvxkzz+t28nQSzvXG6YZvHAj0+Mu5cuDdZO8e51lTHg5jXf+cwibvi3kxY9z6oV/QYkbq5+FLTkubrqiGw///SsiQgJ5YsyvfrIvY5dvoXunIJZPiq3X/uXBI3yTX8Lkq3vVa5/3Xhaf5biYMLAHw/pEeNvT9hXx0c4Cqj21rLnnqp+85pnEPbMJgN/95leMiHTQK/zn/yBoi1ptiF/qaE/G70YS+9Q/W7oUETZ8U8CGbwrO+v5rn+/ntc/312vLP+bmd+9nn3bsbX85uZ5/aeq3hAYHkLztEPtcZd72zXtcvP/V9wDYA6ys+Xw/1/bpzO2DLiS3uJxe4cE/PHStiq9yj/JV7lGWT6r7rOt4JbW1hve+gVND/On1O70TxGebTWtoku3eNdvoHmrnDzdHAVDtqWX+eyf7+dQ/vmHV5n18/nh8A2eCbw+XcqmjPdY2fN9Bqw1xgLB2NrYkDmfooo9buhSRJnG2xy2cOjz04g/DO+uzDrM+6/BZz9U7cR1j+3fzhv8JW/cVcfiYm0B/v3rX89QaeGoN/CywdV8x6394vs+JtRLfH61gTdp+Xv50H18/eSN3rvoCR0ggH+2s+2HWt2sIA3p2YsM3h0nallvvmiXuGo5X1nD4WAU9OgVjDzg5RLqnoJTenduxM7+E3760hdkj+jB7RN+f/HsyDIPSyho62M8+IW1WrW51ypkUlri56umNzXY9kbaub5f29SaLf4kre3XivmEX0aNTMJ/udrI09dt6799wWQSv3X0V+5zHefafu5l+wyXUeAwucbSnfaA/VTW1/DVtv3dv3vR58aZ6zHObW51yJg4T/YOJtAa+CnCA7QeOeG/+OpOKKg/LP8nxhvu6Mzzx81Rvph/kO1cZj4/uR+f2gdR4DLbkuFjx6V7+fv9gyqpqWJ+Vz4SBPRu9pLN34joGXBjKe9OHNr5jPtImQhzqloTpAVkirc+pz+NvjOc37gEgJfN7LnW0J6fw5A+cL3OPeuccTiw1fec/h3i3gDxaXkXHoAAsFguHj7kJb2/j28OldZ89eJRrF3/MJ4/cQFbeMfr3COXm5VvIO1pBxu9G+qSvZ9JmQhzgqbFRZ5woEpG26dQAh/qTxj9ue3xUPxZ+sItgm5W+XUL4KvcoVj8LntqTI9KHjlTwxHtZ9e4wBth1uIR+XRt/r8DP0aaeLHTnNb3Yv2hMS5chIia08INdQN3G6F/lHgWoF+An/DjAARKe28yH2U2zafo5fRNPT09n1qxZ9OnTB4C+ffty7733MnfuXDweDxERESxdutS7mbKISFv39Ppd3rt6femcv4lfddVVrF27lrVr1/K73/2OF154gUmTJvHmm2/Sq1cvkpOTfVmnT+1fNIa4yyIaPlBExEcOFpc3yXl9NpySnp5OfHzd4vy4uDjS0tJ8deomser/DOLuob1bugwRkV/knEM8JyeHBx54gDvuuIMtW7ZQUVHhHT4JDw/H6Ty/n+nt52fh9zddTsqDQxl3ZfOtWRcR8aVzGhPv3bs3Dz74IKNGjSI3N5epU6fi8Xi87zfj/UO/2BU9QnlmfCh9HO1584uDHChqml95RESawjl9E+/SpQujR4/GYrFw4YUX0rlzZ44dO4bb7QagoKAAh8NcTyK7//pL+MfMa/lg1rCWLkVEpNHOKcRTUlJYtWoVAE6nk6KiIm699VZSU+seqr9hwwaGDTNfGIbYA4i8oAP7F43hv26+vKXLERFp0DkNpwwfPpxHHnmEjRs3Ul1dzZNPPklkZCSPPfYYSUlJdOvWjbFjx/q61mY1dXBvpg7uzX9v3sfO/FLeyfh5z5sWEWkO5xTi7du3Z8WKFae1v/rqq7+4oPPNvcMuxjAMloy7gr99cZC0vUWsy2qaRfsiIj9Xm7rt/lxZLBasFphyTS8mX30hDzn7sCWniM/3ukjdcfZnRIuINDWF+M9ksVi41BHCpY4Q/s+Q3mTnHePf+4v5IOswX+xv/EN4RER8QSH+C0V170hU947cPfQiPtvjYp/rOKs/+479WqooIs1AIe5D1/bpzLV9OjN1cG+++b6EWsPg7W25rPlhQ10REV9TiDeRX3Wre+xkVPeOLLjpcmoNgw07Csg/VqHVLiLiMwrxZmD1s2DFwpgrTj7B7NkJMRSWuCkqq+KzPS4qazy892Uee51lP3EmEZH6FOItyNHBjqODncgL6r61Pzi8DzWeWmoNvLuFWP0sfJCdT6C/H0VlVfx7fzGl7ho9HkBEAIX4ecffWncTbXSPjt62E0MzP3a0vAqLxULekQqy844R0SGQL74rxkLdg+tf+3w/fbu0J/+Ym1J3TXOULyLNTCFuYqHBdU+N7BgU4A36uMtOPrPmyd+efHRA0fFK/K1+lLqrOVhcTgd7ANl5xygqq6JbqJ1t++s2ovXUGrybkUdNbd1vBCJyflOItxHh7QOBusDv0SkYqJt0PeGWAScfx7votiuo/SHBSytraB/oj6fWYE9hKZ2CbewvKqO80kPnkEC+PHiEjkEBBNusbN1XTJWnlqPlVThLK+nZKZg9hcfJyjuGzepHlae2Xk3tbFZqag0qa+q3i0jjKcTljPz8LEBd6EPd2Pzl3epCv1tokPe4/j1Dva8bs/VUtacWP4uF4+4aOgbXnbu4rAo/C9QasKeglNBgG/nHKjAMuKhzux+Gg6opddewfFMOdw+9iI5BAWzaVUiQzYqztJJ/7izg+r4RHCmrIvPQMW4f2BM/P/gg+zBllTVUewx6dArios7t2HW4lKAAK/nHKqj2GHSw+1PlqcVdrR8mYj4KcWlWAT+M+Z8IcICwdif3Yr364nAALusa4m3r3bmd9/Vtp2zg8duYbg1eb+Gtdb9VFJdX0fmH30ZOVVVTi82/rqai45UE+PtRcMxN5/aBBAdaCfS3kltcTpDNSuf2gZRV1mAAx901GBgUHa+issZDz07BuI5XUVNbS1g7G/ucZdgDrHQI8ie3uIKwdgFUewz2u8ro3imI749WEHlBB0rdNXyyq5BuoUH07RKC87ibAT07AfDCxj3s+L6EQ0fKSRzVj12HS3kj/SBXXRTG9X0jWJr6LQAPjejLgeIyLuho51+7XWTlHavXx5ieoWT+sLGvtD4Woxl3cDh06BDx8fFs3LiRHj20m45ISymvqsFm9aPWwPtD7FRHyqooKHVzQYcgag2DwAA/io5XEdbOxrYDR6itNYjr52DzHiffucp47fP9zB8TSef2gZRU1GD1s/D29lw+zyni4ZF9qak1iI90sLuglDtXfQHAlb06MfnqCwm2WXl960E+y3Gdtd47rupJ6o4CisuqmuzvpHTf9vsAAAcOSURBVKn17xnK+zOG/uzPNZSbCnEROS8cLa/yTtYD1HhqqayppV1g/QEDwzA4dKSCnmHBZzxPba1BTa1BRbWHYJuVAKsf6fuKCG8fSMegAAID/DAMCPT3470v89iVX8KatAPMiu/DQyP7su7rfNK/K+KvaQeI7+egT5cQCkrc3DKgO1072lm1+TtG/qoLBhAREkh4OxvFZVV8nXeM3YdLWbv1APYAP8LbBXJd3878a7eLiJDAcwpwUIiLiDToeGUNwQFW71zQ+aSh3NSYuIi0ee0DzRuFPq/86aefJjMzE4vFwrx587jiiit8fQkREfmBT0P8iy++4MCBAyQlJbF3717mzZtHUlKSLy8hIiKnOKeNks8mLS2NESNGAHDJJZdw7Ngxjh8/7stLiIjIKXwa4i6Xi06dOnn/HBYWhtPp9OUlRETkFD4N8R9rxoUvIiJtkk9D3OFw4HKdXLBfWFhIRESELy8hIiKn8OnE5tChQ3nxxReZOHEiO3bswOFw0L59e+/7Ho8HgMOHD/vysiIirdaJvDyRnz/m0xCPjY3l8ssvZ+LEiVgsFn7/+9/Xe//E+PjkyZN9eVkRkVbP6XTSq1ev09qb9Y5Nt9tNdnY2ERERWK3W5rqsiIhpeTwenE4nUVFR2O32095v1hAXERHfatLVKSIi0rRM8cCA1nIr/+7du5k+fTp33XUXU6ZMIT8/n7lz5+LxeIiIiGDp0qXYbDZSUlJYs2YNfn5+TJgwgfHjx1NdXU1iYiLff/89VquVhQsX0rNnT3bt2sWTTz4JwGWXXcYf/vCHlu3kjyxZsoTt27dTU1PD/fffT3R0dKvuc0VFBYmJiRQVFVFZWcn06dPp169fq+7zCW63m9/85jdMnz6dwYMHt+o+p6enM2vWLPr06QNA3759uffee1umz8Z5Lj093fiP//gPwzAMIycnx5gwYUILV3RuysrKjClTphjz58831q5daxiGYSQmJhrr1683DMMwnn32WeONN94wysrKjBtvvNEoKSkxKioqjDFjxhhHjhwx3n33XePJJ580DMMwNm/ebMyaNcswDMOYMmWKkZmZaRiGYTz88MPGpk2bWqB3Z5aWlmbce++9hmEYRnFxsXH99de3+j6vW7fOeOWVVwzDMIxDhw4ZN954Y6vv8wnLli0zbr31VuOdd95p9X3eunWrMXPmzHptLdXn8344pbXcym+z2Vi5ciUOx8mNjNPT04mPjwcgLi6OtLQ0MjMziY6OJiQkBLvdTmxsLBkZGaSlpTFy5EgAhgwZQkZGBlVVVeTl5Xl/MzlxjvPFoEGDeP755wHo0KEDFRUVrb7Po0eP5r777gMgPz+fLl26tPo+A+zdu5ecnBxuuOEGoPX/t30mLdXn8z7EW8ut/P7+/qfNLFdUVGCz1T0EPzw8HKfTicvlIiwszHvMif6e2u7n54fFYsHlctGhQwfvsSfOcb6wWq0EB9c9uD85OZnrrruu1ff5hIkTJ/LII48wb968NtHnxYsXk5iY6P1zW+hzTk4ODzzwAHfccQdbtmxpsT6bYkz8VEYrXUxztn79nPbz9e/mo48+Ijk5mdWrV3PjjTd621tzn9966y127tzJo48+Wq/G1tjn999/n/79+9OzZ88zvt8a+9y7d28efPBBRo0aRW5uLlOnTq13M05z9vm8/ybemm/lDw4Oxu12A1BQUIDD4Thjf0+0n/ipXF1djWEYREREcPToyQ1wT5zjfLJ582ZWrFjBypUrCQkJafV9zs7OJj8/H4DIyEg8Hg/t2rVr1X3etGkTGzduZMKECbz99tv8+c9/bvX/zl26dGH06NFYLBYuvPBCOnfuzLFjx1qkz+d9iA8dOpTU1FSAM97Kb2ZDhgzx9m3Dhg0MGzaMmJgYsrKyKCkpoaysjIyMDAYOHMjQoUP58MMPAfjkk0+4+uqrCQgI4OKLL2bbtm31znG+KC0tZcmSJbz88suEhoYCrb/P27ZtY/Xq1UDdUGB5eXmr7/Nzzz3HO++8w9///nfGjx/P9OnTW32fU1JSWLVqFVB3J2VRURG33npri/TZFDf7PPPMM2zbts17K3+/fv1auqSfLTs7m8WLF5OXl4e/vz9dunThmWeeITExkcrKSrp168bChQsJCAjgww8/ZNWqVVgsFqZMmcJvf/tbPB4P8+fPZ//+/dhsNhYtWsQFF1xATk4OCxYsoLa2lpiYGB5//PGW7qpXUlISL774IhdddJG3bdGiRcyfP7/V9tntdvPEE0+Qn5+P2+3mwQcfJCoqiscee6zV9vlUL774It27d+faa69t1X0+fvw4jzzyCCUlJVRXV/Pggw8SGRnZIn02RYiLiMiZnffDKSIicnYKcRERE1OIi4iYmEJcRMTEFOIiIiamEBcRMTGFuIiIiSnERURM7P8DI6rMrEWOXWkAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "----\n",
            " tisuthe Arseds, TARS afle. The s if coronavirus, and havela?\n",
            "Is and the Cariny, Tuttrle Wasler infection, but tes To thas and that the simentiffrone hrapts on a dot tratsinf cases as atyor acliag, and \n",
            "----\n",
            "iter 49900, loss 31.809082\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q4Ln2wisWdoN"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}