2_6_Attention/1_2. Attention Basics, solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [SOLUTION] Attention Basics\n",
    "In this notebook, we look at how attention is implemented. We will focus on implementing attention in isolation from a larger model. That's because when implementing attention in a real-world model, a lot of the focus goes into piping the data and juggling the various vectors rather than the concepts of attention themselves.\n",
    "\n",
    "We will implement attention scoring as well as calculating an attention context vector.\n",
    "\n",
    "## Attention Scoring\n",
    "### Inputs to the scoring function\n",
    "Let's start by looking at the inputs we'll give to the scoring function. We will assume we're in the first step in the decoging phase. The first input to the scoring function is the hidden state of decoder (assuming a toy RNN with three hidden nodes -- not usable in real life, but easier to illustrate):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "dec_hidden_state = [5,1,20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize this vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAJAAAAEYCAYAAACz0n+5AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAARsUlEQVR4nO2dfZBV5X3HP19Qx4SVF0vFFYhmqiGNtqAijSNtMCIiQbEdhsIfEawpNhNrnKZTrdUocZI4dmKGSI1uhQlmLEmjQZkEEeKYUdKovAR8A4QoDrvlJQZ5WRNCF379456Vy3Jfdu9zz332nP19mDN7z3PO2ee57Gd+z/Oce57flZnhOLXSL3YDnGzjAjlBuEBOEC6QE4QL5AThAjlBuEA5QtJISc9LelPSG5K+nJSfLmmVpK3JzyFlrp+dnLNV0uxu1en3gfKDpGag2czWSzoNWAdcB8wB9prZfZJuB4aY2W1drj0dWAuMBSy59mIze79SnR6BcoSZ7TSz9cnrg8AmYDgwDVicnLaYglRduQpYZWZ7E2lWAZOr1ekC5RRJ5wAXAi8Dw8xsZ3JoFzCsxCXDgR1F+61JWUVOCmtmt/A+8hgq3pmneT3+v7mHe24C5hYVtZhZy3GVSE3Ak8CtZnZAOlatmZmkuv1NGiEQ257f1ohqej3nXn7u8QUqfV4l7Ki1AC3ljks6mYI8j5vZj5Pi3ZKazWxnMk7aU+LSNmBC0f4I4OfV2uNdWEQk9Xir8vsELAQ2mdkDRYeWAZ2zqtnA0yUufxaYJGlIMkublJRVxAWKiWrYKnMZ8Hngs5I2JNsU4D7gSklbgYnJPpLGSnoUwMz2AvcCa5Lta0lZRRrShTmlqRZReoqZraa8ZleUOH8t8IWi/UXAop7U6QLFpL7+RMEFiki9I1AMXKCYZN8fFygmeYhAPgtzgvAIFJE8RCAXKCbZ98cFiooL5ITgXZgTRvb9cYFion7ZN8gFioh3YU4Y2ffHBYqJRyAnjOz74wJFxQVyQvAuzAkj+/64QDHxCOSEkX1/XKCYKAcGuUAxyb4/LlBMfAzkhJGCP5IWAVOBPWZ2QVL2Q2BUcspgYJ+ZjSlx7XbgIHAE6DCzsdXqc4EiklIE+h6wAHiss8DM/raozm8B+ytcf7mZvdfdylygmKTgj5m9kKR2ObG6grEzgM/Wqz5flRGT+q+Nr8ZfArvNbGuZ4waslLRO0twy5xyHR6CI1NKFJX/YivmBKjALWFLh+Hgza5N0BrBK0mYze6HSL3SBYlJLfiCrnB+obFXSScDfABdX+N1tyc89kpYC44CKAnkXFpF65weqwkRgs5m1lmnLgCQxJ5IGUMgP9Hq1X+oCxSSFMZCkJcAvgVGSWiXdmByaSZfuS9JZkpYnu8OA1ZI2Aq8APzWzFdXqy30XdsMdN/CRUz9Cv3796N+vP/PvmB+7SR+SxkP1ZjarTPmcEmX/C0xJXr8NjO5pfbkXCOCb//RNBjUNit2ME+gTd6IlfZJCnuHOlK9twDIz25Rmw/oE2fen8hhI0m3ADyi81VeSTcCSJON5r0cSd82/i1u+cQvPvPhM7OYcT+PvA9WdahHoRuB8M/u/4kJJDwBvkCRr7M3c/8/3M3TIUPYd2Med8+9k5JkjueC8C2I3C8jH4xzVZmFHgbNKlDcnx0oiaa6ktZLWtrT0+JZFXRk6ZCgAgwcO5tIxl7LlnS1R23McfSAC3Qo8l6SH7UyD/zHgXODmchd1udllsRKNH/rDIY7aUT566kc59IdDrN+0nlmfKzlJiULuB9FmtkLSJyjckSweRK8xsyNpNy6U9w+8z9cf/joAR44e4TOXfIax51d9QqFxZN+f6rMwMzsKvNSAttSd5j9uZsFdC2I3oyy5j0BOymTfHxcoJh6BnDCy748LFBOPQE4Y2ffHBYqKC+SE4F2YE0b2/XGBYuIRyAkj+/64QDHxCOSEkX1/XKCY5OGBMhcoJjlYVOUCRcTHQE4Y2fcnD0E0u6SxtFnSIkl7JL1eVHaPpDZJG5JtSplrJ0vaImlbd1fduEAxSeeh+u8Bk0uUf9vMxiTb8q4HJfUH/gO4GvgUMEvSp6pV5gJFJI0IlKRj2VtDc8YB28zsbTM7TGE94LRqF7lAMWnssp6bJb2adHFDShwfzrGVNwCtHFtIURYXKCK1RKDiNXfJ1p1MYt8F/gQYA+wEvlWv9+CzsJg0KMGUme3+sErpP4GflDitDRhZtD8iKauIR6CINCrBlKTmot2/pnTiqDXAeZI+LukUCvmEllX73R6BYpJOnuglwARgqKRW4G5ggqQxFJJobgduSs49C3jUzKaYWYekm4Fngf7AIjN7o1p9LlBM0knzW2rt9sIy536YYCrZXw6cMMWvhAsUEf8owwkj+/64QDHJQwTyWZgThEegiOQhArlAMcm+Py5QTDwCOWFk3x8XKCoukBNCGl910GhcoIj4GMgJI/v+NEagcy8/txHVZA6PQN2kfX97I6rp9TQNajq+IPv+eBcWE49AThjZ98cFiolHICeM7Pvjj3M4YXgEioh3YU4Y2ffHBYqJRyAnjOz74wLFJI0IJGkRMBXYY2YXJGX/DlwDHAZ+DdxgZvtKXLsdOAgcATrMrOrXO/osLCaNyw+0CrjAzP4ceAv41wrXX57kEOrWd4O6QBFpVH4gM1tpZh3J7ksUEifUBRcoJnG+9vvvgGfKHDNgpaR13Uwb42OgqNQgRPKHLf7jtiQpX7pz7b8BHcDjZU4Zb2Ztks4AVknanES0srhAEallEF1LfqCkrjkUBtdXmJmV+d1tyc89kpZSSHtXUSDvwmLSoC5M0mTgX4Brzex3Zc4ZIOm0ztfAJErnEToOFygiKaX5XQL8EhglqVXSjcAC4DQK3dIGSQ8n554lqTOdyzBgtaSNwCvAT81sRbX6vAuLSBqrMmrND2RmbwOje1qfCxQTvxPthJCHz8J8DOQE4REoInmIQC5QTLLvjwsUFRfICcG7MCeM7PvjAsXEI5ATRvb9cYFi4hHICSP7/rhAMfEI5ISRfX9coKi4QE4IeejCcv9p/Lx75zHxqonMmDkjdlNOJM6qjLqSe4Gu+dw1PDj/wdjNKEmjvjM1TXIv0EUXXcSggYNiN6M0fTkCSbqhng3pi6iGf72NkAg0r9wBSXMlrZW0tqWlx0uY+gzqpx5vvY2KszBJr5Y7RGEZSEm6LH4zzxNdht7nQ4+pNo0fBlwFvN+lXMD/pNKiPkRvHBT3lGpd2E+AJjN7t8u2Hfh56q2rA3fceQdzbpzD9ne3c/XUq3nq6adiN+kYKQyiJS2StEfS60Vlp0taJWlr8nNImWtnJ+dslTS7W2+hzDLpeuJdWELToKbjFPjZXT/r8X/+xHsnVtRI0l8B7cBjRQmm7gf2mtl9km4HhpjZbV2uOx1YC4ylkKVjHXCxmXXtfY4j99P43kyj8gMB04DFyevFwHUlLr0KWGVmexNpVnFioqoT8I8yYtK4IdAwM9uZvN5F6QnQcGBH0X5rUlYRj0ARqSUCFd8iSbZuJYLqJEntUrdxi0egmNQQgWrMD7RbUrOZ7ZTUDOwpcU4bMKFofwTdmCh5BIpIAz8LWwZ0zqpmA0+XOOdZYJKkIcksbVJSVhEXKCbpTONL5Qe6D7hS0lZgYrKPpLGSHgUws73AvcCaZPtaUlYR78IiksaNxDL5gQCuKHHuWuALRfuLgEU9qc8Fikn2b0S7QFFxgZwQeuPjGT3FBYpJ9v1xgWKSh0/jXaCYZN8fFygmHoGcMLLvjwsUE49AThjZ98cFiklvXGXRU1ygmGTfHxcoJj4GcsLIvj8uUEw8AjlhZN8fFygmHoGcMLLvjwsUE38eyAkj+/64QFFxgZwQfBDthJF9f3xhYUzqvTJV0ihJG4q2A5Ju7XLOBEn7i875ash78AgUkzpHIDPbAowBkNSfwnr3pSVOfdHMptajThcoIimPga4Afm1m76ZZiXdhMUk3T/RMYEmZY5dK2ijpGUnn19T2BBcoImnlB5J0CnAt8KMS1a4Hzjaz0cCDwFMh78G7sJiklx/oamC9me0ucf2BotfLJT0kaaiZvdfz1jRIoKZBTY2oJnOkOAaaRZnuS9KZwG4zM0njKPRCv621ooYINE9lk9r3Ke62u48vSMEfSQOAK4Gbisr+AcDMHgamA1+U1AH8HphpAal6vQuLSBoP1ZvZB8AfdSl7uOj1AmBBvepzgWKSgzvRLlBE8vBZmE/jnSA8AkUkDxHIBYpJ9v1xgWLiEcgJI/v+uEBRcYGcELwLc8LIvj8uUEw8AjlhZN8fFygmHoGcMLLvjwsUE49AThjZ98cFiolHICeM7PvjzwM5YXgEioh3YU4Y2ffHBYqJf9WBE4R3YU4Y6Sws3A4cBI4AHWY2tstxAfOBKcDvgDlmtr7W+lygiKQYgS6vsNb9auC8ZPsL4LvJz5rwaXxM0k3vUo5pwGNW4CVgsKTmWn+ZCxSTdAQyYKWkdaVSvwDDgR1F+61JWU14FxaRWrqwRIpiMVqSlC+djDezNklnAKskbTazFwKbWhYXKCYp5Acys7bk5x5JS4FxQLFAbcDIov0RSVlNeBcWkRSytA6QdFrna2AS8HqX05YB16vAp4H9Zraz1vfgESgm9Z+EDQOWJqKdBPyXma3okh9oOYUp/DYK0/gbQip0gSJS72m8mb0NjC5RXpwfyIAv1atOFygm2b8RnT+BBo4YyHWPXUfTsCbMjPUt63n5Oy9z6pBTmf7D6Qw+ZzD7tu/jiRlPcGjfoahtzcNHGbkbRB/tOMrKr6zkofMfYuGnF3LJly5h6J8OZfzt43nnuXdY8IkFvPPcO4y/fXzspuaCqgJJ+qSkKyQ1dSmfnF6zaqd9Vzu7frULgMPth/nNpt8wcPhARk0bxcbFGwHYuHgjo64bFbOZQP1nYTGoKJCkW4CngX8EXpc0rejwN9JsWD0YdPYgmi9spvXlVpqGNdG+qx0oSNY0rBekHo7zUUZdqTYG+nvgYjNrl3QO8ISkc8xsPr3y7Rzj5AEnM+PJGay4dQWHDx4+4XhAZtv60av/B7tHNYH6mVk7gJltlzSBgkRnU+HtF99uf+SRR+rT0h7Q76R+zHhyBq89/hqbl24GoH13O01nFqJQ05lNfLDng4a3qyu9sUvqKdXGQLsljencSWSaCgwF/qzcRWbWYmZjzWzs3LmlPs9Ll2sXXst7m97jpW+/9GHZW8veYvTswi2S0bNHs+XpLQ1v1wn0gS7seqCjuMDMOijcCm98aOkGIy8byejrR7P71d3c9KtCsvbn7niO1fetZvp/T+fCGy9k/7v7+dGMUt9D0ljyEIEqCmRmrRWO/aL+zQlnxy92lP1qhe9P/H6DW1OF7PuTvxuJWcIfqneCyH0X5qRM9v1xgaLiAjkheBfmhJF9f1ygmHgEcsLIvj8uUEyUA4NcoJhk3x8XKCY+BnLCyL4/LlBMPAI5YWTfn/ytysgUdX6gTNJISc9LelPSG5K+XOKcCZL2S9qQbF8NeQsegSKSQhfWAXzFzNYna+TXSVplZm92Oe9FM5tajwo9AsWkzhHIzHZ2pqszs4PAJgJy/3QHFygitawLkzRX0tqireRD58kqmguBl0scvlTSRknPSDo/5D14FxaTFPIDASSLQJ8EbjWzA10OrwfOTpZqTQGeopAvsSY8AkUkjZWpkk6mIM/jZvbjrsfN7EDRUq3lwMmShtb6HlygmNR/FiZgIbDJzB4oc86ZyXlIGkfBgd/W+ha8C4tICrOwy4DPA69J2pCU3QF8DD7MEzQd+KKkDuD3wEwLWKbrAsWkzvHfzFZTJU6Z2QJgQb3qdIEi4o9zOGFk3x8XKCb+YaoTRvb9cYFi4hHICSP7/rhAMfEI5ISRfX9coJh4BHLCyL4/LlBUXCAnBO/CnDCy748LFJM8RCD1ioztDUDS3C7fLerUgb70RGLjM573AfqSQE4KuEBOEH1JIB//pECfGUQ76dCXIpCTArkXSNJkSVskbZN0e+z25I1cd2GS+gNvAVcCrcAaYFaJbBVOjeQ9Ao0DtpnZ22Z2GPgBMK3KNU4PyLtAw4EdRfutpJzupK+Rd4GclMm7QG3AyKL9EUmZUyfyLtAa4DxJH5d0CjATWBa5Tbki149zmFmHpJuBZ4H+wCIzeyNys3JFrqfxTvrkvQtzUsYFcoJwgZwgXCAnCBfICcIFcoJwgZwgXCAniP8H3cTVc2cEZOAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 108x324 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# Let's visualize our decoder hidden state\n",
    "plt.figure(figsize=(1.5, 4.5))\n",
    "sns.heatmap(np.transpose(np.matrix(dec_hidden_state)), annot=True, cmap=sns.light_palette(\"purple\", as_cmap=True), linewidths=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our first scoring function will score a single annotation (encoder hidden state), which looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotation = [3,12,45] #e.g. Encoder hidden state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIYAAAEYCAYAAACZYo4WAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAO4klEQVR4nO3df5BV5X3H8feXXVBWNgoVESWCo6bEQkODoTimDXElUmNikmYyEYOZjiOaKa2pIUoyZhLalDEzJiTTHymrQdPUX9SYxGEalYoMTYhIUaooUdFiI/IjtKgwjou7fPvHPQuX9dk9e+6ee55z735eM3fYe+7ecx8uH57n/Pqex9wdkb5GxG6AlJOCIUEKhgQpGBKkYEiQgiFBCkYTMrMWM3vSzFYnz+8ws/82sy3JY0baOlrr3kqJ4TpgG/CuqmVfdvf7BrsC9RhNxswmAR8FbhvKehSM5vNd4AbgcJ/lf2tmT5nZcjM7Lm0lRQwlOuZ+lB3z7C7L/N3YFVwDLKxa1OnunQBmdimw1903m9mcqt/5CrAbGAV0AjcCfz3Q5xSyjXHw9f1FfEzpjTlxbJ8lFvy9gbgf7qTyjxtyAfBxM7sEOB54l5n9i7t/Lnm9y8xuBxanfY6GkpjMsj8G4O5fcfdJ7j4F+Cyw1t0/Z2YTKx9nBnwC2JrWNO2VRJW9x6jRnWY2PvnALcC1aW9QMKKqXzDcfR2wLvn5wqzvVzCiKqzHyEzBiCllmyEmBSOq8gZDeyUSpB4jJg0lEqZgSIDXEIyioqRgxKShRMIUDAlSMCTEynu0QMGISj2GBCkYEqK9EglTMCRIwZAgBUNCSryNUd4daYlKPUZU6jEkJOfygaOrfUdR85lmttHMtpvZvWY2Km0dCkZUVsNjUHqLmnt9C1ju7mcD+4Gr0lagYESVfzD6FjUnRUYXAr2V7j+kUnQ0IG1jxFSfvZLvUilqbk+e/w7wmrt3J89fAU5PW4l6jKiy9xhmttDM/rPqcaTAubqoeagtU48RVS1FzZ6pqBn4HnCSmbUmvcYkYGfa56jHiMixzI8B1xcuar4CeBT4dPJrnwd+ltY2BSOmOu2uBtwIXG9m26lsc/wg7Q0aSqIqrKj5JWBWlvcrGFGV98inghFTiU+iNXUwurq6uPqaL3Do0CF6enro6LiQaxdeHbtZVcq7idfUwRg1ahT/9I9/T1tbG293d3PV1Qu54PzzmT59WuymlV5qMMxsKnAZR4+W7QQecPdt/b+rHMyMtrY2ALq7u+nu7i7XsF7ioWTAvszMbgTuofJ1Pp48DLjbzJbUv3lD19PTw+VXLGDuxX/C7FmzmD6tTL1F3U6iDVlaj3EV8Hvu/nb1QjP7DvAMcHO9GpaXlpYW7r7zRxw4cIAv3XAj2198kbPPOit2sxIN2mNQubvsaYHlE3nnnWePqD6e39nZ39HbYrW3t3PezJls+NVjsZtyVHEHuDJL6zG+CDxiZi8Av0mWnQGcDSzq7019jud7rBvA7t+/n9bWVtrb23nrrbfYuPFxPn/lgihtCStvjzFgMNz9QTN7D5WjZtUbn5vcvafejRuqffv28fWlf0PP4R78sHPRRR388R99MHazqpQ3GFbA9JrReoyyGXPi2GOScHj1tMxf/ohLtxaSpqY+jlF+5e0xFIyoFAwJUjAkpMRHPhWMqBQMCVIwJMBLPJSU94IAiUo9RlTqMSQo39PuZna8mT1uZv9lZs+Y2dJkuWZqbij5b2N0ARe6+0EzGwn8wsx+nryWaaZmBSOqfIPhlRNfB5OnI5NHTSfDNJREVZdq9xYz2wLsBda4+8bkpUwzNSsYMdmIzI+BipoB3L3H3WdQqVGdZWbTqMzUPBX4ADCOSmXagDSURJV7UXP1771mZo8C89z9lmSxZmpuCDlf2mdm483spOTn0cBc4Neaqbnh5L5XMhH4oZm1UPlPv8rdV5vZWs3U3FBy3yt5CviDwHLN1NxYynvkU8GIqcQn0RSMqBQMCVIwJKS8uVAw4ipvMhSMqBQMCdFeiYTUMrd7URSMqBQMCVIwJETbGBKmYEiQgiEhGkokTMGQoPJeWalgxFTioaS8kZWoCukxxpw4toiPaUDl7TEKCcabLz9SxMeUXtvkjmMXaCiRsMKq3TWFd2PJvXa1t9r9fcAMYJ6ZzUZTeDeYnCvRvCJU7Z55Cm8FI6r6V7sDL6IpvBtN9mBkrXanUuWemQ5wRVVItfv5aArvBlNMtfs2apjCWz1GRHW45rO/avdngXvM7JvAk2gK77IrrNpdU3g3lBIf+VQwolIwJEjBkCAFQ0LKmwsFI67yJkPBiErBkBDtrkpYec9IKBgxqceQMAVDghQMCVIwJETbGBKmYEiQgiEhGkokTMGQgJrmvSyIghFVeXuM8h6sHw7yLx94t5k9ambPJkXN1yXLv2FmO6um8L4krWnqMaLKvcfoBr7k7k+YWTuw2czWJK8tr5pmM5WCEVXu5QO7gF3JzwfMbBuDqFMN0VASU85DybGrtilUakx6p/BelEzhvdLMxqa9X8GIKv+iZgAzGwP8GPiiu78BfB84i8o9M3YB305rWdMNJd/49o9Y/9jTjDupnftu/RoAyzvvZ/1jTzNyZAuTJo5n6eIFtI9pi9xSqEdRs5mNpBKKO939/uQ9e6pevxVYnfY5TddjfGzubP5h2aJjls1+/1T+9dabWLXiJiZPOoWV9zwUqXV95L9XYlTqUre5+3eqlk+s+rVPMhyn8J75++fw6u7/PWbZ+eede+Tn6VPP5N//48mim9WP3PdKLgAWAE8nN08B+CpwuZnNoHJMbQdwTdqKag6Gmf2Zu99e6/tj+dlDG/jIh2bGbkYi972SX/Sz0n/Luq6hDCVL+3uhegOpszP1Hh+Fue2un9PS0sIlHZkKv+vHRmR/FGTAHsPMnurvJWBCf+/rs4HkZbjP5wMP/4r1G7ey4lvXYaU5q1mWdrxT2lAyAbiYyi0AqxmwoS4tqoNfbnqGO1at4bZb/orRx6fe4rJAjRuM1cAYd9/S9wUzW1ePBg3VkmUr2fzU87z2+kEunv9Vrl3wUW6/92EOHXqbLyz5OwCmv3cKN103P3JLKXMuMPe6n/wtxVBSBm2TO46JwqHNN2X+8kfN/GYhcWq63dXGUt4uQ8GISsGQkNLsHb2TghGVgiEBmttdwjSUSJiCIUEKhoRoKJEwBUOCFAwJUjAkRNsYElbeYDTdVeKSDwUjpuKKmseZ2RozeyH5U5Vo5Za9Ei1Fb1HzucBs4M/N7FxgCfCIu58DPJI8H5CCEdWIGh79c/dd7v5E8vMBKjMong5cRmWGZhjkTM3a+IypjnslfYqaJySV8AC7GeAK/17qMaIqrKj5CK9c5Jt6ral6jKiKKWoG9pjZRHffldSx7k37HPUYMRVU1Aw8QGWGZtBMzY2gsKLmm4FVZnYV8DLwmbQVKRhRFVbUDNCRZV0KRkSucyUSpmBIkIIhQQqGhJQ3FwpGXOVNhoIRlYIhIdpdlTAFQ4IUDAlSMCRkuG9jtE3OdP5mGBnmweCu8n4BhZrf98Kp8n4vGkpiKvAW0FkpGFGpx5CQ4b7xKf0pbzDKO8hJVOoxolKPIQFulvmRJpk+c6+Zba1alnmmZgUjqtyLmgHuAOYFli939xnJI3UqLAUjqvyD4e7rgf8bassUjJjqOFNzgGZqbhz1KWoO0EzNjSX/ouZ+3qOZmhtKQUOJZmpuOPkfxzCzu4E5wMlm9grwdWBOYTM1Sx7yD4a7Xx5Y/IOs61EwoirvkU8FIyadXZUwBUOCFAwJKfFQouMYEqQeI6ry9hgKRky6SlzC1GNIkIIhISXeK1EwItLc7tIPBUNCNJRImIIhQQqGBCkYEqJtDAlTMArXcxj+9NYzmNDezYr5r7LkpxN4/OU22o/rAeDmT+zhvad2RW5leTVtMP5540mcdfIhDnYdPVF1w9zfMu/cgxFb1UcdhhIzWwlcCux192nJsnHAvcAUKleJf8bd9w+0ntTTe2Y21cw6kikbq5eHCmdLYfcbrax7YQyffv/rsZuSorCi5nxnajazv6Qy495fAFvN7LKql5cNppUxLHtwPF++6LeM6PM9Ll97Mh/7/mSWPTieQ91lGN8LK2rOfabmq4GZ7n4wmfn3PjOb4u7fG1QrI3j0+RMYd0IP007rYuOO0UeWX9+xj/Fjeni7x/ja6lPo/OVYFn1oyEXhQ1TYV5h5pua0YIxw94MA7r7DzOZQCcdkBvhbJYW2CwFWrFjBwjH9/Wb+nvif0ax97gTWv3AmXd3Gwa4RLL7/VG751G4ARrU6n5rxBis3pBZ8118N2xjV322iM6lnHRR3dzNLnanZKjM699uItcD17r6lalkrsBK4wt1bBtOWWDeA3bhjNCs3jGXF/FfZe6CFU9p7cIdlD43nuFZn8UX7im3QfD/mi3hz58bUf6C+2k7/w9QvM+ndV1dtfD4HzKmaqXmdu//uQOtI6zGuBLqrF7h7N3Clma1Ia2CZLL5/IvvfbMEdpp7axdJL96S/qe4K+w/TO1PzzQxypuYBe4ycROsxSqdvj/Hqpuw9xmkfGPDLrC5qBvZQKWr+KbAKOINkpmZ3H3ADq2mPYzSG/C8G7qeoGTRTcwPRuRIJUzAkQNd8Sj8UDAkp8TZGeWvkJCr1GFGVt8dQMGIq8VCiYESlYEiQgiFBCoaEaBtDwhQMCVIwJERDiYQpGBKkYEiIhhIJUzAkSMGQgpjZDuAA0AN0u/t5taxHwYipfreM/rC7D6maSsGIqMzXfOoKrpjqM72mAw+b2eZBTtYbpB4jqroUNX/Q3Xea2SnAGjP7dXJrhEwUjKjyn6nZ3Xcmf+41s58As4DMwdBQElW+N04xsxPMrL33Z+AjDGJW5hD1GDHlf+RzAvATq6y3FbjL3R+sZUUKRlT5BsPdXwLel8e6FIyoyru7qmDEpJNoEqZgSJCCISElHkp0HEOC1GNEVd4eo4i79pWCmS3McqPU4W44DSU1n2kcjoZTMCQDBUOChlMwtH2RwbDZ+JRshlOPIRk0fTDMbJ6ZPWdm280sdconqWjqocTMWoDngbnAK8Am4HJ3fzZqwxpAs/cYs4Dt7v6Sux8C7qEyP5ikaPZgnA78pur5K8kySdHswZAaNXswdgLvrno+KVkmKZo9GJuAc8zsTDMbBXyWyvxgkqKpT7u7e7eZLQIeAlqAle7+TORmNYSm3l2V2jX7UCI1UjAkSMGQIAVDghQMCVIwJEjBkCAFQ4L+Hx82PtLP1tEZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 108x324 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's visualize the single annotation\n",
    "plt.figure(figsize=(1.5, 4.5))\n",
    "sns.heatmap(np.transpose(np.matrix(annotation)), annot=True, cmap=sns.light_palette(\"orange\", as_cmap=True), linewidths=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IMPLEMENT: Scoring a Single Annotation\n",
    "Let's calculate the dot product of a single annotation. Numpy's [dot()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html) is a good candidate for this operation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "927"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def single_dot_attention_score(dec_hidden_state, enc_hidden_state):\n",
    "    # TODO: return the dot product of the two vectors\n",
    "    return np.dot(dec_hidden_state, enc_hidden_state)\n",
    "    \n",
    "single_dot_attention_score(dec_hidden_state, annotation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Annotations Matrix\n",
    "Let's now look at scoring all the annotations at once. To do that, here's our annotation matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotations = np.transpose([[3,12,45], [59,2,5], [1,43,5], [4,3,45.3]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And it can be visualized like this (each column is a hidden state of an encoder time step):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVoAAAD4CAYAAACt8i4nAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAWq0lEQVR4nO3de5xVdbnH8c8zG1DmIjMIjigk3hJvpWGmUl5AkrwEaZqhREVMqZWmpmR1FE0j62i9jueYk6jEEbx78FhSHgTJCzdNEcTSUEMSEJxRhtvM7HnOH7ONkWD2jDO//Vuu+b5fr/Vi77Vmr/XMctbXZ//Wb8+YuyMiIuEUxS5ARCTtFLQiIoEpaEVEAlPQiogEpqAVEQmsWwGOoWkNItJW1uE9TLO2Z85o7/jx2qAQQUtdzapCHCbxSit2h2kF+e+afKOdundqYleRCKW9KqhbtyJ2GYlQuuuATthL8q6xggStiEjBmIJWRCQwBa2ISGAKWhGRsCx5k6kUtCKSMsnraJMX/SIiKaOOVkTSRbMORERCU9CKiATl7QjaQkWyglZE0kWzDkREQtPQgYhIYApaEZGwNOtARCQ0Ba2ISFi6GSYiEpo6WhGRwBS0IiKBKWhFRMJK4KyD5I0ai4ikjDpaEUkXzToQEQkteUMHCloRSRkFrYhIWAm8GaagFZGUUdCKiASmoBURCcrbMetAf2FBROQDUUcrIhKYglZEJKxOnHVgZq8B64Es0OjuR5hZb+BuYCDwGnCWu9e0tp9UB+2WLVsYf953qa9vIJvNMmzocXxr/Ndjl1VQQ3+5NyU7NVFkTqYIHqj6Oy+t6sGVv6tkY30Re5Y38IvTV1G6U1PsUgtq4jU/4U9PPEnvigruuWta7HKiy2azjPn6+fTt24df/eLa2OV0UKd3tCe4+9oWzycAs9x9kplNyD2/vLUdpDpoe/Towa9vupHi4mIaGhsZV/Vthhz9KQ495ODYpRXUlLEr6F28NUh/+L+7c/nwtzhy4Cbu+/Mu3PpkBRcNXRexwsI77ZRTOOvML3LlVVfHLiURpt/zIAMHfoQNGzbGLqUTBB86GAkcn3s8BZhDnqBN3oeCO5GZUVxcDEBjYyONjY0kcfym0F5b151P7rUJgCH7bOSPy0ojV1R4n/jE4fTaZZfYZSTC6jVv8cRT8xl12smxS+kcZm1ezKzKzBa1WKq22ZsDfzSzZ1psq3T3N3OPVwGV+UrK29Ga2SCaE3zP3KqVwEPuvqxN33Rk2WyWc79axYo3VnLWGaM49JCDYpdUWAbjpvbHDL40+B2+NPgd9u9bz6y/lHDioA3MfLGUN9/tHrtKiejff/lfXHjBeDZsTEM3C+1ppty9Gqhu5Us+7e4rzWw34FEze2mb17uZeb7jtNrRmtnlwF00V74gtxgwPTc2kXiZTIbpUyfzyEP3suTFZbzyt+WxSyqo6V9bwYPf/Du/OWcldy4sZ+HrPbl25CqmLSzn9OqPsGFLET0yeX9OJKXmPjmPiopyDhz00dildCJrx9I6d1+Z+3cN8CBwJLDazPoB5P5dk28/+TraccDB7t7wvm/D7AZgKTBpey/KtdhVALfccgujz/x8vjqCKysr44jBh/PUvAXst+8+scspmMpdGgHYtSTL8EF1LF65M+OOqeG2MSsBeHVdd+a83PWGDqTZ84uXMPeJp3ny6QXU19dTt2EjP7rqp/zkqh/ELu2D66RZB2ZWAhS5+/rc488CVwMPAWNpzr+xwIx8+8oXtE3AHsDr26zvl9u2Xdu0415XsypfHUHU1NTSrVuGsrIyNm/ewvwFixg7ZnSUWmLYWG80OZTu5GysN578WzHnH7eOdRsy7FqSpcnh5rm7cvYRtbFLlUi+c943+M553wBg0bPPMXXavR/ukAU68T5MJfCgNQd3N2Cau880s4XAPWY2juZsPCvfjvIF7UXALDN7GViRW/cRYD/g2x+s9sJZu3YdV15zHdlsE+7OicOO59hPHxO7rIJZt6EbF9y9BwDZJjj1kPUcu99GpswrZ9rCcgCGH1jHGYe9G7HKOK740Y9Z9Myz1NbW8rlTT+Ob48czamT8d17SCTrpF3+7+3Lg49tZvw4Y1q6S3FsfnzOzIprHJVreDFvo7tk2HiNaR5s0pRW7wzTNegBgtFP3TqtzvLuM0l4V1K1bkf8Lu4DSXQd0+AJpevjQNt90KDr1hYJckHlnHbh7EzCvALWIiHSC5DUzqf7Agoh0RQpaEZGw9BcWRERCU9CKiATVnl/8XSjJq0hEJGXU0YpIymjoQEQkMAWtiEhYmnUgIhJa8m49KWhFJF3U0YqIhKagFREJTEErIhKWhg5EREJT0IqIBKZZByIiYWnoQEQkNAWtiEhgCloRkbCSl7MKWhFJm+QlrYJWRNIlgb/4W0ErIqni6mhFREJT0IqIBKagFREJSx9YEBEJLXk3w5JXkYhIR5i1fWnT7ixjZn82s4dzz/c2s/lm9oqZ3W1mPfLtQ0ErIilj7Vja5EJgWYvnPwNudPf9gBpgXL4dKGhFJGU6L2jNrD9wCnBr7rkBQ4H7cl8yBRiVbz8KWhFJmU7taH8JXAY05Z7vCtS6e2Pu+RvAnvl2oqAVkXRpxxitmVWZ2aIWS9XW3dipwBp3f6ajJRVk1kFpxe6FOMyHw2iPXUFilPaqiF1CYpTuOiB2CSnS9uld7l4NVO9g8xDg82Z2MrAzsAvwK6DczLrlutr+wMp8xylI0G5c8adCHCbxigd8hrratbHLSITS8j5kHx0au4xEyAx/jLqaVbHLSIROaco6aR6tu/8A+EHzLu144FJ3P8fM7gW+CNwFjAVm5NuXhg5EJGU6fdbBti4HLjazV2ges52c7wX6wIKIpEznfzLM3ecAc3KPlwNHtuf1CloRSRd9BFdEJDQFrYhIYMm79aSgFZF00dCBiEhY+gsLIiLBKWhFRMLS0IGISGi6GSYiEpg6WhGRsJKXswpaEUmb5CWtglZEUkZBKyISlmYdiIiEplkHIiJhqaMVEQlNQSsiEpiCVkQkMAWtiEhYppthIiKBqaMVEQlLsw5EREJT0IqIBOWxC9gOBa2IpIw6WhGRsDTrQEQkNHW0IiJhadaBiEhoCtrgrvr57cydv5je5WXcd+vVANx4y73Mnfc83btl6L/Hbkz8/tcoKy2OXGlhrVq9mn+76hrefrsGM/jCqJGMPvus2GUVXLbJOfP6t6jsVcTN5/XhR3fWsPTv9bjDwN26ce2YCkp2St4YXyhbtmxh/Hnfpb6+gWw2y7Chx/Gt8V+PXVYHJS9oU/cTddpJQ/jPn170vnVHDT6Ie2+dyD2/mche/Su5bfrv4xQXUSaT4XsXfof77r6TOyZXc+99D7B8+auxyyq4qbPr2Ldya38x4fRePPiDSv7nikr6VWSY9viGiNUVXo8ePfj1TTdy13/fxrSpk3nq6QW8sGRp7LI6xqztS6u7sZ3NbIGZPW9mS81sYm793mY238xeMbO7zaxHvpJSF7SDP/ZRepWVvG/d0UccTLdMBoBDD9yH1W/VxCgtqr59+nDgoAMAKCkpYe+Be7HmrbciV1VYq2qyPL50C2ccs/Xno7Rn8yXg7mxuSOTwXlBmRnFx87u7xsZGGhsbSWJH2D5F7VhatQUY6u4fBw4DRpjZUcDPgBvdfT+gBhjXloo+EDP72gd9bUwzZj7BkCMPiV1GVP/4x5u89NeXOeTgg2OXUlCT7q/l0lG7ULRNjlwxtYZjr1jFq6sbOOe4ku2/OMWy2SxfHjOO4Z8bxVFHHsGhhxwUu6QOsnYsO+bN6nJPu+cWB4YC9+XWTwFG5auoIx3txB1tMLMqM1tkZouqq6s7cIjOdeudD5PJZDh52FGxS4lm48aNfH/CD7n0e9+ltLTrhMqcFzbRuyzDwR/513d5142pYM61u7PP7t155JlNEaqLK5PJMH3qZB556F6WvLiMV/62PHZJHdOOoYOWWZVbqt6/K8uY2XPAGuBR4G9Arbs35r7kDWDPfCW1ejPMzBbvaBNQuaPXuXs18F7C+sYVf8pXR3AP/eFJ5s5bzC0/vwTrau8PcxoaG/n+hB/yuRGfZegJx8cup6CeXV7P7Bc2MXfpZrY0OBs2O5dNeZvrx/YGIFNknDy4J5MfXc/pR3ed/wG1VFZWxhGDD+epeQvYb999YpfTAW2/vrfJqu1tzwKHmVk58CAw6INUlG/WQSVwEs3jEC0Z8NQHOWAMTy5Ywh13z+TWGy6j5847xS4nCnfnmp/8lL0H7sW5o8+OXU7BXTyyFxeP7AXAgr9u4fZZ6/nZVyp4/a1G9urbDXfnscWb2buye+RKC6umppZu3TKUlZWxefMW5i9YxNgxo2OX1UGd30i5e62ZzQaOBsrNrFuuq+0PrMz3+nxB+zBQ6u7PbbvBzOa0v9zwJlxbzTPP/4Xad+o46ezv862xn+f26b+nvqGR8y6/AWi+Ifaji8ZErrSwnnt+Mb97ZCb77bcvXz53LAAXnPdNPj3kmMiVxePePD5bt6kJBw7YsztXfqk8dlkFtXbtOq685jqy2SbcnROHHc+xn/6Q/0x00jtWM+sLNORCticwnOYbYbOBLwJ3AWOBGXn35R78d90kYuggCYoHfIa62rWxy0iE0vI+ZB8dGruMRMgMf4y6mlWxy0iE0ordO5yS9c/8uM2h1mPwNTs8npl9jOabXRma72fd4+5Xm9k+NIdsb+DPwLnuvqW146TuAwsi0tV1Tkfr7ouBw7ezfjlwZHv2paAVkXRJ4M1uBa2IpIyCVkQkKFfQiogEpl/8LSISmjpaEZGwdDNMRCQ0Ba2ISGAKWhGRwBS0IiJhJXDWQfIqEhFJGXW0IpIumnUgIhKaglZEJDAFrYhIWAm8GaagFZGUUUcrIhKYglZEJCzNOhARCU1BKyISmIJWRCQo16wDEZHQ1NGKiASmoBURCSt5OaugFZG0SV7SKmhFJGUUtCIiYWnWgYhIaMnraJMX/SIiHWLtWFrZi9kAM5ttZi+a2VIzuzC3vreZPWpmL+f+rchXkYJWRNLFrO1L6xqBS9z9IOAo4AIzOwiYAMxy9/2BWbnnrZfk7h38rvIKfgARSY0Ov+/f/NIdbc6cnQd9tc3HM7MZwE255Xh3f9PM+gFz3P2A1l5bkDHa7MxjCnGYxMuMeIoNa1+LXUYilPQZqHORU9JnINmZR8cuIxEyI57u+E4C3Awzs4HA4cB8oNLd38xtWgVU5nu9hg5EJGXaPkZrZlVmtqjFUvUvezMrBe4HLnL3d1tu8+YhgbwdtGYdiEjKtH30wd2rgeod7smsO80he6e7P5BbvdrM+rUYOliT7zjqaEUkXTrpZpiZGTAZWObuN7TY9BAwNvd4LDAjX0nqaEUkZTptHu0QYAzwgpk9l1t3BTAJuMfMxgGvA2fl25GCVkRkO9z9CXac2sPasy8FrYikin7xt4hIcMn7CK6CVkRSRkErIhKW/ty4iEhoCloRkbB0M0xEJDR1tCIigSloRUQCU9CKiISlWQciIqEpaEVEwkrgrIPkVSQikjLqaEUkZTR0ICISmIJWRCQszToQEQlNQSsiElYCZx0oaEUkVVwdrYhIaApaEZGwdDNMRCQ0Ba2ISGC6GSYiEpaGDkREQlPQFky2yTnzF+9Q2auIm7+5C1fcWcfCVxoo7dn8H+G60aUc2D+13/52nXLGVygp7klRURGZTIY7b7spdknR6Fy8d428m7tGynLXSGOLa6Sky10joaT2LE59fDP7Vmao2+z/XHfpyGJOOmyniFXFd8t/XE9Fea/YZSRCVz8XO75GekSsqhMkcOggeaPGnWBVbZbHl9ZzxtE7xy5FJJFW1Tbx+NIGzjg6jY2HtWMpjLxBa2aDzGyYmZVus35EuLI6ZtIDG7l0ZAlF25zHX/1uI6Mm1TLpgQ3UN/r2X5xiZnDB965g9Ncv4P4Zv49dTlRd/VxMemADl44s3sE18s6H/BopasdSGK0OHZjZd4ELgGXAZDO70N1n5DZfB8wMXF+7zVlST+9S4+AB3VjwcsM/13/v1GL67GI0ZOHKuzZw6/9t4vwRxRErLbzbbr6B3fr24e2aWs67aAID9xrA4MMOjV1WFF35XDRfI0VtuEY2c/6InhEr/YA6cejAzG4DTgXWuPshuXW9gbuBgcBrwFnuXtPafvJF+nhgsLuPAo4HfmxmF75XQyvFVZnZIjNbVF1dnfeb6UzPvtrA7CUNnDixhkumrGf+yw1c9tv19O1VhJnRo5vxhU/txAuvNxa0riTYrW8fAHpXlHPCsUNY+uJLkSuKpyufi2dfbWT2knpOnFjLJVPqctdIXYqukU4dOrgD2Pbd+wRglrvvD8zKPW9VvpthRe5eB+Dur5nZ8cB9ZrZXa1W6ezXwXsJ6duYd+eroNBefVsLFp5UAsODlBm5/bBPXf6WMt95pom+vItydWS/Us3+/TMFqSoJNmzbT1NRESUkxmzZtZt6CZxj/tXNilxVFVz8XF59WzMWnNb+ba75GNnP9V0pTdI10Xkfr7nPNbOA2q0fS3HgCTAHmAJe3tp98QbvazA5z9+dyB60zs1OB24AP1fusy6au5+06xx0G7Znhyi+V5n9Riqx7u4ZLrpgIQLYxy4jPnsCQoz4Zuao4dC6277KpddtcIx/SobV2DB2YWRVQ1WJVda5RbE2lu7+Ze7wKqMx7HPcdD3ibWX+g0d1XbWfbEHd/Mt8BAM/OPKYNX5Z+mRFPsWHta7HLSISSPgN1LnJK+gwkO/Po2GUkQmbE0x1uRzf+Y1Gb7+IV73FE3uPlOtqHW4zR1rp7eYvtNe5e0do+Wu1o3f2NVra1JWRFRAor/C/+Xm1m/dz9TTPrB6zJ94JUzqMVka4s+Dzah4CxucdjgRmtfC2goBWRlHGszUs+ZjYdeBo4wMzeMLNxwCRguJm9DJyYe96q1H4EV0S6qk6ddfDlHWwa1p79KGhFJF0S+LsOFLQikjIKWhGRsNTRioiEpqAVEQlMQSsiEpiCVkQkLI3RioiElrzPYSloRSRd1NGKiISmoBURCUxBKyISloYORERCU9CKiASmoBURCUtDByIioSloRUSCastfTig0Ba2IpIuGDkREQtNHcEVEAlNHKyISloYORERCU9CKiASmoBURCUtDByIioWnWgYhIWAnsaJMX/SIiKaOOVkRSJnkdrbl77BoKwsyq3L06dh1JoHOxlc7FVjoX4XSloYOq2AUkiM7FVjoXW+lcBNKVglZEJAoFrYhIYF0paDX2tJXOxVY6F1vpXATSZW6GiYjE0pU6WhGRKBS0IiKBpT5ozWyEmf3FzF4xswmx64nJzG4zszVmtiR2LTGZ2QAzm21mL5rZUjO7MHZNsZjZzma2wMyez52LibFrSqNUj9GaWQb4KzAceANYCHzZ3V+MWlgkZnYsUAf81t0PiV1PLGbWD+jn7s+aWRnwDDCqK/5cmJkBJe5eZ2bdgSeAC919XuTSUiXtHe2RwCvuvtzd64G7gJGRa4rG3ecCb8euIzZ3f9Pdn809Xg8sA/aMW1Uc3qwu97R7bklv9xVJ2oN2T2BFi+dv0EUvKNk+MxsIHA7Mj1xKNGaWMbPngDXAo+7eZc9FKGkPWpEdMrNS4H7gInd/N3Y9sbh71t0PA/oDR5pZlx1WCiXtQbsSGNDief/cOunicuOR9wN3uvsDsetJAnevBWYDIyKXkjppD9qFwP5mtreZ9QDOBh6KXJNElrsBNBlY5u43xK4nJjPra2blucc9ab5x/FLUolIo1UHr7o3At4E/0HzD4x53Xxq3qnjMbDrwNHCAmb1hZuNi1xTJEGAMMNTMnsstJ8cuKpJ+wGwzW0xzY/Kouz8cuabUSfX0LhGRJEh1RysikgQKWhGRwBS0IiKBKWhFRAJT0IqIBKagFREJTEErIhLY/wMeI0NKb63cIwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's visualize our annotation (each column is an annotation)\n",
    "ax = sns.heatmap(annotations, annot=True, cmap=sns.light_palette(\"orange\", as_cmap=True), linewidths=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IMPLEMENT: Scoring All Annotations at Once\n",
    "Let's calculate the scores of all the annotations in one step using matrix multiplication. Let's continue to us the dot scoring method\n",
    "\n",
    "<img src=\"images/scoring_functions.png\" />\n",
    "\n",
    "To do that, we'll have to transpose `dec_hidden_state` and [matrix multiply](https://docs.scipy.org/doc/numpy/reference/generated/numpy.matmul.html) it with `annotations`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([927., 397., 148., 929.])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def dot_attention_score(dec_hidden_state, annotations):\n",
    "    # TODO: return the product of dec_hidden_state transpose and enc_hidden_states\n",
    "    return np.matmul(np.transpose(dec_hidden_state), annotations)\n",
    "    \n",
    "attention_weights_raw = dot_attention_score(dec_hidden_state, annotations)\n",
    "attention_weights_raw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at these scores, can you guess which of the four vectors will get the most attention from the decoder at this time step?\n",
    "\n",
    "## Softmax\n",
    "Now that we have our scores, let's apply softmax:\n",
    "<img src=\"images/softmax.png\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.19202922e-001, 7.94715151e-232, 5.76614420e-340, 8.80797078e-001],\n",
       "      dtype=float128)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def softmax(x):\n",
    "    x = np.array(x, dtype=np.float128)\n",
    "    e_x = np.exp(x)\n",
    "    return e_x / e_x.sum(axis=0) \n",
    "\n",
    "attention_weights = softmax(attention_weights_raw)\n",
    "attention_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even when knowing which annotation will get the most focus, it's interesting to see how drastic softmax makes the end score become. The first and last annotation had the respective scores of 927 and 929. But after softmax, the attention they'll get is 0.119 and 0.880 respectively.\n",
    "\n",
    "# Applying the scores back on the annotations\n",
    "Now that we have our scores, let's multiply each annotation by its score to proceed closer to the attention context vector. This is the multiplication part of this formula (we'll tackle the summation part in the latter cells)\n",
    "\n",
    "<img src=\"images/Context_vector.png\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3.57608766e-001, 4.68881939e-230, 5.76614420e-340,\n",
       "        3.52318831e+000],\n",
       "       [1.43043506e+000, 1.58943030e-231, 2.47944200e-338,\n",
       "        2.64239123e+000],\n",
       "       [5.36413149e+000, 3.97357575e-231, 2.88307210e-339,\n",
       "        3.99001076e+001]], dtype=float128)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def apply_attention_scores(attention_weights, annotations):\n",
    "    # TODO: Multiple the annotations by their weights\n",
    "    return attention_weights * annotations\n",
    "\n",
    "applied_attention = apply_attention_scores(attention_weights, annotations)\n",
    "applied_attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize how the context vector looks now that we've applied the attention scores back on it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVoAAAD4CAYAAACt8i4nAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAiTUlEQVR4nO3deXgUVdrG4d+bBBUSCMmYYBQkAVFAVFREHEQUBB03RFFHUJDxMyr7KLiNG+LCOCAuiALKuIyCijoqMy6oKCKKAoOMCIoLgohJGLYkSCCd8/3RbUwkpBPS1d0Uz31ddZGuqj79dtE8OZw6VW3OOURExDsJsS5ARMTvFLQiIh5T0IqIeExBKyLiMQWtiIjHkqLwGprWICI1ZXVu4Vmreeb0dXV/vRqIRtBStHlDNF4m7qWkplO0eWOsy4gLKalpOhYhKalpFBesjHUZcSE5o1UEWolKdtZKVIJWRCRqTEErIuIxBa2IiMcUtCIi3rL4m0yloBURn4m/Hm38Rb+IiM+oRysi/qJZByIiXlPQioh4ytUiaKMVyQpaEfEXzToQEfGahg5ERDymoBUR8ZZmHYiIeE1BKyLiLZ0MExHxmnq0IiIeU9CKiHhMQSsi4q04nHUQf6PGIiI+ox6tiPiLZh2IiHhNQwciIh6zWizVtGK2n5l9YmafmdkyMxsdWv+EmX1nZktCS/twFalHKyL+ErmTYSVAN+dckZnVA+aZ2euhbaOcczNr2pCCVkR8JjJB65xzQFHoYb3Q4nanLQ0diIjP1HzowMxyzWxhhSW3UktmiWa2BMgHZjvnFoQ23WVmS81sgpntG64i9WhFxFdcLWYdOOemAFOq2R4A2ptZY+BlM2sH3Aj8BOwTeu71wB3VvY56tCLiM5E5GVaRc24TMAc43Tm3zgWVAH8HOoZ7voJWRHwmYrMOMkI9WcysPtADWGFmWaF1BpwLfB6uIg0diIi/RG7WQRbwpJklEuyUPu+cm2Vm75pZBsGkXgJcFa6hPT5o53/0EePG30+gLMC5vc5h4ID+lbbPfPElnp/5IokJidRvUJ+bb7yBFi1yAFi58mvuuuevFBcXYwnG009MY999w45rR10gEODSAQPJyMjggQnjK20bf9/9LFy0CIBt27axYeNG3n/37Rq3/e833uDJp57GOUhu0IAbr7+OQw9tRUlJCVdceTXbt28nEAjQvXs3rsq9AoC1a3/kxptvZvPmLbRpfRhjRt9OvXr1IveGoyD4uZlAoKysys+NH5WUbOf/hlzP9u07CATK6H5KZ66+vF+lfV7999vcP2kamfv/DoCLzj+L3mefFoty6yBisw6WAkdXsb5bbdvao4M2EAgw9t7xTJr4AE0yM7l0wJ/o2qVLeZACnH7aafQ5/zwA3p/7Affd/wATH7yf0tJSbr7tdsbcfhuHHtqKTZs2k5QUn4dj+oznyM7Opri4eKdt114zovznGc89z5dffVWrtg868ECmPvoIjRo14sP587nznnt46u/T2GeffXh00kQaNGjAjtJSLr8il84nnMARR7TjwYkP0+/iizmtZw/uvuev/POVV7mgz/l1fZtRE/zcjGPSxAdDn5uBO31u/Giffeox+YG7adCgfvDv9Orr6Hz8sRzZrnWl/Xp268IN11wdoyojQVeGRdSyZV/QrGlTmh50EPXq1aNnz1N5b+7cSvukpCSX//zzzz9jof9WfLzgE1odcgiHHtoKgMaNU0lMTIxe8TWUl5fPvA/nc26vc8Lu++ZbszmtZ4/yx089/Q8uHTCQi/r249EpU6t8zlFHHkmjRo0AOKJdO/LzCwAwMxo0aABAaWkppaWlYOCc49OFC+ne7RQAzjrzDN57f26VbcernT83PXb63PhR8O+0PhD6Ow0Eyv89+IpZzZcoCduFM7PWQC/goNCqtcCrzrnlXhZWE/kFBTRpkln+uElmJp8vW7bTfs+/MJN/PDuD0h07eHTSRABWr16NmTF46Ag2btrIaT16MKD/JVGrvabGT5jA8KFDKN66c2+2onXr1rH2xx85rkMHAD76eAGr16zhqSem4Zzjz9eOYvHi/3DMMTv9T6jcP199jd+f0Kn8cSAQ4JL+l7Hmhx+4sM/5HNGuHRs3baJhw4blvf/MJpkUFBRE4J1GT00/N34UCATod/kI1qxdx4W9z+SIww/baZ9335/P4s+W0bzZgVw79AoOaJIRg0rrIv5+eVTbozWz64EZBCv/JLQYMN3MbvC+vMi48II+vPryTIYOGcRj0/4OQGkgwJIln3HnmNt5fOpk5rz3Pp988mmMK61s7gfzSEtLo02b1mH3ffOt2Zza7ZTyXvnHCxbw8YIF9L2kP/0uHcCq779n9Zo1u3z+pwsX8cqrrzJsyJDydYmJiUx/5mlen/Uqn3/xBV9/803d35TEVGJiIjOeeIg3XnqCZcu/4utvV1XaflLnjsx6YRrPPzmR4zscza13TYhNoXUS+elddRWuR3s5cLhzbkfFlWZ2H7AMGFvVk0JXV+QCTJ48mb4X9YlAqTvLzMggLy+//HFefj4ZGbv+7Xtazx7c89e/AcFezNFHtyetcWMAOnc+gRVffknHjsd5Uuvu+GzpUuZ+8AEfzp/P9pLtFBUXc/Ott3HnHaN32vet2W9z/XUjyx875xg4YADnn9e70n7PvzCTl//5CgAP3n8fGRkZrFy5kjF33c1D90+gcePUndpu2LAhHY49lvkffcyl/fpSWFhIaWkpSUlJ5OdVf8zjUW0/N37UsGEKHY45kvkfL+aQFtnl6xunNir/uffZPXnwkb/HoLo6isPhkHBjtGXAgVWszwptq5JzbopzroNzrkNubu6udquztm3bsGbNGtau/ZEdO3bw1ltv07VLl0r7rF79ay9u3ocfcnCzZgCc0Ol4vv7mG37eto3S0lIWL/4POTnxdTJk6OBBvD7rNWa98k/uvmsMx3XoUGXIfrdqFVsKt3DkEUeUrzuhUydeee01tm7dCkB+fj4bNmzgwgv6MP2Zp5n+zNNkZGSw7qefGHn9jYwZfRvNmx9c/vyNGzdSWFgIBGczLFjwCdnNm2NmdDj2WN55dw4As/71b7p2rXzM493On5vZO31u/Gjjxs0UFgYv3d9WUsLHn/6H7OZNK+1TsH5D+c/vz1tAdvNmUa0xMva8Hu0I4B0zWwn8klgHA4cAQ3b1pGhJSkriulHXMmTYCAJlZfQ6+yxatmzBI5On0LZNG7qe1IXnXpjJJ598SlJSEg0bNWT0bbcA0KhRIy7pezH9B/wJM6Pz70+gy4mdY/yOaib4/lrT9aSTAHjrrdn07NGj0omNEzodz3erVnHZ5cEpWQ3q12fMHbeTnp5eqa2pjz3O5s2bGRvq6ScmJvKPp55g/fr13DZ6DIGyAK7Mceqp3Tmpy4kADBs6mJv+cguTHp3MYYceyrnnhD9RF0+Cn5uRDBk2vNLnxu8K/reB2+4KTmlzZWX06NaFkzp35JHH/kHb1q3oeuLxzJj5Ku/P+4TExARSGzVk9F9GxLrs2ovDG39b8AY11exglkDwErOKJ8M+DV0DXBOuaPOG8HvtBVJS0ynavDHWZcSFlNQ0HYuQlNQ0igtWxrqMuJCc0arO3cyyWUfU+A5bCWf9Nyrd2rCzDpxzZcDHUahFRCQC4m+MNj5n6IuI7DYFrYiIt+Jw1oGCVkR8RkErIuKp2tz4O1riryIREZ9Rj1ZEfEZDByIiHlPQioh4S7MORES8Fn+nnuKvIhGRuojQjb/NbD8z+8TMPjOzZWY2OrQ+x8wWmNnXZvacme0TriQFrYj4TMTu3lUCdHPOHQW0B043s07AX4EJzrlDgI0EbydbLQWtiPhMZILWBRWFHtYLLQ7oBswMrX+S4FeOV0tBKyL+UouhAzPLNbOFFZbcyk1ZopktAfKB2cA3wCbnXGlolx/49c6Gu6STYSLiMzWfdeCcmwJMqWZ7AGhvZo2Bl4Hw3ytVBQWtiPhM5P+j7pzbZGZzgBOAxmaWFOrVNiV4j+4oVyQiEkuRm3WQEerJYmb1gR7AcmAO8MsXIQ4AXglXknq0IuIzEbtgIQt40swSCXZKn3fOzTKzL4AZZnYn8B/g8XANKWhFxGciE7TOuaXA0VWs/5bg13vVmIJWRPwl/q7AVdCKiN/EX9IqaEXEX+Lwxt8KWhHxFacerYiI1xS0IiIeU9CKiHhLN/4WEfGaToaJiHhLPVoREa8paEVEPKagFRHxmIJWRMRbe+sYbUpqejReZo+QkpoW6xLiho7Fr5IzWsW6BB/ZS4O2aMO6aLxM3EtJz6Jo88ZYlxEXUlLTdCxCUlLTKF6/KtZlxIXk/bPr3sje2qMVEYkeBa2IiMcUtCIi3orDoYP4u1ZNRKROrBZLNa2YNTOzOWb2hZktM7PhofW3m9laM1sSWs4IV5F6tCLiMxHrP5YC1zrnFptZQ2CRmc0ObZvgnBtX04YUtCLiLxEaOnDOrQPWhX4uNLPlwEG705aGDkTEVxxW46WmzCyb4DfiLgitGmJmS81smpmFnRCuoBURn6n5GK2Z5ZrZwgpL7k6tmaUALwIjnHNbgEeAlkB7gj3e8eEq0tCBiPhLLYYOnHNTgCm7bsrqEQzZZ5xzL4Wek1dh+1RgVrjXUY9WRHwmoRbLrpmZAY8Dy51z91VYn1Vht97A5+EqUo9WRHwmYvNoOwOXAv81syWhdTcBF5tZe8ABq4ArwzWkoBURf4lQzjrn5u2itX/Xti0FrYj4TPxdGaagFRGfUdCKiHgrDu91oKAVEZ+Jv8lUCloR8Rf1aEVEvKagFRHxmIJWRMRjCloREW+ZToaJiHhMPVoREW9p1oGIiNcUtCIinnKxLqAKCloR8Rn1aEVEvKVZByIiXlOPVkTEW5p1ICLiNQWtp0bf+Vc+mP8R6WmNef6ZJ3a537IvVjAwdxB333Erp3Y7OWr11dboMXfywbwPSU9L4/kZz1a5z8JFixh/3/2UlpbSuHFjpk5+pMbtf7xgAQ89PIkdO0qpVy+J4UOH0vG4DgAMGTaC9evXEwgEOLp9e66/biSJiYnMfvsdpkx9jO9WreKpv0+jbds2EXmv0Tb/o48YN34CgbIyzu11DgMH9I91SZ77KS+fW8f8jf9t3IQB5/U6g74X9t5pv4WLP2PcA4+GPlOpPPbwuOgXWycKWk+dfebpXHhBb2674+5d7hMIBHhw0mQ6dTwuipXtnrPPPJMLL+jDbbffUeX2wsJCxt77Nx564H6yDjiADRs21Kr9xo0bc//4cWRkZPD1N98wZNgI3vjXawCMvfsuUlKScc5x3Q038vY773Jazx4c0rIFf7t3LHffM7bO7y9WAoEAY+8dx6SJD9IkM5NLBwyka5cutGiRE+vSPJWYmMifh+bS5rBWFBdvpd/lQ+h03DG0yGlevk9hYRH3jJ/IxPF3kXVAJhs2bopdwbsrQkMHZtYMeApoQnDW2BTn3ANmlg48B2QT/HLGC51zG6trK/5Oz9XBMUcfRWqjhtXu89wLL9H95JNIS2scnaLq4Jhjjia1UaNdbn/9zTfpdvLJZB1wAADp6enl2/79+uv0v+xPXNzvUu66ZyyBQGCn57c+7DAyMjIAaNmiBSUlJWzfvh2AlJRkAEoDAXbs2FH+2c3JySG7efOd2tqTLFv2Bc2aNqXpQQdRr149evbswXtz58a6LM9l7P872hzWCoDk5AbkNG9GfsH6Svu8PnsO3bp2JuuATADS94B/JzuLzNeNA6XAtc65tkAnYLCZtQVuAN5xzrUC3gk9DlvRbjGzgbv73FjJzy9gzvvz6HNer1iXEhGrV69hS2EhuVddTb/+A5j1r+CXc3733Xe8NfttHn9sCtOfeZqEhARef+PNatt65905tD7sUPbZZ5/ydYOHDqfHaX+gQYNkunfr5ul7iab8ggKaNMksf9wkM5OCgoIYVhR9P677iS9XfkO7w1tXWv/96h/YUljEFUNG0fdPg5n1+uwYVVgXVotl15xz65xzi0M/FwLLgYOAXsCTod2eBM4NV1Fdhg5GA3+vaoOZ5QK5AJMnT6Zvn7Pr8DKRM+7+iQwbnEtCgj868oFAgOUrVvDowxPZVlLCwMv/jyPateOTTxeyfMWX9B8Q/F1YUlJCelraLtv55ptveXDiwzz80AOV1j/80AOUlJRw86238enChXQ6/nhP349Ex9atPzPyL2O4dthVpCQnV9oW/EytZPKDf2VbSQmXXTmCIw5vQ/ODm8ao2t1Qi6GDilkVMsU5N6WK/bKBo4EFQBPn3LrQpp8IDi1Uq9qgNbOlu9pUXeOhQn8p1hVtWLerXaNq+YovufGW4Hjnps2b+fCjBSQmJnJK1y4xrmz3ZGZmkpqaSv369alfvz7HtD+ar1auxDnHWWeewdDBgyrt/+6c95j62OMA3PKXm2jbtg15efmMvO567rj9Vpo13fkf07777kvXk07i/bkf+CZoMzMyyMvLL3+cl59fPoTidztKSxn5lzGc0bMb3U8+caftTTIzSE1tRP36+1G//n4c0/4Ivvr62z0raGtxMuw3WVV1a2YpwIvACOfcFqsQ5M45Z2Zhr/oN17VrAvQHzq5i+V+4xuPNay/NYNbLzzHr5efofkpXbhg5Yo8NWYCTT+rCkiWfUVpays/btvH5smXk5GTT8bjjeOfdd8tPjm3evJl169bR7ZSTmf7M00x/5mnatm1DYWEhw/98DUOHDKL9UUeVt7t161YK1gfH7kpLS5n34Yd7/LhsRW3btmHNmjWsXfsjO3bs4K23ZtO1y577Oagp5xx33HMfOc2bcckfz69yn65dTmDJ0mWUlgZCn6kV5GQfHOVK6yoyQwcAZlaPYMg+45x7KbQ6z8yyQtuzgPxdPf8X4YYOZgEpzrklVRTwXtgqo+ymW+9g4eIlbNq0mT+c04cr/28gpaWlAHvkuOxNN9/CwkWL2bRpE38462yuvOKKX9/P+eeRk5PD70/oxB/7XUKCJXBur3M4pGVLAAZddSWDhw6nzJWRlJTEDaNGkZWVVan9555/gTU//MDUx6Yx9bFpQHC4wDnHNdeOYvuO7bgyR4djj+H884LTgN6d8x5/Gz+ejRs3Mfyaazi01aE7DTnEu6SkJK4bNZIhw4YTKCuj19ln0bJli1iX5bklS5fxrzfe4ZCWOfxxwNUADLlyID+Fevd9ep9Fi+yD+f3xHbhowFUkmHHu2adzSIvsGFa9GyI368CAx4Hlzrn7Kmx6FRgAjA39+UrYtpzz/F43cTN0EGsp6VkUba52FsheIyU1TcciJCU1jeL1q2JdRlxI3j+7zim5fdEtNQ61fY4ds8vXM7MTgQ+A/wJlodU3ERynfR44GPie4PSuaudW+moerYhIpC5YcM7Nq6ax7rVpS0ErIv6iex2IiHhNQSsi4imnoBUR8Zhu/C0i4jX1aEVEvKWTYSIiXlPQioh4TEErIuIxBa2IiLficNZB/FUkIuIz6tGKiL9o1oGIiNcUtCIiHlPQioh4Kw5PhiloRcRn1KMVEfGYglZExFtxOOsg/gYzRETqJKLfgjvNzPLN7PMK6243s7VmtiS0nBGuHQWtiPhM5IIWeAI4vYr1E5xz7UPLv8M1oqEDEfEVF8FZB865uWaWXdd21KMVEZ+peY/WzHLNbGGFJbeGLzLEzJaGhhbSwu2soBURn6l50DrnpjjnOlRYptTgBR4BWgLtgXXA+HBPUNCKiL9EdIh2Z865POdcwDlXBkwFOoZ7joJWRHzG26Q1s6wKD3sDn+9q31/oZJiI+Ezk5tGa2XTgZGB/M/sBuA042czaAw5YBVwZrh0FrYj4S2RnHVxcxerHa9uOglZEfCb+rgxT0IqIzyhoRUS8FYf3OohK0KakZ4XfaS+Rkhp2bvNeQ8fiV8n7Z8e6BB/ZS4O2eN2SaLxM3EvOak/R5o2xLiMupKSm6ViEpKSmwbPxFw4x0dfVvQ3d+FtExGvx90tLQSsiPqOgFRHx1t56MkxEJHriL2jjb9RYRMRn1KMVEV+J5I2/I0VBKyI+E39DBwpaEfEZBa2IiLc060BExGsKWhERb+lkmIiI19SjFRHxWPwFbfz1sUVE6iRyX85oZtPMLN/MPq+wLt3MZpvZytCfYe/3qaAVEX8xq/kS3hPA6b9ZdwPwjnOuFfBO6HG1FLQi4jOR69E65+YCG36zuhfwZOjnJ4Fzw7WjoBURf7GEGi9mlmtmCyssuTV4hSbOuXWhn38CmoR7gk6Gicheyzk3BZhSh+c7Mwv7tRDq0YqIz0Ru6GAX8swsCyD0Z364JyhoRcRnPA/aV4EBoZ8HAK+Ee4KCVkT8JYKzDsxsOvARcJiZ/WBmlwNjgR5mthI4NfS4WhqjFRGfidwFC865i3exqXtt2lHQioi/6F4HIiLecnF4Ca6CVkR8RkErIuIt3fhbRMRrCloREY/pZJiIiLc0dCAi4jUFrefOvGgIyQ32IyEhgcTERJ6Zck+V+y1b8TWXDbqFe24dzqknd4pylXUXCAS4dMBAMjIyeGDC+Bo956e8PG69fTQbNmzAMHr3Ppe+f7wIgEmPTub9uXNJsATS0tMYfestZGRk8N2qVYy+405WfPklg66+iv6X9PPybUXN/I8+Ytz4CQTKyji31zkMHNA/1iVFXaAMzp96ME0aljK574+s2ZjENS9msWlrIocfWMK9vdexT2Ksq/QH3wUtwOQJt5LWuNEutwcCZTww+Vk6HXdkFKuKrOkzniM7O5vi4uIaPycxMZE/Dx9Gm9atKS4u5pL+l9GpY0datMih/yWXMOiqK4NtP/ccUx+bxk03Xk9qo0aMGnkN7733vldvJeoCgQBj7x3HpIkP0iQzk0sHDKRrly60aJET69Ki6qkFjWm5/3aKSoJjmuPezuCyTps4s10ht87KZObiVPoetznGVe6GOBw6iL9R4yiY8dLrdD/peNIbp8a6lN2Sl5fPvA/nc26vc8rXLV++giuuvJp+/QcweOhwCtav3+l5GfvvT5vWrQFITk4mJyeb/ILgjYdSUpLL9/v5523l//tKT0/n8LZtSUryz+/kZcu+oFnTpjQ96CDq1atHz549eG/u3FiXFVU/bUnivZUp9DkmGKTOwcffNeC0toUA9D5qC+98mRLLEuvA85vK1FrYoDWz1mbW3cxSfrP+t1/vEBfMYPCou+ibewMvvvb2TtvzCzYwZ96nXNCrRwyqi4zxEyYwfOgQEhKCH5QdpaXcO2489469m2eeepJe55zFpEcerbaNH3/8kRVffkW7w9uVr3t40iOccdY5vPHGm1x9ZU3uf7xnyi8ooEmTzPLHTTIzKSgoiGFF0Xf3GxmMOrWA0EeIjT8n0Gi/AEmhRDigUSl5W/bUX64JtViio9ojaWbDgMHAcuBxMxvunPvllmB3A294XF+tTXvoDjIz0tmwcTNXj7yT7IMP5Nij2pZvHzfxCYbl9iUhYc/szM/9YB5paWm0adOahYsWAfD999/zzbffMGjIMAACZWXsv//vdtnG1q1bGXXDjYy8ZkSlnuzgQVczeNDVTHviSZ57YSZX5V7h7ZuRmJjzVTLpyQHaHVjCglX1Y11O5MXh0EG4X1lXAMc654rMLBuYaWbZzrkHqKbfHfo6iFyAyZMn0+/sjpGqN6zMjHQA0tNSOeXEjixb/k2loP3iy2+58Y4HAdi0eQvzFvyHxMRETulyXNRqrIvPli5l7gcf8OH8+Wwv2U5RcTGTp0ylRU4Lnpj2WKV9f8rL48/XjATg/PN60+f889hRWsqo62/kD6edRrdTTqnyNf5w+mkMH3GNb4M2MyODvLxf79Wcl59PRkZGDCuKrsWr6/Pul8nMXZlDSalRVJLAXW9ksmVbIqVlkJQQHFpo0qg01qXupj0vaBOcc0UAzrlVZnYywbBtTjXv5jdfD+GK1y2pe6U18PPP2yhzjuQG9fn55218vHApV/Q/v9I+s2ZMLP/5tnsm0eWEY/aYkAUYOngQQwcPAmDhokU8/Y9nufvOMfS56GKWLv0vRx55BDtKS1n9/WpatmzB9GeeLn+uc44xY+4iJyebS/r1rdTu6tWrOfjggwF4//25ZGc3j96birK2bduwZs0a1q79kczMDN56azZ3jbkj1mVFzbWnrufaU4Nj+AtW1Wfa/DTGn/cTw17I4s0vGnJmu0Je/qwR3Q4rinGlu2vPC9o8M2vvnFsCEOrZngVMA47wurja+t/GzVx7yzggOLPg9O6d6Xx8e2a+MhuAPnvwuGx16tWrx71j7+Zv4+6jqKiIQCDAxRdfRMuWLSrtt+Szz/jX669zyCEtubjfpUBwuODEzr/noYcn8f33q7EEI+uAA7jphusBWL/+f1x62WUUFxdjlsD0GTN4YcaMSkMOe5qkpCSuGzWSIcOGEygro9fZZ+10rPZGo05dz59nZnH/u7+jTVYJFxy9JdYl7Z44HDow53b9vWJm1hQodc79VMW2zs65D2vwGlHr0ca75Kz2FG3eGOsy4kJKapqORUhKaho8G3/hEBN9XZ0PxNYfF4b9ssRfNDiwQ1QOfLU9WufcD9Vsq0nIiohEl278LSLitch1Us1sFVAIBAj+777D7rSjoBURX/HgGxZOcc7tfAVQLShoRcRn4m+8O/4GM0RE6iKCXzcOOOAtM1sUuj5gt6hHKyI+U/MebcWLq0KmhK4D+MWJzrm1ZpYJzDazFc65Wt8YQ0ErIv5Si3m0v7m4qqrta0N/5pvZy0BHoNZBq6EDEfGZyNy9y8ySzazhLz8DPYHPd6ci9WhFxGcidjKsCfCyBXvIScCzzrndupGWglZEfCYyQeuc+xY4KhJtKWhFxF/i8F4HCloR8Zn4O/WkoBURf1GPVkTEawpaERGPKWhFRLyloQMREa8paEVEPKagFRHxloYORES8pqAVEfGUB9+wUGcKWhHxFw0diIh4TZfgioh4TD1aERFvaehARMRrCloREY8paEVEvKWhAxERr8XfrIP4q0hEpC7Mar6EbcpON7MvzexrM7thd0tS0IqIVMHMEoGHgT8AbYGLzazt7rSloBURn7FaLNXqCHztnPvWObcdmAH02p2KojFGa8lZ7aPwMmGKMMt1zk2JdR0pqWmxLkHHooJ4ORb0dbGuIH6ORR2lpKbV+GyYmeUCuRVWTalwDA4C1lTY9gNw/O7UtDf1aHPD77LX0LH4lY7Fr/a6Y+Gcm+Kc61Bh8eQXzd4UtCIitbEWaFbhcdPQulpT0IqIVO1ToJWZ5ZjZPsAfgVd3p6G9aR7tHj/2FEE6Fr/SsfiVjkUFzrlSMxsCvAkkAtOcc8t2py1zLvaD8CIifqahAxERjyloRUQ85vugjdQldH5gZtPMLN/MPo91LbFkZs3MbI6ZfWFmy8xseKxrihUz28/MPjGzz0LHYnSsa/IjX4/Rhi6h+wroQXCy8afAxc65L2JaWIyY2UlAEfCUc65drOuJFTPLArKcc4vNrCGwCDh3b/xcmJkByc65IjOrB8wDhjvnPo5xab7i9x5txC6h8wPn3FxgQ6zriDXn3Drn3OLQz4XAcoJXAe11XFBR6GG90OLf3leM+D1oq7qEbq/8ByVVM7Ns4GhgQYxLiRkzSzSzJUA+MNs5t9ceC6/4PWhFdsnMUoAXgRHOuS2xridWnHMB51x7glc+dTSzvXZYySt+D9qIXUIn/hIaj3wReMY591Ks64kHzrlNwBzg9BiX4jt+D9qIXUIn/hE6AfQ4sNw5d1+s64klM8sws8ahn+sTPHG8IqZF+ZCvg9Y5Vwr8cgndcuD53b2Ezg/MbDrwEXCYmf1gZpfHuqYY6QxcCnQzsyWh5YxYFxUjWcAcM1tKsGMy2zk3K8Y1+Y6vp3eJiMQDX/doRUTigYJWRMRjCloREY8paEVEPKagFRHxmIJWRMRjCloREY/9P+Z6SqsmkMlDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's visualize our annotations after applying attention to them\n",
    "ax = sns.heatmap(applied_attention, annot=True, cmap=sns.light_palette(\"orange\", as_cmap=True), linewidths=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Contrast this with the raw annotations visualized earlier in the notebook, and we can see that the second and third annotations (columns) have been nearly wiped out. The first annotation maintains some of its value, and the fourth annotation is the most pronounced.\n",
    "\n",
    "# Calculating the Attention Context Vector\n",
    "All that remains to produce our attention context vector now is to sum up the four columns to produce a single attention context vector\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 3.88079708,  4.0728263 , 45.26423912], dtype=float128)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def calculate_attention_vector(applied_attention):\n",
    "    return np.sum(applied_attention, axis=1)\n",
    "\n",
    "attention_vector = calculate_attention_vector(applied_attention)\n",
    "attention_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIYAAAEWCAYAAACjaO9mAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAPR0lEQVR4nO3df5BV9XnH8fezq2gpAkEjoqQh0zAVAyOOwGiYqMWfFVAzbY0/0E7HETsjltQqaKfW7KCpnWlEMxksFIWMQ8RMjNOGadoahDDqVMmCKEKTIJiJDMKYYVOFEbK7T/84Z+Xu5bt79l72nO+5dz+vmTvsPXfvOd9ZPvucc+6e53zN3RGp1hJ7AFJOCoYEKRgSpGBIkIIhQQqGBCkYTcjMWs1sq5mtS5+vNrM9ZvZm+piatY6Tch+lxLAQ2AmMrFh2v7v/YKArKKJiuB6fPnoxw2t9ZP2wzWw8MBtYmfW9/SmkYvz2tx8XsZnSGzVqRK/nZrls5glgEXBa1fJHzewfgPXAA+5+pL+V6BgjIrN6HjbfzH5W8Zh/bH02Bzjg7u1Vm3oQOBeYDowBFmeOrYC/lbgqRmLUqBG9asRJJ2XvGqp1dtJnnTGzfwRuAzqBU0mOMX7o7vMqvucy4D53n9PfdlQxIqqnYvTH3R909/HuPgG4CXjZ3eeZ2bhke2bADcD2rLHprCSinI4xQtaY2WcBA94E/irrDdqVFKh6V3LKKbXvSo4c6XtXMphUMSIqsGLUTMGIqMzB0MGnBKliRFTmiqFgRGRWz4F/MWlSMCJSxZAgBUOCFAwJainxOaGCEZEqhgQpGBKkYEiQgiFBCoYEKRgSpGBIkIIhQQqGBJU5GCX+UFZiUsWISBVDgga7r+TYeo/rdv+Cmb1uZrvM7HkzG5a1DgUjoryCwbFu9x7/BCx19y8CB4E7slagYESURzCqu93T7rNZQM8tEL5L0o3WLwUjIjOv49F3U3PqCZJu9+70+elAh7t3ps/fB87JGpsOPiOq5+DT3VcAK8LrO9btnjYv103BiCiHs5KZwHVmdi3Hut2fBEab2Ulp1RgP7M1akXYlERXU7X4rsAH4s/Tb/gL4t6yxKRgR5XhWUm0xcK+Z7SI55ng66w1Nsys5cuQId911J0ePHqWrq4vLL7+c+fN7d/vv27ePJUva6Og4yMiRo2hrW8LYsWMjjTjfi4HdfSOwMf16NzCjlvc3TTCGDRvGsmX/wvDhw+ns/B133nkHF188kylTpnz6PU8+uZRrr53NnDlz2bz5DZYt+w5tbUuijVmffBbAzBg+fDgAnZ2ddHZ2HveD37NnD9OnTwdg2rTpbNr006KH2UuBu5KaZQbDzM41s8Vm9u30sdjMJhUxuFp1dXVx6603c/XVVzJjxkVMnjyl1+sTJ05kw4aXAdi4cQOHDh2io6MjwkgTDRsMM1sMrCXppH0jfRjwnJk9kP/watPa2sqaNc+xbt2P2bFjO+++u6vX6wsX/g1btmxh3rxb2LKlnTPPPJPW1tZIoy13MPq91ZKZ/QL4krv/rmr5MOAdd5/Yx/vmA/MBli9ffuHXvnbL4I14gFauXMGpp57KvHm3B18/fPgwN974p6xb9+PCxlR9q6Xzzuuuud19x46WQuKRtSvpBs4OLB/HsY9cj+PuK9x9mrtPmz+/+hPbfBw8eJCPPvoIgE8++YTXX3+dz39+Qq/v6eg4SHd3MuzVq1cxd+51hYytEWWdlXwdWG9mvwR+nS77A+CLwIIcx1WzDz/8kLa2h+nu7qK727niiiv4ylcuYfnyp5g06TwuueRS2tvbWbbsO4BxwQUXsGhR3L1hmc9KMu/aZ2YtJOfAPX942QtsdveuAW5Dd+1LVe9KJk+ufVeyfXsxu5LMzzHcvRv4nwLGMuSUuWI0zQdcjUjBkCAFQ4IUDAlSMCRIwZCg+u7zWQwFIyJVDAlSMCRIwZAgBUOCyhyMprm0rxEN9oU6Znaqmb1hZtvM7B0za0uXr9YU3g0kh6vEjwCz3P1jMzsZeMXMeq5EqmkKbwWjiXhyDUXPNQ4np4+6PizRriSinLrdW83sTeAA8JK7v56+9KiZvWVmS83slKz1KBgR1ROMrG53d+9y96kkPaozzGwydUzhrV1JRIPd7V71fR1mtgG4xt3/OV18xMxWAfdlvV8VI6Iczko+a2aj069/D7gS+F9N4d1gcvgcYxzwXTNrJfml/767rzOzl2udwlvBiGiwg+HubwEXBJbPqnVdCkZEZf7kU8GISMGQIAVDghQMCdKlfRKkiiFBCoYEKRgSpGBIUJmDoT+iSZAqRkRlrhgKRkQKhgTlecvoE6VgRKSKIUFDPhijRo0oYjMNZ8gHo6NDt3MEGD269y/IkA+GhCkYEqRgSFCZg1HiM+nmV2C3u6bwbiQ59K72dLufD0wFrjGzi9AU3o1lsIPhiVC3u6bwbiR5TOFd3e0OvIum8G4seTQ1p9OFTE17WF8k6XKvmYIRUZ5nJRXd7hejKbyHtj663XdSxxTeqhgRFdjtvgNYa2aPAFsZSlN4N6ICu92H7hTejajMn3wqGBEpGBKkYEiQgiFBCoYE6SpxCVLFkCAFQ4IUDAlSMCRIwZAgBUOCyhyMEp9JS0yqGBHpPp8SVOZdiYIRkYIhQQqGBCkYElTmYOh0NaIcmpo/Z2YbzGxH2tS8MF3+DTPbWzGF97VZY1PFiCiHitEJ/K27bzGz04B2M3spfW1pxTSbmZquYnR1dXHbbbdw770Lj3tt69Yt3H77LXz5yzNYv/4nEUbXWw5NzfvcfUv69UckzUaZfaohTReM559/jgkTJgRfGzv2LB56qI2rrrqm2EH1IYfbIFSs2yaQ9Jj0TOG9IJ3C+xkz+0zW+5sqGPv37+fVV1/h+utvCL5+9tlnM3HiRFpaynHUV08wsrrdk/XaCOAF4Ovu/n/AU8AfktwzYx/wrayx1X2MYWZ/6e6r6n1/HpYu/RYLFizk8OFDsYcyIHl0u5vZySShWOPuP0zfs7/i9X8F1mVt50QqRls/g/s01StWZE5DPiheeWUTY8Z8hkmTJhWyvcHQ0lL7oz/pFN1PAzvd/fGK5eMqvu2rnOgU3mb2Vl8vAWP7el9Vqr2I+3xu27aNTZs28dprr3LkyFEOHfqYhx/+e9raHsl92/XK4axkJnAb8HZ68xSAvwNuNrOpJHfXeQ+4K2tFWbuSscDVJPdtqmTAawMebgHuvvse7r77HgDa23/GmjXPljoUkEtT8ysk/zfV/qPWdWXtStYBI9z9V1WP94CNtW4shuXLn2LTpp8CsGPHO8yZ8yesX/8THnvsm9x0059HHVueZyUnPDb33K8JKGRX0ghGjx7R67/2oYeO1vzDX7JkWCHx0CefEZX5byUKRkQKhgQpGBKkaz4lSBVDghQMCVIwJEjBkCAFQ4IUDAlSMCRIwZAgBUOCFAwJUjAkqMzBaKr2ARk8qhgR6ZbREqRdiQQV2O0+xsxeMrNfpv8OrRbFRpPDVeI93e7nARcBd5vZecADwHp3nwisT5/3S8GIqMBu9+tJpu6GAU7hrWOMiPI8xqjqdh/r7vvSlz6gny7CHqoYEeUxt3uy3uO63T/lSSNR5sWmqhgRFdXtDuw3s3Huvi9tcD6QtR1VjIhyOCsJdrsD/04ydTdoCu/yK7Db/THg+2Z2B/Ar4MasFSkYERXY7Q5weS3rUjAiKvMnnwpGRAqGBCkYEqRgSFCZg6HPMSRIFSOiMlcMBSOiIR+M0aNHFLGZhjPkg1HmH0CRqm+QWOafi3YlEeliYAlSxZAgBUOCFAwJUjAkSPf5lCBVDAlSMCRIwZAgBUOCyhyMEn8oK/VIJ9w9YGbbK5Z9o9a53RWMiHKaE201EJqKeqm7T00fmZPnaVcSUR67EnfflDY0nxBVjIjqqRgDaWruw9Cd273R1BMMd1/h7tMqHgOZCru4ud3lxBV1VlL03O5ygnI6+AxsZ5Dndpd85VExzOw54DLgDDN7H3gYuGyw53aXHOV0VnJzYPHTta5HwYiozJ98KhgRKRgSpKvEJUgVQ4J0aZ8EqWJIkIIhQWUORomPiyUmVYyIylwxFIyIFAwJUjAkSMGQoDIHo2nPSlpaYMsW+NGPkuerVsHu3bB1a/I4//y444PiLtSpR9NWjIULYedOGDny2LL774cXXog3pmqqGAU75xyYPRtWrow9kv6VuWJkBsPMzjWzy9N5tiqXh5paSuGJJ2DRIuju7r380Udh2zZ4/HEYNizK0Hpp2GCY2V+TTJN0D7DdzK6vePmbeQ6sXrNnw4EDyfFFpQcfhHPPhenTYcwYWLw4zvgqNWwwgDuBC939BpILTB/qmf0X+pxJp1dTzIoVA2l7GDwzZ8J118GePbB2LcyaBc8+Cx98kLx+9GhyIDpjRqHDCipzMMyr70raa+D2jrt/qeL5COAHwA5glrtPHcA2PNZB1qWXwn33wdy5cNZZx8KxdCl88klSRYrk3vuXafPmwzVfkDF9+vB+f5pm9gwwBzjg7pPTZWOA54EJJFeJ3+juB/tbT1bF2J9edg6Au3+cbvQMYErGe0tlzRp46y14+2044wx45JHYI8qtYqzm+KbmmqfwzqoY44FOd/8g8NpMd391AAONVjHKprpitLfXXjEuvLD/igGQNjWvq6gYPwcuq5h3daO7/1F/6+j3cwx3f7+f1wYSCulHgRcDawrvxuI1P06g2z3ZoqbwLr96drFZU3j3QVN4N5ICT1c1hXcjKbCpWVN4N5ICm5pBU3g3jjKfxisYESkYEqRgSJCCIUEKhgQpGBKkYEiQgiFBCoYEKRgSpGBIkIIhYQqGhJQ4FwpGTNqVSJCCIUEtLboBrASoYkiQgiFBCoYEKRgSlFP7wHvAR0AXSd/xtHrWo2BElGPF+GN3//BEVqBgRFTmXYlaFCOqp0VxAE3NDvy3mbXX2vDca2z93R9jkOj+GKnq+2P85jcf1/zDP/30EVl31DnH3fea2ZnAS8A97r6p1u2oYkSUR1Ozu+9N/z0AvAjUdbcxBSOiwQ6Gmf2+mZ3W8zVwFQOYrjtEB58R5bCLHQu8aMmKTwK+5+7/Wc+KFIwm4u67gUG5S7qCEVGZD8qLOCspBTObn96mSAZgKB181n1OPxQNpWBIDRQMCRpKwdDxRQ2GzMGn1GYoVQypQdMHw8yuMbOfm9kuM8u8674kmnpXYmatwC+AK4H3gc3Aze6+I+rAGkCzV4wZwC533+3uR4G1wPUZ7xGaPxjnAL+ueP5+ukwyNHswpE7NHoy9wOcqno9Pl0mGZg/GZmCimX3BzIYBN5FM0SAZmvrP7u7eaWYLgP8CWoFn3P2dyMNqCE19uir1a/ZdidRJwZAgBUOCFAwJUjAkSMGQIAVDghQMCfp/HQAGoCw/4oUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 108x324 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's visualize the attention context vector\n",
    "plt.figure(figsize=(1.5, 4.5))\n",
    "sns.heatmap(np.transpose(np.matrix(attention_vector)), annot=True, cmap=sns.light_palette(\"Blue\", as_cmap=True), linewidths=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the context vector, we can concatinate it with the hidden state and pass it through a hidden layer to produce the the result of this decoding time step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "CVND",
   "language": "python",
   "name": "cvnd"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}