transformer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "s_qNSzzyaCbD"
   },
   "source": [
    "##### Copyright 2019 The TensorFlow Authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "jmjh290raIky"
   },
   "outputs": [],
   "source": [
    "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "J0Qjg6vuaHNt"
   },
   "source": [
    "# Transformer model for language understanding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AOpGoE2T-YXS"
   },
   "source": [
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://www.tensorflow.org/beta/tutorials/text/transformer\">\n",
    "    <img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />\n",
    "    View on TensorFlow.org</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/r2/tutorials/text/transformer.ipynb\">\n",
    "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />\n",
    "    Run in Google Colab</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/r2/tutorials/text/transformer.ipynb\">\n",
    "    <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />\n",
    "    View source on GitHub</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/r2/tutorials/text/transformer.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "M-f8TnGpE_ex"
   },
   "source": [
    "This tutorial trains a <a href=\"https://arxiv.org/abs/1706.03762\" class=\"external\">Transformer model</a> to translate Portuguese to English. This is an advanced example that assumes knowledge of [text generation](text_generation.ipynb) and [attention](nmt_with_attention.ipynb).\n",
    "\n",
    "The core idea behind the Transformer model is *self-attention*—the ability to attend to different positions of the input sequence to compute a representation of that sequence. Transformer creates stacks of self-attention layers and is explained below in the sections *Scaled dot product attention* and *Multi-head attention*.\n",
    "\n",
    "A transformer model handles variable-sized input using stacks of self-attention layers instead of [RNNs](text_classification_rnn.ipynb) or [CNNs](../images/intro_to_cnns.ipynb). This general architecture has a number of advantages:\n",
    "\n",
    "* It make no assumptions about the temporal/spatial relationships across the data. This is ideal for processing a set of objects (for example, [StarCraft units](https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/#block-8)).\n",
    "* Layer outputs can be calculated in parallel, instead of a series like an RNN.\n",
    "* Distant items can affect each other's output without passing through many RNN-steps, or convolution layers (see [Scene Memory Transformer](https://arxiv.org/pdf/1903.03878.pdf) for example).\n",
    "* It can learn long-range dependencies. This is a challenge in many sequence tasks.\n",
    "\n",
    "The downsides of this architecture are:\n",
    "\n",
    "* For a time-series, the output for a time-step is calculated from the *entire history* instead of only the inputs and current hidden-state. This _may_ be less efficient.   \n",
    "* If the input *does* have a  temporal/spatial relationship, like text, some positional encoding must be added or the model will effectively see a bag of words. \n",
    "\n",
    "After training the model in this notebook, you will be able to input a Portuguese sentence and return the English translation.\n",
    "\n",
    "<img src=\"https://www.tensorflow.org/images/tutorials/transformer/attention_map_portuguese.png\" width=\"800\" alt=\"Attention heatmap\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tell TF not to consume all GPU memory so you can run more than one notebook at once\n",
    "import os\n",
    "os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gpustat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JjJJyJTZYebt"
   },
   "outputs": [],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals\n",
    "\n",
    "import tensorflow_datasets as tfds\n",
    "import tensorflow as tf\n",
    "\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fd1NWMxjfsDd"
   },
   "source": [
    "## Setup input pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "t4_Qt8W1hJE_"
   },
   "source": [
    "Use [TFDS](https://www.tensorflow.org/datasets) to load the [Portugese-English translation dataset](https://github.com/neulab/word-embeddings-for-nmt) from the [TED Talks Open Translation Project](https://www.ted.com/participate/translate).\n",
    "\n",
    "This dataset contains approximately 50000 training examples, 1100 validation examples, and 2000 test examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8q9t4FmN96eN"
   },
   "outputs": [],
   "source": [
    "examples, metadata = tfds.load('ted_hrlr_translate/pt_to_en', with_info=True,\n",
    "                               as_supervised=True)\n",
    "train_examples, val_examples = examples['train'], examples['validation']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RCEKotqosGfq"
   },
   "source": [
    "Create a custom subwords tokenizer from the training dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KVBg5Q8tBk5z"
   },
   "outputs": [],
   "source": [
    "tokenizer_en = tfds.features.text.SubwordTextEncoder.build_from_corpus(\n",
    "    (en.numpy() for pt, en in train_examples), target_vocab_size=2**13)\n",
    "\n",
    "tokenizer_pt = tfds.features.text.SubwordTextEncoder.build_from_corpus(\n",
    "    (pt.numpy() for pt, en in train_examples), target_vocab_size=2**13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4DYWukNFkGQN"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokenized string is [7915, 1248, 7946, 7194, 13, 2799, 7877]\n",
      "The original string: Transformer is awesome.\n"
     ]
    }
   ],
   "source": [
    "sample_string = 'Transformer is awesome.'\n",
    "\n",
    "tokenized_string = tokenizer_en.encode(sample_string)\n",
    "print ('Tokenized string is {}'.format(tokenized_string))\n",
    "\n",
    "original_string = tokenizer_en.decode(tokenized_string)\n",
    "print ('The original string: {}'.format(original_string))\n",
    "\n",
    "assert original_string == sample_string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o9KJWJjrsZ4Y"
   },
   "source": [
    "The tokenizer encodes the string by breaking it into subwords if the word is not in its dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bf2ntBxjkqK6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7915 ----> T\n",
      "1248 ----> ran\n",
      "7946 ----> s\n",
      "7194 ----> former \n",
      "13 ----> is \n",
      "2799 ----> awesome\n",
      "7877 ----> .\n"
     ]
    }
   ],
   "source": [
    "for ts in tokenized_string:\n",
    "    print ('{} ----> {}'.format(ts, tokenizer_en.decode([ts])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bcRp7VcQ5m6g"
   },
   "outputs": [],
   "source": [
    "BUFFER_SIZE = 20000\n",
    "BATCH_SIZE = 64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kGi4PoVakxdc"
   },
   "source": [
    "Add a start and end token to the input and target. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UZwnPr4R055s"
   },
   "outputs": [],
   "source": [
    "def encode(lang1, lang2):\n",
    "    lang1 = [tokenizer_pt.vocab_size] + tokenizer_pt.encode(\n",
    "        lang1.numpy()) + [tokenizer_pt.vocab_size+1]\n",
    "\n",
    "    lang2 = [tokenizer_en.vocab_size] + tokenizer_en.encode(\n",
    "        lang2.numpy()) + [tokenizer_en.vocab_size+1]\n",
    "  \n",
    "    return lang1, lang2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6JrGp5Gek6Ql"
   },
   "source": [
    "Note: To keep this example small and relatively fast, drop examples with a length of over 40 tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2QEgbjntk6Yf"
   },
   "outputs": [],
   "source": [
    "MAX_LENGTH = 40"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "c081xPGv1CPI"
   },
   "outputs": [],
   "source": [
    "def filter_max_length(x, y, max_length=MAX_LENGTH):\n",
    "    return tf.logical_and(tf.size(x) <= max_length,\n",
    "                          tf.size(y) <= max_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Tx1sFbR-9fRs"
   },
   "source": [
    "Operations inside `.map()` run in graph mode and receive a graph tensor that do not have a numpy attribute. The `tokenizer` expects a string or Unicode symbol to encode it into integers. Hence, you need to run the encoding inside a `tf.py_function`, which receives an eager tensor having a numpy attribute that contains the string value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Mah1cS-P70Iz"
   },
   "outputs": [],
   "source": [
    "def tf_encode(pt, en):\n",
    "    return tf.py_function(encode, [pt, en], [tf.int64, tf.int64])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9mk9AZdZ5bcS"
   },
   "outputs": [],
   "source": [
    "train_dataset = train_examples.map(tf_encode)\n",
    "train_dataset = train_dataset.filter(filter_max_length)\n",
    "# cache the dataset to memory to get a speedup while reading from it.\n",
    "train_dataset = train_dataset.cache()\n",
    "train_dataset = train_dataset.shuffle(BUFFER_SIZE).padded_batch(\n",
    "    BATCH_SIZE, padded_shapes=([-1], [-1]))\n",
    "train_dataset = train_dataset.prefetch(tf.data.experimental.AUTOTUNE)\n",
    "\n",
    "\n",
    "val_dataset = val_examples.map(tf_encode)\n",
    "val_dataset = val_dataset.filter(filter_max_length).padded_batch(\n",
    "    BATCH_SIZE, padded_shapes=([-1], [-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_fXvfYVfQr2n"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0710 09:44:53.060396 46915574880000 backprop.py:842] The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.string\n",
      "W0710 09:44:53.061331 46915574880000 backprop.py:842] The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.string\n",
      "W0710 09:44:53.063553 46915572778752 backprop.py:842] The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.string\n",
      "W0710 09:44:53.064394 46915572778752 backprop.py:842] The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.string\n",
      "W0710 09:44:53.066137 46915574880000 backprop.py:842] The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.string\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: id=311410, shape=(64, 40), dtype=int64, numpy=\n",
       " array([[8214, 1259,    5, ...,    0,    0,    0],\n",
       "        [8214,  299,   13, ...,    0,    0,    0],\n",
       "        [8214,   59,    8, ...,    0,    0,    0],\n",
       "        ...,\n",
       "        [8214,   95,    3, ...,    0,    0,    0],\n",
       "        [8214, 5157,    1, ...,    0,    0,    0],\n",
       "        [8214, 4479, 7990, ...,    0,    0,    0]])>,\n",
       " <tf.Tensor: id=311411, shape=(64, 40), dtype=int64, numpy=\n",
       " array([[8087,   18,   12, ...,    0,    0,    0],\n",
       "        [8087,  634,   30, ...,    0,    0,    0],\n",
       "        [8087,   16,   13, ...,    0,    0,    0],\n",
       "        ...,\n",
       "        [8087,   12,   20, ...,    0,    0,    0],\n",
       "        [8087,   17, 4981, ...,    0,    0,    0],\n",
       "        [8087,   12, 5453, ...,    0,    0,    0]])>)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pt_batch, en_batch = next(iter(val_dataset))\n",
    "pt_batch, en_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nBQuibYA4n0n"
   },
   "source": [
    "## Positional encoding\n",
    "\n",
    "Since this model doesn't contain any recurrence or convolution, positional encoding is added to give the model some information about the relative position of the words in the sentence. \n",
    "\n",
    "The positional encoding vector is added to the embedding vector. Embeddings represent a token in a d-dimensional space where tokens with similar meaning will be closer to each other. But the embeddings do not encode the relative position of words in a sentence. So after adding the positional encoding, words will be closer to each other based on the *similarity of their meaning and their position in the sentence*, in the d-dimensional space.\n",
    "\n",
    "See the notebook on [positional encoding](https://github.com/tensorflow/examples/blob/master/community/en/position_encoding.ipynb) to learn more about it. The formula for calculating the positional encoding is as follows:\n",
    "\n",
    "$$\\Large{PE_{(pos, 2i)} = sin(pos / 10000^{2i / d_{model}})} $$\n",
    "$$\\Large{PE_{(pos, 2i+1)} = cos(pos / 10000^{2i / d_{model}})} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WhIOZjMNKujn"
   },
   "outputs": [],
   "source": [
    "def get_angles(pos, i, d_model):\n",
    "    angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))\n",
    "    return pos * angle_rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1Rz82wEs5biZ"
   },
   "outputs": [],
   "source": [
    "def positional_encoding(position, d_model):\n",
    "    angle_rads = get_angles(np.arange(position)[:, np.newaxis],\n",
    "                            np.arange(d_model)[np.newaxis, :],\n",
    "                            d_model)\n",
    "  \n",
    "    # apply sin to even indices in the array; 2i\n",
    "    sines = np.sin(angle_rads[:, 0::2])\n",
    "  \n",
    "    # apply cos to odd indices in the array; 2i+1\n",
    "    cosines = np.cos(angle_rads[:, 1::2])\n",
    "  \n",
    "    pos_encoding = np.concatenate([sines, cosines], axis=-1)\n",
    "  \n",
    "    pos_encoding = pos_encoding[np.newaxis, ...]\n",
    "    \n",
    "    return tf.cast(pos_encoding, dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1kLCla68EloE"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 50, 512)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEKCAYAAAD+XoUoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOydeXhU1fnHP+feWZOZ7CtJIOyLIouoIFZFcd+3iv5ssWq11mqt1rq12qq1WlvtZl1Lq1bFrSpS3NG6giyisgiENZCQfZ3Meu/5/XHvJJMQwgAJEjyf5znP3e/cDMOZM+97vt9XSClRKBQKxbcD7Zt+AIVCoVDsPVSnr1AoFN8iVKevUCgU3yJUp69QKBTfIlSnr1AoFN8iVKevUCgU3yL6tNMXQmwUQnwlhFgmhFhs78sSQrwthFhrLzP78hkUCoXim0IIMUsIUS2EWL6D40II8RchRJkQ4kshxMSEYzPtfnKtEGJmbz3T3hjpT5NSjpdSTrK3bwLelVIOB961txUKhWJ/5F/AiT0cPwkYbrfLgYfAGhwDtwOHAYcCt/fWAPmbCO+cATxhrz8BnPkNPINCoVD0OVLKD4D6Hk45A3hSWiwAMoQQhcAJwNtSynopZQPwNj1/eSSNozdu0gMSeEsIIYFHpJSPAvlSykoAKWWlECKvuwuFEJdjffOBcBycIzXqUtIoGViIa30Z6/QURjkjeAtz+XxzE+MLPdRvqqWlZDDNDS0cVOBi8+qtpDs0XKNHsXp9Ba5UP2MKU2hasZaWmEluthfHwCGUVQdoa2wE08Dh9ZGVlcoAvxsaqwhsa6QlZBCVEoeAFF3D43ehe1w409PA4ydsCloiBq2hKKGwQSxqYMYimLEo0jSttyGufBYChIbQNITQELqO0HQ0TUcIgdCwlwJNE2hCoOsCXQg0DXtp7deEdUtNCOu28fX4y2DtB+uY/b52vMed3u8u7/92/yA7Ob6T/bt95g5Oaw7HSHcKpNDQIm2sbYHU8o0UTDiAVVuaSK8pJ++gA1i5vpIxqVFaagKEBg+lurKG8SOKqFy2AkNC8ahiVjc7aGuow5+bw/A0jcavN9AUM8n0OPAPHkCTlsLW2jZi4RBGOIjmcOH2+8hL95DpcSIC9YTrGwk3hWmLmUSlRGKNqBxC4NIELpeG0+vEkeJG87jR3ClIhwupOTAlxExJxJREDZOIYRKNSSKGiWGYSFNimhJpgpTSbiaYJtL+bElpf8akiYSOz5u97LSPnajw+7lKXwbraqWUubt7vZZWLImFkn2tFUDiyY/a/VyyFAHlCdtb7H072r/H9HWnP1VKWWF37G8LIb5O9kL7jXsUQEvJkecEffxrzInc8rebGXj+6ZyReTBPFm7ioFuvwPeTN/jopuE8e+UTvPubJ3n7pff59OclXHPkTZyQncrA/77HkRf8hoGHTOPTWyfy3wNP4L2aNq48bSx5f5nN6Q8u4PNXXyEWaiVvzFQunDGZ244dAi/fx6I//Jf/fV3HtlCMLKfOxAwPI48ZROaIIvJOPBF5wDTWtTn4cFMD/1tdzdoNDdRXttBStYlQQxXRYCtmLII0DQA0hwvN4cLp9eHwpOJKTceZmo4rJRW3x4nL68Dh0nF7nLi9DlI8DjJSnPg8TvxuBz6P1bxOnRSnjiYEboeGx6Hh1Kx1p6bh1EX7UhcC3f5Np9tfEJpIWMf6Moh/icT3QceXhCY6978d53bulbUkvxy0rt8yO2BHp729vpETil1EHV68mxdz2vs6h/7s+9z48cdMvPktTn/wWn787geMO/8e/ju5kvcf/oSVf3qev/zuMT558w7uyh5PU9TkD7N+z1HvZbDkhaeZesVlzD3exdypFzNvWyvnlGYz7ak7mOc9mFtmLaZ63RoaNy4nNbeEEUccwY9PGcV5Y3LRP32ejbNfoez1MpbVBakIRTEkuDRBjktncKqT4pI08sfmkXPQEPyjRuAadhBmVglhXz5tUZPaoEFFS5itzSG2NAbZ0hCksjFIY0uYUCBKOBglEowRCccwDZNoqA0jHMSMRTBiEWuQEY3YnzUTaRpI08C0P3fSMNo/g/Fl1/We9vUnosv+uWmPbhAL4Rh5erKvFUoIXe8O3X3CZQ/795g+De9IKSvsZTXwMlZsqsr++YK9rO7LZ1AoFIpdQgiEpifVeoEtQEnCdjFQ0cP+PabPOn0hRKoQwh9fB44HlgNzgHgmeibwal89g0KhUOw6ov0X+c5aLzAH+L49i2cy0GSHv98EjhdCZNoJ3OPtfXtMX4Z38oGX7Z//DuAZKeUbQohFwPNCiEuBzcB5ffgMCoVCsWvYI/3euZV4FjgayBFCbMGakeMEkFI+DMwDTgbKgDbgB/axeiHEncAi+1Z3SCl7SggnTZ91+lLK9cC4bvbXAcfuyr1Ss7O5bGwx/82ezPc2P4f7/ScYdN8Gnn7kOmruP4qBUxy89/Nfc9wVU7jt9c8Zc+QhrPrrvRR4HIw99wD+uHAT0UATEyYUYi6ex1dNYfLdDoqOHM8XNUGqy5uIhVrRHC7S8/MYW5SOu2UbVWvKaaxspTVmWs+ha/jT3aTkpZFakI0ju4CAw0tjqI2Gtgh1rREiwVh7vDUeV+0aI7WStxqa09XxU1EINIeG7tCsZKwGQhO4HBq6ptlx+Y4Wj4nrwmpWYrcjfh9fJsbEO63v4L3uLobeNU7fdXtH+5NP6ib/LHEG/momBxb8iIfev5uHrv8bc05Lo6bhBE56aCFP/uw7vPggXPzUUsafchzv3vUjjr7sUO6Yt5oB447EfPtxasIGEzM8xMafQvnfnsaTnssZE4oILnyKlc1hfA6NvLF5yIFjWbK0kebaBkINVQB4MwvIyEmhJN2DI1BLtGozbdWtNIViBAwTw4686gK8uobPoeFOc+NK8+JM9aKl+BEuL9KVQsSQdjNpixqEYibBiEEkZhKJmRgxK5lrGhLTTtiaZkdot/0zZmz/OWs/x+jfMfq9jcD6P9obSCkv2MlxCVy1g2OzgFm98iAJ9HUiV6FQKPoXQqD10kh/X0R1+gqFQtGF3grv7IuoTl+hUCgS6cWY/r6I6vQVCoUiAYFAczi/6cfoM/qFy+YIvyTryVd4876zeWDmY1z6ickzNx5NjsvBtQ9+yq8uPYR5W5spvO431K5ZxK9OP4BP3trA1NJ0Bs28iA8+2YwzNZ0Zk0rY+vp8qsIxxqS5SD3sGD7eVE9TxQYAXKnpZOb7GJPrQ2xbS2NZBTVhg6Bh4tIE6U6NlGwvKQXZuPNyMFOzaI2Y1AdjVDeHCQWjRMKxDtFMNNIpuRZP2moJ83zjU790h4am2UpcO6GrawKHnbh1OTQ7qWsnc+PJ28Sk7g4yrF2TuTtKxMbpKszqbZIVZvXEw/9ZzZbF7/DyqhrmPvg47xx5IbUX383C2c8z5uMHueC04Syd8waP/N8EFtQHKf7pLWxd+h4nTx/Gikfnku7UmDiliHc2NNKwaTnpJaM5ZnAWW95bSlU4Rr7bQcGkYTS4slm6qYFA9WYigSZ0lxdvZh7D8/0UpbnRW6oJbK2htSpAU9QkkpBkdWkCry7weBy4Up24/Kk401LQUtMwXV6kw03EVuLGk7ihmJXEDUZiRGKmpcK1FblmzFLnJiZuzW4mCnQVZil2kb07T3+vo0b6CoVC0YX+2qEng+r0FQqFIhEhem3K5r6I6vQVCoUiAcH+PdLvFzH9bV9v5jtXP4v2y+8TlZIX/v4Uw9+4j5k3HM2Gj+ZwUVYNWS6dJzZIXKnpHJ1Sy/LmEOMuPYKGEcdSsXwx2cMmMq00nQ3vrMOQUDKxgNigg3lvVTXBugp0l5eU7AGMHJRBSZqT6KavadrUTE3YaDfPynLp+PJTSS3IRs8uxEzJpDViUtcWoT4QIRyMEQ3HMCJB22GzG2FWQhxfa4/xC3RdQ9PtpSYQQrTH8F0OrT22b8XzrVh+PK4PcYFWh0irvdkSKctErXMsPdFsbXfoq5h/MtzzyIXc/+cbuOGGoyicMJ1X1jdw9l3zSckewOyr/s0BD/6dcEs9g5bOZoDHwWv1aRiRID/9TikLPt7C5Cwvo783jVmfbCQaaKJoZDGlooHyj8sJGpJhPifp48dT1hCioryJSGsDZiyCKzUdf5aX4QU+crwOzOrNtG6toa0uSFPUaI/pJwqznKku3OlunGkp6Kl+tFQ/0pmC6XC3i7NCMZOwLcxqixiEE4RZRszEjJlWXD8e0+/y2eowUzO7fb+SNVtTAEJDd7iSav0RNdJXKBSKRMT+PdJXnb5CoVAkIFDz9BUKheJbxf7c6feLmL5Dg8byVfx11jKuf+Qi3L5MHrvuRRzX/Ym04hF8ftUNnHFMKX989gtKJ0+j8qE/WDH4GZfz7PIqAjXlDB5bgnfth3y5uYksl07J0WMoa5ZsXd9AJNCEJz0HX34JEwZlkCEDtKxZR/OWZppjVtzT59BI9zhIyfPhzM3HkVOA4c2gOWxQ1xahrjVMOBglGgoRiwQ7FU6JIzS93WxN6HZs3+lC07X2SllCE1Zsvz2er3drtpYY1+9kwJZgthanOyO0rnPlNdF1Pn9H8ZT4Nd3da1fZ0+Ipca71nctpr/+WL2fey/x7Tub7Rw5k0yevcd3157GoIcRvPo8w+IiT+ej6xzh52iDufukrsodNZODWT1nVEubAs0bjmv59Vnxeie7yMv3gIswv3mXt1hZcmmDA2DwcYyaztLKZ+qpWIoEmANzpOWTkpjI0KxW/2UascoNVXa0pTMiecw9WDsijCdtszYXL78HlT0GkpCG8fqTTTThmtsf026Im4Zhhma0ZltmaaVixfCkTzNbsz1WnZmwfr99dVJwfNU9foVAovl2o8I5CoVB8axBCoDn758ycZFCdvkKhUCSiDNcUCoXi28X+3On3i0RuzgHDufe+azi5KI1XDryM2391ERWhKOc8tJAZF5/MC+9sYML9v2bjJ2/yk3MOZOFjCzgyJ4Xlooh/v1OG7vLyve8Mpvq1lykPRhnhc5H5naP5aHMDDVsrkKZBau5Asgv8jM3z46hdT8OacqpaIgQNiS4gzaGRmp9KamE2enYB+HNoCRvUtkWoaQ7TErCqZhnhIGY0sp0RVlezNc3RUTVLj1fMcmjt4ixdE7gTDdYSjNcSK2WBtR4XbSUiEpKzcWHWniZid0RvV83aGc/c9zfuuuNtZl77EIGrz2fi668z9OgzuamwgnNGZfP4429x7w8P5b+raxn/2xtY++EHjJ82jnUPPowuBIMu+i7Lgn5qVi8hvXgEZx1YSOXb77OxLUKOS6dwUinBrCF8sraWQM1mpGmgOVykZBdRmu9nUIYHvbmSti0VtFS0Uh8x2iusuTRhm61peF067nQ3rrRUXGmpaP4MpMuLdKYkJHENwjGDkGGJs+Jma0ZM2uIsaRutdTZbSyRRfJVotqaqZu0emj2xYmetP9IvOn2FQqHYWwhhzaJLpiV5vxOFEKuFEGVCiJu6Of6AEGKZ3dYIIRoTjhkJx+b0xt+nwjsKhULRBV3vnfGwEEIHHgSOA7YAi4QQc6SUK+PnSCl/lnD+1cCEhFsEpZTje+VhbNRIX6FQKBIR9OZI/1CgTEq5XkoZAWYDZ/Rw/gXAs73wV+yQftHpr6wKccGSv3P80rlc+8sn+FH4Iy45bzRLX36J+4/JI2JK3hEjAfjBqFQ+qG1j/EUT+eN7ZWxc+gWZpQdyyogcyl77gqAhGT46B0ZN5a0V22it2ojmcJFRmE/pwHSGZnqIlH1JfVkd20IxIqbEq2uW2VpeCr6iXBy5RRip2bRGLbO16pYw4WDMKqBiC7PM6A7EWQmxfat4isP+AFmxeaGBpncumNIptp8oyrJj+3o8bt+D2VriMk53//jJfiASzda+idDmaVdfwQ+mD0Z3e3lw9kqO+sMnvHLz0bxxwjUc88LvqV//BaeYK3Bpgs+zD6OtroI7TxnDZ/9ZxcQMD23jTuWxBZtoq6ugaMxIDsyAze+vpSlqMsznInfKBNY1hFm3sZFQQxWAbbbm44CiNPJTHMjqzbSUVxOo7lxARRdWXN/nELjT3FbL8KGn+tBS/JjOFEynx47pW0ZrcbO1cMwSZkUiBoZhtsfyDdtwLR7PTyygsiOztZ7i+UqEtWMsl81e6/SLgPKE7S32vu1fV4hBwGBgfsJujxBisRBigRDizN38kzqhwjsKhULRCbEr1d1yhBCLE7YflVI+2ulm2yO72QcwA3hRSpn4jTxQSlkhhBgCzBdCfCWlXJfsw3WH6vQVCoUiETu8kyS1UspJPRzfApQkbBcDFTs4dwZwVeIOKWWFvVwvhHgfK96/R51+vwjvKBQKxd6kF8M7i4DhQojBQggXVse+3SwcIcRIIBP4NGFfphDCba/nAFOBlV2v3VX6xUg/3NLIXde+yGetJxFta+bf597NhRXLcJ96F2U//SFnHVzIT59cwsBDj6PxsTsBGHTl1Xxy3yaat6xh0nkXkF+9jFdW15Pu1Bh07Cg2RVNZV1ZHqKkGb2Y+OUV+DhuaTa4jQsua1TRtaqbZnnftc2jkuB34BvhxFxRgpmZjpmTS3BClxjZbiwSjRMMR22wtukOzNc3htEzWEszWdLvFC6LH5+nrmsCldxRS6Wq2Fp+fb8XwrdfZkdlae1wfO3fQvl9sP8d+HzdbA3jC/SabnnyVOeEogbIXmPXys6Qaz/PalmY2tw6l5LBTWPCjX3PqwYVc99wyMkoPZEJkDU80hLh8xhj+83UtH366Gc3h4oiJRYgv3+LrNfXoAgaNzMZ10JEs3NJE3bYWIoEmHB6fbbaWwrDsVNK1KNHKjbRuqaW1IUTA6Ijpe3UNj2YVUHH5nLjT3LjSUtD8mWipacRcXqtwij1HP96CEYNg1CqiEjdbMwyrSSm3N1pL0mytuwIqPZ33bUcI0B29k6iSUsaEED8B3gR0YJaUcoUQ4g5gsZQy/gVwATBbSpkY+hkNPCKEMLEG6PckzvrZXfpFp69QKBR7k96sCielnAfM67Lvti7bv+7muk+Asb32IDaq01coFIoEhOi/attkUJ2+QqFQdGEXErn9DtXpKxQKRRf2506/X8zeKSzK58icFBY9929+fuslfNEU5qSHFnLmJWfz7PMrOfzh21g9/3V+POMgFvzxXaZme1mVMoptyz9CaDoXHT2Emldms6Y1zAifi7zpx/K/jfXUbNjSbrY2cUg24wvTcNaU0bBqE9saQ7TGzASzNUuYpdvCrJaYoKo1wrbGEE2tEcLBGLFgK0Y4iNGlalZXs7XOIi3RyWxN1zUcDg23Q7OqZnUxXEs0W9MTkrldE7i7arbWiyHMPjdbA7jxe7M48tK/kn33DzlqwZsMnHIqj9w7nzMGpXPnA6/z2ysn8+KCLUz+0w0sf/t9Djr2UNY/8AcAhv/wQma9u45tK5aQVjyCCyYWUfX6m6wLRMh1OyiaOoRg3kg+WltDc+VGzFgEtz/TMlsr9DMsOwW9aSttGzfSUmmZrQUNK+mvC9orZvncDjyZHtwZftwZfrRUP9Jpma3Fq2a1RU3aosmbrcH2CdeuZmu7g0riJiC6ETruoPVH1EhfoVAoEhAINEe/GA/vFqrTVygUikQEKpGrUCgU3yZ6c8rmvka/+A2TG6rllOVvMvK4c7hRfMLlM8awcPbzPHpiAa0xk7e9E5CmwZUH+HinOsBhPziEe95dQzTQRNaQcZw1OpfVLy0haEhGj82Dsccw98vKdrO1zKIBHFqayYgsL5G1y6hdXbOd2Zq/0NduthbSvTSFjXaztVBblHAwihEJYkRCOzVb0x2udrM1zaF1MlsTCUKsrmZrrniBld0wW+s6cNmZ2VrP8f9v1mwNYObxQ9CcLh54bClT/7SU1399HLoQHD/vz9SuWcS5WGZrywqPIlBTzh/OOpCPnv2KiRkegpPOYv3SrwnUlFNy4BgmZMK6N1bSFDUZ7XeRP/VgyhrCrFnf0G625s0sIC0nnYNKMshPcUDVRlrKq2mtbKU+YhI0LE1NvHhKt2ZrvgxMtw/T6SFkm61ZBVSseH5bxOjRbC3+udqZ2Zq5E9GWit/3jGW4llzrj/T5YwshdCHE50KIufb2YCHEQiHEWiHEc7Y0WaFQKPYNhKqctaf8FFiVsH0v8ICUcjjQAFy6F55BoVAokkSg6VpSrT/Sp08thCgGTgEet7cFcAzwon3KE0CveEQrFApFbyD285F+Xydy/wT8AvDb29lAo5QyZm/3VFDgcuByAB86hz3wFQt/cywP54/joq2fk3L+A6y45GJmTCvl0scXMWTqidT8+VfoAkp+fB0f3vk1qbklDJ00itzyBTy3qo4sl87gE8dSFvKwbo1ltpaSPYD8gemML/CTp7XRuHwFjesbaYhacU+fQyM3xYm/OB13QQGGL5emsEFTyGBba5jq5hChQIRIMEg01Iq5gzn6yZqtxeP5Loe+c7O19vnEoGu9a7bWvt3lXrtLb5qtATT/9Tk+SndTXf0Ks16Zjah5nCtuO4F7Kgcw5Mgz+OB7v+ScY0q5+sklZA+byNiGJTzWEOTqS8bz7PJqGjYuR3d5OX7yQFg8l1VlDegCBh6Yi2vCND4tb6S2oplwSz0Ojw9/XiFZ+amMzPWRLsJEy9fQsrmGpvrtzdZ8Do10p447zYUnw4s7w2eZrfkyiLm87XP0W8IdZmutoVifmK3FUXH8XUOJs3YDIcSpQLWUckni7m5O7baggJTyUSnlJCnlJC96nzyjQqFQdEUIthdF7qD1R/pypD8VOF0IcTLgAdKwRv4ZQgiHPdrvqaCAQqFQfCP01w49GfpspC+lvFlKWSylLMUqHDBfSvl/wHvAufZpM4FX++oZFAqFYlcRJDfK769fDN+EOOtGYLYQ4i7gc+Af38AzKBQKRbcIAa792IZhr/xlUsr3pZSn2uvrpZSHSimHSSnPk1KGd3Z9ZoqTFfNeYPHRx1ARijL9nv9x3fXn8eTctUx6/E+U/W8ut198MPP/+gHTC/18YhRT9dUHFI8/lCunD6di9jOsC0Q4MM1N7vEn8/a6Wmo3bECaBr78wUwZnsOgdBf6ttXUrdhARXO43Wwt06njH2ALs/IHYqZm0xI2qQ6E2dYYoiUQIWKbrZnRCEa0Z7M1zRZmWeIsbTuzNZdtttZ1ROFyaD2arSXSndlaTwjRex+EvTX2OeMHd1N33qkMf+Mtxp76XR58eBH1F/+O+//4ArN+dgQvLa9m0oP3sPKdt5l22mGs/O0fcWmCYVf9iFlvrsGIBMksPZCLJhaz5dV5rAtEGOBxMvCoUTRlDOWdlVU0V65Hmgae9Bwy832MLMlgWFYKjoZyWjdsprm8hfqIQatdYc2lCTyaIN2p4XXpeDM9uDP9uDP9aP4MpMsyWwsZknCso2pWIF41KxIjGDG6NVuLTxDoarrW1WzNTPjsJZu8VUnezggBDk0k1fojyoZBoVAoEhDs3zF91ekrFApFIqL/xuuTYf8NXCkUCsVuYI30taRaUvcT4kQhxGohRJkQ4qZujl8shKgRQiyz22UJx2baljVrhRAze+Pv6xedvmv4CKZfcRnPfFbB9Xefxop5L3BTYQWZTp0/b/bhSk3nnLRqPq4LMvnGE7htzgrMWISzjx3KWaNzWPn850RMycjJRcTGHMOcJVtprdqIw+MjZ2A+k0uzcFWtJrxiIbVf17EtZGBIW5jl1kkr9uMfmI+WN5AALqoCYaoDltlasCVCONRhtta1kIXoGstPLJ6ia2i6tdQdAkeiuVp7IRWtPW7f1WwNLNFUMmZr8XFLPH7fk4tgb5ut9UWxiZKDj+bpDzcz+fq5fHLjFEb73Zx993za6ioYv+SflHidzG4eQLilnt+fOpq355ZxbJ6P8pKpbFi8FH/hUIZMHMUIrY6y19fQGjMZl+Eh5ztT+aq6jfXr6mmrq0BoOqm5Ayka4OegknQKUh0YFWU0b6xsL6ASF2a57OIpaU4rnm8VUPGh+zPQ/RmYLh+Gw0M4JgnFtjdba4sYGHFRVswWaMVMjFgMaRjbxe07hFpmp/cmLtpq396NOP+3nd6avSOE0IEHgZOAMcAFQogx3Zz6nJRyvN3iDgZZwO3AYcChwO1CiMw9/dv6RaevUCgUewtNWIOuZFoSHAqU2RNYIsBs4IwkH+UE4G0pZb2UsgF4Gzhxt/6oBFSnr1AoFF3QbcuTnTUgRwixOKFd3uVWRUB5wvaOrGfOEUJ8KYR4UQhRsovX7hIqkatQKBQJxG0YkqRWSjmpp9t1s6+r9cxrwLNSyrAQ4kdYRpTHJHntLtMvRvpfb6plzpRWLjlhCEtOvYWSw07hjROu4fvXTOWPf3+XCaedxPJf3EyBx4Hv4l+y6sPPySw9kEsPKUZ88DQLy5sp8ToZfvYUFlW2sXl1LeGWelJyBjBkaBZj81KJrV1K/Zerqd3QYbaW5tDJzvTgL87EVTQI05dDY9hgW0uYyuYQlY1BQm1RIm0BosGezdaEpm1nthY3WdMdlk1rvGiKy6Hb5mkd8f3uzNYSC6LHl12LpiSG07vG1jXR+fiemq3taeR+V0L/y28cxS9/ewpVX33Ax4cfx8Vz72DDR3M4bMZ3ef7yfzDjJ4dz++OLGDj5ZLI/msWa1ggHX30kf/pwI81b1lB80AS+d/QQIvOf5vOtLfgcGiVHFKONPZr/ra+jbmst0UATrtR00vNzmDgokzG5PnzheqIbV9G8qZ765jDNMctsTRfg1a05+u50F55MD57MVDwZfjRfBiIlHelOJRQzCRkmrRHLYK01HGs3WwtGDGJRAzNmYhoSU8rtzNbMBLM1FZ/vO3pRkbsFKEnY3s56RkpZl6BXegw4ONlrd4d+0ekrFArF3qKXxVmLgOF28SgXliXNnM6vJwoTNk+no/7Im8DxQohMO4F7vL1vj1DhHYVCoUhAIHrNhkFKGRNC/ASrs9aBWVLKFUKIO4DFUso5wDVCiNOBGFAPXGxfWy+EuBPriwPgDill/Z4+k+r0FQqFIoFdjOnvFCnlPGBel323JazfDNy8g2tnAbN67WFQnb5CoVB0Yn+3YegXMX1pxPjLET9h8PNzmXnz08y5/The29JM6q0PUfP1Av4582Befbk/C/YAACAASURBVG0tJx81kMe+qqd+/ReMPHwcA8o/Ye0/X6IiFOPgQh+px5zDi19UUL9hJULTySgZwbTReRTqbTQtW0bNF5vY3BYjaJi4NNEuzEorLcQ5oBTDl0tD0KqYtaU+SKAlQjgYJRZstcRZ3Zmt6Tq6LcxKFGlZCdwEgVb73F99u7nAui3KcmoCp9Zhtqa1J207hFnQYbjWLtKiZ4FU4oegPQHczXk9Cbr2Ng+MOI3nj7yeX9xxNc9/Vc09beMYcuQZvPGjQ1lQHyTn9ofZvGAeP//+RD6++SlKvE5yLr2Bee+Uobu8nHbUYM4alcOa5z6gPBhlaKqLQdMnsEVkMn/5NloqygDwZg8gu9DH2MI0Bmd4cNRvomndVho3NVETNggaHWZrqbpVMcuT4bHM1jL8uLPS0dOzMd2pmK5UgrHOZmutoRhtttlaOGJgGpJY1Ogkzuq2ala7QMtM2mwt2X3felQRFYVCofj2EPfT319Rnb5CoVB0QXX6CoVC8S1B28+LqPSLTn94aT7BslaOuPUtWrdtJP3h6zljUDpnP7KQgnHTyH/7z1SEYky481p+8OxynKnpXH/SKDY9ci1L3tqAVxeMOH00W/1D+XjZxwRqynH7sygYlMmU4ky0TZ9R/XkZtavrqI3EMCRkuTQKPA7Si9NIHViEzBxAQ9ik0o7nVzYFCbaGCQejREOtmLFoJ3HWdsVTnC4rtu+04vkOp27F8+MCLVuYpWsCl965kIpT03DqWrswK7F4ynaCK0QnYVbigCXRbK3rQGZX4/W9bba2q+mCXLfOldf9kdprC1l79kiO+t0TfDb7JtZecg5nDsnkome+ICV7AJcMinHzmjpmHDeY12s9VHzxATkjDmHmwcVkbfyYNz8qx5AwelgmaUefwmubm9i2sZFgQxW6y4s/fxBjS7MYmZNCYYpGZPEKmtZtpWVbgKbo9mZrqV5Hu9maJzsNLT0bzZ9BzO0nikbYiNEWNWiJdBRPaQ1bcf240ZphmEhT2ssOIVaiMAt2EKPvwWxNkSS9PHtnX6NfdPoKhUKxtxBsX41uf0J1+gqFQtGFvrAD31dQnb5CoVAkILBqVuyv9Itshdi8jp/P+zUbPprDJT+/jL/fM5/j5/2ZJf95mVuvPJK3fvYs0/NSWTHgSDZ+Np+Bh0zjxAKTL5/7ii+aQoxL91B87pm8vraOyjWbMGMR0opGcPiYPEZmuwktX0DNylq2VrfRFO0oiJ5W5CdtcAGOAYMx/Pk0hAy2NofY1hSkrilEKBAlGmjCCAcxdmC2Zs3Ld3YqiO5w6t0WRO80L1/r8PTuWhBdFx3FU7qarXVXEF0TotuY+Y4GM4m795bZ2q4yo3wxgyYfzx0zZ5H12EsITSP1weuZ9cIqpr/4O96fPZcpZ5/A+ttuIGJKDrr1Cu6ds5JooIkxU4YzJLCWitnPsrw5zACPgyHHjSJQPJHXl1dSv3kdZiyCJz2HrEI/EwdlUORz4qxbT6BsLQ3rG9kWitEcMzFk4hx9DU+mh5ScFDzZ6Xiy09HSs5HeNKTbRzBqEopJWiOGZbYWitESjrUXRI9FTHuOvmyP65uxyHZGfpBcQfSdxfNVvH8HCKz8WRKtP6JG+gqFQpGAAJxJlkLsj6hOX6FQKBLY38M7qtNXKBSKRET/Dd0kg+r0FQqFIoGdeVX1d/pF4KqmKcxlW0bynR/8gD+VlpOqa9xTOQCn18cPc6p4syrA9DvP4KezlxELtnLRqaNoe/FvfFwXJGhIxh89kNjE05n96SYay1fh8PjIH1LEtOE5eKtXU/XZSiq3tLC5LUrElPgcGkVeBxmD0kgfWoReMJiA8FDREqaiMUhVY4hgS4RwKEos1IoRCWF2Y7amOzuSuHHTNYfL2WGypluma45EszVbmNWRxLVGHbqgc0LXTt4mmq21G6wlVM/qlJRlexFWd2Zr3ZF43XbCrh1c05f/cYZf8QJf/vowRvvdTLvlTW791cU8cu98BnicPGmMIdRUy6wLxjHnmeWcVJLG5hEn8fUHn+IvHMp1xw6n9oV/svL5ZTRFTQ7OSaHg5BNYVNHKyq9rCNSUIzQdX/5ghgzKYFx+Gp7GzRibV9G4tpzmLS3URzqbrfkcGpkuh53E9ePJTsORkYXuz8B0+TAcHoIxSSBi0BKO0RKxK2ZFDNoiBpEEcVbcaM2IxdqFWdLsXDHLaman96SrMKvTMZW03SXi/9921vojaqSvUCgUCQgBTr1fjId3C9XpKxQKRQL7e3hHdfoKhULRhf4aukmGfvEbpiDfx/MPPMJbZ2Uwa/r1XP3gBdz/xxc45eKz+HTm9YzwuTAu+CVfvf0BeWOmctVhxSz92zu0xkxG+FyMuPA43tnQyIbllUQDTfgKShk7Oo8JhT4iyz9m25KtbAhEaYhacc9Mp052birpg/NwFQ/BSC+gLmhQ2RJmU10bgeYwba0Rwi3NRIOtxCJBzFik/Xnjwqx2cZbTLpzi9nY2WXNoaLYwKx7Hd3cRaGnCMlxz6JodywenLraL7SfG8TU6i7HiRmtxNEGX491/wvfWBIbdGVS1Vm3gueHTuOiLF9my6A2uMT5BF4If3n8utz3wNqOOOx3nU79mXSDCEXecxa3/XUVL5TqGTT6UabkxVvx7IZ9VtJDu1Bh6whDkuOOZu6KKmg1biAaa8KTnkl2Sx+HDcyjNcCHLVxFas5yGtTXUNIXahVm6AJ9DI8ul28IsL57sdFLyMtHSssGXjfT4CcZMgjHTiuVHbGFWKEZLKGoJs6JWMw1LmGUanYunmOb2BVS6I1lhlmLHCETnXFkPLan7CXGiEGK1EKJMCHFTN8evE0KsFEJ8KYR4VwgxKOGYIYRYZrc5Xa/dHdRIX6FQKBLpRZdNIYQOPAgcB2wBFgkh5kgpVyac9jkwSUrZJoS4Evg9cL59LCilHN8rD2PTL0b6CoVCsbewYvrJtSQ4FCiTUq6XUkaA2cAZiSdIKd+TUrbZmwuA4l78c7ZDdfoKhUKRQNyGIZkG5AghFie0y7vcrggoT9jeYu/bEZcCrydse+z7LhBCnNkbf1+/6PRbs4oYdtTpvDxpBqtawnw85Sra6ir41+mDeO7TLZz94ylc++pKWqs2cvwp43DPf5wP1tZTmuLksHH5OI79Pk8s2ETD+i/QHC7yho7gxAPyyWmroHbBUqrL6mmIGgQNa45+gceao585ogTnwBEE3Zlsa42wuaGNysYgwdYIoUDEnqMf7HaOvtB1a46+s2OOvu5w2PH87gui60K0x/NdDg2XruHUO+boO9vj+h1Ga4lz9DsZrok9K4iu7SDmn+wc/b7ms3//nHWBKN95chvHX3EJs86+mytuO4G1J95A9cqP+ddVh/Pa7XOZnOXFOPsXfPj6EryZBfz4lFGEXn2YT8saqAjFmJjhYdDpx7CqRePjLyppqVwHQGpuCQMGZjCxMJ20UC3hsi+p/3oTDRsa2RYyaI11LYiukZLjxZvtIyUvE2dGBnpmLqbHj+H2EYiahGKmFc+35+i3hq15+sFQjFg0cX6+iWnK9s9V1zn6sH1B9F2do69i/j0gsP5/JdGAWinlpIT26PZ32w7Z7csKcREwCbgvYfdAKeUk4ELgT0KIoXv65/VZpy+E8AghPhNCfCGEWCGE+I29f7AQYqEQYq0Q4jkhhKuvnkGhUCh2lfhgqZcSuVuAkoTtYqBiu9cUYjpwK3C6lDIc3y+lrLCX64H3gQm7/YfZ9OVIPwwcI6UcB4wHThRCTAbuBR6QUg4HGrB+zigUCsU+gv1rOomWBIuA4fZg1wXMADrNwhFCTAAewerwqxP2Zwoh3PZ6DjAVSEwA7xZ91ulLi1Z702k3CRwDvGjvfwLolTiVQqFQ9Aa9OdKXUsaAnwBvAquA56WUK4QQdwghTrdPuw/wAS90mZo5GlgshPgCeA+4p8usn92iT6ds2tOVlgDDsKYtrQMa7TcCekhq2AmRywGyC4pI6csHVSgUChtha2F6CynlPGBel323JaxP38F1nwBje+1BbPo0kSulNOw5psVYU5dGd3faDq59NJ4caWiJsvTOo/mgto2f3XA0l/3mVQ6b8V1WXT6TLJdO4a/+wpsvfUDWkHHcfvxwlv7+BbaFYhx+YC5jL53Gp3UaXy6pINiwDV9BKSPH5HJ4STqxrz6gYuF6ylqj7Ym5TKdOYbaXzOG5eEqHYqQPoC4Yo7wpyKa6NpobQ7S1hAkHWokEmjAioe2SuIkGa/Gl5nSh6RoOp241l7VMFGR1SuI6OpK2mhYXYtEu2OqopNU5eQs9VMQSImlh1p6SvHBl9+6/adox3DL/Xpa88DSvHquzpjVM/cW/44J73mPQ4acxctE/WVAf5KQbp3P72+uoXbOIwZOPYMaoDL76x3uUB6P4HBqjjh6EY8qZvLx8GxVrtxJqqsHtzyKrpISpw3MYluVBbFlJ/fINNKyupKY2SEPUIGLKBGGWhi/TQ2p+Kt7cTFzZWeiZeWj+LEuYFbWEWU128rY1bAmzgpEY4QRhVixqWhWzpGyvlmXGIp2EWaCSsHuD+KSInbX+yF4RZ0kpG4UQ7wOTgQwhhMMe7Xeb1FAoFIpvEu0bm5fW9/Tl7J1cIUSGve4FpmPFtN4DzrVPmwm82lfPoFAoFLuKQI30d5dC4Ak7rq9hJTDmCiFWArOFEHdhyY//0YfPoFAoFLvMflw4q09n73wppZwgpTxISnmglPIOe/96KeWhUsphUsrzEuek7giH18f8A47gZz87gqor76d2zSLe+NGhPPXyav7v4vFc/+YmGjcu56jTppC3+DneXVLJAI+DcZcfg/fUy3jk4w3UrF6C5nCRO2wMZ44vojBaQ+3HC6haXkNV2More3VBkddB5pAMskaV4iodRTg11xJmNQbZVBugrdmK50cDTRiRILFwN2ZrCcIszeFCd3lxuNw4XPp2wiyvS++2eEpcmOXU7GYLs5xahzCrvYhKgjCrvZAK1rG42VoyxVP6izAL4I01dZy0KI8pF32fpw+byTXXHsHZd89n86dzeeRnR/DfKx5nXLqHtGvu4z//WYLbn8UVp4/BmPs3PllWhVcXjEt3M/y8aayJZfDWkq00b10DgC+/lILSDKYMyiQ71kBkzefUrdpK3doGtoVinYRZaQ6dLJdOal4qqXl+UvIy0TPz0DPzML3pmG4/bVGTYNSkKRyjJWLQ1BZtj+tHwp2FWfGlNHsuntKdMKu7mL8SZu0GSY7y9/uRvhDicKA08Rop5ZN98EwKhULxjSFIeg5+vySpTl8I8RQwFFgGxIcJElCdvkKh2O/Yn8M7yY70JwFjpJTdTq9UKBSK/Yn9uM9PutNfDhQAlX34LAqFQvGNs7+XS0w2kZsDrBRCvCmEmBNvfflgiRxYks7r5c1UXf1nzrr5P0y+8P9Ye8k5+BwaA3//OC8+8x5ZQ8Zx3+ljWPrbJ6gIxTj6oDxSTr+cBS2pLFq4hba6CnwFpYwZm8+RgzIwv3qfrZ+UsbolQmvMxKsLclwOCrO9ZI/Mwzt0OEZmCdVtMTY2BFlfE2gXZkUDTUkJsxwuL7rLm7QwK7ElI8yKJ2qhszBrRz9N9xdhFsDd8+/mw3/+k/fO8rG0MUTw5w+y4aM5DJxyKlNWPss71QHOvPFYbnp9LVXLP2DI4cfwg4Ny+fyv81gXiDAxw8NBx5TiPHoGLy2vZOsaS7zn9meRPaiUaaPzGJ2TgrZ1JbXL1lC3toHq6gC1kZ6FWe68HEuYlZ6D6U2nLSYJJAizmkNRWkIxWkNRW5hlJW/jwizDMC1BVjSyA2GWmfR7pBK2u49K5MKv+/IhFAqFYl+iX3jO7yZJdfpSyv8JIfKBQ+xdnyW6wSkUCsX+gujFcon7Ikl9oQkhvgt8BpwHfBdYKIQ4t+erFAqFon+iwjuWuf8h8dG9ECIXeIcOi+Q+pemrVdx42/eZ9PNnaNi4nHUPncVNN67i6quncMVr66lf/wUX3vATcj95glmfVVCa4mTiNSfyYZOXBz8oo3rVIjSHi/zhB3DewcUURSqpfP8jKr6qbhdm5bgcFHkdZA/LJPuAIbiGHEAgNZet29rYUN/WSZgVSUKYpbu97UZrfSXMiouxEoVZiRWzEoVZiQOX/i7MAjjmk3ym/fBSHj/4In7+q+M54ra3GHLkGTx1/ZG8OPFwDsn0kHLNH3jxsqfwpOfy03MPJPrifby/dBs+h8b44wYz7PzjWBVNZ97CNTRuXA6Av3AoRUMyOaI0i5xoHaHlC6hZvoXq6gBbgz0Ls1ILs9Ez8xAZeZgevyXMCho7EWYZ2wmzdlQxK1F8lYwwqztUnH/nCFR4B0DrEs6pY/9+XxQKxbeYvprksC+QbKf/hhDiTeBZe/t8uvhDKxQKxX5BDzPg9geSTeTeIIQ4B6tclwAelVK+3KdPplAoFN8AAujFGir7HEmHaKSUL0kpr5NS/mxvd/ithsmHZ95G05Y1nHHVJSw57UwGeJxk3PE4c56cS96Yqdx/2igW/OpJtoViTDu8GOfp1/CHd9aydEE5wYZtpBWPYOLEQo4alEF0yVuUf7i2fY6+z6ExMMVBUV4K2WMG4B02iljWQKoCMTY2WnP0m+qDBJpDRFrqiYUCREOB7eL58Tn6usvbvnS43B3z8xPm6HtdOm47rp/i0u34vhXPt+L3Gg5da5+j79S7KddmR9a1hNh++/N0Y7S2K3P0d/fn7d6Yow+w6PlneG1sOeXBKKsuvIuti+bxyi3TGP7GfXxcF+Tc+87lypeWU/P1AkZOO5aLhrpY9Id5lAejTM7yMnzmmejTvse/FpVTvmI9oaYaPOm55A4exAljCxiTmwIbl1Hz+VpqV9exNRjrVDwl3amT5dLwZ6fgH+AjpSAbd14uenaBZbSWkkkgJmmNmtQHozSFYjS0RWhsi9LYFiEYsozWYvZc/XghlZ6Lp5g9GqjtzGhNkTxCiKRaf6THTl8I8ZG9bBFCNCe0FiFE8955RIVCodh7WBMhkmtJ3U+IE4UQq4UQZUKIm7o57hZCPGcfXyiEKE04drO9f7UQ4oTe+Pt6DO9IKY+wl/7eeDGFQqHoD/TWGN6uJ/IgcBxWTfBFQog5XQqcXwo0SCmHCSFmAPcC5wshxgAzgAOAAcA7QogRUso9+hmX7Dz9p5LZp1AoFP2fbkKpO2hJcChQZtcRiQCzgTO6nHMG8IS9/iJwrLBiR2cAs6WUYSnlBqDMvt8ekWxM/4DEDSGEAzh4T19coVAo9jl2rYhKjhBicUK7vMvdioDyhO0t9r5uz7FrhzcB2Uleu8v0GN4RQtwM3AJ4E2L4AogAj+7piydL0bACrvjZg/zkliu4Z0yAH/9gM/c8ciFnPr6IQE05115/Ptqzd/Hf5TUcmOZm3PUX8PL6AMsXrKOubCkOj4+SA8dw4aQS8hrXsvHtD9m4spaKUAxdQL7bQdEAP1nDM8k5aCiOwQfS5Mxgc32AsppWNla30toYItzSTCTQRCwSbBfQxNEcLnSnC93lQXPayVy3NyGJq7UvXXbS1uty4NI7G605NctkTRfYCdwO87Wuwqz4QCPRdK07h8BEo7VkhVldr09kR+ObvelM+Js//II7jzuRW57/KYN+8RQHn/d/+B++gcfue4/TitOoOf1G3pz5F/yFQ7lrxngaHruT99bUkevWGX/eAXDk//FRRRvzF26msXwVQtNJLxnNqFE5HFWaTWZrOa2fL6D6iy1U1gWpjXQIs7y6RppDI9/jxD/AR2pBBr6iXPTsQkR6HkZKJoYzhda2GC1hg6ZQjKZwtF2Y1RqKdSRuDUksYmDETIxYrN1orSdhFtBJmJUsKrmbHEJKRPLvVa2UclJPt+tmX1eL+h2dk8y1u0yPI30p5e/seP59Uso0u/mllNlSypv39MUVCoViX0RIM6mWBFuAkoTtYqBiR+fYUZR0oD7Ja3eZnc3eGWWvviCEmNi17emLKxQKxb6HBGkm13bOImC4EGKwEMKFlZjtaks/B5hpr58LzLcLVs0BZtizewYDw7E80PaInYmzrgMuB/7YzTEJHLOnD6BQKBT7HL1UJFBKGRNC/AR4E9CBWVLKFUKIO4DFUso5wD+Ap4QQZVgj/Bn2tSuEEM8DK4EYcNWeztyBnU/ZvNxeTtvTF9oT1kdScKfncGfqEl6Ycg8nF/hYe+INfHb+7Qw76nRuHu/l1YteJWJKpp83mpbDL+LPf/uUmlULMCJB8sZM5fjJAzlqUDptLzzEpvfWs6Y1QsSUZLl0Bqc6yRubS+aIYjwjxxPLGUJla4y1dW18XdlMc4MlzAq3WsKseNw1juZwtYuz2ounuL04XM4OQZarQ6DlcmikuLopoqJr7TF8h73ekzBLa4/TJxZT2bnRWifBVjfvd1+LTnrj9jPeuItP/G5ulccQrPsX719zKXflXE5rzOTaF3/Pdx5ZSEvlOk648occ59jIq/fPpyZscM6obEovvYTX1jcze3E5W1esIBpowpdfyoDhRZw8tpDROR6MTxdStfhrar+uazdaM2TcaE0j162Tmp+Cv9CHrygXV34hem4RZmoWUYeXtohBa8QSZjWHYzS1RWkMWsKscDhGNGxYwqyIYRVOiRdPiYuzEg3XTKOTMMvsRoS1M2GWiufvAlImO4pP8nZyHl1sa6SUtyWsh7AcjLu79rfAb3vtYUh+yuZ5Qgi/vf5LIcR/hBATevNBFAqFYl+hF2P6+xzJTtn8lZSyRQhxBHAC1pzSh/vusRQKheKbQoIZS671Q5Lt9OO/DU8BHpJSvgq4+uaRFAqF4htE0puJ3H2OZK2VtwohHgGmA/cKIdzsRT/9puoalj14KX8ZcQgb2yL89eunGXHPezi9Pv5+1RTW3XwZ71QHOLXQz4ibb+G2jzdRtnApRiRISvYAhk0axoUTi3CtfJcVcxeyclMTNeEYLk1Q4nVSODyL3HFDSBsxBFEymtqYkzV1zaysaKaiJkBLfZBwUw3RQFN74ZR4jFRoOkLTcbi96C4Putsqhq67vAkGa/YcfZeGu91gzYHXmWC0Fp+jL0R7ARVrXUNv36d1O0c/Xgx9R6HyeIzfWu8waUtkb83R7610wT2//x9/aVzMJdN/ya/uuY7PjjsZXQguOW80s52T+PK/v2fAwSfw4LljWXn1+bxX08Zov5sJVx7NtkFH8OAzy9i4spqWinU4PD6yh45l6rhCpg7MwFPxJVULF1L1xTY2NIepjcQw7Lyez6GR63aQm+4hrTgNX1EOqUW56LlFSH8OZkomLRGTQNS05uaHY9S1RahrjdDUFqE1ZMfzox1Ga4ZhzdGPz8k37Nh+ohYksXAKsMtz9BW7goRdKEDf30i24/4uVvb5RCllI5AF3NBnT6VQKBTfIPtzTD9ZP/02IcQ64ATb6e1DKeVbfftoCoVC8Q3RTzv0ZEh29s5PgaeBPLv9WwhxdV8+mEKhUHwjSAmmkVzrhyQb078UOExKGQAQQtwLfAr8ta8eTKFQKL4p+mvoJhmSjekLOmbwYK/vNXet1KxstBsuJGCYXHPZRK75ys/mT+cy/aIzmLzhNZ5/ejkDPA6+c+cZfCyG8sK81TRtXkVa8QgKxx7KpUcNZZRWT9XcOWz6sJyNbVEMaRmtDclLoeDgItLHj8c95lBCGQPZ1BRidU2rJcyqC9LW1EykrckSZsU6G60JTUdzutAcTjRngjDLqeN0O3C6Oxuuee0krkvvEGZ5XTpOzRJjxZO4Tt1K7FrrCYZrWocwS9gVs+L/QN0Js7pLnPZktJYozNpXk7gAv7j2cMb+8iOKDzme64Jv8/SCrVx+63EM++d/uOWBd9CdLm687DBy3v4rr7+6Fl3AUceVkj7jamYt2cqaxRuo+XoxZixCevEIBo3O5dQD8hkomggtfpdtC9eyZX0jVeEYQcOqluXVBZlOnQKPTlqxn/RBmfgH5uPIH4iePQAzNZuAqdMcMWiNGNS2RWkIRqlvjdAUjNLYFiUcjBKLGO3JXCuJ2yHMajdbMzoLsxKJJ3GVMKuv6FUbhn2OZEf6/wQWCiHiZRLPxJIOKxQKxf5HP+3QkyHZRO79Qoj3gSOwBnw/kFJ+3pcPplAoFN8IvWzDsK+xMz99D/AjYBjwFfB32+RfoVAo9ksE+3dMf2cj/SeAKPAhcBIwGri2rx+qKyPSJX97ZgV/fOYythx7DU+cewcDp5zKM+eN5J0xP6QqHONH549Bu+BWfvnQQrZ8/j+cqemUTpzA1PEDOG1EFtH//pm1r33JssYQrTGTdKfGMJ+TgvH55B86BueIg4llFrOlJcrK6lZWbG2iviZAa2OQUFMN0UBzuzArTtxkzWGLsZweHw6vD6fHg8vtSCicorfH8y1hVsfS69JtozWREMPv2WhNJMTz48KsrvH8RLozWuuOnuL5O2JvFk5J5JWz72Tzz++n8t37uD9vHGcNz6Lxsnu58KGFVC3/gKkzL+aHxQHePO9Z1gUinDEonTE/v5z5DSm89O5KalcvIhZqJSV7AEVjhjPj0BIOGeCDJe9S8eHnbFtWxYZAlPqIFQ/36ho+h0aBRyej0EdasR//wHw8JSU4CgZi+HKIuPy0BA2aQwZN4RgNwSi1rWHqAhEa2yIEbWFWJBxrN1uLRaKW6Mo28duR0Vq7CdsuxvMVu4OE/Vj8trNOf4yUciyAEOIf7IKXsxCiBHgSKABM4FEp5Z+FEFnAc0ApsBH4rpSyYdcfXaFQKPqAuA3DfsrOZu9E4yu7EdaJAddLKUcDk4Gr7OruNwHvSimHA+/a2wqFQrHP8G1W5I7rUhs3XitXAFJKmbajC6WUlUClvd4ihFiFVdT3DOBo+7QngPeBG3f3D1AoFIre5VucyJVS6r3xIkKIUmACsBDIt78QkFJWCiHydnDN5VhVu0gXDv4+/QieHHwR997y/+3deXxcZdnw8d81+2QhaZIm3ZumC90tUHYstOxYk9zPgwAAIABJREFUpOICPiLqAyK++rz6QZDteX1URFFE0EcQqgiiCMhSFgVKgUIpshVoS6F039IkzdIkk2X23O8f58x0kmaaKW0zmeb6fj7nMzPnnJlzDkzvnLnu+7ru53B6fDx03Vw2XPkVnqkOsKBqCFN/8XOuWbKJNS+9TrSjldHHf4avnjWRM8eXkf/hC3z4j1dYs2E3u+xCa5V5HkZPKWP4cZPwTT+BaMWRNAbjfFgfYOWOVrbsbKNtd5Bgcz3Rjlaiwfbe4/lpCq25fU489jh9p8uB3+faq9Baotia2yH47HH7yfH5PQqtuZ17YvlOR/d4fm9R9T3j+JP/PZPrYe8x+n3G+/f9v7hPBzv0f/33b+G239/AeyedRtwY5i17lOk3v8z2t5cw5sT5PPSNY1jznxfy7M4A04/wcuINn6F20tnc8tf32P7eW8RC7bh8BVRMmc282aM4vaqE/Or3qHv1Varf3MGm5lCy0JrHIZR5nBS5nQwt8lE8toiiccMoHDsCV8VoTFEFXfmltEW6CETiNHZG9iq01tIeIRKKEQ2nFlyLJwurdcUiexVa6xnP3xcdn3+QDdZG/2AQkQLgceD7xphAprMyGWMWAgsBRjp8B2fuMqWU6kuiDMNh6pCWRxYRN1aD/6Ax5gl79S4RGW5vHw7UH8pzUEqp/WMwsWhGy4EQkRIRWSIiG+zHIb3sM0tE3hCRD0VktYhclLLtfhHZIiIr7WVWJsc9ZI2+WLf09wJrjTG/SdmUOvP714CnDtU5KKXUfjP0V8G1TAa1dAKXGmOmAecAd4hIccr2a4wxs+xlZSYHPZThnZOBrwIfiEjiZG4AbgH+ISKXAdtJMyGwUkplg8H01yQ1fQ5qMcasT3leIyL1wFCg5ZMe9JA1+saY5aTv/zt9fz7L5YBhDz/DdV/8OaHWBm685WomPn8rP/vHWqYf4eW0O7/No61DeWzRy7TVbqJ0wtGcdcYELpk5jOKm9Wz9+8N8tGwH69utjtjRfjdHjjmCUSeNp/j4E+kaO4ttbVF2BsKsrgmwdmcrLQ0ddOxuJtTaQKQzQDwS7DZbVs9O3ERiltV560qZNcuJx+60LfC58bv3JGZ5XA67A9eJy7mnE9chexdaEwGnXUStZyduX4XW+urE7WkgF1pLOPrzF/O5F27hpg/quX3R9zj3sVq2LH+awuHjufN7pyD3XMfj/9pIicfJOV/9FL5LbuT6Zzfy8eur6WjYQV7pCAqHT2DWMSO4aNZIRkdqaHvtOXa8+jFbt7SwIxhNFlor8TgZ5nNR5nUxpKqYonFlFI0fiWvEOBxDxxArrCAQd9IajtHQEaGxM0prOEpDIMzuDqszNxKOWUlZ9mxZPTtxeyu0Bj2Sr+JxTcbqD4b9mTmrTERWpLxeaPdHZiKjQS0JInIc1jS1m1JW3ywiP8L+pWCMCfd10EPekauUUrllvzpyG40xs9NtFJEXsRJUe7pxf87I7v/8K/A1Y5JDi64H6rD+ECzE+pXw074+Sxt9pZRKZcwBd9Lu+ShzRrptIrJLRIbbd/lpB7WIyBHAv4D/Nsa8mfLZtfbTsIjcB1ydyTn12+TmSimVG0yP+kfplwPU56AWEfEAi4AHjDGP9tiWGAUpWOXu12Ry0Jy40y+bNpFTvv8YLl8+pyw4l+uL13HHVY/hdwoX/fg81s+8iJt+vYxdHyyjoKKSmXOP4qo5VRSufJr65a/x8RMfsao1RKTLMMLnYnqZn9Enj6H808chk46nJp7HqroA25o7eW9bM011bbTtDhBsqSPaGSAe3jue73B79ornu70e3F4XHu+eCVT8Phcel4PCHvF8v8eJz+XsPnGKnZTlsIutJZKyek6ckmk8P7X42iedOCWdbMbzAV6d28L/PWkxP/z+Sfy2aD7Lb76NqjkX8NXPTuHUTY9zz89eoDXaxSVnjqPyhpv43bt1PPf8x+zevAp3fhHDph3L8HFD+PoJY5lRGCHy0jNsXfwuO1bXs6kjQmvU+gVd5HYywudieKmfgvJ8SiaUUjR+JN7R43CNqCJWNIygw0dLZ5yGjij1HREaOsK0dkapbwvT1B4mFIwSCVqJWVZcP04sEiZuF/BLJmYlk7L2JGYB3QqtJejEKYdQYvTOodfroBYRmQ1caYy5HPgSMAcoFZGv2+/7uj1S50ERGYr1z3olVkXkPuVEo6+UUv3H7E9H7ic/ijFN9DKoxRizArjcfv434G9p3j/vkxxXG32llEpl6K8hm1mhjb5SSnVzeJdhyIlG/6NdIRxbVnHfXddwYVkbD828gppQlO9ceSyRr/+Mb/7vv9n8+vN4C0uYctop/Gz+VKoa32XdHx+k5t063mzooDXaRYnHyYwiL2PnjGHkvONwzZxDo6+cD2raWbGtmW1NHdTuDNDa2Eln004ibc3dCq2ljs93uDy9Tpzi9bvw+N14/S68XheFdky/t4lTfC6ryJo3MUY/ZZx+z4lTeo7VTxfPT9hXPD9VX/H83ou5ZTeeD/D/Tr2Gr80dy4Yr7+Dmb/6KsknH8uQNc5nY9B6Pnfa/rG0L86UZ5Rzzmx/xaEMBf3ziXXZ9sAyHy0P51JOZ++lKPj2+lLljj6Drtb+z/bnl7FhezUeBcHLilCK3gxE+F6OLvJROGEJ+RT7Fk0aTX1WFe8wk4kcMI+wtYndnjKZglNr2MLvaw9S1hGgLx2hqD9PWESEcjBEORYmG9kyckjo+PzWeb43XP7CJUzSef4AO4uidgSgnGn2llOo/eqevlFKDR/+N3skKbfSVUiqFwWD6YfROtmijr5RSqfROP/vCbS38+hffZd4rt7H41iW8uTvIlRdNpeJXf2H+H95izQvP4nB7mHTqPH7y+RkcE9vE5rvu4t1nN7KlI0pDOE6R28GnirxUzRnDmLOPxXvsWbQUjWNNXQdvbt3NO5ua6AyEad7VTkfD9r06cYFkJ67Ll4/D5cGTX4Q7vwhPXj5enxuP35VMzvJ6XRT4XBT43FZyVvK1NXOW154xK5GQ5Uu8diYKrjmSBdeSCVmk78RNSJ0tC3rvxO1ttqyDXWTtUJtXWUzR35/hM1+/A9+QCh786QUU3n0Nz//pDZY2dHLB2CJOuedaXnRO5RcPrGDbmy9iuuKUTzuZE08Zy7dOHMv4IV4cK55i+9OL2fLiFla1hNgVtmbLKnA5GOFzU5nvoWRiCSVHVpA/vJSCiRNwV04hXjyScP5QdgfjNHXGqG0LU99hdeLWtgbpjMRpbY8Q6ogSDlqduJFwjGg4QjwcJB4JJjtxk5222ok7MBiDiUb63i9H5USjr5RS/ad/krOyRRt9pZTq6TD+xaSNvlJKpTLmsA6T5USjP2xkBZetv4+fX72I1mgX37pgEhPue4L5C99hxaKnMPE4R847lx9fPIu5nhq2/ObXvP3IGt5rCRGMGzue7+PIT4+mav7x+E+aT2vpJFbt6uC1zU28saGRhuoAoc4IHQ3VhFsbiXS0Eo8Ek+fg9PiT8XyXvwCny4PLV9Atnu+1k7J8fjcFPhfFeR4KvC68LocVy/c4k/F8a/IUawKVPbF9K5bvEOkWz3c62JOgRe/xfId0j+enJmtlI55/qEP/E//9Ksd9407E4eSBX17K5Md/wu9/8SIN4Tjzhxcy7/4f8kb5qVz353fYuHwJ8UiQ8qknc+JpR/KDuROZ7mig6/332fHYk2x8bgOrGjp7xPNdjC/wMHRaGUOnD6ds5gTcpWV4KifTVTqWaOEwmjpjNHbGqA6EqGsPs3N3kNrWEPWBMJFInFCnFc+PBGNp4/malDUw6egdpZQaLIzBxLXRV0qpQcEYQ1c0lu3TOGS00VdKqVQGvdPPtvJQIzdd+XfG53s46axxjL//Cc79w1u88/iTmHicKWecx88vOZozPNVsufUW3nj4A95pDhE3Vjz/6GIfk08bS9X848mbs4CW0km8X9fBKxsbeX1dAw3VAVpqdxHpbM0onu/JK8Lp9WcUz08UXEsdn58az08dn58Ym++Qgx/Pz3TSlFyI5wPMvuR2HG4Pj/7uCiY//CN+e9MLOAXOH3UEZz783ywrn8vV977N+qXPE48EqZgxh1PmTeaHp09kuuyi/Zm/0Lh6Ixv+uY5VDZ3sCEa7xfMnFXq7xfN9k6bjHFJOV1kl0cJhNHTGqO+IUh0IsbMtxM7dQaqbO6kPhOloixCLxjOK5ycnRNd4/oCijb5SSg0Sxhi6tJ6+UkoNHofz6B2dGF0ppVLZo3cyWQ6EiJSIyBIR2WA/DkmzX1xEVtrL0ynrx4nIW/b7H7EnUe+TNvpKKZUiMXonk+UAXQe8ZIyZCLxkv+5N0Bgzy14+m7L+l8Dt9vubgcsyOWhOhHdqdjRz7NASPr/4N+yq/DSn/3o5q//1JG5/ATPmn80d/3EUR7evYt3/3MbyZzawqjUEwKQCL2PyXBx5VhWV538az4mfoaGwkhXVbSzb2Mhb6xuorw4QqKujs2kn8UiISEdrrzNluXz5dnE1q8ia0+XA63Pjy3d3mymrOM9Ngc+d7MQtSMyc5XaSZ3fkJmbK6q0T1+nY/5myeuvYhf7vxO3PWmz+IcN46Y6Lcd10Obfe/Q4VXheXXn8GFRdexKPRifzkrjfY+u/FAIw45mzOPGMCV51axcTgZnYv+gvrH19B8+YW3msOJpOyitwORvvdjB/iY+jUMspmjKJs5ng8VdNwjJ5Ml7eQUP5QGjqi1HdE2d4aotbuxK1tDVLbEiLUESXUaXXkpiuyFosEMfG4duIOYF3905F7AXCa/fwvwCvAtZm8Uax/yPOA/0h5/4+BP/T1Xr3TV0qpVPaQzQzDO2UisiJluWI/jlRhjKkFsB/L0+znsz/7TRFZYK8rBVqMMYmfG9XAyEwOmhN3+kop1W/2LyO30RgzO91GEXkRGNbLphv344zGGGNqRKQKeFlEPgACvexnMvkwbfSVUiqF4eCN3jHGnJFum4jsEpHhxphaERkO1Kf5jBr7cbOIvAIcBTwOFIuIy77bHwXUZHJOOdHoD8lzc+GaZ/nGC428/eclbFn+NIXDx3PSgnn87sLpjFi9iHd/dT/LX69mfXsEv1OYfoSXGceNoPTIckaeOw/n0Wexw1HKW1tbeHldAx9u3k3jzgCBumqCzXWE25qTha/AiuenJmV58otw5xXhyS/E43fjdDrw5bvx+t14fC78vt7j+X6PE7fDit8n4vk+lxXTt2L7e+L53WL4GcTzEzH0TOL50iPgnsvxfICN913K+2efxQPLtnNCiZ8v3XUpW0/9Dvd9WMc9f32ZXR8sw5NfxJjZp3LRuZO4bPYoKra/Ts2jj7Bu0WpWb2+lORqnIRzHKTDU62S03824YfkMnVrG0JmVDJk6Hs+EmTBsPLHiUXTGDE3tMWrawuwMhNgZCFG9O0hda5DGQJhQZ5RQR4RwMEY83kU0HCMaCiXj+fGYlYy1p8hatFvcfl/x/HRxe43nHwLG0BXplzIMTwNfA26xH5/quYM9oqfTGBMWkTLgZOBXxhgjIkuBLwAPp3t/bzSmr5RSqQx0dXVltBygW4AzRWQDcKb9GhGZLSJ/sveZAqwQkVXAUuAWY8xH9rZrgatEZCNWjP/eTA6aE3f6SinVXwz9U2XTGNMEnN7L+hXA5fbzfwMz0rx/M3Dc/h5XG32llEplSIbZDkc50eh7J07ixDvXsebZRXTFIgw/6gyu+Mpsrj5hOJ0P3Myy3y1h2ZYWGsJxhnqdHDvEz8Rzqhg7fw6eyinEJ89hbWsXy7Y1snRtPVu3NLN7Vzvtu7YkC6z1nADd6fXj8vhx5x+B21eQnADdl+fB63fhsMfpe/0uCvPcFPpcFPk9FNpx/AKfi3yPC5/LmhTF67Lj+j3i+KkToCfG5juw4/d2XH9fY/OhR+G1lP9uBxLPH6ix/ITHRh3F601BLj5mOHMevp0HA6P42c0v07jpQ9pqN1E4fDyT55zIf517JAsmFsOyB9nwyD/Z8PxmVqZMgO5xCBVeF+Py3YyqKrbG588cT+GUKXiqphErGUs4r5SGjhjBqEkWWKtuDlLdHKQ+EKKlLZwcnx8NxQmHopguQzTUSTy8Z2z+viZMAauh0bH5A4HRMgyfhIj8WUTqRWRNyrqM0o6VUipr9m+cfs45lB259wPn9FiXadqxUkplhTGGeCSW0ZKLDlmjb4xZBuzusfoCrHRh7McFKKXUgGLs8FvfSy7q75h+t7RjEUmXdoydznwFwMhRo3tNaVNKqYNOZ87KDmPMQmAhgHvIGFP/9CMM+9RcRk8emSywtv67V/Pak+uTBdamFHo5akopE87/FEPPPpeuKafSGHexYnt7rwXWwm3NxCPBZKfYvgqsWTNj2bNk+dy4PI60BdYKfC57diyrwJpVVC19gbVEElay07aPhKzeOnDh8C6w1tPOYIwf3XQu3u/dxoWPrGb5U38jUL0ep8fPyGPP615g7Z5fs/7xFaxe08CmjgjtsS48DqHAJWkLrDlHTSI6ZAzNcRdNrdYMWa3hWNoCa+FgzErGCseIhjox8XjaAmuJpKzUDlzIrMCaduD2AwMmnlFFg5zU341+RmnHSimVLQbTX1U2s6K/M3ITacewH2nDSinVbwyYLpPRkosO2Z2+iDyEVSu6TESqgf/BSjP+h4hcBmwHvniojq+UUp+EMRCPHL5htEPW6Btjvpxm015px31+VjzGqZf9J3d+aSbj3J203Ptjnv/tUpbtaqc12sUwn4tjy/KYcO4Exnz2dFyzz6HWU8E7WwJsawmydG091dta2F3bTEfDdsKtjUSD7XtNluJwe3D78nH5C5KxfI/fb8fzXcnJUvL8bjwuB8V5nr2KqyUSslKLqznEiuNbMX0rlu+Q7glZB7O4Guw7lp/6nlS5EMtPuHrdk9y9I4/bfvAsNe8uxuUvoGrOBQyrLObqcyZz1ggn8aX38tEjS1j38jbWBMLUhawhdiUeq7jaUK+TiqpiymdUUDZzPPmTJuMZP4PYkFG0eYppDMatGH5biJpAiNbOKLWtIWpbggTshKxwKLonnm8XV+uyC6vFuxVX2zshSydLGaCM0Zi+UkoNJl3a6Cul1CChQzaVUmrwMEBXjnbSZkIbfaWUSmWMduRm28TKchbPi7H+2ktY9nYtr27aTUM4TonHydkV+Uw8vZKqBafiOfEzNBZWsqKmnWUbt/HW+gY6AmGaatvoaNhOqHlXt4qaPZOxHC6PNUOWXVHT63Pjy3fj8bvxeJ34/O5kMpbH5aDQuycZy+92kud2JjtwU5OxEh256SpqOh3pO3CBbutg7w7cbusO8w7chBm3bWLrvxcDMOKYs5PJWJVHuOG1v7P5tmfY+Owm3msOJitqFrkd3ZKx8ivyKZ02jiOmTsZdNZ2u0rF05A+loTNGfZM9M1ZgTzJWWyi2V0XNSDhGNBxJzo6VmoylHbi5yWhyllJKDSLa6Cul1GCiGblKKTV49FNGbibzi4jIXBFZmbKERGSBve1+EdmSsm1WJsfNiTt92b6Z35/wLda2hQEY5nNx/qgj9k7G2hlg6VubWLmxicaaAIG6GiKdrWmTsdz+AlwpyVhOrz9tMlahz0VRSjKWx+VIm4zldlqxfK+djOV0kNVkrFwurJbO9neWMvaEs/jcWRO58oQxjGp4n7q7r2HDxzvSJmNVledRPrWMsumjKZ0xAceQ8r2Tseo6k8lY1buD1LUG2dUSItQZJRaJp03Gitnx/EQylrVoLD8XGfptnH5ifpFbROQ6+/W13c7FmKXALLD+SAAbgRdSdrnGGPPY/hw0Jxp9pZTqN8bQ1T+jdy7AKlUD1vwir9Cj0e/hC8BzxpjOAzmohneUUiqFMdadfibLAeo2vwiQdn4R28XAQz3W3Swiq0XkdhHxZnJQvdNXSqke9mNWrDIRWZHyeqE9FwgAIvIi9DoH1I37cz52KfoZwOKU1dcDdYAHa+6Ra4Gf9vVZOdHoN7SGafXFuWBsESUTSxh//tEMOeN8wuNOYE1DkFfWNbF07WpqtrfQXNfSrahaIqYK4HB5cHr9aYuquTwOvD57ohS/mwKfa6+iagU+Fz6X0yqg5rRi+W5nYmx+96JqTofgwIrXOx3seS59x/Ghx1h9e126OH7Pbanv6SmTWP5AjOOneuxP13H6MCH20gNs+PZLvPmKFcdvj3URjBv8TqEyz82EAg/DJ5ZQPqOCkmnjKJg8Ffe4acRKxtDlLaQ2DE3BGNt3tbEzEKKmJUh1c5D6QGivompdsS4i4RjxSDA5Lr+vompAcsw+aBw/J5j9uotvNMbMTv9R5ox020Rkf+YX+RKwyBgTTfnsWvtpWETuA67O5IQ1vKOUUqnscfqZLAdof+YX+TI9Qjv2HwrEuvtbAKzJ5KA5caevlFL9xdBvBdd6nV9ERGYDVxpjLrdfVwKjgVd7vP9BERmK9aN+JXBlJgfVRl8ppVIZQzxy6Bt9Y0wTvcwvYoxZAVye8norMLKX/eZ9kuNqo6+UUimMgS6jZRiyalh5Adc+eCMy60yC/lJW7+pk2ZYmlr68gobqAM11TXTUbyfS3rxXEpY4nLj8Bbh9+bjzi3D7CnDnF+HLz0smX3n9bjw+F06Xg6I8N4U+N0V2Qpbf40x23iYKqrkdkkzAspKxZK/OW03COrTKr7mER96oZm1bhN2ROE6xkrBG+NwcWeihbFIJ5TOGUTZzAv5J03CNnUJ8yCjanAU0BmPU7Y7QGrY6b3c27+m8bWsLE+qMEgnG7GJqe5KwTFe8W+et1XGrSViHo7g2+kopNTgY4DCut6aNvlJK9aR3+kopNUh0GYjozFnZ1VE6kv/TOIuPFq6jMxDeZwzf6fHjyS/C5cvHk19kFVZLE8MvyHPbSVduiv3uZBG13mL4vh5JWE57YpS+YvjOlMlNNIZ/8Pz5Xxso8TgZn+9m3oQShk4rY+jMSvLKh+wVw98ajFHXFmHnlhA7AzXJQmptodg+Y/jJ+H0GhdTSxe01hp+bNLyjlFKDhMFoeEcppQYL7chVSqlBRhv9LNu2fRd/u/WubrFQp8ePy+vHP6SiW/E0r9+Lx29NaO71uXG6BF+a4ml+j5N8txOvy4rdOwW8LmdyQvOe4+8T8XqnHQzf14TmB1I8TWP3ffvFvZfimzQd56gjiQ0ZTavx0hiMszMSZ3trkNq6MDs/aqS6eTv1gTAdbRHCoSihjqgVtw/HiMdi3SY07238faK/SMffDx7G6OgdpZQaNAw6ekcppQYNjekrpdQgo+EdpZQaJKyYfrbP4tDJiUbf5S9g3Cnzrdmt3I49SVZeF8V5bgp6K5DmdOC1Z7iyEqocfXbQZlogLbVzFjS5KhuudJ5P/fthQst3E+qsIxyMEQlGice7iEWiyQ7amN1Ja+LxZAdtVyya7GTVDlrVG73TV0qpQcIA/TKFSpZoo6+UUikMRkfvKKXUYGGN3tFGP6umjSnm9V+ene3TUAPIY7f/IdunoA5Xh3lHrqPvXQ4+ETlHRNaJyEYRuS4b56CUUr1J3OlnshwIEfmiiHwoIl32ZOjp9uu1vRSRcSLylohsEJFHRMSTyXH7vdEXESdwJ3AuMBX4sohM7e/zUEqpdOIms+UArQEuBJal26GP9vKXwO3GmIlAM3BZJgfNxp3+ccBGY8xmY0wEeBi4IAvnoZRSe+nCKsOQyXIgjDFrjTHr+tit1/ZSrLHg84DH7P3+AizI5LjZiOmPBHakvK4Gju+5k4hcAVxhvwzn+f1r+uHc+ksZ0JjtkziIDrfrgcPvmgbT9Yw9kA9uJLL4HraVZbi7T0RWpLxeaIxZeCDH7yFde1kKtBhjYinrR2bygdlo9HtLKdrrT6b9H24hgIisMMakjXnlGr2ege9wuya9nswZY845WJ8lIi8Cw3rZdKMx5qlMPqKXdWYf6/uUjUa/Ghid8noUUJOF81BKqUPKGHPGAX5EuvayESgWEZd9t59xO5qNmP47wES759kDXAw8nYXzUEqpga7X9tIYY4ClwBfs/b4GZPLLof8bffuv0neBxcBa4B/GmA/7eNvBjJENBHo9A9/hdk16PQOMiHxORKqBE4F/ichie/0IEXkW+mwvrwWuEpGNWDH+ezM6rjmMM8+UUkp1l5XkLKWUUtmhjb5SSg0iA7rRz9VyDSLyZxGpF5E1KetKRGSJnTK9RESG2OtFRH5nX+NqETk6e2feOxEZLSJLRWStnTb+PXt9Tl6TiPhE5G0RWWVfz0/s9b2mtYuI13690d5emc3zT0dEnCLyvoj8036d69ezVUQ+EJGVibHwufqdG0gGbKOf4+Ua7gd6jvW9DnjJTpl+yX4N1vVNtJcrgIFYSSwG/MAYMwU4AfiO/f8iV68pDMwzxnwKmAWcIyInkD6t/TKg2RgzAbjd3m8g+h5WZ19Crl8PwFxjzKyUMfm5+p0bOIwxA3LB6tFenPL6euD6bJ/Xfpx/JbAm5fU6YLj9fDiwzn5+D/Dl3vYbqAvW0LAzD4drAvKA97CyHBsBl70++f3DGjlxov3cZe8n2T73HtcxCqsRnAf8Eyt5J2evxz63rUBZj3U5/53L9jJg7/TpPf04ozTjAarCGFMLYD+W2+tz6jrtUMBRwFvk8DXZoZCVQD2wBNhE+rT25PXY21uxhsgNJHcAP2TPpE/7StPPhesBK8P0BRF51y7LAjn8nRsoBnI9/U+cZpxjcuY6RaQAeBz4vjEmIOkn6R3w12SMiQOzRKQYWARM6W03+3FAX4+IzAfqjTHvishpidW97JoT15PiZGNMjYiUA0tE5ON97Jsr15R1A/lO/3Ar17BLRIYD2I/19vqcuE4RcWM1+A8aY56wV+f0NQEYY1qAV7D6KopFJHEjlHrOyeuxtxcBu/v3TPfpZOCzIrIVqwrjPKw7/1y9HgCMMTX2Yz3WH+bjOAy+c9k2kBv9w61cw9NYqdLQPWX6aeBSe/TBCUBr4ufrQCHWLf29wFrUyuu3AAACqElEQVRjzG9SNuXkNYnIUPsOHxHxA2dgdYCmS2tPvc4vAC8bO3A8EBhjrjfGjDLGVGL9O3nZGPMVcvR6AEQkX0QKE8+Bs7Dqz+fkd25AyXanwr4W4DxgPVa89cZsn89+nPdDQC0QxboDuQwrZvoSsMF+LLH3FaxRSpuAD4DZ2T7/Xq7nFKyfyquBlfZyXq5eEzATeN++njXAj+z1VcDbwEbgUcBrr/fZrzfa26uyfQ37uLbTgH/m+vXY577KXj5M/PvP1e/cQFq0DINSSg0iAzm8o5RS6iDTRl8ppQYRbfSVUmoQ0UZfKaUGEW30lVJqENFGX2WdiMTtSoof2pUvrxKRT/zdFJEbUp5XSkq1U6UGO2301UAQNFYlxWlYhdzOA/7nAD7vhr53UWpw0kZfDSjGSrm/AviunV3pFJFbReQdu076twBE5DQRWSYii0TkIxG5W0QcInIL4Ld/OTxof6xTRP5o/5J4wc7CVWpQ0kZfDTjGmM1Y381yrGzmVmPMscCxwDdFZJy963HAD4AZwHjgQmPMdez55fAVe7+JwJ32L4kW4PP9dzVKDSza6KuBKlE18Sysmiorsco5l2I14gBvG2M2G6ti5kNY5SJ6s8UYs9J+/i7WXAdKDUoDubSyGqREpAqIY1VQFOC/jDGLe+xzGnuXzk1XUySc8jwOaHhHDVp6p68GFBEZCtwN/N5YhaEWA9+2SzsjIpPsqosAx9lVWB3ARcBye300sb9Sqju901cDgd8O37ix5uP9K5Ao4fwnrHDMe3aJ5wZggb3tDeAWrJj+Mqya6wALgdUi8h5wY39cgFK5Qqtsqpxkh3euNsbMz/a5KJVLNLyjlFKDiN7pK6XUIKJ3+kopNYhoo6+UUoOINvpKKTWIaKOvlFKDiDb6Sik1iPx/LFDs+ztFgooAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pos_encoding = positional_encoding(50, 512)\n",
    "print (pos_encoding.shape)\n",
    "\n",
    "plt.pcolormesh(pos_encoding[0], cmap='RdBu')\n",
    "plt.xlabel('Depth')\n",
    "plt.xlim((0, 512))\n",
    "plt.ylabel('Position')\n",
    "plt.colorbar()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a_b4ou4TYqUN"
   },
   "source": [
    "## Masking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "s42Uydjkv0hF"
   },
   "source": [
    "Mask all the pad tokens in the batch of sequence. It ensures that the model does not treat padding as the input. The mask indicates where pad value `0` is present: it outputs a `1` at those locations, and a `0` otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "U2i8-e1s8ti9"
   },
   "outputs": [],
   "source": [
    "def create_padding_mask(seq):\n",
    "    seq = tf.cast(tf.math.equal(seq, 0), tf.float32)\n",
    "  \n",
    "    # add extra dimensions so that we can add the padding\n",
    "    # to the attention logits.\n",
    "    return seq[:, tf.newaxis, tf.newaxis, :]  # (batch_size, 1, 1, seq_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "A7BYeBCNvi7n"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=311427, shape=(3, 1, 1, 5), dtype=float32, numpy=\n",
       "array([[[[0., 0., 1., 1., 0.]]],\n",
       "\n",
       "\n",
       "       [[[0., 0., 0., 1., 1.]]],\n",
       "\n",
       "\n",
       "       [[[1., 1., 1., 0., 0.]]]], dtype=float32)>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = tf.constant([[7, 6, 0, 0, 1], [1, 2, 3, 0, 0], [0, 0, 0, 4, 5]])\n",
    "create_padding_mask(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Z0hzukDBgVom"
   },
   "source": [
    "The look-ahead mask is used to mask the future tokens in a sequence. In other words, the mask indicates which entries should not be used.\n",
    "\n",
    "This means that to predict the third word, only the first and second word will be used. Similarly to predict the fourth word, only the first, second and the third word will be used and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dVxS8OPI9uI0"
   },
   "outputs": [],
   "source": [
    "def create_look_ahead_mask(size):\n",
    "    mask = 1 - tf.linalg.band_part(tf.ones((size, size)), -1, 0)\n",
    "    return mask  # (seq_len, seq_len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yxKGuXxaBeeE"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=311443, shape=(3, 3), dtype=float32, numpy=\n",
       "array([[0., 1., 1.],\n",
       "       [0., 0., 1.],\n",
       "       [0., 0., 0.]], dtype=float32)>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = tf.random.uniform((1, 3))\n",
    "temp = create_look_ahead_mask(x.shape[1])\n",
    "temp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xluDl5cXYy4y"
   },
   "source": [
    "## Scaled dot product attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vsxEE_-Wa1gF"
   },
   "source": [
    "<img src=\"https://www.tensorflow.org/images/tutorials/transformer/scaled_attention.png\" width=\"500\" alt=\"scaled_dot_product_attention\">\n",
    "\n",
    "The attention function used by the transformer takes three inputs: Q (query), K (key), V (value). The equation used to calculate the attention weights is:\n",
    "\n",
    "$$\\Large{Attention(Q, K, V) = softmax_k(\\frac{QK^T}{\\sqrt{d_k}}) V} $$\n",
    "\n",
    "The dot-product attention is scaled by a factor of square root of the depth. This is done because for large values of depth, the dot product grows large in magnitude pushing the softmax function where it has small gradients resulting in a very hard softmax. \n",
    "\n",
    "For example, consider that `Q` and `K` have a mean of 0 and variance of 1. Their matrix multiplication will have a mean of 0 and variance of `dk`. Hence, *square root of `dk`* is used for scaling (and not any other number) because the matmul of `Q` and `K` should have a mean of 0 and variance of 1, so that we get a gentler softmax.\n",
    "\n",
    "The mask is multiplied with -1e9 (close to negative infinity). This is done because the mask is summed with the scaled matrix multiplication of Q and K and is applied immediately before a softmax. The goal is to zero out these cells, and large negative inputs to softmax are near zero in the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LazzUq3bJ5SH"
   },
   "outputs": [],
   "source": [
    "def scaled_dot_product_attention(q, k, v, mask):\n",
    "    \"\"\"Calculate the attention weights.\n",
    "    q, k, v must have matching leading dimensions.\n",
    "    k, v must have matching penultimate dimension, i.e.: seq_len_k = seq_len_v.\n",
    "    The mask has different shapes depending on its type(padding or look ahead) \n",
    "    but it must be broadcastable for addition.\n",
    "  \n",
    "    Args:\n",
    "      q: query shape == (..., seq_len_q, depth)\n",
    "      k: key shape == (..., seq_len_k, depth)\n",
    "      v: value shape == (..., seq_len_v, depth_v)\n",
    "      mask: Float tensor with shape broadcastable \n",
    "            to (..., seq_len_q, seq_len_k). Defaults to None.\n",
    "    \n",
    "    Returns:\n",
    "      output, attention_weights\n",
    "    \"\"\"\n",
    "\n",
    "    matmul_qk = tf.matmul(q, k, transpose_b=True)  # (..., seq_len_q, seq_len_k)\n",
    "  \n",
    "    # scale matmul_qk\n",
    "    dk = tf.cast(tf.shape(k)[-1], tf.float32)\n",
    "    scaled_attention_logits = matmul_qk / tf.math.sqrt(dk)\n",
    "\n",
    "    # add the mask to the scaled tensor.\n",
    "    if mask is not None:\n",
    "        scaled_attention_logits += (mask * -1e9)  \n",
    "\n",
    "    # softmax is normalized on the last axis (seq_len_k) so that the scores\n",
    "    # add up to 1.\n",
    "    attention_weights = tf.nn.softmax(scaled_attention_logits, axis=-1)  # (..., seq_len_q, seq_len_k)\n",
    "\n",
    "    output = tf.matmul(attention_weights, v)  # (..., seq_len_q, depth_v)\n",
    "\n",
    "    return output, attention_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FiqETnhCkoXh"
   },
   "source": [
    "As the softmax normalization is done on K, its values decide the amount of importance given to Q.\n",
    "\n",
    "The output represents the multiplication of the attention weights and the V (value) vector. This ensures that the words we want to focus on are kept as is and the irrelevant words are flushed out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n90YjClyInFy"
   },
   "outputs": [],
   "source": [
    "def print_out(q, k, v):\n",
    "    temp_out, temp_attn = scaled_dot_product_attention(\n",
    "        q, k, v, None)\n",
    "    print ('Attention weights are:')\n",
    "    print (temp_attn)\n",
    "    print ('Output is:')\n",
    "    print (temp_out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yAzUAf2DPlNt"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are:\n",
      "tf.Tensor([[0. 1. 0. 0.]], shape=(1, 4), dtype=float32)\n",
      "Output is:\n",
      "tf.Tensor([[10.  0.]], shape=(1, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "np.set_printoptions(suppress=True)\n",
    "\n",
    "temp_k = tf.constant([[10,0,0],\n",
    "                      [0,10,0],\n",
    "                      [0,0,10],\n",
    "                      [0,0,10]], dtype=tf.float32)  # (4, 3)\n",
    "\n",
    "temp_v = tf.constant([[   1,0],\n",
    "                      [  10,0],\n",
    "                      [ 100,5],\n",
    "                      [1000,6]], dtype=tf.float32)  # (4, 2)\n",
    "\n",
    "# This `query` aligns with the second `key`,\n",
    "# so the second `value` is returned.\n",
    "temp_q = tf.constant([[0, 10, 0]], dtype=tf.float32)  # (1, 3)\n",
    "print_out(temp_q, temp_k, temp_v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zg6k-fGhgXra"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are:\n",
      "tf.Tensor([[0.  0.  0.5 0.5]], shape=(1, 4), dtype=float32)\n",
      "Output is:\n",
      "tf.Tensor([[550.    5.5]], shape=(1, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# This query aligns with a repeated key (third and fourth), \n",
    "# so all associated values get averaged.\n",
    "temp_q = tf.constant([[0, 0, 10]], dtype=tf.float32)  # (1, 3)\n",
    "print_out(temp_q, temp_k, temp_v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UAq3YOzUgXhb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are:\n",
      "tf.Tensor([[0.5 0.5 0.  0. ]], shape=(1, 4), dtype=float32)\n",
      "Output is:\n",
      "tf.Tensor([[5.5 0. ]], shape=(1, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# This query aligns equally with the first and second key, \n",
    "# so their values get averaged.\n",
    "temp_q = tf.constant([[10, 10, 0]], dtype=tf.float32)  # (1, 3)\n",
    "print_out(temp_q, temp_k, temp_v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aOz-4_XIhaTP"
   },
   "source": [
    "Pass all the queries together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6dlU8Tm-hYrF"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are:\n",
      "tf.Tensor(\n",
      "[[0.  0.  0.5 0.5]\n",
      " [0.  1.  0.  0. ]\n",
      " [0.5 0.5 0.  0. ]], shape=(3, 4), dtype=float32)\n",
      "Output is:\n",
      "tf.Tensor(\n",
      "[[550.    5.5]\n",
      " [ 10.    0. ]\n",
      " [  5.5   0. ]], shape=(3, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "temp_q = tf.constant([[0, 0, 10], [0, 10, 0], [10, 10, 0]], dtype=tf.float32)  # (3, 3)\n",
    "print_out(temp_q, temp_k, temp_v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kmzGPEy64qmA"
   },
   "source": [
    "## Multi-head attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fz5BMC8Kaoqo"
   },
   "source": [
    "<img src=\"https://www.tensorflow.org/images/tutorials/transformer/multi_head_attention.png\" width=\"500\" alt=\"multi-head attention\">\n",
    "\n",
    "\n",
    "Multi-head attention consists of four parts:\n",
    "*    Linear layers and split into heads.\n",
    "*    Scaled dot-product attention.\n",
    "*    Concatenation of heads.\n",
    "*    Final linear layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JPmbr6F1C-v_"
   },
   "source": [
    "Each multi-head attention block gets three inputs; Q (query), K (key), V (value). These are put through linear (Dense) layers and split up into multiple heads. \n",
    "\n",
    "The `scaled_dot_product_attention` defined above is applied to each head (broadcasted for efficiency). An appropriate mask must be used in the attention step.  The attention output for each head is then concatenated (using `tf.transpose`, and `tf.reshape`) and put through a final `Dense` layer.\n",
    "\n",
    "Instead of one single attention head, Q, K, and V are split into multiple heads because it allows the model to jointly attend to information at different positions from different representational spaces. After the split each head has a reduced dimensionality, so the total computation cost is the same as a single head attention with full dimensionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BSV3PPKsYecw"
   },
   "outputs": [],
   "source": [
    "class MultiHeadAttention(tf.keras.layers.Layer):\n",
    "    def __init__(self, d_model, num_heads):\n",
    "        super(MultiHeadAttention, self).__init__()\n",
    "        self.num_heads = num_heads\n",
    "        self.d_model = d_model\n",
    "    \n",
    "        assert d_model % self.num_heads == 0\n",
    "    \n",
    "        self.depth = d_model // self.num_heads\n",
    "    \n",
    "        self.wq = tf.keras.layers.Dense(d_model)\n",
    "        self.wk = tf.keras.layers.Dense(d_model)\n",
    "        self.wv = tf.keras.layers.Dense(d_model)\n",
    "    \n",
    "        self.dense = tf.keras.layers.Dense(d_model)\n",
    "        \n",
    "    def split_heads(self, x, batch_size):\n",
    "        \"\"\"Split the last dimension into (num_heads, depth).\n",
    "        Transpose the result such that the shape is (batch_size, num_heads, seq_len, depth)\n",
    "        \"\"\"\n",
    "        x = tf.reshape(x, (batch_size, -1, self.num_heads, self.depth))\n",
    "        return tf.transpose(x, perm=[0, 2, 1, 3])\n",
    "    \n",
    "    def call(self, v, k, q, mask):\n",
    "        batch_size = tf.shape(q)[0]\n",
    "    \n",
    "        q = self.wq(q)  # (batch_size, seq_len, d_model)\n",
    "        k = self.wk(k)  # (batch_size, seq_len, d_model)\n",
    "        v = self.wv(v)  # (batch_size, seq_len, d_model)\n",
    "    \n",
    "        q = self.split_heads(q, batch_size)  # (batch_size, num_heads, seq_len_q, depth)\n",
    "        k = self.split_heads(k, batch_size)  # (batch_size, num_heads, seq_len_k, depth)\n",
    "        v = self.split_heads(v, batch_size)  # (batch_size, num_heads, seq_len_v, depth)\n",
    "    \n",
    "        # scaled_attention.shape == (batch_size, num_heads, seq_len_q, depth)\n",
    "        # attention_weights.shape == (batch_size, num_heads, seq_len_q, seq_len_k)\n",
    "        scaled_attention, attention_weights = scaled_dot_product_attention(\n",
    "            q, k, v, mask)\n",
    "    \n",
    "        scaled_attention = tf.transpose(scaled_attention, perm=[0, 2, 1, 3])  # (batch_size, seq_len_q, num_heads, depth)\n",
    "\n",
    "        concat_attention = tf.reshape(scaled_attention, \n",
    "                                      (batch_size, -1, self.d_model))  # (batch_size, seq_len_q, d_model)\n",
    "\n",
    "        output = self.dense(concat_attention)  # (batch_size, seq_len_q, d_model)\n",
    "        \n",
    "        return output, attention_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0D8FJue5lDyZ"
   },
   "source": [
    "Create a `MultiHeadAttention` layer to try out. At each location in the sequence, `y`, the `MultiHeadAttention` runs all 8 attention heads across all other locations in the sequence, returning a new vector of the same length at each location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Hu94p-_-2_BX"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([1, 60, 512]), TensorShape([1, 8, 60, 60]))"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "temp_mha = MultiHeadAttention(d_model=512, num_heads=8)\n",
    "y = tf.random.uniform((1, 60, 512))  # (batch_size, encoder_sequence, d_model)\n",
    "out, attn = temp_mha(y, k=y, q=y, mask=None)\n",
    "out.shape, attn.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RdDqGayx67vv"
   },
   "source": [
    "## Point wise feed forward network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gBqzJXGfHK3X"
   },
   "source": [
    "Point wise feed forward network consists of two fully-connected layers with a ReLU activation in between."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ET7xLt0yCT6Z"
   },
   "outputs": [],
   "source": [
    "def point_wise_feed_forward_network(d_model, dff):\n",
    "    return tf.keras.Sequential([\n",
    "        tf.keras.layers.Dense(dff, activation='relu'),  # (batch_size, seq_len, dff)\n",
    "        tf.keras.layers.Dense(d_model)  # (batch_size, seq_len, d_model)\n",
    "    ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mytb1lPyOHLB"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 50, 512])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_ffn = point_wise_feed_forward_network(512, 2048)\n",
    "sample_ffn(tf.random.uniform((64, 50, 512))).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7e7hKcxn6-zd"
   },
   "source": [
    "## Encoder and decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yScbC0MUH8dS"
   },
   "source": [
    "<img src=\"https://www.tensorflow.org/images/tutorials/transformer/transformer.png\" width=\"600\" alt=\"transformer\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MfYJG-Kvgwy2"
   },
   "source": [
    "The transformer model follows the same general pattern as a standard [sequence to sequence with attention model](nmt_with_attention.ipynb). \n",
    "\n",
    "* The input sentence is passed through `N` encoder layers that generates an output for each word/token in the sequence.\n",
    "* The decoder attends on the encoder's output and its own input (self-attention) to predict the next word. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QFv-FNYUmvpn"
   },
   "source": [
    "### Encoder layer\n",
    "\n",
    "Each encoder layer consists of sublayers:\n",
    "\n",
    "1.   Multi-head attention (with padding mask) \n",
    "2.    Point wise feed forward networks. \n",
    "\n",
    "Each of these sublayers has a residual connection around it followed by a layer normalization. Residual connections help in avoiding the vanishing gradient problem in deep networks.\n",
    "\n",
    "The output of each sublayer is `LayerNorm(x + Sublayer(x))`. The normalization is done on the `d_model` (last) axis. There are N encoder layers in the transformer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ncyS-Ms3i2x_"
   },
   "outputs": [],
   "source": [
    "class EncoderLayer(tf.keras.layers.Layer):\n",
    "    def __init__(self, d_model, num_heads, dff, rate=0.1):\n",
    "        super(EncoderLayer, self).__init__()\n",
    "\n",
    "        self.mha = MultiHeadAttention(d_model, num_heads)\n",
    "        self.ffn = point_wise_feed_forward_network(d_model, dff)\n",
    "\n",
    "        self.layernorm1 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
    "        self.layernorm2 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
    "    \n",
    "        self.dropout1 = tf.keras.layers.Dropout(rate)\n",
    "        self.dropout2 = tf.keras.layers.Dropout(rate)\n",
    "    \n",
    "    def call(self, x, training, mask):\n",
    "\n",
    "        attn_output, _ = self.mha(x, x, x, mask)  # (batch_size, input_seq_len, d_model)\n",
    "        attn_output = self.dropout1(attn_output, training=training)\n",
    "        out1 = self.layernorm1(x + attn_output)  # (batch_size, input_seq_len, d_model)\n",
    "    \n",
    "        ffn_output = self.ffn(out1)  # (batch_size, input_seq_len, d_model)\n",
    "        ffn_output = self.dropout2(ffn_output, training=training)\n",
    "        out2 = self.layernorm2(out1 + ffn_output)  # (batch_size, input_seq_len, d_model)\n",
    "    \n",
    "        return out2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AzZRXdO0mI48"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 43, 512])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_encoder_layer = EncoderLayer(512, 8, 2048)\n",
    "\n",
    "sample_encoder_layer_output = sample_encoder_layer(\n",
    "    tf.random.uniform((64, 43, 512)), False, None)\n",
    "\n",
    "sample_encoder_layer_output.shape  # (batch_size, input_seq_len, d_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6LO_48Owmx_o"
   },
   "source": [
    "### Decoder layer\n",
    "\n",
    "Each decoder layer consists of sublayers:\n",
    "\n",
    "1.   Masked multi-head attention (with look ahead mask and padding mask)\n",
    "2.   Multi-head attention (with padding mask). V (value) and K (key) receive the *encoder output* as inputs. Q (query) receives the *output from the masked multi-head attention sublayer.*\n",
    "3.   Point wise feed forward networks\n",
    "\n",
    "Each of these sublayers has a residual connection around it followed by a layer normalization. The output of each sublayer is `LayerNorm(x + Sublayer(x))`. The normalization is done on the `d_model` (last) axis.\n",
    "\n",
    "There are N decoder layers in the transformer.\n",
    "\n",
    "As Q receives the output from decoder's first attention block, and K receives the encoder output, the attention weights represent the importance given to the decoder's input based on the encoder's output. In other words, the decoder predicts the next word by looking at the encoder output and self-attending to its own output. See the demonstration above in the scaled dot product attention section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9SoX0-vd1hue"
   },
   "outputs": [],
   "source": [
    "class DecoderLayer(tf.keras.layers.Layer):\n",
    "    def __init__(self, d_model, num_heads, dff, rate=0.1):\n",
    "        super(DecoderLayer, self).__init__()\n",
    "\n",
    "        self.mha1 = MultiHeadAttention(d_model, num_heads)\n",
    "        self.mha2 = MultiHeadAttention(d_model, num_heads)\n",
    "\n",
    "        self.ffn = point_wise_feed_forward_network(d_model, dff)\n",
    " \n",
    "        self.layernorm1 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
    "        self.layernorm2 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
    "        self.layernorm3 = tf.keras.layers.LayerNormalization(epsilon=1e-6)\n",
    "    \n",
    "        self.dropout1 = tf.keras.layers.Dropout(rate)\n",
    "        self.dropout2 = tf.keras.layers.Dropout(rate)\n",
    "        self.dropout3 = tf.keras.layers.Dropout(rate)\n",
    "    \n",
    "    \n",
    "    def call(self, x, enc_output, training, \n",
    "             look_ahead_mask, padding_mask):\n",
    "        # enc_output.shape == (batch_size, input_seq_len, d_model)\n",
    "\n",
    "        attn1, attn_weights_block1 = self.mha1(x, x, x, look_ahead_mask)  # (batch_size, target_seq_len, d_model)\n",
    "        attn1 = self.dropout1(attn1, training=training)\n",
    "        out1 = self.layernorm1(attn1 + x)\n",
    "    \n",
    "        attn2, attn_weights_block2 = self.mha2(\n",
    "            enc_output, enc_output, out1, padding_mask)  # (batch_size, target_seq_len, d_model)\n",
    "        attn2 = self.dropout2(attn2, training=training)\n",
    "        out2 = self.layernorm2(attn2 + out1)  # (batch_size, target_seq_len, d_model)\n",
    "    \n",
    "        ffn_output = self.ffn(out2)  # (batch_size, target_seq_len, d_model)\n",
    "        ffn_output = self.dropout3(ffn_output, training=training)\n",
    "        out3 = self.layernorm3(ffn_output + out2)  # (batch_size, target_seq_len, d_model)\n",
    "    \n",
    "        return out3, attn_weights_block1, attn_weights_block2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ne2Bqx8k71l0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 50, 512])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_decoder_layer = DecoderLayer(512, 8, 2048)\n",
    "\n",
    "sample_decoder_layer_output, _, _ = sample_decoder_layer(\n",
    "    tf.random.uniform((64, 50, 512)), sample_encoder_layer_output, \n",
    "    False, None, None)\n",
    "\n",
    "sample_decoder_layer_output.shape  # (batch_size, target_seq_len, d_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SE1H51Ajm0q1"
   },
   "source": [
    "### Encoder\n",
    "\n",
    "The `Encoder` consists of:\n",
    "1.   Input Embedding\n",
    "2.   Positional Encoding\n",
    "3.   N encoder layers\n",
    "\n",
    "The input is put through an embedding which is summed with the positional encoding. The output of this summation is the input to the encoder layers. The output of the encoder is the input to the decoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jpEox7gJ8FCI"
   },
   "outputs": [],
   "source": [
    "class Encoder(tf.keras.layers.Layer):\n",
    "    def __init__(self, num_layers, d_model, num_heads, dff, input_vocab_size, \n",
    "                 rate=0.1):\n",
    "        super(Encoder, self).__init__()\n",
    "\n",
    "        self.d_model = d_model\n",
    "        self.num_layers = num_layers\n",
    "    \n",
    "        self.embedding = tf.keras.layers.Embedding(input_vocab_size, d_model)\n",
    "        self.pos_encoding = positional_encoding(input_vocab_size, self.d_model)\n",
    "    \n",
    "    \n",
    "        self.enc_layers = [EncoderLayer(d_model, num_heads, dff, rate) \n",
    "                           for _ in range(num_layers)]\n",
    "  \n",
    "        self.dropout = tf.keras.layers.Dropout(rate)\n",
    "        \n",
    "    def call(self, x, training, mask):\n",
    "\n",
    "        seq_len = tf.shape(x)[1]\n",
    "    \n",
    "        # adding embedding and position encoding.\n",
    "        x = self.embedding(x)  # (batch_size, input_seq_len, d_model)\n",
    "        x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))\n",
    "        x += self.pos_encoding[:, :seq_len, :]\n",
    "\n",
    "        x = self.dropout(x, training=training)\n",
    "    \n",
    "        for i in range(self.num_layers):\n",
    "            x = self.enc_layers[i](x, training, mask)\n",
    "    \n",
    "        return x  # (batch_size, input_seq_len, d_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8QG9nueFQKXx"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(64, 62, 512)\n"
     ]
    }
   ],
   "source": [
    "sample_encoder = Encoder(num_layers=2, d_model=512, num_heads=8, \n",
    "                         dff=2048, input_vocab_size=8500)\n",
    "\n",
    "sample_encoder_output = sample_encoder(tf.random.uniform((64, 62)), \n",
    "                                       training=False, mask=None)\n",
    "\n",
    "print (sample_encoder_output.shape)  # (batch_size, input_seq_len, d_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "p-uO6ls8m2O5"
   },
   "source": [
    "### Decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZtT7PKzrXkNr"
   },
   "source": [
    " The `Decoder` consists of:\n",
    "1.   Output Embedding\n",
    "2.   Positional Encoding\n",
    "3.   N decoder layers\n",
    "\n",
    "The target is put through an embedding which is summed with the positional encoding. The output of this summation is the input to the decoder layers. The output of the decoder is the input to the final linear layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "d5_d5-PLQXwY"
   },
   "outputs": [],
   "source": [
    "class Decoder(tf.keras.layers.Layer):\n",
    "    def __init__(self, num_layers, d_model, num_heads, dff, target_vocab_size, \n",
    "                 rate=0.1):\n",
    "        super(Decoder, self).__init__()\n",
    "\n",
    "        self.d_model = d_model\n",
    "        self.num_layers = num_layers\n",
    "    \n",
    "        self.embedding = tf.keras.layers.Embedding(target_vocab_size, d_model)\n",
    "        self.pos_encoding = positional_encoding(target_vocab_size, self.d_model)\n",
    "    \n",
    "        self.dec_layers = [DecoderLayer(d_model, num_heads, dff, rate) \n",
    "                           for _ in range(num_layers)]\n",
    "        self.dropout = tf.keras.layers.Dropout(rate)\n",
    "    \n",
    "    def call(self, x, enc_output, training, \n",
    "             look_ahead_mask, padding_mask):\n",
    "\n",
    "        seq_len = tf.shape(x)[1]\n",
    "        attention_weights = {}\n",
    "    \n",
    "        x = self.embedding(x)  # (batch_size, target_seq_len, d_model)\n",
    "        x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))\n",
    "        x += self.pos_encoding[:, :seq_len, :]\n",
    "    \n",
    "        x = self.dropout(x, training=training)\n",
    "\n",
    "        for i in range(self.num_layers):\n",
    "            x, block1, block2 = self.dec_layers[i](x, enc_output, training,\n",
    "                                                   look_ahead_mask, padding_mask)\n",
    "      \n",
    "            attention_weights['decoder_layer{}_block1'.format(i+1)] = block1\n",
    "            attention_weights['decoder_layer{}_block2'.format(i+1)] = block2\n",
    "    \n",
    "        # x.shape == (batch_size, target_seq_len, d_model)\n",
    "        return x, attention_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "a1jXoAMRZyvu"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([64, 26, 512]), TensorShape([64, 8, 26, 62]))"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_decoder = Decoder(num_layers=2, d_model=512, num_heads=8, \n",
    "                         dff=2048, target_vocab_size=8000)\n",
    "\n",
    "output, attn = sample_decoder(tf.random.uniform((64, 26)), \n",
    "                              enc_output=sample_encoder_output, \n",
    "                              training=False, look_ahead_mask=None, \n",
    "                              padding_mask=None)\n",
    "\n",
    "output.shape, attn['decoder_layer2_block2'].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "y54xnJnuYgJ7"
   },
   "source": [
    "## Create the Transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uERO1y54cOKq"
   },
   "source": [
    "Transformer consists of the encoder, decoder and a final linear layer. The output of the decoder is the input to the linear layer and its output is returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PED3bIpOYkBu"
   },
   "outputs": [],
   "source": [
    "class Transformer(tf.keras.Model):\n",
    "    def __init__(self, num_layers, d_model, num_heads, dff, input_vocab_size, \n",
    "                 target_vocab_size, rate=0.1):\n",
    "        super(Transformer, self).__init__()\n",
    "\n",
    "        self.encoder = Encoder(num_layers, d_model, num_heads, dff, \n",
    "                               input_vocab_size, rate)\n",
    "\n",
    "        self.decoder = Decoder(num_layers, d_model, num_heads, dff, \n",
    "                               target_vocab_size, rate)\n",
    "\n",
    "        self.final_layer = tf.keras.layers.Dense(target_vocab_size)\n",
    "    \n",
    "    def call(self, inp, tar, training, enc_padding_mask, \n",
    "             look_ahead_mask, dec_padding_mask):\n",
    "\n",
    "        enc_output = self.encoder(inp, training, enc_padding_mask)  # (batch_size, inp_seq_len, d_model)\n",
    "    \n",
    "        # dec_output.shape == (batch_size, tar_seq_len, d_model)\n",
    "        dec_output, attention_weights = self.decoder(\n",
    "            tar, enc_output, training, look_ahead_mask, dec_padding_mask)\n",
    "    \n",
    "        final_output = self.final_layer(dec_output)  # (batch_size, tar_seq_len, target_vocab_size)\n",
    "    \n",
    "        return final_output, attention_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tJ4fbQcIkHW1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([64, 26, 8000])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_transformer = Transformer(\n",
    "    num_layers=2, d_model=512, num_heads=8, dff=2048, \n",
    "    input_vocab_size=8500, target_vocab_size=8000)\n",
    "\n",
    "temp_input = tf.random.uniform((64, 62))\n",
    "temp_target = tf.random.uniform((64, 26))\n",
    "\n",
    "fn_out, _ = sample_transformer(temp_input, temp_target, training=False, \n",
    "                               enc_padding_mask=None, \n",
    "                               look_ahead_mask=None,\n",
    "                               dec_padding_mask=None)\n",
    "\n",
    "fn_out.shape  # (batch_size, tar_seq_len, target_vocab_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wsINyf1VEQLC"
   },
   "source": [
    "## Set hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zVjWCxFNcgbt"
   },
   "source": [
    "To keep this example small and relatively fast, the values for *num_layers, d_model, and dff* have been reduced. \n",
    "\n",
    "The values used in the base model of transformer were; *num_layers=6*, *d_model = 512*, *dff = 2048*. See the [paper](https://arxiv.org/abs/1706.03762) for all the other versions of the transformer.\n",
    "\n",
    "Note: By changing the values below, you can get the model that achieved state of the art on many tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lnJn5SLA2ahP"
   },
   "outputs": [],
   "source": [
    "num_layers = 4\n",
    "d_model = 128\n",
    "dff = 512\n",
    "num_heads = 8\n",
    "\n",
    "input_vocab_size = tokenizer_pt.vocab_size + 2\n",
    "target_vocab_size = tokenizer_en.vocab_size + 2\n",
    "dropout_rate = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xYEGhEOtzn5W"
   },
   "source": [
    "## Optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GOmWW--yP3zx"
   },
   "source": [
    "Use the Adam optimizer with a custom learning rate scheduler according to the formula in the [paper](https://arxiv.org/abs/1706.03762).\n",
    "\n",
    "$$\\Large{lrate = d_{model}^{-0.5} * min(step{\\_}num^{-0.5}, step{\\_}num * warmup{\\_}steps^{-1.5})}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iYQdOO1axwEI"
   },
   "outputs": [],
   "source": [
    "class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n",
    "    def __init__(self, d_model, warmup_steps=4000):\n",
    "        super(CustomSchedule, self).__init__()\n",
    "    \n",
    "        self.d_model = d_model\n",
    "        self.d_model = tf.cast(self.d_model, tf.float32)\n",
    "\n",
    "        self.warmup_steps = warmup_steps\n",
    "    \n",
    "    def __call__(self, step):\n",
    "        arg1 = tf.math.rsqrt(step)\n",
    "        arg2 = step * (self.warmup_steps ** -1.5)\n",
    "    \n",
    "        return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7r4scdulztRx"
   },
   "outputs": [],
   "source": [
    "learning_rate = CustomSchedule(d_model)\n",
    "\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate, beta_1=0.9, beta_2=0.98, \n",
    "                                     epsilon=1e-9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "f33ZCgvHpPdG"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'Train Step')"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEGCAYAAACtqQjWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXxU9b34/9c7O1kJSdgSMBDCEkTRprjWDRe0C7ettlDvvVxrpbdq12/rcvu93l5/9Xu17a22Veu17q0tUKot7XVDsWqrIhEFWUQyA0LYMmEJJCwhyfv3x/kEhjiTTJKZzCTzfj4eeeTMWT7nPRPIO5/z+Zz3EVXFGGOMiYaUeAdgjDFm8LCkYowxJmosqRhjjIkaSyrGGGOixpKKMcaYqEmLdwDxVFxcrOXl5fEOwxhjBpS33367QVVLQm1L6qRSXl5OTU1NvMMwxpgBRUQ+DLfNLn8ZY4yJGksqxhhjosaSijHGmKixpGKMMSZqLKkYY4yJmpgmFRGZJSIbRKRWRG4JsT1TRBa67ctFpDxo261u/QYRuSxo/SMiUi8ia8Kc87sioiJSHIv3ZIwxJryYJRURSQXuAy4HqoC5IlLVabdrgb2qOgG4G7jLHVsFzAGmArOA+117AI+5daHOOQa4BNgS1TdjjDEmIrHsqcwAalXVr6otwAJgdqd9ZgOPu+XFwEwREbd+gaoeUdVNQK1rD1V9FdgT5px3AzcBg7Kev6qyaMVWmo60xjsUY4wJKZZJpRTYGvS6zq0LuY+qtgKNQFGEx55ARD4DbFPVVd3sN19EakSkJhAIRPI+Esa7W/dx0x9Wc/Pi1fEOxRhjQoplUpEQ6zr3IMLtE8mxxxsRyQa+D9zWXVCq+qCqVqtqdUlJyCoDCWvLnoMALF2/K86RGGNMaLFMKnXAmKDXZcD2cPuISBpQgHdpK5Jjg1UA44BVIrLZ7b9SREb2If6E4ws0A9DS2s5Wl2CMMSaRxDKprAAqRWSciGTgDbwv6bTPEmCeW74SWKbe842XAHPc7LBxQCXwVrgTqep7qjpcVctVtRwvKZ2uqjuj+5biyxdoQlwf7tk1O+IbjDHGhBCzpOLGSG4EngfWA4tUda2I3O7GPwAeBopEpBb4DnCLO3YtsAhYBzwH3KCqbQAi8jvgDWCSiNSJyLWxeg+Jxh9o5vyJJUwdnc+zawZVvjTGDBIxrVKsqs8Az3Rad1vQ8mHgqjDH3gHcEWL93AjOW97TWBNde7uyqaGJsyuK+Hj5MH78/AZ2NB5iVMGQeIdmjDHH2B31A8T2xkMcPtrO+JIcZp3sDRU9Z70VY0yCsaQyQPjdIH1FSS4VJblMHpnHn1d1NXfBGGP6nyWVAcIXaAJgfEkOALOnl7Jyyz4+3N0cz7CMMeYEllQGCH+gmbysNEpyMwGYPX00IvDHd6y3YoxJHJZUBghfoInxJbmIm1M8eugQzhxXxNPv1OHNwjbGmPizpDJA+APNVBTnnLDus6eXsnn3Qd7Zui9OURljzIksqQwATUda2bn/MBXDc09Yf/nJI8lMS+GP72yLU2TGGHMiSyoDwCY382t8p55KXlY6l1SN4M+rtnOktS0eoRljzAksqQwA/gZv5lfnngrAVdVj2HvwKC+stSKTxpj4s6QyAPjqm0gROKko+yPbPjGhmLLCIfx2uT2XzBgTf5ZUBgBfQzNlhdlkpqV+ZFtKijB3xlje8O/G7+5lMcaYeLGkMgD46puoKMkJu/2q6jLSUoQFK7aG3ccYY/qDJZUE196ubN7dzPiSj46ndBiel8XFU0aw+O06G7A3xsSVJZUE11FIsqKLpALwpTPGsqe5xYpMGmPiypJKgut42uP4Li5/AZw7oZhxxTk88vfNdoe9MSZuLKkkuI7B9+56KikpwjXnlLNq6z5WbtnbH6EZY8xHWFJJcL5AE3lZaRTnZnS775UfK6NgSDoPvbapHyIzxpiPsqSS4PyB5hMKSXYlOyONuTPG8vzanWzdc7AfojPGmBNZUklw/kBzl9OJO5t39kmkiPDY65tjF5QxxoQR06QiIrNEZIOI1IrILSG2Z4rIQrd9uYiUB2271a3fICKXBa1/RETqRWRNp7Z+LCLvi8hqEXlaRIbG8r31h2OFJLsZTwk2qmAIV0wbxcIVW2k8eDSG0RljzEfFLKmISCpwH3A5UAXMFZGqTrtdC+xV1QnA3cBd7tgqYA4wFZgF3O/aA3jMretsKXCyqp4CfADcGtU3FAebjj1COPKeCsDXLqig6Ugrj75uYyvGmP4Vy57KDKBWVf2q2gIsAGZ32mc28LhbXgzMFG/wYDawQFWPqOomoNa1h6q+CuzpfDJVfUFVW93LN4GyaL+h/nb8EcKR91QApozK5+IpI3j075s5cNh6K8aY/hPLpFIKBNcNqXPrQu7jEkIjUBThsV35MvBsqA0iMl9EakSkJhAI9KDJ/ucPhC8k2Z1vzJxA46Gj/PrND2MQmTHGhBbLpBJqulLnu/LC7RPJsaFPKvJ9oBV4MtR2VX1QVatVtbqkpCSSJuPGF2hmzLDQhSS7c0rZUM6fWMJDr23iYEtr9wcYY0wUxDKp1AFjgl6XAdvD7SMiaUAB3qWtSI79CBGZB3wKuFoHwW3lvkDTRx7M1RNfv2gCe5pbePJNK4tvjOkfsUwqK4BKERknIhl4A+9LOu2zBJjnlq8ElrlksASY42aHjQMqgbe6OpmIzAJuBj6jqgP+Jo32dmVTQ3OPZn51Vl0+jHMnFPPLV3w2tmKM6RcxSypujORG4HlgPbBIVdeKyO0i8hm328NAkYjUAt8BbnHHrgUWAeuA54AbVLUNQER+B7wBTBKROhG51rV1L5AHLBWRd0XkgVi9t/6wbd8hjrS293iQvrObZ01mT3MLv3rVH6XIjDEmvLRYNq6qzwDPdFp3W9DyYeCqMMfeAdwRYv3cMPtP6FOwCcbf0LvpxJ1NKyvgk6eM4qG/beKfziqnJC8zGuEZY0xIdkd9gvLV9246cSjfvXQSLa3t/GLZxj63ZYwxXbGkkqD8DZEXkuzOuOIcvvjxMfx2+RY2ux6QMcbEgiWVBOXV/IqskGQkvjmzksy0FH74v+uj0p4xxoRiSSVB+QJN3T6YqyeG52fx9ZmVvLh+F3/dUB+1do0xJpgllQTUdKSVXfuP9Gk6cSjXnFPOuOIcbv/zOlpa26PatjHGgCWVhHT8aY/R66kAZKalctunq/A3NPOYFZs0xsSAJZUE5D/2XPro9lQALpw0nJmTh/OzFzeys/Fw1Ns3xiQ3SyoJyNeHQpKRuO3TVbSp8u9/WsMgqGZjjEkgllQSkL8PhSQjcVJRDt++eCJL1+3i2TU7Y3IOY0xysqSSgHyBpqgP0nd27bnjOLk0n9v+tNaeEGmMiRpLKgmmo5BkX6oTRyItNYW7Pn8Kew+2cMcz62J6LmNM8rCkkmA6CklWDI9tTwVg6ugC5p83nkU1dbxs964YY6LAkkqCOfYI4Rj3VDp8c2Ylk0bkcdPi1exuOtIv5zTGDF6WVBJMLKcTh5KVnso9c6bTePAotz71ns0GM8b0iSWVBONvaCI/SoUkIzVlVD43zZrEC+t2sahma7+d1xgz+FhSSTC++mbGR7GQZKS+fM44zq4o4j//vO7YHf3GGNNTllQSjL8h9tOJQ0lJEf77C6eSmZbC9U+u5FBLW7/HYIwZ+CypJJADh4+ya/+RqFYn7olRBUO4+4vT2bDrAP/3j3a3vTGm5yypJJBNUXqEcF9cMGk4X7+okj+srGPhChtfMcb0TEyTiojMEpENIlIrIreE2J4pIgvd9uUiUh607Va3foOIXBa0/hERqReRNZ3aGiYiS0Vko/teGMv3Fgu+Y9WJ+//yV7BvzqzkE5XF3LZkLWu2NcY1FmPMwBKzpCIiqcB9wOVAFTBXRKo67XYtsFdVJwB3A3e5Y6uAOcBUYBZwv2sP4DG3rrNbgJdUtRJ4yb0eUPyBZlIExsaokGSkUlOEe744neKcDK57oob6A1bN2BgTmVj2VGYAtarqV9UWYAEwu9M+s4HH3fJiYKZ4055mAwtU9YiqbgJqXXuo6qvAnhDnC27rceAfovlm+oM/0MzYGBaS7Imi3Ex+Na+afQePMv+Jtzl81AbujTHdi2VSKQWCL8rXuXUh91HVVqARKIrw2M5GqOoO19YOYHionURkvojUiEhNIBCI8K30D+8RwvG99BVs6ugC7pkznXe37uOmxatt4N4Y061YJpVQN1p0/q0Ubp9Iju0VVX1QVatVtbqkpCQaTUZFmyskGc9B+lAumzqSm2ZNYsmq7fxiWW28wzHGJLhYJpU6YEzQ6zJge7h9RCQNKMC7tBXJsZ3tEpFRrq1RwICqkLjdFZJMpJ5Kh6+dX8HnTi/lp0s/YJHNCDPGdCGWSWUFUCki40QkA2/gfUmnfZYA89zylcAy9a6xLAHmuNlh44BK4K1uzhfc1jzgT1F4D/2mvwtJ9oSIcOfnTuG8iSXc8tRqlq7bFe+QjDEJKmZJxY2R3Ag8D6wHFqnqWhG5XUQ+43Z7GCgSkVrgO7gZW6q6FlgErAOeA25Q1TYAEfkd8AYwSUTqRORa19adwCUishG4xL0eMDoKSfZHyfveyEhL4ZdXn860sqHc+NuVvLUp1FwJY0yyk2QefK2urtaampp4hwHA959+jz+v2s6q/7i03+t+9cSe5haufOB1AgeOsHD+WVSNzo93SMaYfiYib6tqdahtdkd9gvAHmqkY3v+FJHtqWE4GT3x5BrmZaVz90Jus37E/3iEZYxKIJZUE4Qs0Mb44MS99dVZWmM3vrjuTzLRUrn5oORt2Hoh3SMaYBGFJJQEcOHyU+gPxKyTZG+XFOfxu/pmkpwpf+tWbbNxlicUYY0klIRwbpE/A6cRdGVecw2+vO5PUFGHur+xSmDEmwqQiIueKyDVuucRN8zVR4m/oKCQ5cHoqHSpKcvnd/DNJS0nhi//zBm9/aLPCjElm3SYVEfkP4GbgVrcqHfhNLINKNv5AM6kpEvdCkr1VUZLL4q+dRVFuJlc/tJy/bhhQ950aY6Iokp7KZ4HPAM0AqrodyItlUMnGF2hiTOGQhCgk2Vtlhdks+upZjC/O5bonavjzqu4KIBhjBqNIkkqLu8tdAURk4F2jSXD+QPOAG08JpSQvkwVfPZPTxhTyjQXv8OCrPitCaUySiSSpLBKR/wGGish1wIvAQ7ENK3m0tSv+huYBNfOrK/lZ6Txx7QyuOHkU/++Z9/m3p9dwtK093mEZY/pJWnc7qOpPROQSYD8wCbhNVZfGPLIksX3fIVoStJBkb2Wlp/KLuadxUlE29//VR93eg9x39enkZ6XHOzRjTIxFMlB/l6ouVdXvqep3VXWpiNzVH8Elg0R5hHC0paQIN82azI+uPIU3fLv5/P2vs7mhOd5hGWNiLJLLX5eEWHd5tANJVj53j8pgufzV2Reqx/DEtTMINB3h0/f+jZfWW4VjYwazsElFRL4mIu/hVQNeHfS1CVjdfyEObv5AEwVD0inKyYh3KDFzdkUxf77xXE4qyubax2v46QsbaGu3AXxjBqOuxlR+CzwL/BeuJL1zQFXtDrco8R4hnJPwhST7asywbBb/69n8+x/X8PNltayqa+SeL06ncBAnU2OSUdieiqo2qupmVZ2rqh8Ch/CmFeeKyNh+i3CQ8weaB0whyb7KSk/lR1eewh2fPZnXfQ1c8fPXeNO/O95hGWOiKJKB+k+7B19tAl4BNuP1YEwfdRSSrBg+OMdTQhERrj7jJJ762jlkpacy91dv8tMXNtBq046NGRQiGaj/IXAm8IGqjgNmAn+PaVRJoqOQZLL0VIJNKyvgL18/l8+fXsbPl9XyxQffZOueg/EOyxjTR5EklaOquhtIEZEUVX0ZmB7juJJCRyHJCUnUUwmWk5nGT646lZ/PPY0Pdh7gip+9xqKarXYXvjEDWCRJZZ+I5AKvAk+KyM+A1tiGlRx89a6Q5LDkTCodPnPqaJ755ieYMjqfmxav5l8eXcH2fYfiHZYxphciSSqzgYPAt4HnAB/w6VgGlSz8DU2MHZZNRpo91mbMsGwWXHcm//mZqby1aQ+X3f0qC1dssV6LMQNMt7/NVLVZVdtVtVVVHwfuA2ZF0riIzBKRDSJSKyK3hNieKSIL3fblIlIetO1Wt36DiFzWXZsiMlNEVorIuyLyNxGZEEmM8eSrb2Z8cXL3UoKlpAjzzi7n+W+dR9XofG7+w3v88yNv2Z34xgwgXd38mO9+sd8rIpeK50bAD3yhu4ZFJBUvAV0OVAFzRaSq027XAntVdQJwN3CXO7YKmANMxUtg94tIajdt/hK4WlWn491j838j+wjio61d2bR78BSSjKaxRdn87rozuX32VN7Zso9L73mVn724kSOtbfEOzRjTja56Kr/GKyD5HvAV4AXgKmC2qs6OoO0ZQK2q+lW1BViAdykt2Gzgcbe8GJgp3l2As4EFqnpEVTcBta69rtpUIN8tFwAJ/UCPjkKSg63mV7SkpAj/fFY5L/2f87m0agR3v/gBs+55jb9tbIh3aMaYLnR1R/14VZ0GICIPAQ3AWFU9EGHbpcDWoNd1wBnh9lHVVhFpBIrc+jc7HVvqlsO1+RXgGRE5hFdR+cxQQYnIfGA+wNix8buHs9YVkhxM1YljYUR+Fvd+6XS+UB3gtj+t4R8fXs6nThnFrVdMoXTokHiHZ4zppKueytGOBVVtAzb1IKEAhKo70nnUNdw+PV0P3kSCK1S1DHgU+GmooFT1QVWtVtXqkpKSkIH3h457VAbic+nj4byJJTz3rfP41sWVLF23i4t+8ld+/Pz7NB2xiYjGJJKuksqpIrLffR0ATulYFpH9EbRdB4wJel3GRy9JHdtHRNLwLlvt6eLYkOtFpAQ4VVWXu/ULgbMjiDFufK6Q5DCrfRWxrPRUvnXxRJZ99wIuP3kk973s48Kf/JWFK7ZYgUpjEkRXtb9SVTXffeWpalrQcn6444KsACpFZJyIZOANvC/ptM8SYJ5bvhJY5h5dvASY42aHjQMqgbe6aHMvUCAiE11blwDrI/kA4sWfJIUkY6F06BDumXMaT19/NmOHZXPzH97jkz9/jb9uqLcpyMbEWbdPfuwtN0ZyI/A8kAo8oqprReR2oEZVlwAPA78WkVq8Hsocd+xaEVkErMO70fIGdwmOUG269dcBfxCRdrwk8+VYvbdo8AWaOX9i/C6/DQanjS1k8b+exTPv7eTO59bzL4+u4OPlhXz30kmcMb4o3uEZk5Qkmf+yq66u1pqamn4/74HDR5n2gxe4adYkrr8g4W+nGRBaWttZWLOVe5dtZNf+I3yispjvXjqJU8cMjXdoxgw6IvK2qlaH2ma3csfB8UF6m/kVLRlpKfzTmSfxyvcu5PtXTGHt9v3Mvu/vXPdEDavr9sU7PGOSRswuf5nwjj+X3mZ+RVtWeirXnTeeuWeM5ZG/beJXr/lZum4Xn6gs5oYLJ3DGuGE2jmVMDEXyPJUDQbPAOr62isjTIjK+P4IcbPwBKyQZa7mZaXxjZiWv33IRN8+azPod+5nz4Jtc+cAbLHt/lw3oGxMjkfRUfoo3nfe3ePeJzAFGAhuAR4ALYhXcYOULWCHJ/pKXlc7XLqjgmnPKWVSzlf95xc+XH6th8sg8vnr+eD45bbT9HIyJokj+N81S1f9R1QOqul9VH8S7yXAhUBjj+AYl7xHC1kvpT1npqfzzWeX89XsX8JOrTuVoWzvfXriKc+5axi9e2sjupiPxDtGYQSGSpNIuIl8QkRT3FVxM0q4h9FBHIcmK4TZIHw/pqSlc+bEyln77fB675uNMGZXPfy/9gLPuXMbNi1fz/s5I7us1xoQTyeWvq4GfAffjJZE3gX8UkSHAjTGMbVDattcrJGk9lfhKSREumDScCyYNZ+OuAzz6+maeWlnHwpqtnF1RxD+eeRKXVI0gPdUujRnTE90mFVX1E/6hXH+LbjiDn889Qth6KomjckQe/++z0/jepZP43Yot/OaND7n+yZUU52byheoy5s4Yy5hh2fEO05gBoduk4upqXQeUB++vqgl9x3qi8tW76sTWU0k4hTkZXH/BBL56XgWvfFDPb5dv4YFXfPzyFR+fqCzhSzPGMnPKcOu9GNOFSC5//Ql4DXgRsKck9ZG/oZmh2VZIMpGlpggXTR7BRZNHsH3fIRau2MrCFVv519+8TUleJv8wfTSfO72MKaMiKYFnTHKJJKlkq+rNMY8kSfjqmxhfbIUkB4rRQ4fw7Usm8vWLJvDyhgC/r9nKY69v5levbaJqVD6fO72U2dNLKcnLjHeoxiSESJLKX0TkClV9JubRJAF/gxWSHIjSUlO4pGoEl1SNYE9zC39etZ2nVtbxw/9dz389+z7nTyzhc6eXcvGUEWSlp8Y7XGPiJpKk8k3g30TkCN6DuwTQCMvfmyD7Dx8lcOCI1fwa4IblZDDv7HLmnV3Oxl0HeOqdbTy9chvL3q8nJyOVi6tG8Mlpozh/UgmZaZZgTHKJZPZXXn8Ekgw6CkmOt5pfg0bliDxunjWZ7146iTf9u/nL6u08u2Ynf3p3O3mZaVwydQSfOmUU504osTv3TVIIm1REZLKqvi8ip4farqorYxfW4OQ/VkjSeiqDTWqKcM6EYs6ZUMzts0/mdd9u/rJqO8+v3clTK7eRn5XGZVNHcvm0kZxdUWyXyMyg1VVP5TvAfOC/Q2xT4KKYRDSI+QJNrpCk3fMwmKWnpnD+xBLOn1jCHZ+dxt9qA/xl1Q6eXbOT379dR3ZGKudPLOGSqhFcNHk4Q7NtJqAZPMImFVWd775f2H/hDG7+QLMVkkwyGWkpx6YnH2lt4w3fbl5Yt4sX1+3i2TU7SU0RZpQPOzYJwG6yNANdRE9+FJGz+ejNj0/ELqz+0d9Pfrz07lcYOyybh+Z9vN/OaRJTe7uyelsjS9ft5IW1u9joboqdPDLPlY8p4WMnFdqNliYhdfXkx0juqP81UAG8y/GbHxUY8EmlP7W1K5t3H+SCScPjHYpJACkpwvQxQ5k+Zijfu2wymxuaWbpuFy+u38VDr/l54BUfuZlpnDOhiAsmDef8iSWMHjok3mEb061IphRXA1Xai6caicgsvGKUqcBDqnpnp+2ZeMnpY8Bu4IuqutltuxW4Fi+RfUNVn++qTfHuJvwhcJU75peq+vOexhwrHYUk7WmPJpTy4hyuO2881503ngOHj/L32t288kGAVzbU8/zaXQBMHJHLBZOGc15lCdXlhTbYbxJSJEllDd5DuXb0pGERSQXuAy4B6oAVIrJEVdcF7XYtsFdVJ4jIHOAu4IsiUoX3MLCpwGjgRRGZ6I4J1+a/AGOAyaraLiIJ1SXoeITweJv5ZbqRl5XOrJNHMuvkkagqG+ub+OuGel75IMCjf9/Eg6/6yUhLofqkQs6uKOLsCcWcUlpAml0qMwkgkqRSDKwTkbeAY08yUtXPdHPcDKDWVTlGRBYAs4HgpDIb+IFbXgzc63ocs4EFqnoE2CQita49umjza8CXVLXdxVcfwXvrNz6bTmx6QUSYOCKPiSPymH9eBc1HWnnTv5vXfd7XT174AF74gNzMNM4YN4yzKoo4Z0Ixk0bkkZJipYBM/4skqfygl22XAluDXtcBZ4TbR1VbRaQRKHLr3+x0bKlbDtdmBV4v57NAAO+S2cbOQYnIfLyp0owdO7bn76qXfAErJGn6LiczjZlTRjBzyggA9jS38IZvN6/7Gnjdt5uX3vf+lirKyeCM8cP4eLn3NWVUPqmWZEw/6DKpuEtY/66qF/ei7VD/gjuPy4TbJ9z6UP37jjYzgcOqWi0inwMeAT7xkZ29xyE/CN7sr9ChR58/0GTl7k3UDcvJ4JOnjOKTp4wCYPu+Q7zh283ffQ0s9+/hmfd2ApCbmcbpJxUyo7yQj5cP49QxQ21MxsREl0lFVdtE5KCIFKhqYw/brsMb4+hQBmwPs0+diKQBBcCebo4Nt74O+INbfhp4tIfxxpS/oZkLrJCkibHRQ4fw+Y+V8fmPlQFeklmxeY/3tWmvd7kMyEhN4ZSyAqrLhzFjXCHTxxRaL9pERSSXvw4D74nIUqC5Y6WqfqOb41YAlSIyDtiGN/D+pU77LAHmAW8AVwLLVFVFZAnwWxH5Kd5AfSXwFl4PJlybf8S7y/8R4HzggwjeW7/oKCRpg/Smv40eOoTZ073y/AD7DrZQs3kvKzbv4a3Ne9z0Za/DXl6UzfQxQzltbCHTxwxlyqh8u1HX9FgkSeV/3VePuDGSG4Hn8ab/PqKqa0XkdqBGVZcADwO/dgPxe/CSBG6/RXgD8K3ADaraBhCqTXfKO4EnReTbQBPwlZ7GHCsdhSRtOrGJt6HZGVxcNYKLq7wxmUMtbayq28e7W/fx7pZ9vO7bzR/f9Tr/GWkpTCstcInGu6emdOgQexaQ6VJEd9QPVv11R/0f3q7j//x+FS9+53wm2LPpTQJTVXY0Hubdrft4Z8te3tmyj/e2NXKktR2AkrxMTikt4GT3Na20gBH5mZZokkxf76ivBP4LqAKyOtar6vioRTjI+RuskKQZGESE0UOHMHroEK6Y5g3+H21r5/0dB3hn617edUnm5Q31tLu/R4tzMzm5NJ9pQYlmVEGWJZokFcnlr0eB/wDuBi4EriH07CwThq++mZOskKQZoNJTU5hWVsC0sgL++Sxv3cGWVtZt38+abY28t837/uoHgWOJpigng6mlBUwrzWfKqHwmj8ynvCjbbtBMApEklSGq+pKIiKp+CPxARF7DSzQmAv6GJnswlxlUsjPSqC4fRnX5sGPrDrW0sX6nSzR1jby3rZEHahtoc5kmMy2FiSPymDwyz0s0o/KYMjKfQpt1NqhENPtLRFKAjW6QfBuQUCVQEllbu7K54SAXWiFJM8gNyUjl9LGFnD628Ni6w0fbqK1v4v2dB3h/x37e33mAZe/X8/u3647tMyI/k8kjjyeZyaPyqCjJtQrNA1QkSeVbQDbwDeD/w7sENi+WQQ0mdXsP0tLWbj0Vk5Sy0lOPDeoHCxw4wvs79/P+jgOsd9/f8O2mpc2bEJCeKowrzqFyeB4Vw3OpHJ5L5YhcxhXnkJlmN20msjSI/NMAABRrSURBVEieUb8CwLv6pdfEPqTB5fh0Ypv1ZUyHkrxMSvJK+ETl8RuCj7a1s6mhmfWuR7NxVxPrduzn2TU7jo3VpAicVJTDhKBEM6Ekj4rhOWRnRPI3som1SGZ/nYV3P0kuMFZETgW+qqrXxzq4wcCqExsTmfTUlGPFM2cHrT98tI1NDc1srG+idtcBNtY3sbG+iZffr6e1/fgtEWWFQ6gcnktFSS7jSnIYV5zD+OJcm/LczyJJ7fcAl+Hd/Y6qrhKR82Ia1SBihSSN6Zus9FSmjPJmkQU72tbOh7ub2bir6Vii2bjrAK/7dh+7rwYgOyOV8qIcxpXkML7YSzYdCacgO72/386gF1F/UVW3dsr0beH2NSfyB5rs0pcxMZCemsKE4XlMGJ7H5UHr29uVHfsPsynQzKaGJvwNzWxqaGbNtkaefe/4pTTwCnKOC0o044pzGDssm7FF2eRnWcLpjUiSylb3jHoVkQy8Afv1sQ1r8PAFmrlwkhWSNKa/pKQIpUOHUDp0COdWFp+wraW1nS17DrKpwUs4m1zCeW1jgMVBM9IAhmanc9KwbMYMy+akomzGHlvOYWR+lj1KIIxIksq/4j2+txSvEvALgI2nRKDx0FEamo5QYaVZjEkIGWkpTBie68oljThhW9ORVrbsPsiWPc1s2XOQD3cfZMueg7y3rZHn1uw8YfwmIzWFssIhJyScjh5O6dAh5CVxLyeS2V8NwNXB60TkW3hjLaYL/o5BenuOijEJLzczjarR+VSNzv/Itta2dnY0Hj4h2XQkn5Vb9nLgcOsJ++dnpVFWmE1poddjKiv0vkqHeusKs9MH7eSB3s7B+w6WVLrVMZ3YZn4ZM7ClpaYwxl3+OmfCidtUlcZDR48lm237DrFt7yG27TvElt0Heb22geaWE4ehszNSvUt0nZJNWeEQyoYOoTg3c8A+Drq3SWVgvtt+5gs0kZYinFRkhSSNGaxEhKHZGQzNzuDUMUM/sr0j6dTtPUSdSzZe0jlI3d5DvLt1H/sOHj3hmIzUFEYWZDGyIIvRBVmMLBjCqGOvhzCyIIuinIyETDy9TSrJWy+/B/yBZsYOy7ZyE8YkseCk07myQIemI61s33eIur0H2bb3EHX7DrGz8TA79h3m7S172dm4g6NtJ/7aTU8VRuQfTzIdSWeUS0CjCrLi0uMJm1RE5AChk4cAQ2IW0SDiFZK0S1/GmK7lZqYdu/EzlPZ2ZXdzi5doGg+xc/9htu87zM7GQ8eef/PcmsPHytx0SEvxEs/IgixG5Gd6y/lZjMjP4uyKIobnZ4U8X1+ETSqqGvrdmYhYIUljTLSkpIgrbZPJtLLQvR1VZU9zCzsaD7Oj8XjC8ZYP8/7OA7yyIXBsfOeJL8/o36Ri+qajkKTd+GiM6Q8iQlFuJkW5mWEvswEcOHyUXfuPMKog+gkFLKnEzPGaXzad2BiTOPKy0mN6H01MR5BFZJaIbBCRWhG5JcT2TBFZ6LYvF5HyoG23uvUbROSyHrT5CxFpitV7ipRNJzbGJKOYJRURSQXuAy7He779XBGp6rTbtcBeVZ2A97jiu9yxVcAcYCowC7hfRFK7a1NEqoGPzumLA1+gmUIrJGmMSTKx7KnMAGpV1a+qLcACOKGiNe714255MTBTvNtMZwMLVPWIqm4Cal17Ydt0CefHwE0xfE8R8wVs5pcxJvnEMqmUAluDXte5dSH3UdVWoBEo6uLYrtq8EViiqju6CkpE5otIjYjUBAKBHr2hnvAHmqmw8RRjTJKJZVIJdcdN5/tewu3To/UiMhq4CvhFd0Gp6oOqWq2q1SUlsake3FFI0noqxphkE8ukUgeMCXpdBmwPt4+IpAEFwJ4ujg23/jRgAlArIpuBbBGpjdYb6SkrJGmMSVaxTCorgEoRGeeewzIH9/TIIEuAeW75SmCZqqpbP8fNDhsHVAJvhWtTVf9XVUeqarmqlgMH3eB/XPg6nktvJe+NMUkmZvepqGqriNwIPA+kAo+o6loRuR2oUdUlwMPAr12vYg9eksDttwhYB7QCN6hqG0CoNmP1HnrL7wpJjh1mhSSNMcklpjc/quozwDOd1t0WtHwYbywk1LF3AHdE0maIfeLaRfAHmhlbZIUkjTHJx37rxYAv0MT4Yrv0ZYxJPpZUoqy1rZ0Pdx+kYrgN0htjko8llSir23vIKyRpPRVjTBKypBJl/gYrJGmMSV6WVKKso5Cklbw3xiQjSypR5gs0UZidTqEVkjTGJCFLKlHmCzRbL8UYk7QsqUSZP9Bk4ynGmKRlSSWKGg8epaGpxQpJGmOSliWVKPK5mV92+csYk6wsqUTR8UcI2+UvY0xysqQSRVZI0hiT7CypRJEv0GSFJI0xSc1++0WR36YTG2OSnCWVKGlta2fz7mYbTzHGJDVLKlFSt/cQR9vUCkkaY5KaJZUo6SgkaSXvjTHJzJJKlPjq3XRi66kYY5KYJZUo8Tc0MSwnwwpJGmOSWkyTiojMEpENIlIrIreE2J4pIgvd9uUiUh607Va3foOIXNZdmyLypFu/RkQeEZH0WL63znz1zYwvtktfxpjkFrOkIiKpwH3A5UAVMFdEqjrtdi2wV1UnAHcDd7ljq4A5wFRgFnC/iKR20+aTwGRgGjAE+Eqs3lso/gYrJGmMMbHsqcwAalXVr6otwAJgdqd9ZgOPu+XFwEwREbd+gaoeUdVNQK1rL2ybqvqMOsBbQFkM39sJOgpJ2j0qxphkF8ukUgpsDXpd59aF3EdVW4FGoKiLY7tt0132+ifguT6/gwj5jj1C2JKKMSa5xTKpSIh1GuE+PV0f7H7gVVV9LWRQIvNFpEZEagKBQKhdeuz4I4Tt8pcxJrnFMqnUAWOCXpcB28PtIyJpQAGwp4tju2xTRP4DKAG+Ey4oVX1QVatVtbqkpKSHbyk0nyskOcYKSRpjklwsk8oKoFJExolIBt7A+5JO+ywB5rnlK4FlbkxkCTDHzQ4bB1TijZOEbVNEvgJcBsxV1fYYvq+P8AeaOMkKSRpjDGmxalhVW0XkRuB5IBV4RFXXisjtQI2qLgEeBn4tIrV4PZQ57ti1IrIIWAe0AjeoahtAqDbdKR8APgTe8Mb6eUpVb4/V+wvmCzTbeIoxxhDDpALejCzgmU7rbgtaPgxcFebYO4A7ImnTrY/pewmnta2dD3c3M3PK8Hic3hhjEopdr+mjY4UkradijDGWVPrKF+h4Lr3N/DLGGEsqfXTsufRWSNIYYyyp9JUvYIUkjTGmgyWVPvIHrJCkMcZ0sKTSR75Akw3SG2OMY0mlDxoPHmV3c4tVJzbGGMeSSh90FJK0nooxxngsqfSBr76jOrH1VIwxBiyp9Im/oZn0VCskaYwxHSyp9IGvvomxw6yQpDHGdLDfhn3gb7BCksYYE8ySSi91FJK0QXpjjDnOkkovbXWFJG2Q3hhjjrOk0kv+gE0nNsaYziyp9JJVJzbGmI+ypNJL/kAzRTkZDM22QpLGGNPBkkov+QJNNp5ijDGdWFLpJa86sY2nGGNMMEsqvbDvYAu7m1uoGG49FWOMCRbTpCIis0Rkg4jUisgtIbZnishCt325iJQHbbvVrd8gIpd116aIjHNtbHRtxmyww2dPezTGmJBillREJBW4D7gcqALmikhVp92uBfaq6gTgbuAud2wVMAeYCswC7heR1G7avAu4W1Urgb2u7Zg4Np14uCUVY4wJFsueygygVlX9qtoCLABmd9pnNvC4W14MzBQRcesXqOoRVd0E1Lr2QrbpjrnItYFr8x9i9cZ8AVdIsnBIrE5hjDEDUiyTSimwNeh1nVsXch9VbQUagaIujg23vgjY59oIdy4ARGS+iNSISE0gEOjF24Lyomw+e1opaVZI0hhjThDL34oSYp1GuE+01n90peqDqlqtqtUlJSWhdunWnBlj+dGVp/bqWGOMGcximVTqgDFBr8uA7eH2EZE0oADY08Wx4dY3AENdG+HOZYwxJsZimVRWAJVuVlYG3sD7kk77LAHmueUrgWWqqm79HDc7bBxQCbwVrk13zMuuDVybf4rhezPGGBNCWve79I6qtorIjcDzQCrwiKquFZHbgRpVXQI8DPxaRGrxeihz3LFrRWQRsA5oBW5Q1TaAUG26U94MLBCRHwLvuLaNMcb0I/H+yE9O1dXVWlNTE+8wjDFmQBGRt1W1OtQ2m75kjDEmaiypGGOMiRpLKsYYY6LGkooxxpioSeqBehEJAB/28vBivPtjEo3F1TMWV89YXD2TqHFB32I7SVVD3j2e1EmlL0SkJtzsh3iyuHrG4uoZi6tnEjUuiF1sdvnLGGNM1FhSMcYYEzWWVHrvwXgHEIbF1TMWV89YXD2TqHFBjGKzMRVjjDFRYz0VY4wxUWNJxRhjTNRYUukFEZklIhtEpFZEbumH820WkfdE5F0RqXHrhonIUhHZ6L4XuvUiIj93sa0WkdOD2pnn9t8oIvPCna+bWB4RkXoRWRO0LmqxiMjH3HutdceGegBbpHH9QES2uc/tXRG5Imjbre4cG0TksqD1IX+27nELy128C92jF7qLaYyIvCwi60VkrYh8MxE+ry7iiuvn5Y7LEpG3RGSVi+0/u2pPvMdjLHTnXy4i5b2NuZdxPSYim4I+s+lufX/+208VkXdE5C+J8FmhqvbVgy+8kvs+YDyQAawCqmJ8zs1Acad1PwJuccu3AHe55SuAZ/GehnkmsNytHwb43fdCt1zYi1jOA04H1sQiFrzn5pzljnkWuLwPcf0A+G6Ifavczy0TGOd+nqld/WyBRcAct/wA8LUIYhoFnO6W84AP3Lnj+nl1EVdcPy+3rwC5bjkdWO4+i5DtAdcDD7jlOcDC3sbcy7geA64MsX9//tv/DvBb4C9dffb99VlZT6XnZgC1qupX1RZgATA7DnHMBh53y48D/xC0/gn1vIn3RMxRwGXAUlXdo6p7gaXArJ6eVFVfxXv2TdRjcdvyVfUN9f61PxHUVm/iCmc2sEBVj6jqJqAW7+ca8mfr/mK8CFgc4j12FdMOVV3plg8A64FS4vx5dRFXOP3yebl4VFWb3Mt096VdtBf8WS4GZrrz9yjmPsQVTr/8LEWkDPgk8JB73dVn3y+flSWVnisFtga9rqPr/5DRoMALIvK2iMx360ao6g7wfkkAw7uJL5ZxRyuWUrcczRhvdJcfHhF3makXcRUB+1S1tbdxuUsNp+H9hZswn1enuCABPi93OeddoB7vl66vi/aOxeC2N7rzR/3/Qee4VLXjM7vDfWZ3i0hm57giPH9vf5b3ADcB7e51V599v3xWllR6LtR1zljPyz5HVU8HLgduEJHzutg3XHzxiLunsUQ7xl8CFcB0YAfw3/GIS0RygT8A31LV/V3tGue4EuLzUtU2VZ0OlOH9tTyli/b6LbbOcYnIycCtwGTg43iXtG7ur7hE5FNAvaq+Hby6i3b65bOypNJzdcCYoNdlwPZYnlBVt7vv9cDTeP/RdrkuM+57fTfxxTLuaMVS55ajEqOq7nK/CNqBX+F9br2JqwHv8kVap/XdEpF0vF/cT6rqU2513D+vUHElwucVTFX3AX/FG5MI196xGNz2ArzLoDH7fxAU1yx3KVFV9QjwKL3/zHrzszwH+IyIbMa7NHURXs8lvp9Vd4Mu9vWRQbE0vMG1cRwfvJoaw/PlAHlBy6/jjYX8mBMHe3/klj/JiQOEb+nxAcJNeIODhW55WC9jKufEAfGoxQKscPt2DFZe0Ye4RgUtfxvvujHAVE4cmPTjDUqG/dkCv+fEwc/rI4hH8K6N39NpfVw/ry7iiuvn5fYtAYa65SHAa8CnwrUH3MCJg8+LehtzL+MaFfSZ3gPcGad/+xdwfKA+vp9Vb36pJPsX3syOD/Cu9X4/xuca736Yq4C1HefDuxb6ErDRfe/4hynAfS6294DqoLa+jDcIVwtc08t4fod3aeQo3l8y10YzFqAaWOOOuRdX9aGXcf3anXc1sIQTf2l+351jA0GzbML9bN3P4S0X7++BzAhiOhfvcsFq4F33dUW8P68u4orr5+WOOwV4x8WwBritq/aALPe61m0f39uYexnXMveZrQF+w/EZYv32b98dewHHk0pcPysr02KMMSZqbEzFGGNM1FhSMcYYEzWWVIwxxkSNJRVjjDFRY0nFGGNM1FhSMaaHRKQoqCrtTjmxsm+k1XgfFZFJPTjnKBF5xlXJXSciS9z68SIyp7fvxZhosynFxvSBiPwAaFLVn3RaL3j/v9pDHtjz8zwMrFTV+9zrU1R1tYhcDNyoqhEVbDQm1qynYkyUiMgEEVkjIg8AK4FRIvKgiNSI9wyO24L2/ZuITBeRNBHZJyJ3ul7IGyIyPETzowgqOKiqq93incCFrpf0DdfeT8V79sdqEfmKO9/F4j1D5Y+up3OfS3zGRJUlFWOiqwp4WFVPU9VteOVYqoFTgUtEpCrEMQXAK6p6KvAG3h3Xnd0LPC4iy0Tk3zpqh+GVeXlZVaer6s+B+XhFBmfgFTm8QUTGun3PAL4FTMMr0hiPRzaYQc6SijHR5VPVFUGv54rISryeyxS8pNPZIVV91i2/jVfD7ASq+gxeBeGHXRvviEhRiLYuBa5xJdqXA0OBSrftTVXdrKpteAUIz+3pmzOmO2nd72KM6YHmjgURqQS+CcxQ1X0i8hu8+kudtQQttxHm/6Wq7gaeBJ4UkefwkkJzp90Er4DgSyes9MZeOg+g2oCqiTrrqRgTO/nAAWB/0FP/ekVEZorIELecj1c5dotrPy9o1+eB6ztKn4vIpI7jgDNFZKyIpAJfAP7W23iMCcd6KsbEzkpgHV7lWT/w9z609XHgXhE5ivfH4C9V9R03hTlVRFbhXRq7DxgLvOvG4es5PnbyOt6Dt6biPQ9kSR/iMSYkm1JsTBKwqcemv9jlL2OMMVFjPRVjjDFRYz0VY4wxUWNJxRhjTNRYUjHGGBM1llSMMcZEjSUVY4wxUfP/AzB8Asq4ga6tAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "temp_learning_rate_schedule = CustomSchedule(d_model)\n",
    "\n",
    "plt.plot(temp_learning_rate_schedule(tf.range(40000, dtype=tf.float32)))\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.xlabel(\"Train Step\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YgkDE7hzo8r5"
   },
   "source": [
    "## Loss and metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oxGJtoDuYIHL"
   },
   "source": [
    "Since the target sequences are padded, it is important to apply a padding mask when calculating the loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MlhsJMm0TW_B"
   },
   "outputs": [],
   "source": [
    "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n",
    "    from_logits=True, reduction='none')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "67oqVHiT0Eiu"
   },
   "outputs": [],
   "source": [
    "def loss_function(real, pred):\n",
    "    mask = tf.math.logical_not(tf.math.equal(real, 0))\n",
    "    loss_ = loss_object(real, pred)\n",
    "\n",
    "    mask = tf.cast(mask, dtype=loss_.dtype)\n",
    "    loss_ *= mask\n",
    "  \n",
    "    return tf.reduce_mean(loss_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "phlyxMnm-Tpx"
   },
   "outputs": [],
   "source": [
    "train_loss = tf.keras.metrics.Mean(name='train_loss')\n",
    "train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(\n",
    "    name='train_accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aeHumfr7zmMa"
   },
   "source": [
    "## Training and checkpointing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UiysUa--4tOU"
   },
   "outputs": [],
   "source": [
    "transformer = Transformer(num_layers, d_model, num_heads, dff,\n",
    "                          input_vocab_size, target_vocab_size, dropout_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZOJUSB1T8GjM"
   },
   "outputs": [],
   "source": [
    "def create_masks(inp, tar):\n",
    "    # Encoder padding mask\n",
    "    enc_padding_mask = create_padding_mask(inp)\n",
    "  \n",
    "    # Used in the 2nd attention block in the decoder.\n",
    "    # This padding mask is used to mask the encoder outputs.\n",
    "    dec_padding_mask = create_padding_mask(inp)\n",
    "  \n",
    "    # Used in the 1st attention block in the decoder.\n",
    "    # It is used to pad and mask future tokens in the input received by \n",
    "    # the decoder.\n",
    "    look_ahead_mask = create_look_ahead_mask(tf.shape(tar)[1])\n",
    "    dec_target_padding_mask = create_padding_mask(tar)\n",
    "    combined_mask = tf.maximum(dec_target_padding_mask, look_ahead_mask)\n",
    "  \n",
    "    return enc_padding_mask, combined_mask, dec_padding_mask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Fzuf06YZp66w"
   },
   "source": [
    "Create the checkpoint path and the checkpoint manager. This will be used to save checkpoints every `n` epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hNhuYfllndLZ"
   },
   "outputs": [],
   "source": [
    "checkpoint_path = \"./checkpoints/train\"\n",
    "\n",
    "ckpt = tf.train.Checkpoint(transformer=transformer,\n",
    "                           optimizer=optimizer)\n",
    "\n",
    "ckpt_manager = tf.train.CheckpointManager(ckpt, checkpoint_path, max_to_keep=5)\n",
    "\n",
    "# if a checkpoint exists, restore the latest checkpoint.\n",
    "if ckpt_manager.latest_checkpoint:\n",
    "    ckpt.restore(ckpt_manager.latest_checkpoint)\n",
    "    print ('Latest checkpoint restored!!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0Di_Yaa1gf9r"
   },
   "source": [
    "The target is divided into tar_inp and tar_real. tar_inp is passed as an input to the decoder. `tar_real` is that same input shifted by 1: At each location in `tar_input`, `tar_real` contains the  next token that should be predicted.\n",
    "\n",
    "For example, `sentence` = \"SOS A lion in the jungle is sleeping EOS\"\n",
    "\n",
    "`tar_inp` =  \"SOS A lion in the jungle is sleeping\"\n",
    "\n",
    "`tar_real` = \"A lion in the jungle is sleeping EOS\"\n",
    "\n",
    "The transformer is an auto-regressive model: it makes predictions one part at a time, and uses its output so far to decide what to do next. \n",
    "\n",
    "During training this example uses teacher-forcing (like in the [text generation tutorial](./text_generation.ipynb)). Teacher forcing is passing the true output to the next time step regardless of what the model predicts at the current time step.\n",
    "\n",
    "As the transformer predicts each word, *self-attention* allows it to look at the previous words in the input sequence to better predict the next word.\n",
    "\n",
    "To prevent the model from peaking at the expected output the model uses a look-ahead mask."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LKpoA6q1sJFj"
   },
   "outputs": [],
   "source": [
    "EPOCHS = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iJwmp9OE29oj"
   },
   "outputs": [],
   "source": [
    "# The @tf.function trace-compiles train_step into a TF graph for faster\n",
    "# execution. The function specializes to the precise shape of the argument\n",
    "# tensors. To avoid re-tracing due to the variable sequence lengths or variable\n",
    "# batch sizes (the last batch is smaller), use input_signature to specify\n",
    "# more generic shapes.\n",
    "\n",
    "train_step_signature = [\n",
    "    tf.TensorSpec(shape=(None, None), dtype=tf.int64),\n",
    "    tf.TensorSpec(shape=(None, None), dtype=tf.int64),\n",
    "]\n",
    "\n",
    "@tf.function(input_signature=train_step_signature)\n",
    "def train_step(inp, tar):\n",
    "    tar_inp = tar[:, :-1]\n",
    "    tar_real = tar[:, 1:]\n",
    "  \n",
    "    enc_padding_mask, combined_mask, dec_padding_mask = create_masks(inp, tar_inp)\n",
    "  \n",
    "    with tf.GradientTape() as tape:\n",
    "        predictions, _ = transformer(inp, tar_inp, \n",
    "                                     True, \n",
    "                                     enc_padding_mask, \n",
    "                                     combined_mask, \n",
    "                                     dec_padding_mask)\n",
    "        loss = loss_function(tar_real, predictions)\n",
    "\n",
    "    gradients = tape.gradient(loss, transformer.trainable_variables)    \n",
    "    optimizer.apply_gradients(zip(gradients, transformer.trainable_variables))\n",
    "  \n",
    "    train_loss(loss)\n",
    "    train_accuracy(tar_real, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qM2PDWGDJ_8V"
   },
   "source": [
    "Portuguese is used as the input language and English is the target language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bbvmaKNiznHZ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 0 Loss 4.8740 Accuracy 0.0000\n",
      "Epoch 1 Batch 50 Loss 4.2633 Accuracy 0.0068\n",
      "Epoch 1 Batch 100 Loss 4.1935 Accuracy 0.0156\n",
      "Epoch 1 Batch 150 Loss 4.1367 Accuracy 0.0190\n",
      "Epoch 1 Batch 200 Loss 4.0830 Accuracy 0.0212\n",
      "Epoch 1 Batch 250 Loss 4.0099 Accuracy 0.0231\n",
      "Epoch 1 Batch 300 Loss 3.9261 Accuracy 0.0261\n",
      "Epoch 1 Batch 350 Loss 3.8413 Accuracy 0.0295\n",
      "Epoch 1 Batch 400 Loss 3.7563 Accuracy 0.0320\n",
      "Epoch 1 Batch 450 Loss 3.6814 Accuracy 0.0343\n",
      "Epoch 1 Batch 500 Loss 3.6111 Accuracy 0.0371\n",
      "Epoch 1 Batch 550 Loss 3.5472 Accuracy 0.0404\n",
      "Epoch 1 Batch 600 Loss 3.4869 Accuracy 0.0438\n",
      "Epoch 1 Batch 650 Loss 3.4298 Accuracy 0.0473\n",
      "Epoch 1 Batch 700 Loss 3.3757 Accuracy 0.0509\n",
      "Epoch 1 Loss 3.3741 Accuracy 0.0510\n",
      "Time taken for 1 epoch: 418.88925075531006 secs\n",
      "\n",
      "Epoch 2 Batch 0 Loss 2.9728 Accuracy 0.1187\n",
      "Epoch 2 Batch 50 Loss 2.6246 Accuracy 0.1045\n",
      "Epoch 2 Batch 100 Loss 2.5796 Accuracy 0.1071\n",
      "Epoch 2 Batch 150 Loss 2.5486 Accuracy 0.1101\n",
      "Epoch 2 Batch 200 Loss 2.5220 Accuracy 0.1121\n",
      "Epoch 2 Batch 250 Loss 2.4971 Accuracy 0.1141\n",
      "Epoch 2 Batch 300 Loss 2.4761 Accuracy 0.1157\n",
      "Epoch 3 Batch 250 Loss 2.1514 Accuracy 0.1461\n",
      "Epoch 3 Batch 300 Loss 2.1457 Accuracy 0.1469\n",
      "Epoch 3 Batch 350 Loss 2.1347 Accuracy 0.1476\n",
      "Epoch 3 Batch 400 Loss 2.1308 Accuracy 0.1487\n",
      "Epoch 3 Batch 450 Loss 2.1204 Accuracy 0.1495\n",
      "Epoch 3 Batch 500 Loss 2.1131 Accuracy 0.1505\n",
      "Epoch 3 Batch 550 Loss 2.1055 Accuracy 0.1511\n",
      "Epoch 3 Batch 600 Loss 2.0993 Accuracy 0.1520\n",
      "Epoch 3 Batch 650 Loss 2.0907 Accuracy 0.1529\n",
      "Epoch 3 Batch 700 Loss 2.0860 Accuracy 0.1538\n",
      "Epoch 3 Loss 2.0856 Accuracy 0.1538\n",
      "Time taken for 1 epoch: 100.5058856010437 secs\n",
      "\n",
      "Epoch 4 Batch 0 Loss 2.1574 Accuracy 0.2036\n",
      "Epoch 4 Batch 50 Loss 1.9850 Accuracy 0.1692\n",
      "Epoch 4 Batch 100 Loss 1.9544 Accuracy 0.1693\n",
      "Epoch 4 Batch 150 Loss 1.9421 Accuracy 0.1708\n",
      "Epoch 4 Batch 200 Loss 1.9377 Accuracy 0.1723\n",
      "Epoch 4 Batch 250 Loss 1.9252 Accuracy 0.1734\n",
      "Epoch 4 Batch 300 Loss 1.9168 Accuracy 0.1745\n",
      "Epoch 4 Batch 350 Loss 1.9052 Accuracy 0.1754\n",
      "Epoch 4 Batch 400 Loss 1.8980 Accuracy 0.1766\n",
      "Epoch 4 Batch 450 Loss 1.8872 Accuracy 0.1776\n",
      "Epoch 4 Batch 500 Loss 1.8770 Accuracy 0.1787\n",
      "Epoch 4 Batch 550 Loss 1.8699 Accuracy 0.1796\n",
      "Epoch 4 Batch 600 Loss 1.8623 Accuracy 0.1806\n",
      "Epoch 4 Batch 650 Loss 1.8530 Accuracy 0.1816\n",
      "Epoch 4 Batch 700 Loss 1.8463 Accuracy 0.1828\n",
      "Epoch 4 Loss 1.8464 Accuracy 0.1828\n",
      "Time taken for 1 epoch: 71.18679904937744 secs\n",
      "\n",
      "Epoch 5 Batch 0 Loss 1.9084 Accuracy 0.2324\n",
      "Epoch 5 Batch 50 Loss 1.7326 Accuracy 0.1986\n",
      "Epoch 5 Batch 100 Loss 1.7134 Accuracy 0.1982\n",
      "Epoch 5 Batch 150 Loss 1.7021 Accuracy 0.1995\n",
      "Epoch 5 Batch 200 Loss 1.6962 Accuracy 0.2008\n",
      "Epoch 5 Batch 250 Loss 1.6898 Accuracy 0.2017\n",
      "Epoch 5 Batch 300 Loss 1.6801 Accuracy 0.2028\n",
      "Epoch 5 Batch 350 Loss 1.6740 Accuracy 0.2037\n",
      "Epoch 5 Batch 400 Loss 1.6691 Accuracy 0.2046\n",
      "Epoch 5 Batch 450 Loss 1.6623 Accuracy 0.2054\n",
      "Epoch 5 Batch 500 Loss 1.6540 Accuracy 0.2062\n",
      "Epoch 5 Batch 550 Loss 1.6466 Accuracy 0.2068\n",
      "Epoch 5 Batch 600 Loss 1.6399 Accuracy 0.2073\n",
      "Epoch 5 Batch 650 Loss 1.6324 Accuracy 0.2080\n",
      "Epoch 5 Batch 700 Loss 1.6266 Accuracy 0.2087\n",
      "Saving checkpoint for epoch 5 at ./checkpoints/train/ckpt-1\n",
      "Epoch 5 Loss 1.6264 Accuracy 0.2087\n",
      "Time taken for 1 epoch: 88.9427399635315 secs\n",
      "\n",
      "Epoch 6 Batch 0 Loss 1.6761 Accuracy 0.2622\n",
      "Epoch 6 Batch 50 Loss 1.5292 Accuracy 0.2216\n",
      "Epoch 6 Batch 100 Loss 1.5101 Accuracy 0.2211\n",
      "Epoch 6 Batch 150 Loss 1.4960 Accuracy 0.2216\n",
      "Epoch 6 Batch 200 Loss 1.4961 Accuracy 0.2228\n",
      "Epoch 6 Batch 250 Loss 1.4889 Accuracy 0.2234\n",
      "Epoch 6 Batch 300 Loss 1.4839 Accuracy 0.2245\n",
      "Epoch 6 Batch 350 Loss 1.4798 Accuracy 0.2254\n",
      "Epoch 6 Batch 400 Loss 1.4764 Accuracy 0.2262\n",
      "Epoch 6 Batch 450 Loss 1.4714 Accuracy 0.2267\n",
      "Epoch 6 Batch 500 Loss 1.4634 Accuracy 0.2270\n",
      "Epoch 6 Batch 550 Loss 1.4602 Accuracy 0.2276\n",
      "Epoch 6 Batch 600 Loss 1.4546 Accuracy 0.2281\n",
      "Epoch 6 Batch 650 Loss 1.4475 Accuracy 0.2287\n",
      "Epoch 6 Batch 700 Loss 1.4429 Accuracy 0.2294\n",
      "Epoch 6 Loss 1.4431 Accuracy 0.2294\n",
      "Time taken for 1 epoch: 70.6754732131958 secs\n",
      "\n",
      "Epoch 7 Batch 0 Loss 1.4529 Accuracy 0.2827\n",
      "Epoch 7 Batch 50 Loss 1.3497 Accuracy 0.2400\n",
      "Epoch 7 Batch 100 Loss 1.3350 Accuracy 0.2403\n",
      "Epoch 7 Batch 150 Loss 1.3192 Accuracy 0.2420\n",
      "Epoch 7 Batch 200 Loss 1.3159 Accuracy 0.2432\n",
      "Epoch 7 Batch 250 Loss 1.3110 Accuracy 0.2445\n",
      "Epoch 7 Batch 300 Loss 1.3049 Accuracy 0.2456\n",
      "Epoch 7 Batch 350 Loss 1.2961 Accuracy 0.2462\n",
      "Epoch 7 Batch 400 Loss 1.2916 Accuracy 0.2469\n",
      "Epoch 7 Batch 450 Loss 1.2844 Accuracy 0.2475\n",
      "Epoch 7 Batch 500 Loss 1.2796 Accuracy 0.2479\n",
      "Epoch 7 Batch 550 Loss 1.2734 Accuracy 0.2481\n",
      "Epoch 7 Batch 600 Loss 1.2673 Accuracy 0.2487\n",
      "Epoch 7 Batch 650 Loss 1.2621 Accuracy 0.2494\n",
      "Epoch 7 Batch 700 Loss 1.2563 Accuracy 0.2499\n",
      "Epoch 7 Loss 1.2566 Accuracy 0.2500\n",
      "Time taken for 1 epoch: 59.18611979484558 secs\n",
      "\n",
      "Epoch 8 Batch 0 Loss 1.2509 Accuracy 0.3003\n",
      "Epoch 8 Batch 50 Loss 1.1781 Accuracy 0.2620\n",
      "Epoch 8 Batch 100 Loss 1.1627 Accuracy 0.2599\n",
      "Epoch 8 Batch 150 Loss 1.1488 Accuracy 0.2617\n",
      "Epoch 8 Batch 200 Loss 1.1484 Accuracy 0.2625\n",
      "Epoch 8 Batch 250 Loss 1.1408 Accuracy 0.2631\n",
      "Epoch 8 Batch 300 Loss 1.1403 Accuracy 0.2643\n",
      "Epoch 8 Batch 350 Loss 1.1329 Accuracy 0.2646\n",
      "Epoch 8 Batch 400 Loss 1.1291 Accuracy 0.2648\n",
      "Epoch 8 Batch 450 Loss 1.1278 Accuracy 0.2655\n",
      "Epoch 8 Batch 500 Loss 1.1254 Accuracy 0.2665\n",
      "Epoch 8 Batch 550 Loss 1.1216 Accuracy 0.2668\n",
      "Epoch 8 Batch 600 Loss 1.1174 Accuracy 0.2671\n",
      "Epoch 8 Batch 650 Loss 1.1139 Accuracy 0.2677\n",
      "Epoch 8 Batch 700 Loss 1.1101 Accuracy 0.2682\n",
      "Epoch 8 Loss 1.1101 Accuracy 0.2682\n",
      "Time taken for 1 epoch: 94.94312858581543 secs\n",
      "\n",
      "Epoch 9 Batch 0 Loss 1.1144 Accuracy 0.3267\n",
      "Epoch 9 Batch 50 Loss 1.0554 Accuracy 0.2771\n",
      "Epoch 9 Batch 100 Loss 1.0403 Accuracy 0.2771\n",
      "Epoch 9 Batch 150 Loss 1.0315 Accuracy 0.2775\n",
      "Epoch 9 Batch 200 Loss 1.0305 Accuracy 0.2786\n",
      "Epoch 9 Batch 250 Loss 1.0239 Accuracy 0.2787\n",
      "Epoch 9 Batch 300 Loss 1.0225 Accuracy 0.2794\n",
      "Epoch 9 Batch 350 Loss 1.0168 Accuracy 0.2798\n",
      "Epoch 9 Batch 400 Loss 1.0147 Accuracy 0.2800\n",
      "Epoch 9 Batch 450 Loss 1.0132 Accuracy 0.2801\n",
      "Epoch 9 Batch 500 Loss 1.0115 Accuracy 0.2805\n",
      "Epoch 9 Batch 550 Loss 1.0083 Accuracy 0.2803\n",
      "Epoch 9 Batch 600 Loss 1.0057 Accuracy 0.2804\n",
      "Epoch 9 Batch 650 Loss 1.0041 Accuracy 0.2809\n",
      "Epoch 9 Batch 700 Loss 1.0023 Accuracy 0.2810\n",
      "Epoch 9 Loss 1.0023 Accuracy 0.2810\n",
      "Time taken for 1 epoch: 64.87301301956177 secs\n",
      "\n",
      "Epoch 10 Batch 0 Loss 1.0015 Accuracy 0.3350\n",
      "Epoch 10 Batch 50 Loss 0.9756 Accuracy 0.2865\n",
      "Epoch 10 Batch 100 Loss 0.9551 Accuracy 0.2876\n",
      "Epoch 10 Batch 150 Loss 0.9462 Accuracy 0.2885\n",
      "Epoch 10 Batch 200 Loss 0.9461 Accuracy 0.2895\n",
      "Epoch 10 Batch 250 Loss 0.9426 Accuracy 0.2903\n",
      "Epoch 10 Batch 300 Loss 0.9371 Accuracy 0.2904\n",
      "Epoch 10 Batch 350 Loss 0.9338 Accuracy 0.2910\n",
      "Epoch 10 Batch 400 Loss 0.9320 Accuracy 0.2913\n",
      "Epoch 10 Batch 450 Loss 0.9290 Accuracy 0.2917\n",
      "Epoch 10 Batch 500 Loss 0.9280 Accuracy 0.2917\n",
      "Epoch 10 Batch 550 Loss 0.9261 Accuracy 0.2913\n",
      "Epoch 10 Batch 600 Loss 0.9247 Accuracy 0.2912\n",
      "Epoch 10 Batch 650 Loss 0.9220 Accuracy 0.2915\n",
      "Epoch 10 Batch 700 Loss 0.9219 Accuracy 0.2917\n",
      "Saving checkpoint for epoch 10 at ./checkpoints/train/ckpt-2\n",
      "Epoch 10 Loss 0.9219 Accuracy 0.2917\n",
      "Time taken for 1 epoch: 66.08506345748901 secs\n",
      "\n",
      "Epoch 11 Batch 0 Loss 0.8845 Accuracy 0.3574\n",
      "Epoch 11 Batch 50 Loss 0.8918 Accuracy 0.2987\n",
      "Epoch 11 Batch 100 Loss 0.8791 Accuracy 0.2974\n",
      "Epoch 11 Batch 150 Loss 0.8695 Accuracy 0.2986\n",
      "Epoch 11 Batch 200 Loss 0.8706 Accuracy 0.2995\n",
      "Epoch 11 Batch 250 Loss 0.8684 Accuracy 0.3002\n",
      "Epoch 11 Batch 300 Loss 0.8656 Accuracy 0.3000\n",
      "Epoch 11 Batch 350 Loss 0.8633 Accuracy 0.2998\n",
      "Epoch 11 Batch 400 Loss 0.8617 Accuracy 0.3002\n",
      "Epoch 11 Batch 450 Loss 0.8602 Accuracy 0.3005\n",
      "Epoch 11 Batch 500 Loss 0.8587 Accuracy 0.3001\n",
      "Epoch 11 Batch 550 Loss 0.8576 Accuracy 0.2998\n",
      "Epoch 11 Batch 600 Loss 0.8559 Accuracy 0.2997\n",
      "Epoch 11 Batch 650 Loss 0.8555 Accuracy 0.2998\n",
      "Epoch 11 Batch 700 Loss 0.8547 Accuracy 0.3000\n",
      "Epoch 11 Loss 0.8547 Accuracy 0.3000\n",
      "Time taken for 1 epoch: 71.3841483592987 secs\n",
      "\n",
      "Epoch 12 Batch 0 Loss 0.8170 Accuracy 0.3691\n",
      "Epoch 12 Batch 50 Loss 0.8343 Accuracy 0.3070\n",
      "Epoch 12 Batch 100 Loss 0.8265 Accuracy 0.3053\n",
      "Epoch 12 Batch 150 Loss 0.8160 Accuracy 0.3071\n",
      "Epoch 12 Batch 200 Loss 0.8123 Accuracy 0.3075\n",
      "Epoch 12 Batch 250 Loss 0.8103 Accuracy 0.3078\n",
      "Epoch 12 Batch 300 Loss 0.8091 Accuracy 0.3078\n",
      "Epoch 12 Batch 350 Loss 0.8073 Accuracy 0.3078\n",
      "Epoch 12 Batch 400 Loss 0.8062 Accuracy 0.3087\n",
      "Epoch 12 Batch 450 Loss 0.8057 Accuracy 0.3090\n",
      "Epoch 12 Batch 500 Loss 0.8036 Accuracy 0.3084\n",
      "Epoch 12 Batch 550 Loss 0.8047 Accuracy 0.3083\n",
      "Epoch 12 Batch 600 Loss 0.8041 Accuracy 0.3081\n",
      "Epoch 12 Batch 650 Loss 0.8026 Accuracy 0.3082\n",
      "Epoch 12 Batch 700 Loss 0.8024 Accuracy 0.3082\n",
      "Epoch 12 Loss 0.8022 Accuracy 0.3082\n",
      "Time taken for 1 epoch: 60.4772629737854 secs\n",
      "\n",
      "Epoch 13 Batch 0 Loss 0.8153 Accuracy 0.3560\n",
      "Epoch 13 Batch 50 Loss 0.7856 Accuracy 0.3117\n",
      "Epoch 13 Batch 100 Loss 0.7813 Accuracy 0.3116\n",
      "Epoch 13 Batch 150 Loss 0.7707 Accuracy 0.3123\n",
      "Epoch 13 Batch 200 Loss 0.7670 Accuracy 0.3136\n",
      "Epoch 13 Batch 250 Loss 0.7626 Accuracy 0.3132\n",
      "Epoch 13 Batch 300 Loss 0.7616 Accuracy 0.3133\n",
      "Epoch 13 Batch 350 Loss 0.7592 Accuracy 0.3137\n",
      "Epoch 13 Batch 400 Loss 0.7584 Accuracy 0.3139\n",
      "Epoch 13 Batch 450 Loss 0.7560 Accuracy 0.3140\n",
      "Epoch 13 Batch 500 Loss 0.7570 Accuracy 0.3141\n",
      "Epoch 13 Batch 550 Loss 0.7575 Accuracy 0.3140\n",
      "Epoch 13 Batch 600 Loss 0.7563 Accuracy 0.3136\n",
      "Epoch 13 Batch 650 Loss 0.7552 Accuracy 0.3137\n",
      "Epoch 13 Batch 700 Loss 0.7540 Accuracy 0.3137\n",
      "Epoch 13 Loss 0.7540 Accuracy 0.3138\n",
      "Time taken for 1 epoch: 65.22053170204163 secs\n",
      "\n",
      "Epoch 14 Batch 0 Loss 0.7302 Accuracy 0.3706\n",
      "Epoch 14 Batch 50 Loss 0.7322 Accuracy 0.3183\n",
      "Epoch 14 Batch 100 Loss 0.7276 Accuracy 0.3170\n",
      "Epoch 14 Batch 150 Loss 0.7221 Accuracy 0.3184\n",
      "Epoch 14 Batch 200 Loss 0.7230 Accuracy 0.3198\n",
      "Epoch 14 Batch 250 Loss 0.7214 Accuracy 0.3197\n",
      "Epoch 14 Batch 300 Loss 0.7195 Accuracy 0.3200\n",
      "Epoch 14 Batch 350 Loss 0.7161 Accuracy 0.3198\n",
      "Epoch 14 Batch 400 Loss 0.7158 Accuracy 0.3200\n",
      "Epoch 14 Batch 450 Loss 0.7139 Accuracy 0.3201\n",
      "Epoch 14 Batch 500 Loss 0.7152 Accuracy 0.3201\n",
      "Epoch 14 Batch 550 Loss 0.7146 Accuracy 0.3198\n",
      "Epoch 14 Batch 600 Loss 0.7133 Accuracy 0.3196\n",
      "Epoch 14 Batch 650 Loss 0.7116 Accuracy 0.3193\n",
      "Epoch 14 Batch 700 Loss 0.7126 Accuracy 0.3196\n",
      "Epoch 14 Loss 0.7127 Accuracy 0.3197\n",
      "Time taken for 1 epoch: 65.89196228981018 secs\n",
      "\n",
      "Epoch 15 Batch 0 Loss 0.6992 Accuracy 0.3799\n",
      "Epoch 15 Batch 50 Loss 0.7055 Accuracy 0.3246\n",
      "Epoch 15 Batch 100 Loss 0.6949 Accuracy 0.3231\n",
      "Epoch 15 Batch 150 Loss 0.6859 Accuracy 0.3244\n",
      "Epoch 15 Batch 200 Loss 0.6841 Accuracy 0.3255\n",
      "Epoch 15 Batch 250 Loss 0.6850 Accuracy 0.3260\n",
      "Epoch 15 Batch 300 Loss 0.6844 Accuracy 0.3258\n",
      "Epoch 15 Batch 350 Loss 0.6823 Accuracy 0.3261\n",
      "Epoch 15 Batch 400 Loss 0.6808 Accuracy 0.3257\n",
      "Epoch 15 Batch 450 Loss 0.6813 Accuracy 0.3260\n",
      "Epoch 15 Batch 500 Loss 0.6813 Accuracy 0.3258\n",
      "Epoch 15 Batch 550 Loss 0.6810 Accuracy 0.3254\n",
      "Epoch 15 Batch 600 Loss 0.6799 Accuracy 0.3247\n",
      "Epoch 15 Batch 650 Loss 0.6796 Accuracy 0.3248\n",
      "Epoch 15 Batch 700 Loss 0.6804 Accuracy 0.3248\n",
      "Saving checkpoint for epoch 15 at ./checkpoints/train/ckpt-3\n",
      "Epoch 15 Loss 0.6803 Accuracy 0.3249\n",
      "Time taken for 1 epoch: 59.448163747787476 secs\n",
      "\n",
      "Epoch 16 Batch 0 Loss 0.6580 Accuracy 0.4004\n",
      "Epoch 16 Batch 50 Loss 0.6697 Accuracy 0.3304\n",
      "Epoch 16 Batch 100 Loss 0.6597 Accuracy 0.3289\n",
      "Epoch 16 Batch 150 Loss 0.6522 Accuracy 0.3297\n",
      "Epoch 16 Batch 200 Loss 0.6511 Accuracy 0.3303\n",
      "Epoch 16 Batch 250 Loss 0.6515 Accuracy 0.3308\n",
      "Epoch 16 Batch 300 Loss 0.6510 Accuracy 0.3304\n",
      "Epoch 16 Batch 350 Loss 0.6496 Accuracy 0.3305\n",
      "Epoch 16 Batch 400 Loss 0.6503 Accuracy 0.3307\n",
      "Epoch 16 Batch 450 Loss 0.6507 Accuracy 0.3312\n",
      "Epoch 16 Batch 500 Loss 0.6504 Accuracy 0.3309\n",
      "Epoch 16 Batch 550 Loss 0.6499 Accuracy 0.3305\n",
      "Epoch 16 Batch 600 Loss 0.6496 Accuracy 0.3300\n",
      "Epoch 16 Batch 650 Loss 0.6487 Accuracy 0.3296\n",
      "Epoch 16 Batch 700 Loss 0.6483 Accuracy 0.3296\n",
      "Epoch 16 Loss 0.6482 Accuracy 0.3297\n",
      "Time taken for 1 epoch: 71.1401207447052 secs\n",
      "\n",
      "Epoch 17 Batch 0 Loss 0.5888 Accuracy 0.3522\n",
      "Epoch 17 Batch 50 Loss 0.6391 Accuracy 0.3324\n",
      "Epoch 17 Batch 100 Loss 0.6328 Accuracy 0.3320\n",
      "Epoch 17 Batch 150 Loss 0.6243 Accuracy 0.3329\n",
      "Epoch 17 Batch 200 Loss 0.6249 Accuracy 0.3343\n",
      "Epoch 17 Batch 250 Loss 0.6251 Accuracy 0.3345\n",
      "Epoch 17 Batch 300 Loss 0.6237 Accuracy 0.3347\n",
      "Epoch 17 Batch 350 Loss 0.6223 Accuracy 0.3350\n",
      "Epoch 17 Batch 400 Loss 0.6216 Accuracy 0.3349\n",
      "Epoch 17 Batch 450 Loss 0.6205 Accuracy 0.3350\n",
      "Epoch 17 Batch 500 Loss 0.6203 Accuracy 0.3350\n",
      "Epoch 17 Batch 550 Loss 0.6216 Accuracy 0.3345\n",
      "Epoch 17 Batch 600 Loss 0.6219 Accuracy 0.3340\n",
      "Epoch 17 Batch 650 Loss 0.6211 Accuracy 0.3338\n",
      "Epoch 17 Batch 700 Loss 0.6211 Accuracy 0.3336\n",
      "Epoch 17 Loss 0.6211 Accuracy 0.3337\n",
      "Time taken for 1 epoch: 59.676570892333984 secs\n",
      "\n",
      "Epoch 18 Batch 0 Loss 0.5836 Accuracy 0.4004\n",
      "Epoch 18 Batch 50 Loss 0.6096 Accuracy 0.3377\n",
      "Epoch 18 Batch 100 Loss 0.6033 Accuracy 0.3366\n",
      "Epoch 18 Batch 150 Loss 0.5976 Accuracy 0.3371\n",
      "Epoch 18 Batch 200 Loss 0.5968 Accuracy 0.3389\n",
      "Epoch 18 Batch 250 Loss 0.5973 Accuracy 0.3387\n",
      "Epoch 18 Batch 300 Loss 0.5961 Accuracy 0.3385\n",
      "Epoch 18 Batch 350 Loss 0.5961 Accuracy 0.3389\n",
      "Epoch 18 Batch 400 Loss 0.5953 Accuracy 0.3390\n",
      "Epoch 18 Batch 450 Loss 0.5945 Accuracy 0.3391\n",
      "Epoch 18 Batch 500 Loss 0.5957 Accuracy 0.3392\n",
      "Epoch 18 Batch 550 Loss 0.5956 Accuracy 0.3386\n",
      "Epoch 18 Batch 600 Loss 0.5955 Accuracy 0.3379\n",
      "Epoch 18 Batch 650 Loss 0.5949 Accuracy 0.3377\n",
      "Epoch 18 Batch 700 Loss 0.5954 Accuracy 0.3379\n",
      "Epoch 18 Loss 0.5954 Accuracy 0.3379\n",
      "Time taken for 1 epoch: 70.79017782211304 secs\n",
      "\n",
      "Epoch 19 Batch 0 Loss 0.5863 Accuracy 0.4009\n",
      "Epoch 19 Batch 50 Loss 0.5825 Accuracy 0.3405\n",
      "Epoch 19 Batch 100 Loss 0.5787 Accuracy 0.3403\n",
      "Epoch 19 Batch 150 Loss 0.5740 Accuracy 0.3409\n",
      "Epoch 19 Batch 200 Loss 0.5746 Accuracy 0.3419\n",
      "Epoch 19 Batch 250 Loss 0.5729 Accuracy 0.3420\n",
      "Epoch 19 Batch 300 Loss 0.5726 Accuracy 0.3418\n",
      "Epoch 19 Batch 350 Loss 0.5719 Accuracy 0.3419\n",
      "Epoch 19 Batch 400 Loss 0.5721 Accuracy 0.3418\n",
      "Epoch 19 Batch 450 Loss 0.5720 Accuracy 0.3417\n",
      "Epoch 19 Batch 500 Loss 0.5720 Accuracy 0.3414\n",
      "Epoch 19 Batch 550 Loss 0.5723 Accuracy 0.3414\n",
      "Epoch 19 Batch 600 Loss 0.5727 Accuracy 0.3408\n",
      "Epoch 19 Batch 650 Loss 0.5725 Accuracy 0.3407\n",
      "Epoch 19 Batch 700 Loss 0.5722 Accuracy 0.3406\n",
      "Epoch 19 Loss 0.5721 Accuracy 0.3407\n",
      "Time taken for 1 epoch: 65.01259756088257 secs\n",
      "\n",
      "Epoch 20 Batch 0 Loss 0.5533 Accuracy 0.4038\n",
      "Epoch 20 Batch 50 Loss 0.5625 Accuracy 0.3459\n",
      "Epoch 20 Batch 100 Loss 0.5565 Accuracy 0.3445\n",
      "Epoch 20 Batch 150 Loss 0.5500 Accuracy 0.3443\n",
      "Epoch 20 Batch 200 Loss 0.5508 Accuracy 0.3450\n",
      "Epoch 20 Batch 250 Loss 0.5506 Accuracy 0.3446\n",
      "Epoch 20 Batch 300 Loss 0.5502 Accuracy 0.3449\n",
      "Epoch 20 Batch 350 Loss 0.5491 Accuracy 0.3454\n",
      "Epoch 20 Batch 400 Loss 0.5490 Accuracy 0.3460\n",
      "Epoch 20 Batch 450 Loss 0.5492 Accuracy 0.3460\n",
      "Epoch 20 Batch 500 Loss 0.5493 Accuracy 0.3456\n",
      "Epoch 20 Batch 550 Loss 0.5505 Accuracy 0.3452\n",
      "Epoch 20 Batch 600 Loss 0.5508 Accuracy 0.3447\n",
      "Epoch 20 Batch 650 Loss 0.5515 Accuracy 0.3448\n",
      "Epoch 20 Batch 700 Loss 0.5519 Accuracy 0.3445\n",
      "Saving checkpoint for epoch 20 at ./checkpoints/train/ckpt-4\n",
      "Epoch 20 Loss 0.5517 Accuracy 0.3445\n",
      "Time taken for 1 epoch: 64.86167097091675 secs\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(EPOCHS):\n",
    "    start = time.time()\n",
    "  \n",
    "    train_loss.reset_states()\n",
    "    train_accuracy.reset_states()\n",
    "  \n",
    "    # inp -> portuguese, tar -> english\n",
    "    for (batch, (inp, tar)) in enumerate(train_dataset):\n",
    "        train_step(inp, tar)\n",
    "    \n",
    "        if batch % 50 == 0:\n",
    "            print ('Epoch {} Batch {} Loss {:.4f} Accuracy {:.4f}'.format(\n",
    "                epoch + 1, batch, train_loss.result(), train_accuracy.result()))\n",
    "      \n",
    "    if (epoch + 1) % 5 == 0:\n",
    "        ckpt_save_path = ckpt_manager.save()\n",
    "        print ('Saving checkpoint for epoch {} at {}'.format(epoch+1,\n",
    "                                                             ckpt_save_path))\n",
    "    \n",
    "    print ('Epoch {} Loss {:.4f} Accuracy {:.4f}'.format(epoch + 1, \n",
    "                                                  train_loss.result(), \n",
    "                                                  train_accuracy.result()))\n",
    "\n",
    "    print ('Time taken for 1 epoch: {} secs\\n'.format(time.time() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QfcsSWswSdGV"
   },
   "source": [
    "## Evaluate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "y6APsFrgImLW"
   },
   "source": [
    "The following steps are used for evaluation:\n",
    "\n",
    "* Encode the input sentence using the Portuguese tokenizer (`tokenizer_pt`). Moreover, add the start and end token so the input is equivalent to what the model is trained with. This is the encoder input.\n",
    "* The decoder input is the `start token == tokenizer_en.vocab_size`.\n",
    "* Calculate the padding masks and the look ahead masks.\n",
    "* The `decoder` then outputs the predictions by looking at the `encoder output` and its own output (self-attention).\n",
    "* Select the last word and calculate the argmax of that.\n",
    "* Concatentate the predicted word to the decoder input as pass it to the decoder.\n",
    "* In this approach, the decoder predicts the next word based on the previous words it predicted.\n",
    "\n",
    "Note: The model used here has less capacity to keep the example relatively faster so the predictions maybe less right. To reproduce the results in the paper, use the entire dataset and base transformer model or transformer XL, by changing the hyperparameters above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5buvMlnvyrFm"
   },
   "outputs": [],
   "source": [
    "def evaluate(inp_sentence):\n",
    "    start_token = [tokenizer_pt.vocab_size]\n",
    "    end_token = [tokenizer_pt.vocab_size + 1]\n",
    "  \n",
    "    # inp sentence is portuguese, hence adding the start and end token\n",
    "    inp_sentence = start_token + tokenizer_pt.encode(inp_sentence) + end_token\n",
    "    encoder_input = tf.expand_dims(inp_sentence, 0)\n",
    "  \n",
    "    # as the target is english, the first word to the transformer should be the\n",
    "    # english start token.\n",
    "    decoder_input = [tokenizer_en.vocab_size]\n",
    "    output = tf.expand_dims(decoder_input, 0)\n",
    "    \n",
    "    for i in range(MAX_LENGTH):\n",
    "        enc_padding_mask, combined_mask, dec_padding_mask = create_masks(\n",
    "            encoder_input, output)\n",
    "  \n",
    "        # predictions.shape == (batch_size, seq_len, vocab_size)\n",
    "        predictions, attention_weights = transformer(encoder_input, \n",
    "                                                     output,\n",
    "                                                     False,\n",
    "                                                     enc_padding_mask,\n",
    "                                                     combined_mask,\n",
    "                                                     dec_padding_mask)\n",
    "    \n",
    "        # select the last word from the seq_len dimension\n",
    "        predictions = predictions[: ,-1:, :]  # (batch_size, 1, vocab_size)\n",
    "\n",
    "        predicted_id = tf.cast(tf.argmax(predictions, axis=-1), tf.int32)\n",
    "    \n",
    "        # return the result if the predicted_id is equal to the end token\n",
    "        if tf.equal(predicted_id, tokenizer_en.vocab_size+1):\n",
    "            return tf.squeeze(output, axis=0), attention_weights\n",
    "    \n",
    "        # concatentate the predicted_id to the output which is given to the decoder\n",
    "        # as its input.\n",
    "        output = tf.concat([output, predicted_id], axis=-1)\n",
    "\n",
    "    return tf.squeeze(output, axis=0), attention_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CN-BV43FMBej"
   },
   "outputs": [],
   "source": [
    "def plot_attention_weights(attention, sentence, result, layer):\n",
    "    fig = plt.figure(figsize=(16, 8))\n",
    "  \n",
    "    sentence = tokenizer_pt.encode(sentence)\n",
    "  \n",
    "    attention = tf.squeeze(attention[layer], axis=0)\n",
    "  \n",
    "    for head in range(attention.shape[0]):\n",
    "        ax = fig.add_subplot(2, 4, head+1)\n",
    "    \n",
    "        # plot the attention weights\n",
    "        ax.matshow(attention[head][:-1, :], cmap='viridis')\n",
    "\n",
    "        fontdict = {'fontsize': 10}\n",
    "    \n",
    "        ax.set_xticks(range(len(sentence)+2))\n",
    "        ax.set_yticks(range(len(result)))\n",
    "    \n",
    "        ax.set_ylim(len(result)-1.5, -0.5)\n",
    "        \n",
    "        ax.set_xticklabels(\n",
    "            ['<start>']+[tokenizer_pt.decode([i]) for i in sentence]+['<end>'], \n",
    "            fontdict=fontdict, rotation=90)\n",
    "    \n",
    "        ax.set_yticklabels([tokenizer_en.decode([i]) for i in result \n",
    "                            if i < tokenizer_en.vocab_size], \n",
    "                           fontdict=fontdict)\n",
    "    \n",
    "        ax.set_xlabel('Head {}'.format(head+1))\n",
    "  \n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lU2_yG_vBGza"
   },
   "outputs": [],
   "source": [
    "def translate(sentence, plot=''):\n",
    "    result, attention_weights = evaluate(sentence)\n",
    "  \n",
    "    predicted_sentence = tokenizer_en.decode([i for i in result \n",
    "                                              if i < tokenizer_en.vocab_size])  \n",
    "\n",
    "    print('Input: {}'.format(sentence))\n",
    "    print('Predicted translation: {}'.format(predicted_sentence))\n",
    "  \n",
    "    if plot:\n",
    "        plot_attention_weights(attention_weights, sentence, result, plot)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YsxrAlvFG8SZ"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: este é um problema que temos que resolver.\n",
      "Predicted translation: this is a problem that we have to do with a problem with it .\n",
      "Real translation: this is a problem we have to solve .\n"
     ]
    }
   ],
   "source": [
    "translate(\"este é um problema que temos que resolver.\")\n",
    "print (\"Real translation: this is a problem we have to solve .\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7EH5y_aqI4t1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: os meus vizinhos ouviram sobre esta ideia.\n",
      "Predicted translation: my neighbors heard of this idea is over .\n",
      "Real translation: and my neighboring homes heard about this idea .\n"
     ]
    }
   ],
   "source": [
    "translate(\"os meus vizinhos ouviram sobre esta ideia.\")\n",
    "print (\"Real translation: and my neighboring homes heard about this idea .\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "J-hVCTSUMlkb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: vou então muito rapidamente partilhar convosco algumas histórias de algumas coisas mágicas que aconteceram.\n",
      "Predicted translation: so i 'm going to really share with you some stories of some magic things that happened .\n",
      "Real translation: so i 'll just share with you some stories very quickly of some magical things that have happened .\n"
     ]
    }
   ],
   "source": [
    "translate(\"vou então muito rapidamente partilhar convosco algumas histórias de algumas coisas mágicas que aconteceram.\")\n",
    "print (\"Real translation: so i 'll just share with you some stories very quickly of some magical things that have happened .\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_1MxkSZvz0jX"
   },
   "source": [
    "You can pass different layers and attention blocks of the decoder to the `plot` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "t-kFyiOLH0xg"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: este é o primeiro livro que eu fiz.\n",
      "Predicted translation: this is the first book i did it .\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABHgAAAItCAYAAABCYc09AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzde5xkdXnv+88z3TPMADMiOIri5uotaMDBOSqIhmi8xGPOCUZf7pDseMeEeJe4JXHHaGKOMdnusyUxiiIaNVE0eoJJQOKFi6AotwFRMEbBuwZvjAPMpec5f9Tq6ZqmZ+juWr+1+lf9eb9e9aqu1dXPerq66turn15rVWQmkiRJkiRJqteKvhuQJEmSJEnSaBzwSJIkSZIkVc4BjyRJkiRJUuUc8EiSJEmSJFXOAY8kSZIkSVLlHPBIkiRJkiRVzgGPJEmSJElS5RzwSJIkSZIkVc4BjyRJkiRJUuUc8EiSJEmSJFVusu8GtPRFxLHAY5ubl2bmpj77kTTezBxJXTJzJHXN3FEp7sGjvYqIlwEfAO7dXN4fES/ptytJ48rMkdQlM0dS18wdlRSZ2XcPWsIi4jrg+Mzc0tzeD/hcZh7Tb2eSxpGZI6lLZo6krpk7Ksk9eHR3Apgauj3VLJOkEswcSV0ycyR1zdxRMWNzDp6ICOBjwBmZ+ZW++xkj5wBXRMTHmtu/DpzdYz/SkmHuFGHmSHtg5hRh5kh7YOYUY+6omLE5RCsinszghfGhzHxV3/2Mk4g4DjiRwWT5ksy8pueWpCXB3CnDzJHmZuaUYeZIczNzyjF3VMo4DXjOBd4NvBU4OjN39NxS9SJiBXBdZj6s716kpcjcaZeZI+2dmdMuM0faOzOnfeaOShuLc/BExL2Ah2bmBcAngZN7bmksZOZOYFNEHNp3L9JSY+60z8yR9szMaZ+ZI+2ZmVOGuaPSxmLAA/wO8A/Nx+cAz++xl3FzX+CGiPhURJw3fem7KfUrIk6OiP377qNn5k4ZZo7uwswBzJxSzBzdhZkDmDklmTu6i7ZyZywO0YqI64GnZOZ3mtubgKdl5rf67ax+EfFLcy3PzIu77kVLQ0QcBdwIvCQz3953P30xd8owczSbmTNg5pRh5mg2M2fAzCnH3NFsbeZO9QOeiDgAeFZmvmNo2ROBWz1ZldS+iHgjkMCTMvORfffTB3NH6o6ZY+ZIXTJzzBypa23mTvWHaGXmT4EvzVr2b8C+/XQ0HiLis8315oi4beiyOSJu67s/9SMiJoBnAn8B/Cwiju25pV6YO+0zczQXM2fAzGmfmaO5mDkDZk4Z5o7m0nbuVD/gaZw5z2Wap8w8sblem5nrhi5rM3Nd3/2pN08FLs/MzQzeVeEFPffTJ3OnRWaO9sDMmWHmtMjM0R6YOTPMnJaZO9qDVnNnspWWehIRxwMnAOsj4pVDn1oHTPTT1fiJiBOBB2bmOc0Z9ddm5jf67ku9eD7wP5uPPwb8WUS8KjO39dhTp8yd8swcDTFzzJzizBwNMXPMnE6YOxrSau7UvgfPKmB/BoOqtUOX24Bn9NjX2IiI1wH/HTijWbQKeH9/HakvzfHYB2TmpQCZeSfwEeDxvTbWPXOnIDNH08ycXcycgswcTTNzdjFzCjN3NK1E7ozDSZYngA9lpoFTQERcC2wArs7MDc2y6zLzmH47k/pj7pRj5kh3ZeaUY+ZId2XmlGXuqKSqD9ECyMypiDiw7z7G2LbMzIhIgIjYr++G1L2IOG5vn8/Mq7vqZSkwd4oyc2TmzGLmFGXmyMyZxcwpztxRsdypfsDTuCYizgM+DGyZXpiZH+2vpbFxbkS8AzggIl4IPA94Z889qXvTx4WuBjYCm4AAjgGuAE7sqa8+mTtlmDkCM2cuZk4ZZo7AzJmLmVOOuSMolDvVH6IFEBHnzLE4M/N5nTczhiLiicCTGDzhPtG8TaKWoYj4IPDGzLy+uf0w4PTMfE6vjfXA3CnHzNE0M2eGmVOOmaNpZs4MM6csc0fT2s6dsRjwqLyIWMfQHl+Z+eMe21FPIuLazHz43S2TRmXmCMwcdcfMEZg56pa5I2g/d8biEK2IWM3g7cUeymAXJwCcMI8uIl4EvAG4A9jJYMqcwJEt1F4FPKi5eVNmbh+1por7SkS8i8GZ/hP4beAr/bbUD3OnDDNHs5g5DTOnDDNHs5g5DTOnnFK5Y+ZUq9Xcqf1t0qe9DzgYeDJwMXB/YPMoBSPiPhFxdkSc39w+OiKeP3Kn9TkdeGhmHp6ZR2bmEZnZxkbPScC/A38DvA34akQ8btS6Ku65wA3Ay4CXA19uli1HreaOmbOLmaNhZs4Mt3XKMHM0zMyZYeaU03rumDlVazV3xuIQrYi4JjM3TL+9XESsZHAs4+LfP34QPOcAf5SZx0bEJHBNZv5iW33XICIuAJ6embe3XPcq4JTMvKm5/SDgHzLzEW2uRyql7dwxcwbMHGlubuuUYeZIczNzyimRO2aOpo3FIVrA9O5nP21OSvR94PARa94rM8+NiDMAMnNHREyNWLNGZwCXR8QVwNbphZn50hHrrpwOoKbeV5tfHCOJiMOAB2bmJyNiDTCZmSP9t0EzIuIxwJ8Ah7H7McMj/7ezQm3njpkzUFXmgLlTkpmzG7d1yjBztIuZsxszp5wSuWPmVKrt3BmXAc9ZEXFP4LXAecD+wP8YseaWiDiIwXFwRMSjgZ+NWLNG7wA+DVzP4BjRtlwZEWcz2P0T4LeAq0YpGIO3GTwVOBA4isGupG8HnjBKXe3mbOAVDH5Wy/EX8rC2c8fMGagmc8Dc6YCZM8NtnTLMHA0zc2aYOeWUyB0zp16t5s64HKJ1RGZ+4+6WLbDmccCZwMOALwHrgWdm5qaRmq1MRFyemScUqLsP8PvAiQxOLHYJ8LbM3LrXL9x7zWuBRwJXZOaGZtn1y223z5Ii4orMfFTffSwFbeeOmTNQU+Y0dc2dgsycGW7rlGHmaJiZM8PMKadE7pg59Wo7d8ZlwHN1Zh43a9lVoxxz2LxIpoAHM3iR3ASsGPVFUpuIeCNwC/Bxdt+FcNFv4xcRE8B7M/O3R+9wt7pXZOajho4ZngSuzsxj2lzPchYRbwImgI+y+/Ph6t6a6knbuWPmDNSUOU1tc6cgM2eG2zplmDkaZubMMHPKaTt3zJy6tZ07VR+iFREPYfDWffeIiKcPfWodQ2/nt0ifa0LthqH1XQ0ct+cvGUunNNdnDC0b6W38MnMqItZHxKrM3DZSd7u7OCL+EFgTEU8ETmMQnGrP9HR549CyBBZ9wr3aFMwdM2egpswBc6c0M8dtndLMHA0zc8ycLrSaO2ZO9VrNnaoHPAymv08DDgB+bWj5ZuCFiykYEQcDhzB4Em9gMF2GQajtu/hW65SZRxQqfTNwWUScB2wZWt9bRqj5GuD5DI5nfRHwr8C7RqinWTLzl/vuYQloNXfMnN1Vljlg7hRl5gBu6xRl5miYmQOYOcUVyp2bMXOq1HbujMshWsdn5udaqvVs4DkMJmhfZCaANgPvycyPtrGepS4iHp+Zn541ud9l1MchIl63h7qvH6WuyoqI+wB/DtwvM381Io4Gjs/Ms3turXNt5Y6ZM2DmaC5mzgy3ddpl5mguZs4MM6d9JXPHzKlX27kzLgOeNwN/BtwBXAAcC7w8M98/Qs3fyMx/bKnF6kTE6zPzdRFxzhyfzsx83oj1N2TmNaPUmKPmN2jOyj8sR3xry+YxmKvuSI9BjSLifOAc4I8y89jmONxrluOJ1trOHTOnvsxp6po7BZk5M9zWaZeZc5e6Zg5mzjAzp30lc8fMqVfbuVP7IVrTnpSZr46Ik4FvA88EPgMsOoCA+0fEOgaT5XcyODb0NZl54cjdVqAJnxXA+Zl5boFVvCUi7gt8GPhgZt5wd18wD8PHLa5m8Dw4sIW6/zyr7snAd0ctGhGPBS7PzKmhZcct8RP53Sszz42IMwAyc0dELNe3EW07d8yc+jIHzJ3SzJwZbuu0yMy5CzNnwMyZYea0rHDumDlUmTnQdu5kZvUX4Ibm+p3AU5qPN41Yc1Nz/WTgPAZT66v7/l57eGwvKVj7YOClwGUMjut8bYF1fLZAzRXAp1uocztwMXCfoWVL+jkGXAQcNN0n8Gjg4r776umxaDV3zJxdj0PVmdOsx9xp7/s2c2YeC7d1yjyuZs7cNc2cNHOaazOn/ce2SO6YOfVlTtNfq7kzLnvwfDwibmSwC+FpEbEeuHPEmtPHhj4VOCczN0VE7O0LxtS/RcTpwIfY/YRdi3770KEa3wfeGhGfAV4N/DGDXUEXJSKGz8C/gsHEee1ITc7tgcChLdS5CfhL4KKIeH5mXs7M826peiWDX8hHRcRlwHrgGf221Ju2c8fMGagmc8Dc6YCZM8NtnTLMnLmZOWaOmVNOkdwxc4D6Mgdazp2xOAcPQETcE7gtB28Ttx+wtnmSL7beOQzO9n4Eg+nyBHBRZj6ilYYr0Rx3OVvm6Mdd/gLwLAZP3h8BHwT+MTN/OELNzwzd3MHgbPJ/lZk3jdAqEbGZwTGi0Vx/HzgjRzyGOCKuzszjIuKBDAL+3cDzcvD2kUtWc1zogxk8Hjdl5vaeW+pNm7lj5gzUlDlNXXOnMDNnhts67TNzdtU1cxpmzgwzp4wSuWPm7KpbXeZAu7lT/YAnIvYFHpiZm4aWHQpMZeZ3Rqi7Ang48PXM/GlEHAQckpnXjdy0iIjPA/8AfDgzRz7eskYRcU1mbmg+3o/BybWenplLcs+6Uq+1GpV4LMycssycgZpyx8yZ4bZOfcycATOnTmZOfcycgZoyBwr9TTEGA56VwI3AMZm5pVl2IfCHmXnlCHUD+C3gyMx8Q/NAH5yZXxix3yJ1m9rHAo9tbl46/EQZoeZq4DTgRAbT1UuBt2fmqLtoti4iXrm3z2fmWxZZd/pndkRm/mmbP7M51nVoZn6z7bptKPVaq1GJx8LM2VWzmswBc6ckM2eG2zq76po5Zk4xZs4MM2e32ss6d8ycskq81la02F8vmt2XPsZgl7Tpidf6FoL4bcDxwG82tzcDf7OYQhHxmIiYaLvurHW8DPgAcO/m8v6IeMmodYG/Ax4KnAn8NXA08L4R+jy3ub4+Iq4bulwfEaNO7zcCv8dg189DgN9t+l3LaMeKTv/MTmluj/Qzi4hXN9dnRsRbhy/A6SP0WVTB11p1Cj0WZs5ATZkD5k4xZs4Mt3XMnCFmTiFmzgwzZ9c6lnzumDkDNWYOFHqt5RI4c/SoF+AhDCaqAK8FXtpCzemzWF8ztGxRZ44HTgDOarvurHVcB+w3dHs/4LoW6t6lt1H6Be7bXB8212XEXi9kcGzw9O21wAVL6bnQfO2PmuuXA8+efRm135KXEq+1Wi9tPxZmzp57W6qZ09Q1dwpezJyyj0VNuWPm7Kpv5hS8mDllH4uaMqepseRzx8zZ9bVVZk7Tc6uvtSV5LNpCZeaNEUFEPIjB5PbEFspub6bCCRCDM8fvXGR/l0fE7W3XnSWAqaHbU82yUV0TEY/OzM8DRMSjGLz13qJk5vea61ta6G22Q4FtQ7e3AYe3ULftn9kPIuIw4LnAL7fQX2cKvdaqVOCxMHMGasocMHeKMnNmuK1j5jTMnILMnBlmDlBB7pg5u1SZOdD+a20sBjyNs4F3MZiq/qSFem9lsLvUvSPijQzOSP7axRbLzGtL1B1yDnBFRHysuf3rDB6TUT0K+J2ImD5u8VDgKxFxPYOzvR+zkGIxc8b0u3yqqbduhF7fB3yheQwSOBl47wj1prX9M/tb4ALgSGB497vps8iP9M4dexIRB+cI73wwpO3XWs3afCzMnIGaMgfMnT0yc4pYzts6Zs6AmbMXLeWOmTNjOWcOVJA7Zs4uNWcOtPhaq/4ky9NicAbq7wG/kZmfbKnmQ4AnMHhifCozv7LE6x7HYOIXwCWZeU0LNQ/b2+cLTosXpXkMpk+E1spj0NRt/WcWEX+bmb83cnPzX9+/ZOb/2UKd1l9rtWr7sTBz6sscMHf2si4zp2XLfVvHzBkwc/a6vpFzx8yZsdwzp6m77HPHzNnr+pbcts7YDHgkSZIkSZKWq+rfRUuSJEmSJGm5c8AjSZIkSZJUubEb8ETEqda1bqma1i1ft0Y+f61bY92aei1Zt0a1PcY+f61bY10zZ0ZNPzfrlqtp3Trqjt2ABygVxtatr25NvVq3bj5/rVtj3Zp6LVm3RrU9xj5/rVtjXTNnRk0/N+uWq2ndCuqO44BHkiRJkiRpWaniXbRWxT65mv3mdd/tbGUl+8zrvrlu33n3sH3bFlauml8P9z50/m9df9uPd7DuwMl53fc/vzS/7wsW9jgsRE11a+p13Otu5ie3Zub61psoaNXEmlwzeY953XfbzttZtWJ+ebL1/hPzut/UbVuYWDe/zAHY59s75nW/bVN3sGpizbzrElGkbm7dNq/7LeR5Fivnl6UA23bewaoV8+s3d8zvsQXYnltZGfN8vS3g168ZubC6d7KFbbl1fk/eJWLVijW5ZnLtvO67kOcvE/PLHIBtU7ezamKe20bbt8+/7s47WbVi9bzvX6Jmrpnfc2fb9i2sWjm/7N160PyfYlObtzCxdn5197nl9nnXXVBGrpj//3W35Z2sivk9vrlz57zrLoV8KFGzyswp9PfV1sPn//fVQl4XsW0Br7ctW5jYb351V31vy7zr+jveukup7p7+vpr/1nCPVrMfj4ontF532wkbW68J8OIzzy1S96wHHVmkrlTaJ/Mjt/Tdw0KtmbwHJ9z/t1uv+/U3r2u9JsCRp/+0SN1ctbJI3amvfaP1mpPr79N6TYCp/7y1SN2FDI4WZJ5DuQUr9Q+hAv1esfOTrdcsbc3kWk5Y/6zW6+YB8xsaLdh3flCmbiHbNxzVes2vnVJmM/pBv3tlkbor1ixguL8AO+/cWqQuO6fK1C3givxU3y0sWKm/r776ujJ/X636TpntkSNe/8UidYv9jpcae/r7ykO0JEmSJEmSKueAR5IkSZIkqXIOeCRJkiRJkirngEeSJEmSJKlyDngkSZIkSZIqt6gBT0QcEBGnDd0+KSL+eQ/3fVdEHL3YBiXJzJHUJTNHUtfMHUltWOwePAcAp93tvYDMfEFmfnmR65EkMHMkdcvMkdQ1c0fSyBY74HkTcFREXBsRf9ks2z8iPhIRN0bEByIiACLioojYGBETEfGeiPhSRFwfEa9o5TuQtByYOZK6ZOZI6pq5I2lkk4v8utcAD8vMh8NgF0JgA/BQ4LvAZcBjgM8Ofc3DgUMy82HN1xywyHVLWn7MHEldMnMkdc3ckTSyNk+y/IXM/HZm7gSuBQ6f9fmvA0dGxJkR8RTgtr0Vi4hTI+LKiLhyO1tbbFPSmGg1c2D33Nm28/b2O5ZUs8KZc0f7HUuqnX9fSVqQNgc8wykxxay9gzLzJ8CxwEXA7wPv2luxzDwrMzdm5saV7NNim5LGRKuZ03zNrtxZtWLfFluVNAYKZ86aFluVNCb8+0rSgiz2EK3NwNqFfEFE3AvYlpn/GBH/AbxnkeuWtPyYOZK6ZOZI6pq5I2lkixrwZOaPIuKyiPgScD7wL/P4skOAcyJieq+hMxazbknLj5kjqUtmjqSumTuS2rDYPXjIzFNmLbpo6HMvHvr4pKH7HLfY9Ula3swcSV0ycyR1zdyRNKo2z8EjSZIkSZKkHjjgkSRJkiRJqpwDHkmSJEmSpMo54JEkSZIkSaqcAx5JkiRJkqTKOeCRJEmSJEmq3KLfJr1LsWolk/f7L+0XvvCq9msCZz/mUUXqTtxzR5G6P3jmQ1qvufnw1ksCcMQffb5I3RX771+kbkyWeYlN/eQnReoSUaZulilbUm7bxo5v3NJ63cP/2+rWawI889qvF6n7wWOPKFK3xGsut9zeek2A3FEme2PlqiJ1yZ1lyhZ6HMgKA6KEhNwx1X7d79/afk0gDjygSF3u3Fqk7MRt21qvGVsLvYajzP9fY02Z3z8Ta8tsQ0394IdF6mpIge2+h7zsptZrArz7hvOL1H3O636pSF0VVOzvlfHYHnEPHkmSJEmSpMo54JEkSZIkSaqcAx5JkiRJkqTKOeCRJEmSJEmqnAMeSZIkSZKkyjngkSRJkiRJqlzxAU9EXF56HZI0zcyR1DVzR1KXzBxJe1J8wJOZJ5RehyRNM3Mkdc3ckdQlM0fSnnSxB8/Pm+v7RsQlEXFtRHwpIh5bet2Slh8zR1LXzB1JXTJzJO3JZIfrOgX4RGa+MSImgH07XLek5cfMkdQ1c0dSl8wcSbvpcsDzReDdEbES+P8y89q93TkiTgVOBVg9sbaD9iSNmQVlDszKHbeRJC3c4rd1VuzfQXuSxsziM8ftHGksdfYuWpl5CfA44DvA+yLid+7m/mdl5sbM3LhqYk0nPUoaHwvNnOZrduXOSvYp3qOk8TLSts4Kt3UkLcwomeN2jjSeOhvwRMRhwA8z853A2cBxXa1b0vJj5kjqmrkjqUtmjqTZujxE6yTgDyJiO/Bz4G7/my5JIzgJM0dSt07C3JHUnZMwcyQNKT7gycz9m+v3Au8tvT5Jy5uZI6lr5o6kLpk5kvaks0O0JEmSJEmSVIYDHkmSJEmSpMo54JEkSZIkSaqcAx5JkiRJkqTKOeCRJEmSJEmqnAMeSZIkSZKkyhV/m/Q25PYdTH3/h63XXbHPPq3XBJi69dYidScedFSRuj9+9PbWax79hvZ/XgBTkyuL1N3yhF8oUnfNeVcVqUtEmbqZZepWKFatYvKQQ1uvu/nhB7deE+CDx5b52d3xpGOL1N3vCze3XvN7v/GA1msC3Pttlxepm9u3FamrWiXsnGq/6tatrdcEmLrlp0XqTtx7fZG62w9c3XrNez/gR63XBCB3Fim782e3FambO3YUqauyYuVKJu9z3wKFy2yjPveoxxep++TryvzdduHDD2y9Zk61/zuiqFJ/V/j3yl65B48kSZIkSVLlHPBIkiRJkiRVzgGPJEmSJElS5RzwSJIkSZIkVc4BjyRJkiRJUuUc8EiSJEmSJFVupAFPRBwQEacN3T4pIv559LYk6a7MHEldMnMkdc3ckTSKUffgOQA47W7vJUntMHMkdcnMkdQ1c0fSoo064HkTcFREXBsRf9ks2z8iPhIRN0bEByIiACLiERFxcURcFRGfiIj7jrhuScuPmSOpS2aOpK6ZO5IWbdQBz2uA/8jMh2fmHzTLNgAvB44GjgQeExErgTOBZ2TmI4B3A28ccd2Slh8zR1KXzBxJXTN3JC3aZIGaX8jMbwNExLXA4cBPgYcB/9YMnCeA7+2tSEScCpwKsJp9C7QpaUy0kjnN18/kzsTaQu1KqlyZzFmxf6F2JY2B9v++cjtHGkslBjxbhz6eatYRwA2Zefx8i2TmWcBZAOtWHJStdihpnLSSObB77txjn4PNHUlzKZM5K9ebOZL2pPW/r+6x6j5mjjSGRj1EazMwn/HvTcD6iDgeICJWRsRDR1y3pOXHzJHUJTNHUtfMHUmLNtKAJzN/BFwWEV8aOgnYXPfbBjwD+IuI2ARcC5wwyrolLT9mjqQumTmSumbuSBrFyIdoZeYpsxZdNPS5Fw99fC3wuFHXJ2l5M3MkdcnMkdQ1c0fSYo16iJYkSZIkSZJ65oBHkiRJkiSpcg54JEmSJEmSKueAR5IkSZIkqXIOeCRJkiRJkirngEeSJEmSJKlyI79NeicyyW3bCpTN1muWlN/6bpG6f3Xiha3XfMchT2+9JkDc8q0iddf80xeL1KWy55hm5LZt7Lj5m63XXVOgJkCpZ9qaCzcVqfuR/7i09ZonH3p86zWlruSOKaZ+8rO+25i/KPM/wlhRpu5Br7+59ZoTf3D/1msCxR7bYj+zlauK1M0d24vUddtsILdvZ8d3yvxtUcSKiSJlL3x2mW2Hp2y6rPWan3jkIa3XBNi5ZUuRuuqHe/BIkiRJkiRVzgGPJEmSJElS5RzwSJIkSZIkVc4BjyRJkiRJUuUc8EiSJEmSJFXOAY8kSZIkSVLlFjzgiYiXRsRXIuIDEfF/RcRrFvC1h0fEKQtdp6TlzdyR1CUzR1KXzBxJbZlcxNecBvxqZn6juX3e7DtExGRm7pjjaw8HTgH+fhHrlbR8mTuSumTmSOqSmSOpFQsa8ETE24EjgfMi4t3AT4CNmfniiHgP8GNgA3B1RJwH/O/mSxN4HPAm4Bci4lrgvZn5v9r5NiSNK3NHUpfMHEldMnMktWlBA57M/N2IeArwy5l5a0Q8Z9ZdHgT8SmZORcTHgd/PzMsiYn/gTuA1wOmZ+bQ2mpc0/swdSV0ycyR1ycyR1Ka2T7L84cycaj6+DHhLRLwUOGAPuxTuUUScGhFXRsSV29nacpuSxoi5I6lLZo6kLpk5kuat7QHPlukPMvNNwAuANcDnI+IhCymUmWdl5sbM3LiSfVpuU9IYMXckdcnMkdQlM0fSvC3mJMvzEhFHZeb1wPURcTzwEOBbwNpS65S0vJk7krpk5kjqkpkj6e60vQfPsJdHxJciYhNwB3A+cB2wIyI2RcQrCq5b0vJk7kjqkpkjqUtmjqS9WvAePJl5+NDH7wHe03z8nFn3e8keSjxhoeuUtLyZO5K6ZOZI6pKZI6ktJffgkSRJkiRJUgcc8EiSJEmSJFXOAY8kSZIkSVLlHPBIkiRJkiRVzgGPJEmSJElS5RzwSJIkSZIkVW7Bb5Peiwhi1arWy+bWra3XLCm3bStS913HHN16zQu//p7WawI85bBHFqmbU1NF6hJlyrKzUL/SLKVy5+T7t/9a/sR3r2q9JsCTD9lQpC6ZZeqqSjExwcS6/VuvO/XTn7Ves6SpW39cpO7PHru99ZrvvuWfWq8J8IIjTypSt9i2jtskdYogVhb4+6rQ82zFmtVF6vLVbxYpe+H/3f62w5rzf956TYCtz1pbpO7Uf95apG7u2FGkbkyWGY2U6ndP3INHkiRJkiSpcg54JEmSJEmSKueAR5IkSZIkqXIOeCRJkiRJkirngEeSJEmSJKlyDngkSZIkSZIqt9cBT0QcHhFfamNFEXFzRNyrjVqSxpOZI6lr5o6kLpk5kkpyDx5JkiRJkqTKzWfAMxkR742I6yLiIxGxL0BEPCEiromI6zs6yXoAACAASURBVCPi3RGxz96WT4uINRFxQUS8sMD3I6l+Zo6krpk7krpk5kgqYj4DngcDZ2XmMcBtwGkRsRp4D/CszPxFYBL4vT0tH6q1P/Bx4O8z852tfReSxomZI6lr5o6kLpk5koqYz4DnW5l5WfPx+4ETGYTSNzLzq83y9wKP28vyaf8EnJOZf3d3K42IUyPiyoi4cnveOY82JY2JXjIHZuUOW0f9PiTVo/dtnW15Rxvfh6Q69J45/n0ljaf5DHhyjtuxh/vuafm0y4BfjYi7ux+ZeVZmbszMjStj9TzalDQmeskcmJU77HP3XyBpXPS+rbMq1syjTUljovfM8e8raTzNZ8BzaEQc33z8m8BngRuBwyPiAc3y/wZcvJfl0/4Y+BHwtlEblzS2zBxJXTN3JHXJzJFUxHwGPF8Bnh0R1wEHAn+bmXcCzwU+HBHXAzuBt+9p+ax6LwdWR8Sb2/omJI0VM0dS18wdSV0ycyQVMbm3T2bmzcDRe/jcp4ANC1h++NDN5y6kSUnLg5kjqWvmjqQumTmSSprPHjySJEmSJElawhzwSJIkSZIkVc4BjyRJkiRJUuUc8EiSJEmSJFXOAY8kSZIkSVLl9vouWktGJrl1a99d9C537Kim7pPv9/DWaw5sK1L1VV+7oUjdv3jR7xSpO/mpq4rU1Yxcty9bT/w/Wq+75uIyz7Wdt99epO7EgfcsUnfqRz9uveaxbz6t9ZoA97/fzUXq5gFri9Tlh+0/tgBZ6DlWQtxe4f+vJidg/UHt1/3Zbe3XBMgsVHdnkbIxMdF6zf/6yle1XhNgn8eV2d67Y/3KInXvccNPi9TNm75epC4rov2aWwvULC2TnJrqu4t5K7WdUyzLNm9uveSWx7VeEoDYZ58idaeO/8UidR/8li8Xqfu1Ewo9F6JQPuyh3Qq3gCRJkiRJkjTMAY8kSZIkSVLlHPBIkiRJkiRVzgGPJEmSJElS5RzwSJIkSZIkVc4BjyRJkiRJUuUc8EiSJEmSJFWulwFPRFzex3olLV/mjqQumTmSumTmSIKeBjyZeUIf65W0fJk7krpk5kjqkpkjCfrbg+fnfaxX0vJl7kjqkpkjqUtmjiSAyb4b2JOIOBU4FWA1+/bcjaTlYDh39ll9QM/dSBp3u23rTK7ruRtJ486/r6Txt2RPspyZZ2XmxszcuJJ9+m5H0jKwW+6s2q/vdiSNueHMWTWxpu92JI05/76Sxt+SHfBIkiRJkiRpfhzwSJIkSZIkVc4BjyRJkiRJUuX6epv0/ftYr6Tly9yR1CUzR1KXzBxJ4B48kiRJkiRJ1XPAI0mSJEmSVDkHPJIkSZIkSZVzwCNJkiRJklQ5BzySJEmSJEmVm+y7gfmINatZ8ZCjW6+789ovt16zqBUTfXcwbztP+MUidSeuKPMze+2fvqBI3cn1WaTuAfe8Z5G6rD+wTN2bypQtKW67nX3O/2LrdXe2XrGsqR/9uO8W5u3g//fyInV3FKkK8YMyv4Iv+OaVReo++ZANReqS7edkZm2vNMit25j696/33UbvckepV1z79vvIFX23sCD3WLu2SN3zb7q0SN1imVNCgRzrRImsrPWxWMZy69YidVdcdl2Run99yFVF6j55R0WZsxfuwSNJkiRJklQ5BzySJEmSJEmVc8AjSZIkSZJUOQc8kiRJkiRJlXPAI0mSJEmSVDkHPJIkSZIkSZVrfcATEX8SEac3H78hIn5ljvucFBH/3Pa6JS0/Zo6kLpk5krpk5khaiMmSxTPzj0vWl6RhZo6kLpk5krpk5ki6O63swRMRfxQRN0XEJ4EHDy1/T0Q8o/n4KRFxY0R8Fnh6G+uVtDyZOZK6ZOZI6pKZI2mxRt6DJyIeAfxXYENT72rgqln3WQ28E3g88DXgQ6OuV9LyZOZI6pKZI6lLZo6kUbSxB89jgY9l5u2ZeRtw3hz3eQjwjcz898xM4P13VzQiTo2IKyPiym07bm+hTUljokjmwO65s52tLbYsqWJmjqQumTmSFq2tkyxnS/eZuXPmWZm5MTM3rprcd5FtSRpTrWcO7J47K9lnEW1JGlNmjqQumTmSFqWNAc8lwMkRsSYi1gK/Nsd9bgSOiIijmtu/2cJ6JS1PZo6kLpk5krpk5khatJHPwZOZV0fEh4BrgVuAS+e4z50RcSrwLxFxK/BZ4GGjrlvS8mPmSOqSmSOpS2aOpFG08jbpmflG4I1zLH/O0McXMDheVJJGYuZI6pKZI6lLZo6kxWrrHDySJEmSJEnqiQMeSZIkSZKkyjngkSRJkiRJqpwDHkmSJEmSpMo54JEkSZIkSaqcAx5JkiRJkqTKtfI26aXtnFzBtgPXtF63im9+SKws03Fu29Z6zVX/8f3WawJw0D2LlF3/qW8WqZtbbi9Sl/vdu0jZrz63zOPLq8qULSlWrWLykENbr7vj5jLPNQERheqW+V/IirVri9R94Pt+r0jdo467rUjdOw/et/WaeennWq9ZWqxayeT9/kvrdXfc8q3Wa1apQD5MHnyf1msC5J1bi9SN1fsUqXvCK3+3SN39j7+jSN2fPLj9zJn6pwozZ59VTBx2ROt1p/79663XLClWripSN7e3//dVqe2cmJgoUjenporUPen5LyxSd9+Dyjx3v/rW9n+3A/CbH55zsXvwSJIkSZIkVc4BjyRJkiRJUuUc8EiSJEmSJFXOAY8kSZIkSVLlHPBIkiRJkiRVzgGPJEmSJElS5XoZ8ETE5c314RFxSh89SFo+zBxJXTJzJHXJzJE0rZcBT2ae0Hx4OGAISSrKzJHUJTNHUpfMHEnT+tqD5+fNh28CHhsR10bEK/roRdL4M3MkdcnMkdQlM0fStMme1/8a4PTMfFrPfUhaHswcSV0ycyR1ycyRlrm+Bzx7FBGnAqcC7LP6gJ67kbQcDOfO6om1PXcjadyZOZK6tFvmTK7ruRtJJSzZd9HKzLMyc2Nmbly5cr++25G0DAznzqqJfftuR9KY2z1z1vTdjqQxZ+ZI46/vAc9mwH9ZSeqKmSOpS2aOpC6ZOdIy1/eA5zpgR0Rs8kRgkjpg5kjqkpkjqUtmjrTM9XIOnszcv7neDjyhjx4kLR9mjqQumTmSumTmSJrW9x48kiRJkiRJGpEDHkmSJEmSpMo54JEkSZIkSaqcAx5JkiRJkqTKOeCRJEmSJEmqnAMeSZIkSZKkyvXyNukLFVM7WfmzO1uvm61XLCu3bStUuP1HYuo/b229JsCKffctUnfq51uK1J3Yf78idb/11IOK1J0s8zBUKbdvZ+p7P+i7DS1EgSwb1J0qUnbqJz8pUvfI13y+SN3vvuL4InXvfOTPW6+5/cu1/YYHCFjh/92KKZAPO75f5nfExAEHFKm74wf/WaTuuo+VybLc8OAidQ/87W+1XnPis4W20UvavoP8zvf77qJ3uWN73y30LnfW9Ttzv2vbfw0DTB1xvyJ1H334zUXqfmMPy92SkCRJkiRJqpwDHkmSJEmSpMo54JEkSZIkSaqcAx5JkiRJkqTKOeCRJEmSJEmqnAMeSZIkSZKkyjngkSRJkiRJqpwDHkmSJEmSpMo54JEkSZIkSaqcAx5JkiRJkqTKTfbdwJ5ExKnAqQCrV92j524kLQe75Q779tyNpHG3W+ZMru25G0njbrfMif167kZSCUt2D57MPCszN2bmxpWT/qElqbzdcidW992OpDE3nDmrVritI6ms3TLH7RxpLC3ZAY8kSZIkSZLmp/cBT0S8KyI29t2HpOXBzJHUNXNHUpfMHGn56v0cPJn5gr57kLR8mDmSumbuSOqSmSMtX73vwSNJkiRJkqTROOCRJEmSJEmqnAMeSZIkSZKkyjngkSRJkiRJqpwDHkmSJEmSpMo54JEkSZIkSapcZGbfPdytiPhP4JZ53v1ewK0F2rBufXVr6nXc6x6WmesL9FDMEsidpfBzs+54162p14XWNXMWp6a6NfVq3Trrmjkzavq5WbdcTesurbpz5k4VA56FiIgrM3Ojda1bU6/WrZvPX+vWWLemXkvWrVFtj7HPX+vWWNfMmVHTz8265Wpat466HqIlSZIkSZJUOQc8kiRJkiRJlRvHAc9Z1u22bkT8fNbt50TEX49at6l1UUTcZTe1iHhxRHwtIjIi7rWQmi2wbtm6NSrxWNT2cxv3zPlARNwUEV+KiHdHxMqF1G2BzzEzZ1htj3HVz9+eMufsiNgUEddFxEciYv+F1G2Bdc2cYTX93Maibgu5s8de95Q7Q58/c/b651N3RNZtqe7YnYNH3YuIn2fm/kO3nwNszMwXt1D7IuD0zLxy1vINwE+Ai5p1lTjJlaQlqKfMeSpwfnPz74FLMvNvR12fpKWvp8xZl5m3NR+/BfhhZr5p1PVJqkMfudN8biPwMuDk4fWrHuO4B4+WkIhYHxH/GBFfbC6PaZY/MiIuj4hrmusHN8vXRMQHm/9YfQhYM1fdzLwmM2/u7juRVIOCmfOv2QC+ANy/s29K0pJVMHOmhzvR3Mf/yEoCyuVOREwAfwm8urNvRq2b7LsBjYU1EXHt0O0DgfOaj/838L8y87MRcSjwCeAXgBuBx2Xmjoj4FeDPgd8Afg+4PTOPiYhjgKs7+y4k1aK3zGkOzfpvDP67JWl56CVzIuIc4KnAl4FXtf1NSVrS+sidFwPnZeb3BrNl1cgBj9pwR2Y+fPrG9C6Ezc1fAY4eCol1EbEWuAfw3oh4IIP/Sk2fz+JxwFsBMvO6iLiufPuSKtNn5ryNweFZl7bxjUiqQi+Zk5nPbf6jfibwLOCc1r4jSUtdp7kTEfcDngmc1Pp3ok454FFpK4DjM/OO4YURcSbwmcw8OSIOZ3AunWnuhixpsYplTkS8DlgPvKiVTiWNg6LbOZk51RxS8Qc44JE0UCJ3NgAPAL7WDI72jYivZeYD2mpa3fAcPCrtQga7+wEQEdOT6HsA32k+fs7Q/S8Bfqu578OAY8q3KGmMFMmciHgB8GTgNzNzZ7stS6pY65kTAw+Y/hj4NQaHXkgSFMidzPyXzDw4Mw/PzMMZHNLlcKdCDnhU2kuBjc1Jvb4M/G6z/M3A/xMRlwETQ/f/W2D/ZtfBVzM4meldRMRLI+LbDE50el1EvKvYdyCpJkUyB3g7cB/gcxFxbUT8cZn2JVWmROYEg8MsrgeuB+4LvKHUNyCpOqW2dTQGfJt0SZIkSZKkyrkHjyRJkiRJUuUc8EiSJEmSJFXOAY8kSZIkSVLlHPBIkiRJkiRVzgGPJEmSJElS5RzwSJIkSZIkVc4BjyRJkiRJUuUc8EiSJEmSJFXOAY8kSZIkSVLlHPBIkiRJkiRVzgGPJEmSJElS5RzwSJIkSZIkVc4BjyRJkiRJUuUc8EiSJEmSJFXOAY8kSZIkSVLlHPBIkiRJkiRVzgGPJEmSJElS5RzwSJIkSZIkVc4BjyRJkiRJUuUc8EiSJEmSJFXOAY8kSZIkSVLlHPBIkiRJkiRVbrLvBrT0RcSxwGObm5dm5qY++5E03swcSV0ycyR1zdxRKe7Bo72KiJcBHwDu3VzeHxEv6bcrSePKzJHUJTNHUtfMHZUUmdl3D1rCIuI64PjM3NLc3g/4XGYe029nksaRmSOpS2aOpK6ZOyrJPXh0dwKYGro91SyTpBLMHEldMnMkdc3cUTFjcw6eiAjgY8AZmfmVvvsZI+cAV0TEx5rbvw6c3WM/0pJh7hRh5kh7YOYUYeZIe2DmFGPuqJixOUQrIp7M4IXxocx8Vd/9jJOIOA44kcFk+ZLMvKbnlqQlwdwpw8yR5mbmlGHmSHMzc8oxd1TKOA14zgXeDbwVODozd/TcUvUiYgVwXWY+rO9epKXI3GmXmSPtnZnTLjNH2jszp33mjkobi3PwRMS9gIdm5gXAJ4GTe25pLGTmTmBTRBzady/SUmPutM/MkfbMzGmfmSPtmZlThrmj0sZiwAP8DvAPzcfnAM/vsZdxc1/ghoj4VEScN33puyn1KyJOjoj9++6jZ+ZOGWaO7sLMAcycUswc3YWZA5g5JZk7uou2cmcsDtGKiOuBp2Tmd5rbm4CnZea3+u2sfhHxS3Mtz8yLu+5FS0NEHAXcCLwkM9/edz99MXfKMHM0m5kzYOaUYeZoNjNnwMwpx9zRbG3mTvUDnog4AHhWZr5jaNkTgVs9WZXUvoh4I5DAkzLzkX330wdzR+qOmWPmSF0yc8wcqWtt5k71h2hl5k+BL81a9m/Avv10NB4i4rPN9eaIuG3osjkibuu7P/UjIiaAZwJ/AfwsIo7tuaVemDvtM3M0FzNnwMxpn5mjuZg5A2ZOGeaO5tJ27lQ/4GmcOc9lmqfMPLG5XpuZ64YuazNzXd/9qTdPBS7PzM0M3lXhBT330ydzp0VmjvbAzJlh5rTIzNEemDkzzJyWmTvag1ZzZ7KVlnoSEccDJwDrI+KVQ59aB0z009X4iYgTgQdm5jnNGfXXZuY3+u5LvXg+8D+bjz8G/FlEvCozt/XYU6fMnfLMHA0xc8yc4swcDTFzzJxOmDsa0mru1L4HzypgfwaDqrVDl9uAZ/TY19iIiNcB/x04o1m0Cnh/fx2pL83x2Adk5qUAmXkn8BHg8b021j1zpyAzR9PMnF3MnILMHE0zc3YxcwozdzStRO6Mw0mWJ4APZaaBU0BEXAtsAK7OzA3Nsusy85h+O5P6Y+6UY+ZId2XmlGPmSHdl5pRl7qikqg/RAsjMqYg4sO8+xti2zMyISICI2K/vhtS9iDhub5/PzKu76mUpMHeKMnNk5sxi5hRl5sjMmcXMKc7cUbHcqX7A07gmIs4DPgxsmV6YmR/tr6WxcW5EvAM4ICJeCDwPeGfPPal708eFrgY2ApuAAI4BrgBO7KmvPpk7ZZg5AjNnLmZOGWaOwMyZi5lTjrkjKJQ71R+iBRAR58yxODPzeZ03M4Yi4onAkxg84T7RvE2ilqGI+CDwxsy8vrn9MOD0zHxOr431wNwpx8zRNDNnhplTjpmjaWbODDOnLHNH09rOnbEY8Ki8iFjH0B5fmfnjHttRTyLi2sx8+N0tk0Zl5gjMHHXHzBGYOeqWuSNoP3fG4hCtiFjN4O3FHspgFycAnDCPLiJeBLwBuAPYyWDKnMCRLdReBTyouXlTZm4ftaaK+0pEvIvBmf4T+G3gK/221A9zpwwzR7OYOQ0zpwwzR7OYOQ0zp5xSuWPmVKvV3Kn9bdKnvQ84GHgycDFwf2DzKAUj4j4RcXZEnN/cPjoinj9yp/U5HXhoZh6emUdm5hGZ2cZGz0nAvwN/A7wN+GpEPG7UuiruucANwMuAlwNfbpYtR63mjpmzi5mjYWbODLd1yjBzNMzMmWHmlNN67pg5VWs1d8biEK2IuCYzN0y/vVxErGRwLOPi3z9+EDznAH+UmcdGxCRwTWb+Ylt91yAiLgCenpm3t1z3KuCUzLypuf0g4B8y8xFtrkcqpe3cMXMGzBxpbm7rlGHmSHMzc8opkTtmjqaNxSFawPTuZz9tTkr0feDwEWveKzPPjYgzADJzR0RMjVizRmcAl0fEFcDW6YWZ+dIR666cDqCm3lebXxwjiYjDgAdm5icjYg0wmZkj/bdBMyLiMcCfAIex+zHDI/+3s0Jt546ZM1BV5oC5U5KZsxu3dcowc7SLmbMbM6ecErlj5lSq7dwZlwHPWRFxT+C1wHnA/sD/GLHmlog4iMFxcETEo4GfjVizRu8APg1cz+AY0bZcGRFnM9j9E+C3gKtGKRiDtxk8FTgQOIrBrqRvB54wSl3t5mzgFQx+VsvxF/KwtnPHzBmoJnPA3OmAmTPDbZ0yzBwNM3NmmDnllMgdM6derebOuByidURmfuPuli2w5nHAmcDDgC8B64FnZuamkZqtTERcnpknFKi7D/D7wIkMTix2CfC2zNy61y/ce81rgUcCV2TmhmbZ9cttt8+SIuKKzHxU330sBW3njpkzUFPmNHXNnYLMnBlu65Rh5miYmTPDzCmnRO6YOfVqO3fGZcBzdWYeN2vZVaMcc9i8SKaABzN4kdwErBj1RVKbiHgjcAvwcXbfhXDRb+MXERPAezPzt0fvcLe6V2Tmo4aOGZ4Ers7MY9pcz3IWEW8CJoCPsvvz4eremupJ27lj5gzUlDlNbXOnIDNnhts6ZZg5GmbmzDBzymk7d8ycurWdO1UfohURD2Hw1n33iIinD31qHUNv57dIn2tC7Yah9V0NHLfnLxlLpzTXZwwtG+lt/DJzKiLWR8SqzNw2Une7uzgi/hBYExFPBE5jEJxqz/R0eePQsgQWfcK92hTMHTNnoKbMAXOnNDPHbZ3SzBwNM3PMnC60mjtmTvVazZ2qBzwMpr9PAw4Afm1o+WbghYspGBEHA4cweBJvYDBdhkGo7bv4VuuUmUcUKn0zcFlEnAdsGVrfW0ao+Rrg+QyOZ30R8K/Au0aop1ky85f77mEJaDV3zJzdVZY5YO4UZeYAbusUZeZomJkDmDnFFcqdmzFzqtR27ozLIVrHZ+bnWqr1bOA5DCZoX2QmgDYD78nMj7axnqUuIh6fmZ+eNbnfZdTHISJet4e6rx+lrsqKiPsAfw7cLzN/NSKOBo7PzLN7bq1zbeWOmTNg5mguZs4Mt3XaZeZoLmbODDOnfSVzx8ypV9u5My4DnjcDfwbcAVwAHAu8PDPfP0LN38jMf2ypxepExOsz83URcc4cn87MfN6I9Tdk5jWj1Jij5jdozso/LEd8a8vmMZir7kiPQY0i4nzgHOCPMvPY5jjca5bjidbazh0zp77MaeqaOwWZOTPc1mmXmXOXumYOZs4wM6d9JXPHzKlX27lT+yFa056Uma+OiJOBbwPPBD4DLDqAgPtHxDoGk+V3Mjg29DWZeeHI3VagCZ8VwPmZeW6BVbwlIu4LfBj4YGbecHdfMA/Dxy2uZvA8OLCFuv88q+7JwHdHLRoRjwUuz8ypoWXHLfET+d0rM8+NiDMAMnNHRCzXtxFtO3fMnPoyB8yd0sycGW7rtMjMuQszZ8DMmWHmtKxw7pg5VJk50HbuZGb1F+CG5vqdwFOajzeNWHNTc/1k4DwGU+ur+/5ee3hsLylY+2DgpcBlDI7rfG2BdXy2QM0VwKdbqHM7cDFwn6FlS/o5BlwEHDTdJ/Bo4OK+++rpsWg1d8ycXY9D1ZnTrMfcae/7NnNmHgu3dco8rmbO3DXNnDRzmmszp/3HtkjumDn1ZU7TX6u5My578Hw8Im5ksAvhaRGxHrhzxJrTx4Y+FTgnMzdFROztC8bUv0XE6cCH2P2EXYt++9ChGt8H3hoRnwFeDfwxg11BFyUihs/Av4LBxHntSE3O7YHAoS3UuQn4S+CiiHh+Zl7OzPNuqXolg1/IR0XEZcB64Bn9ttSbtnPHzBmoJnPA3OmAmTPDbZ0yzJy5mTlmjplTTpHcMXOA+jIHWs6dsTgHD0BE3BO4LQdvE7cfsLZ5ki+23jkMzvZ+BIPp8gRwUWY+opWGK9Ecdzlb5ujHXf4C8CwGT94fAR8E/jEzfzhCzc8M3dzB4Gzyf5WZN43QKhGxmcExotFcfx84I0c8hjgirs7M4yLigQwC/t3A83Lw9pFLVnNc6IMZPB43Zeb2nlvqTZu5Y+YM1JQ5TV1zpzAzZ4bbOu0zc3bVNXMaZs4MM6eMErlj5uyqW13mQLu5U/2AJyL2BR6YmZuGlh0KTGXmd0aouwJ4OPD1zPxpRBwEHJKZ143ctIiIzwP/AHw4M0c+3rJGEXFNZm5oPt6Pwcm1np6ZS3LPulKvtRqVeCzMnLLMnIGacsfMmeG2Tn3MnAEzp05mTn3MnIGaMgcK/U0xBgOelcCNwDGZuaVZdiHwh5l55Qh1A/gt4MjMfEPzQB+cmV8Ysd8idZvaxwKPbW5eOvxEGaHmauA04EQG09VLgbdn5qi7aLYuIl65t89n5lsWWXf6Z3ZEZv5pmz+zOdZ1aGZ+s+26bSj1WqtRicfCzNlVs5rMAXOnJDNnhts6u+qaOWZOMWbODDNnt9rLOnfMnLJKvNZWtNhfL5rdlz7GYJe06YnX+haC+G3A8cBvNrc3A3+zmEIR8ZiImGi77qx1vAz4AHDv5vL+iHjJqHWBvwMeCpwJ/DVwNPC+Efo8t7m+PiKuG7pcHxGjTu83Ar/HYNfPQ4Dfbfpdy2jHik7/zE5pbo/0M4uIVzfXZ0bEW4cvwOkj9FlUwddadQo9FmbOQE2ZA+ZOMWbODLd1zJwhZk4hZs4MM2fXOpZ87pg5AzVmDhR6reUSOHP0qBfgIQwmqgCvBV7aQs3ps1hfM7RsUWeOB04Azmq77qx1XAfsN3R7P+C6FurepbdR+gXu21wfNtdlxF4vZHBs8PTttcAFS+m50Hztj5rrlwPPnn0Ztd+SlxKvtVovbT8WZs6ee1uqmdPUNXcKXsycso9FTblj5uyqb+YUvJg5ZR+LmjKnqbHkc8fM2fW1VWZO03Orr7UleSzaQmXmjRFBRDyIweT2xBbKbm+mwgkQgzPH71xkf5dHxO1t150lgKmh21PNslFdExGPzszPA0TEoxi89d6iZOb3mutbWuhttkOBbUO3twGHt1C37Z/ZDyLiMOC5wC+30F9nCr3WqvT/t3fvQZaedZ3Av7/pnlsymYSQsCIICZdAEJHLgIKGjcKWoriKYCHs6oZaTCEiBVWwi7Vb1C7KilqFpbiCESGg4FLColxW3JXaCAkKBphclnA1IBeB3CCEYW7dz/4xPaQnZiZ9eZ/3zNv9+VSlcvr029/3OadPf/s9v3n7nA73hc45Ykqdk+idrnTO7Rzr6JwlOqcjnXM7nZNkAr2jc75tkp2TDP+ztiEGPEv+KMlrc2SqessAeb+bI6dL3aOqXp4jr0j+n9ca1lrb2yN3mdcn+WBVvX3p45/Kkftkvb4vyc9X1dG/W7xPkuuq6pocebX3h60mrG5/xfR/9qmlvN3rWOsfJ/nQ0n3QkjwlyRvWkXfU0N+zVyd5Q/rtEQAAHnVJREFUT5L7JVl++t3RV5Ff1zt3HE9VfUdbxzsfLDP0z9qUDXlf6JwjptQ5id45Lp3TxWY+1tE5R+icExiod3TO7TZz5yQT6B2d821T7pxkwJ+1yb/I8lF15BWo/ynJU1trfz1Q5oOTPCFHHhjvba1dd5LnPjJHJn6V5H2ttY8OkHnfE32+47R4TZbug6MvhDbIfbCUO/j3rKpe3Vr7xXUvbuX7e3dr7ccHyBn8Z22qhr4vdM70OifROyfYl84Z2GY/1tE5R+icE+5v3b2jc2632TtnKXfT947OOeH+TrpjnQ0z4AEAAADYrCb/LloAAAAAm50BDwAAAMDEbbgBT1VdLFdur0y5/XOnyONX7hRzp7TWnrlTNLX72ONX7hRzdc7tpvR9k9svU+40cjfcgCdJrzKWO73cKa1V7rR5/MqdYu6U1tozd4qmdh97/MqdYq7Oud2Uvm9y+2XKnUDuRhzwAAAAAGwqk3gXrW21ve3IqSva9lAOZGu2r2jbmptb8RoOtv3ZVjtWtO29HvL1Fed+7ebFnHHmyuZsX7hm14pzV3M/rMaUcqe01o2e+43ccmNr7ezBF9HRttrRdtbKeudgDmTbCu+Lg/db2XYLt+7L3O5TVrRtkszfsLIeOXTwm9m6bWW3K0m27Duwou0OLu7Pti0r68jVWE1uO7yw4tyT4edi1rlTWutqc/fnmznYDtTgi+io17HOWQ9d2c9wknzj5sM57cz5FW1747Ur/x6fDI/f2rHCjl7Yl21zK+vetn/l9+3J8HMht1+mzrndfb7nthWv4ZabF3O3FT4P+sdez4NW8V071A5ka63weeaWlZ9HsdJjnbawuOLMKf2syV1b7vGeX63st/iM7cip+b56wuC5c6ffbfDMJHnFu/6yS+6Lz31sl9wuqtPJYYsrfwJ3UqhOv+snMJhd7q/bWz836zWs1s46Nd+/48cGz/3SK88dPDNJznzNyg98VmPn33+mS24PCzff0id4Yj9vJB9s7531Elat17HOs/5nn/p9/fn365Lb6/f83P3PGzxz4WOfHDyTaZps52x54uC5r3r35YNnJskvn/v4Lrm1pc+x+pZTVv6PdCu1cOutg2cyXcd7fuVPtAAAAAAmzoAHAAAAYOIMeAAAAAAmzoAHAAAAYOIMeAAAAAAmbk0Dnqo6o6qeu+zjC6vqXcfZ9rVV9ZC1LhBA5wBj0jnA2PQOMIS1nsFzRpLn3uVWSVprz26tfWyN+wFIdA4wLp0DjE3vAOu21gHPK5Lcv6r2VtVvLV23q6reWlUfr6o3VVUlSVVdVlV7qmquqi6tqmur6pqqeuEgtwDYDHQOMCadA4xN7wDrNr/Gr3tJkoe21h6eHDmFMMkjknx3ki8luSLJDyS5fNnXPDzJvVprD136mjPWuG9g89E5wJh0DjA2vQOs25Avsvyh1toXWmuLSfYmOecOn/+HJPerqldV1Y8mufVEYVV1cVVdWVVXHsqBAZcJbBCDdk5ybO8c1DvAsbp2jmMd4E54fgWsypADnuUtsZA7nB3UWrslyfcmuSzJLyV57YnCWmuXtNb2tNb2bM32AZcJbBCDds7S13y7d7bpHeBYXTvHsQ5wJzy/AlZlrX+i9Y0kp63mC6rqrCQHW2tvq6rPJLl0jfsGNh+dA4xJ5wBj0zvAuq1pwNNau6mqrqiqa5P8ZZJ3r+DL7pXk9VV19KyhX1nLvoHNR+cAY9I5wNj0DjCEtZ7Bk9baM+9w1WXLPve8ZZcvXLbNI9e6P2Bz0znAmHQOMDa9A6zXkK/BAwAAAMAMGPAAAAAATJwBDwAAAMDEGfAAAAAATJwBDwAAAMDEGfAAAAAATNya3yZ9I1jct69L7gPmO83NWuuT20NbmPUKYH3m5lKn7x489tQ/PX3wzCSZ/+aBLrk97oMk+eqF9xw88+y3f3zwzCRZuOWWLrndVPXJ7fU7qMd6J/Tr8hgd7os3POL8wTOTZO7sXV1ys9jnm7frkpsGz7ztqfcYPDNJ2jdu65Lb67hX52SSnVPbt2XuvucOnvuUP3js4JlJcsZT+zy32P3Oq7rk1llnDp45t6XPz9rC12/tkttN9XmuvWXb1i65i/v3d8k9HmfwAAAAAEycAQ8AAADAxBnwAAAAAEycAQ8AAADAxBnwAAAAAEycAQ8AAADAxHUf8FTVB3rvA+AonQOMTe8AY9I5wPF0H/C01h7Xex8AR+kcYGx6BxiTzgGOZ4wzeG5b+v89q+p9VbW3qq6tqgt67xvYfHQOMDa9A4xJ5wDHMz/ivp6Z5K9aay+vqrkkp4y4b2Dz0TnA2PQOMCadAxxjzAHP3yd5XVVtTfLnrbW9J9q4qi5OcnGS7NBVwOqtqnOSO/TOll2dlwdsQI51gDGtvXPmd4+wPGBso72LVmvtfUken+SLSf64qn7+Lra/pLW2p7W2Z2u2j7JGYONYbecsfc23e2fblp3d1whsLI51gDGtp3O2zTnOgY1otAFPVd03yVdba3+Y5I+SPHKsfQObj84BxqZ3gDHpHOCOxvwTrQuTvLiqDiW5Lcld/ms6wDpcGJ0DjOvC6B1gPBdG5wDLdB/wtNZ2Lf3/DUne0Ht/wOamc4Cx6R1gTDoHOJ7R/kQLAAAAgD4MeAAAAAAmzoAHAAAAYOIMeAAAAAAmzoAHAAAAYOIMeAAAAAAmrvvbpA+iKrV9++Cx7dDhwTOT5GmPenKX3Pn7ndIl9/Ov3Dl45nf+ap/ZYfvIx7rkdtNan9yqPrm91jtBbeFwFm+6efDc3W8bPjNJ2mKf793BCx7WJfeB//7jg2fedOmtg2d21ennuObmuuS2hYUuuXpnmQndFwtf+WqX3Jrvc2j6X+/9vsEzX3DDBYNnJknaYp/cXib0uE0yvfX2cnghueGmwWN3f/Yeg2cmya4/+2CX3EMXPrJL7pd+YMfgmee8ptNxztR+Jlqf45HF/Z2Oc0bmDB4AAACAiTPgAQAAAJg4Ax4AAACAiTPgAQAAAJg4Ax4AAACAiTPgAQAAAJi4dQ14quqMqnruso8vrKp3rX9ZAP+czgHGpHOAsekdYD3WewbPGUmee5dbAQxD5wBj0jnA2PQOsGbrHfC8Isn9q2pvVf3W0nW7quqtVfXxqnpTVVWSVNWjqupvqurDVfVXVXXPde4b2Hx0DjAmnQOMTe8Aa7beAc9Lknymtfbw1tqLl657RJIXJHlIkvsl+YGq2prkVUme1lp7VJLXJXn5OvcNbD46BxiTzgHGpneANZvvkPmh1toXkqSq9iY5J8nXkjw0yf9ZGjjPJfmnE4VU1cVJLk6SHTmlwzKBDWKQzln6er0D3BWdA4xt+OdXW07tuFxgVnoMeA4su7ywtI9K8v9aa49daUhr7ZIklyTJ7i13b4OuENhIBumc5I69c6beAe5Mn84pnQMc1+DPr06fP1vnwAa03j/R+kaS01aw3SeSnF1Vj02SqtpaVd+9zn0Dm4/OAcakc4Cx6R1gzdY14Gmt3ZTkiqq6dtmLgN3ZdgeTPC3Jb1TVVUn2JnncevYNbD46BxiTzgHGpneA9Vj3n2i11p55h6suW/a55y27vDfJ49e7P2Bz0znAmHQOMDa9A6zVev9ECwAAAIAZM+ABAAAAmDgDHgAAAICJM+ABAAAAmDgDHgAAAICJM+ABAAAAmLh1v036KFpLO3Bg1qtYsYWvfLVPcFWX2Ksf89HBM3/kww8fPJNlWpv1Cja+lrTDh2e9ipnb8r69XXLffG6H3lmcWO90+jn2uGW5xX37Zr2EVen1+D1/2ynDhy4uDJ8JI2kLC1n42tcHz9395r8bPLOnucs+0iX3Y28e/vjpR14+seMcZsIZPAAAAAATZ8ADAAAAMHEGPAAAAAATZ8ADAAAAMHEGPAAAAAATZ8ADAAAAMHGrHvBU1fOr6rqqelNV/euqeskqvvacqnrmavcJbG56BxiTzgHGpHOAocyv4Wuem+RJrbXrlz5+xx03qKr51trhO/nac5I8M8mb17BfYPPSO8CYdA4wJp0DDGJVA56qek2S+yV5R1W9LsktSfa01p5XVZcmuTnJI5J8pKrekeR3lr60JXl8klckOb+q9iZ5Q2vtt4e5GcBGpXeAMekcYEw6BxjSqgY8rbXnVNWPJvmh1tqNVXXRHTY5L8kTW2sLVfXOJL/UWruiqnYl2Z/kJUle1Fp78hCLBzY+vQOMSecAY9I5wJCGfpHlP2utLSxdviLJK6vq+UnOOM4phcdVVRdX1ZVVdeWhHBh4mcAGoneAMekcYEw6B1ixoQc83zx6obX2iiTPTrIzyd9V1YNXE9Rau6S1tqe1tmdrtg+8TGAD0TvAmHQOMCadA6zYWl5keUWq6v6ttWuSXFNVj03y4CSfT3Jar30Cm5veAcakc4Ax6Rzgrgx9Bs9yL6iqa6vqqiTfSvKXSa5OcriqrqqqF3bcN7A56R1gTDoHGJPOAU5o1WfwtNbOWXb50iSXLl2+6A7b/fJxIp6w2n0Cm5veAcakc4Ax6RxgKD3P4AEAAABgBAY8AAAAABNnwAMAAAAwcQY8AAAAABNnwAMAAAAwcQY8AAAAABO36rdJn4mq1Pbtg8e2AwcGz0ySVPWJ3batS+6P3PtRg2f+6vV/N3hmkrz0gY/tkttLO3y4T/CWuT65iwt9cieo5uczd9Y9Bs9dvOnmwTOTpC10+t5Vn38H6NE7n/z94TOT5Pzf+1qX3Np/sEvuwue/1CV3y3nndslduO7THUKHj+ytqrJlx47BcxcPHho8M0lqS59jnbnv+Bddcp/0oNMGz/zOPoc6+epTTu2Se/jLX+mSm9b65NJVzc9l7m53Hzx34eY+vzPTFrvE1lyfY+onPeiCwTMfvbfPfXvlo3d2yU2n3xO9nsPP7d7dJXfh1lu75B6PM3gAAAAAJs6ABwAAAGDiDHgAAAAAJs6ABwAAAGDiDHgAAAAAJs6ABwAAAGDiTjjgqapzquraIXZUVZ+tqrOGyAI2Jp0DjE3vAGPSOUBPzuABAAAAmLiVDHjmq+oNVXV1Vb21qk5Jkqp6QlV9tKquqarXVdX2E11/VFXtrKr3VNUvdLg9wPTpHGBsegcYk84BuljJgOdBSS5prT0sya1JnltVO5JcmuTprbXvSTKf5BePd/2yrF1J3pnkza21PzzRTqvq4qq6sqquPNT2r/JmARM2k85Jju2dg4vfGvI2ASe3mR/rHMyBoW8TcPKafecsen4FG9FKBjyfb61dsXT5T5L8YI6U0vWttU8uXf+GJI8/wfVH/UWS17fW3nhXO22tXdJa29Na27O1dqxgmcAGMZPOSY7tnW1bdq73dgDTMfNjnW3ZflebAxvH7Dtni+dXsBGtZMDT7uTjOs62x7v+qCuSPKmq7mo7YPPSOcDY9A4wJp0DdLGSAc99quqxS5efkeTyJB9Pck5VPWDp+p9L8jcnuP6olya5Kcnvr3fhwIalc4Cx6R1gTDoH6GIlA57rkvy7qro6yZlJXt1a25/kWUn+rKquSbKY5DXHu/4OeS9IsqOqfnOoGwFsKDoHGJveAcakc4Au5k/0ydbaZ5M85Dife2+SR6zi+nOWffis1SwS2Bx0DjA2vQOMSecAPa3kDB4AAAAATmIGPAAAAAATZ8ADAAAAMHEGPAAAAAATZ8ADAAAAMHEnfBetk0XNzWXL7t2D5y7ccMPgmV0tti6xc7t3DZ759MueM3hmkpx/7i1dcuu2fV1yF2/9RpfcLWec3iV38aabu+TmW31ie2qHD2fhhpuGD15cGD6zpzad9Z733A91yV2c7/Or8tan7emS2/77XJfcM572hS65k/uZ6GRx184c/L7vHjx3299eN3hmkizuP9Al9/AXv9Qlt4ev/PjduuTe+MbTuuTe7clf6ZKbLX06J22xS2zNdVjv4eEje2vbtmbxu75j+OAbOxw79dTj8ZBk8ZvDP7d4xxsvGDwzSe595qe75B46715dcuc+9LEuufmue3aJnftyp5HLjXd+tTN4AAAAACbOgAcAAABg4gx4AAAAACbOgAcAAABg4gx4AAAAACbOgAcAAABg4mYy4KmqD8xiv8DmpXeAMekcYEw6B0hmNOBprT1uFvsFNi+9A4xJ5wBj0jlAMrszeG6bxX6BzUvvAGPSOcCYdA6QeA0eAAAAgMmbn/UCjqeqLk5ycZLs2LJrxqsBNoNjeienzHg1wEa3vHO27zhjxqsBNrpjjnO2nT7j1QA9nLRn8LTWLmmt7Wmt7dm2ZeeslwNsAst7Z2u2z3o5wAZ3TOdsPXXWywE2uGM6Z94/ZMFGdNIOeAAAAABYGQMeAAAAgImb1duke1EdYFR6BxiTzgHGpHOAxBk8AAAAAJNnwAMAAAAwcQY8AAAAABNnwAMAAAAwcQY8AAAAABNnwAMAAAAwcfOzXsBKtMOHs3DDDbNexsq11if20MEuuQtfGz73vGd9ePDMJFnokprUfJ8fhbd89v1dcp9+zgVdctvhw11yp6i2b8vcfe87eO7Cp/5h8Mwp2nLKKYNntoVODbHYp9P3n1Fdcrf+6T265Gbhi11i69HfM3zotR8YPrOzLd/cn21/e93guYv79g2e2VX1+bmoubnBMxdu+frgmUly95/Z3yU3O3d2if38n5zbJffez/h0l9z9T3jY4Jnt8j7He13t25+292OzXsXMtQMH+gRvGb5z7vUHVw2emSQL3+rTOQcfc06X3H/8/Q7HDUke8mt95g2t0++143EGDwAAAMDEGfAAAAAATJwBDwAAAMDEGfAAAAAATJwBDwAAAMDEGfAAAAAATNzgA56q+i9V9aKlyy+rqifeyTYXVtW7ht43sPnoHGBMOgcYk84BVmO+Z3hr7aU98wGW0znAmHQOMCadA9yVQc7gqar/VFWfqKq/TvKgZddfWlVPW7r8o1X18aq6PMlPD7FfYHPSOcCYdA4wJp0DrNW6z+Cpqkcl+dkkj1jK+0iSD99hmx1J/jDJDyf5dJK3rHe/wOakc4Ax6RxgTDoHWI8hzuC5IMnbW2v7Wmu3JnnHnWzz4CTXt9Y+1VprSf7krkKr6uKqurKqrjyUAwMsE9ggunROcmzvHFz41oBLBiZsnM5p+wdcMjBho3SO51ewMQ31IsttoG1u37i1S1pre1pre7Zm+xqXBWxQg3dOcmzvbJvbuYZlARtU/86pHWtYFrBBde8cz69gYxpiwPO+JE+pqp1VdVqSn7iTbT6e5Nyquv/Sx88YYL/A5qRzgDHpHGBMOgdYs3W/Bk9r7SNV9ZYke5N8Lsn772Sb/VV1cZJ3V9WNSS5P8tD17hvYfHQOMCadA4xJ5wDrMcjbpLfWXp7k5Xdy/UXLLr8nR/5eFGBddA4wJp0DjEnnAGs11GvwAAAAADAjBjwAAAAAE2fAAwAAADBxBjwAAAAAE2fAAwAAADBxBjwAAAAAEzfI26T3Vlu2ZMvOUwbPXdy3b/DMJElVn9zW+uROSM33ecj2yn30+5/TJfceP7WjS+7u91/fJTdf7hPbU5ufy+GzThs8tz41eOQk1batg2du2bl78MwkWbjhxi65uz93uEvuqZ+4oUvuge9/SJfcOrw4fOgU//lq+7bkgfcdPveq64bP7KnTsU5bWOiS20Nt394ltx3u0znzl53eJbcd6rPem88f/vfP4Ss7Hft3VNu2Zv477z147uHPfX7wzK56PW/roHb2Of5Pp+fEp77/E11yz//bPs/bvvzT53XJPXhGp8fYb9z51VM8BAIAAABgGQMeAAAAgIkz4AEAAACYOAMeAAAAgIkz4AEAAACYOAMeAAAAgImbyYCnqj6w9P9zquqZs1gDsHnoHGBMOgcYk84BjprJgKe19rili+ckUUJAVzoHGJPOAcakc4CjZnUGz21LF1+R5IKq2ltVL5zFWoCNT+cAY9I5wJh0DnDU/Iz3/5IkL2qtPXnG6wA2B50DjEnnAGPSObDJzXrAc1xVdXGSi5NkR50649UAm8ExvbPt9BmvBtjojumcrToH6OuYzpk7bcarAXo4ad9Fq7V2SWttT2ttz7baMevlAJvA8t7ZutVgGejrmGOd+VNmvRxggzumc+Z2zno5QAezHvB8I4nxMTAWnQOMSecAY9I5sMnNesBzdZLDVXWVFwIDRqBzgDHpHGBMOgc2uZm8Bk9rbdfS/w8lecIs1gBsHjoHGJPOAcakc4CjZn0GDwAAAADrZMADAAAAMHEGPAAAAAATZ8ADAAAAMHEGPAAAAAATZ8ADAAAAMHEzeZv0k0bVrFdwcuhxP7Q2fGaSVJ+ZZFtY7JL7gJft65L7mX97SpfcfWfdv0tuXt0ntqdabJn75oHBc9vWbYNnHgnu8xju9TO38PVbhw/92teHz0xS27d3yT3lik90yc2pffrh+p/s89i9/1v3d8mdmjZXOXza8I81/5K3pNdxSQeL+/ocO7RDh7vk3vP113TJzc4dXWLv8xPXD575+XceHDyzu7m5LJ5+6qxXMXu9uqEtDB65cONNg2cm6facuMuxXkc7vtbnsXDLDw7/fOJE/N4HAAAAmDgDHgAAAICJM+ABAAAAmDgDHgAAAICJM+ABAAAAmDgDHgAAAICJM+ABAAAAmDgDHgAAAICJM+ABAAAAmDgDHgAAAICJm5/1Ao6nqi5OcnGS7KhTZ7waYDM4pne27p7xaoCNbnnnbN9++oxXA2x0jnNg4ztpz+BprV3SWtvTWtuzrXbMejnAJnBM78wbLAN9HdM5W3UO0JfjHNj4TtoBDwAAAAArM/MBT1W9tqr2zHodwOagc4Cx6R1gTDoHNq+ZvwZPa+3Zs14DsHnoHGBsegcYk86BzWvmZ/AAAAAAsD4GPAAAAAATZ8ADAAAAMHEGPAAAAAATZ8ADAAAAMHEGPAAAAAATV621Wa/hLlXVDUk+t8LNz0pyY4dlyJ1e7pTWutFz79taO7vDGro5CXrnZPi+yd3YuVNa62pzdc7aTCl3SmuVO81cnXO7KX3f5PbLlHty5d5p70xiwLMaVXVla22PXLlTWqvcafP4lTvF3CmttWfuFE3tPvb4lTvFXJ1zuyl93+T2y5Q7jVx/ogUAAAAwcQY8AAAAABO3EQc8l8gdN7eqbrvDxxdV1e+tN3cp67Kq+menqVXVpVV1fVXtXfrv4SvNHIDcvrlT1OO+mNr3baN3TlXVy6vqk1V1XVU9fzW5A/AY0znLTe0+nvTjd0ad8/5lxzhfqqo/X03uAOTqnOWm9H3bELkD9M5x13qC3nlCVX1kqXcur6oHrCZ3neQOlLvhXoOH8VXVba21Xcs+vijJntba8wbIvizJi1prV97h+kuTvKu19tb17gOYlhl1zrOS/FCSi1pri1V1j9baV9e7P+DkN4vOucM2b0vyF621N653f8A0zOhY55NJfrK1dl1VPTfJY1prF613f4xrI57Bw0mkqs6uqrdV1d8v/fcDS9c/pqo+UFUfXfr/g5au31lV/6Oqrq6qtyTZOdMbAExKx875xSQva60tJonhDpD0P86pqtOS/HCS453BA2wyHXunJdm9dPn0JF/qfmMY3PysF8CGsLOq9i77+Mwk71i6/DtJfru1dnlV3SfJXyU5P8nHkzy+tXa4qp6Y5L8leWqOPIna11p7WFU9LMlHTrDfl1fVS5O8N8lLWmsHhr1ZwElqFp1z/yRPr6qnJLkhyfNba58a/JYBJ6NZHeckyVOSvLe1duuAtwc4+c2id56d5H9V1beS3Jrk+we/VXRnwMMQvtVa+/Zr4Bw9hXDpwycmeUhVHf307qV/jTo9yRuq6oE5Mi3euvT5xyf53SRprV1dVVcfZ5+/kuTLSbblyN8q/sckLxvqBgEntVl0zvYk+1tre6rqp5O8LskFw90k4CQ2i8456hlJXjvEjQAmZRa988IkP9Za+2BVvTjJK3Nk6MOEGPDQ25Ykj22tfWv5lVX1qiT/t7X2lKo6J8llyz59ly8M1Vr7p6WLB6rq9UleNMhqganr0jlJvpDkbUuX357k9eteKbAR9OqcVNXdkzwmR87iAThq8N6pqrOTfG9r7YNLV70lyXuGWjDj8Ro89Pa/k3z7xcDq9ne7Oj3JF5cuX7Rs+/cl+TdL2z40ycPuLLSq7rn0/0ryU0muHXLRwGR16Zwcef2LH166/C+TfHKY5QIT16tzkuRncuQNJfYPtVhgQ+jRO7ckOb2qzlv6+F8luW64JTMWAx56e36SPUsv6vWxJM9Zuv43k/x6VV2RZG7Z9q9Osmvp1MH/kORDx8l9U1Vdk+SaJGcl+bUuqwemplfnvCLJU5d659fjlGXgiF6dkyQ/m+RPO6wZmLbBe6e1djjJLyR5W1VdleTnkry4422gE2+TDgAAADBxzuABAAAAmDgDHgAAAICJM+ABAAAAmDgDHgAAAICJM+ABAAAAmDgDHgAAAICJM+ABAAAAmDgDHgAAAICJ+/83QaiDoaik6AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1152x576 with 8 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Real translation: this is the first book i've ever done.\n"
     ]
    }
   ],
   "source": [
    "translate(\"este é o primeiro livro que eu fiz.\", plot='decoder_layer4_block2')\n",
    "print (\"Real translation: this is the first book i've ever done.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RqQ1fIsLwkGE"
   },
   "source": [
    "## Summary\n",
    "\n",
    "In this tutorial, you learned about positional encoding, multi-head attention, the importance of masking and how to create a transformer.\n",
    "\n",
    "Try using a different dataset to train the transformer. You can also create the base transformer or transformer XL by changing the hyperparameters above. You can also use the layers defined here to create [BERT](https://arxiv.org/abs/1810.04805) and train state of the art models. Futhermore, you can implement beam search to get better predictions."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "s_qNSzzyaCbD"
   ],
   "name": "transformer.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "tensorflow-gpu/2.0.0-beta-py36",
   "language": "python",
   "name": "tensorflow_gpu_2.0.0-beta-py36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "023fd301148a4b6caa974c277bc27cc5": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "0401c5a6be1e46febc1c7a4fbb6d80db": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "info",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_bcd3781e3b434eb0bbd07143a072edad",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_a8589289cea74eb5ae784f1186308762",
       "value": 1
      }
     },
     "041cac226c33477887b2913c9fb8412f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "06f7ea1674594c17965789acb9bb1856": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "09dce7b837e24a9da9c5009680a3234f": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "0aba56b1892b41abad9d5006e3459a41": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "info",
       "description": "Reading...",
       "description_tooltip": null,
       "layout": "IPY_MODEL_64d248a63e1249c885031b9de6e5e13b",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_1fc7afa135e14826977a67ffa355af7b",
       "value": 1
      }
     },
     "0c42d968d5144269af0ca4509c479cc5": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_d70823f2a7184ca996180951dc559dcb",
        "IPY_MODEL_bf4375be38944530990238c7972cd193"
       ],
       "layout": "IPY_MODEL_eaaace21f00448a4add8049ba15a5d9c"
      }
     },
     "0ecb3b94cbfe45de9a131e35d5a0afdc": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "0ee48acefa5943c49bb803446448cece": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "",
       "description": "Writing...",
       "description_tooltip": null,
       "layout": "IPY_MODEL_023fd301148a4b6caa974c277bc27cc5",
       "max": 1803,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_8f4622b3e1e44463b8f1fd3ddeb9701f",
       "value": 0
      }
     },
     "0f83553dc1834e738c9b97d4c15f68d1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "1076fef0a2634a6ea233d1d00ca6ba04": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "info",
       "description": "Reading...",
       "description_tooltip": null,
       "layout": "IPY_MODEL_fa0e541b862a4ec08fb1f53ff7ba8913",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_12d3f8cf41a348aeb1e11a9035bb9f36",
       "value": 1
      }
     },
     "12d3f8cf41a348aeb1e11a9035bb9f36": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "13ccc0ff408441cbbfdd212672564a80": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "143e18c17fe24615a8e19f3143c5be21": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "145f8a85d82441258a8c24e92ee1fc51": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "Extraction completed...",
       "description_tooltip": null,
       "layout": "IPY_MODEL_4c91a126aa6e4c2e8fd9538c7fbfda1c",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_b2de90cc47e9497992e6eff86874246b",
       "value": 1
      }
     },
     "1552a2e2301b43658eee816d3c178f01": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "Dl Size...",
       "description_tooltip": null,
       "layout": "IPY_MODEL_56e357824880464c9d75fec6553b6211",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_3a1fbf9aedc84dc38165ff11d541c335",
       "value": 1
      }
     },
     "1951d6ad41eb439eb2baa3ac3d50aad2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "1eb00fb916ae467e9cdf1205480a096c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_82cb36a0315d4a72af0850c0f5b53e20",
        "IPY_MODEL_3afde188dafa492f9348bbbc603527e1"
       ],
       "layout": "IPY_MODEL_b61e5f71999d494b9f78187b1802ee84"
      }
     },
     "1fc7afa135e14826977a67ffa355af7b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "2482416206a448e68b87d45085c67e68": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "2a2cf39b613b4d52a9f8daf47e94d055": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_cf6ea55d4657419daf329fd882999cc5",
       "placeholder": "​",
       "style": "IPY_MODEL_ae8edf2ca82d480990584f750d272acb",
       "value": "30717/|/| 30717/? [00:00&lt;00:00, 307165.17 examples/s]"
      }
     },
     "2b6f3118b18f4d6a8831ead044ea15eb": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "2ebb46927da046858a567f773ad5c542": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_bc10ac7f76fe4e768ea13de163562cc5",
        "IPY_MODEL_a02c5cbc3d8642d5bd221dfe5aa3ed64"
       ],
       "layout": "IPY_MODEL_09dce7b837e24a9da9c5009680a3234f"
      }
     },
     "3090bf86c7b14da8a78536648d272eb0": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_145f8a85d82441258a8c24e92ee1fc51",
        "IPY_MODEL_ee2dd761dc514e4e86f2c45ec6106f8f"
       ],
       "layout": "IPY_MODEL_8890604db73e4c7f8f068539e109c123"
      }
     },
     "36c9c8aae47544dd9c81999aa120e856": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "3a162cfa6149476dab4c6daf44142e8f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "3a1fbf9aedc84dc38165ff11d541c335": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "3afde188dafa492f9348bbbc603527e1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_2b6f3118b18f4d6a8831ead044ea15eb",
       "placeholder": "​",
       "style": "IPY_MODEL_d3284eaee2f64f59aa267058bc89ab5d",
       "value": " 74% 38450/51785 [00:00&lt;00:00, 182602.50 examples/s]"
      }
     },
     "3b3fcc12923f4ddb87619a40680e71be": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "3d7636b5eefc475db1e1ecf759cab5c3": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "3e20ebf33e5945ef95d2c61ab8617d58": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "3e68ed217980454aae46cd50dfe7174a": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_0ecb3b94cbfe45de9a131e35d5a0afdc",
       "placeholder": "​",
       "style": "IPY_MODEL_3a162cfa6149476dab4c6daf44142e8f",
       "value": "100% 1/1 [00:00&lt;00:00,  8.31 shard/s]"
      }
     },
     "3f2dca820d8842209953c6b5ad995e45": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "",
       "description": "Writing...",
       "description_tooltip": null,
       "layout": "IPY_MODEL_b099be214f96409f9001901076d8b6be",
       "max": 1193,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_1951d6ad41eb439eb2baa3ac3d50aad2",
       "value": 0
      }
     },
     "45558f58b3214275a7f3fbf33146868f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "4606ca24ac1441f58f9898cf0a8fe2d1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_533a3e3e703f47a7b099f274caba1282",
        "IPY_MODEL_73d13e331ee748f6b8bf1da2ff249e69"
       ],
       "layout": "IPY_MODEL_f63a45d0ad38403dae52cab2f8a03dfb"
      }
     },
     "4c91a126aa6e4c2e8fd9538c7fbfda1c": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "4e8b287eed4343ec96952ac5d3800e03": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "4fb5ba3e15564274ae0fad908de4f3d4": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "50b540e7a615420f80ba241cbed86020": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_82dab1131a584373ae83ac19ae207f3e",
        "IPY_MODEL_9a54aed83f42452f82cf0948d8462cc9"
       ],
       "layout": "IPY_MODEL_e5f3a7624c38420d9c141533269a1568"
      }
     },
     "52c4c98b067448ec87956a0f59b1dc6b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "533a3e3e703f47a7b099f274caba1282": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "Dl Completed...",
       "description_tooltip": null,
       "layout": "IPY_MODEL_823236827c694236aafaf60d939a4a69",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_929bf562bae7437e85d4e97038bed3cb",
       "value": 1
      }
     },
     "56e357824880464c9d75fec6553b6211": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "599f1e59bc9245a4bdc326c3e2be11de": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "59bb7162b62540caa4b0dfe887b2159e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_cbb84ab0ffe64eef8204e3ee29440796",
       "placeholder": "​",
       "style": "IPY_MODEL_3b3fcc12923f4ddb87619a40680e71be",
       "value": "901/|/| 901/? [00:00&lt;00:00, 4220.06 examples/s]"
      }
     },
     "5c115d15a5284aa19185c2b01005ea78": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "5c8ff3746b884122bd3a988a0b6300ac": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "info",
       "description": "Reading...",
       "description_tooltip": null,
       "layout": "IPY_MODEL_f3d672b66f0d4b479bbbc43f59e785bc",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_599f1e59bc9245a4bdc326c3e2be11de",
       "value": 1
      }
     },
     "5ea5d37e549f4afd9ff39e3450fb5514": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "64d248a63e1249c885031b9de6e5e13b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "69409db842524d2f868549d4dec69e30": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "6cb2e5214b16490c90e2b9edf4d9059e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "6d0aa2e5b84745d0ae36bd83a63c1166": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_36c9c8aae47544dd9c81999aa120e856",
       "placeholder": "​",
       "style": "IPY_MODEL_c7b1a52bba43471ea3d1b16a223cfabc",
       "value": "  0% 0/1193 [00:00&lt;?, ? examples/s]"
      }
     },
     "6ef8eb3f91bc4520ba239335dcad53b2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_1552a2e2301b43658eee816d3c178f01",
        "IPY_MODEL_83a1a536492247628ae460d4008e794b"
       ],
       "layout": "IPY_MODEL_bced4a94aa1e4bfb96886d8992f6780b"
      }
     },
     "7049fb01858b4c5ea635b9278c46198c": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "7090154702d44fe299d4b4e0f36ae8d2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_e8eeb63c176b44a2835417e2bdf91801",
        "IPY_MODEL_3e68ed217980454aae46cd50dfe7174a"
       ],
       "layout": "IPY_MODEL_f4efab5407684405b05766532960f620"
      }
     },
     "7120d502acc943a49729cc79c8d44273": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "722246e19e7b43eeb0807c93a3a3983a": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_0401c5a6be1e46febc1c7a4fbb6d80db",
        "IPY_MODEL_59bb7162b62540caa4b0dfe887b2159e"
       ],
       "layout": "IPY_MODEL_e86f3d20721f42d4872e61eee254a20f"
      }
     },
     "73374448986441d2801acf8aba0fc764": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "73d13e331ee748f6b8bf1da2ff249e69": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_06f7ea1674594c17965789acb9bb1856",
       "placeholder": "​",
       "style": "IPY_MODEL_45558f58b3214275a7f3fbf33146868f",
       "value": "1/|/100% 1/1 [00:14&lt;00:00,  9.92s/ url]"
      }
     },
     "760021e299cb4c528c69253dfad8fddf": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_3f2dca820d8842209953c6b5ad995e45",
        "IPY_MODEL_6d0aa2e5b84745d0ae36bd83a63c1166"
       ],
       "layout": "IPY_MODEL_cd463a4d27164414a591dbf35cef0adf"
      }
     },
     "7aa8b03fe2b047b7b8cc1b8a06a12ea7": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "823236827c694236aafaf60d939a4a69": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "82cb36a0315d4a72af0850c0f5b53e20": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "",
       "description": "Writing...",
       "description_tooltip": null,
       "layout": "IPY_MODEL_7049fb01858b4c5ea635b9278c46198c",
       "max": 51785,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_f038a2caea44466e96ae8e27bbdbe205",
       "value": 38450
      }
     },
     "82dab1131a584373ae83ac19ae207f3e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "info",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_830e88859020498fbec5892467e8e11a",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_4fb5ba3e15564274ae0fad908de4f3d4",
       "value": 1
      }
     },
     "830e88859020498fbec5892467e8e11a": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "83a1a536492247628ae460d4008e794b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_69409db842524d2f868549d4dec69e30",
       "placeholder": "​",
       "style": "IPY_MODEL_ff19821b27164ffebd2d259d090a95aa",
       "value": "124/|/100% 124/124 [00:14&lt;00:00, 15.44 MiB/s]"
      }
     },
     "85cce9a9248b4bf2aab8690d9f3800e3": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "85d134417b7842c995e6588f53ba9bfa": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_a7a66b4fe4f6479d90b56bc4ad93cdc0",
       "placeholder": "​",
       "style": "IPY_MODEL_7120d502acc943a49729cc79c8d44273",
       "value": "100% 1/1 [00:00&lt;00:00,  2.03 shard/s]"
      }
     },
     "8890604db73e4c7f8f068539e109c123": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "8f4622b3e1e44463b8f1fd3ddeb9701f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "929bf562bae7437e85d4e97038bed3cb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "954082004e53452f9b89e70a365ac824": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_0ee48acefa5943c49bb803446448cece",
        "IPY_MODEL_ce9bc72c56cc491c92106a1a17fcc340"
       ],
       "layout": "IPY_MODEL_2482416206a448e68b87d45085c67e68"
      }
     },
     "95770a6e7d3949ceadb651102af360d7": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "9a54aed83f42452f82cf0948d8462cc9": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_5ea5d37e549f4afd9ff39e3450fb5514",
       "placeholder": "​",
       "style": "IPY_MODEL_ce4bf4e0d0194f089fd096bd6fea8a34",
       "value": "1420/|/| 1420/? [00:00&lt;00:00, 4485.58 examples/s]"
      }
     },
     "9d0995a4a5eb46d38aa79ccc04875dec": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "a02c5cbc3d8642d5bd221dfe5aa3ed64": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_95770a6e7d3949ceadb651102af360d7",
       "placeholder": "​",
       "style": "IPY_MODEL_6cb2e5214b16490c90e2b9edf4d9059e",
       "value": "100% 1/1 [00:00&lt;00:00,  6.00 shard/s]"
      }
     },
     "a7a66b4fe4f6479d90b56bc4ad93cdc0": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "a8589289cea74eb5ae784f1186308762": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "a9a8ccb99bb149f2b5e233da599b7531": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_9d0995a4a5eb46d38aa79ccc04875dec",
       "placeholder": "​",
       "style": "IPY_MODEL_041cac226c33477887b2913c9fb8412f",
       "value": "0/|/| 0/? [00:00&lt;?, ? examples/s]"
      }
     },
     "ae8edf2ca82d480990584f750d272acb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "b099be214f96409f9001901076d8b6be": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "b2de90cc47e9497992e6eff86874246b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "b425419eedf74a67a4a1ff879c69233e": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "b61e5f71999d494b9f78187b1802ee84": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "ba7c85918a7a49f5a6a06dca541fd070": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": ""
      }
     },
     "bc10ac7f76fe4e768ea13de163562cc5": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "",
       "description": "Shuffling...",
       "description_tooltip": null,
       "layout": "IPY_MODEL_52c4c98b067448ec87956a0f59b1dc6b",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_f5aa95bd91464a7cbfd90d923d214829",
       "value": 1
      }
     },
     "bcd3781e3b434eb0bbd07143a072edad": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "bced4a94aa1e4bfb96886d8992f6780b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "bf4375be38944530990238c7972cd193": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_3e20ebf33e5945ef95d2c61ab8617d58",
       "placeholder": "​",
       "style": "IPY_MODEL_f941837557684ad3b22e2ce2de5495a1",
       "value": "51275/|/| 51275/? [00:10&lt;00:00, 5074.94 examples/s]"
      }
     },
     "c32fb6b39e4c4dea9111b2d328d5df62": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "c60ce00b89004a52926216dfd2ea6989": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_1076fef0a2634a6ea233d1d00ca6ba04",
        "IPY_MODEL_2a2cf39b613b4d52a9f8daf47e94d055"
       ],
       "layout": "IPY_MODEL_d90b4f8f00ec46bcb907296109621967"
      }
     },
     "c7b1a52bba43471ea3d1b16a223cfabc": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "cbb84ab0ffe64eef8204e3ee29440796": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "cd463a4d27164414a591dbf35cef0adf": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "ce4bf4e0d0194f089fd096bd6fea8a34": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "ce8dec3295d84a7fba99b8597723e68b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "ce9bc72c56cc491c92106a1a17fcc340": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_b425419eedf74a67a4a1ff879c69233e",
       "placeholder": "​",
       "style": "IPY_MODEL_5c115d15a5284aa19185c2b01005ea78",
       "value": "  0% 0/1803 [00:00&lt;?, ? examples/s]"
      }
     },
     "cf6ea55d4657419daf329fd882999cc5": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "d3284eaee2f64f59aa267058bc89ab5d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "d70823f2a7184ca996180951dc559dcb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "info",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_c32fb6b39e4c4dea9111b2d328d5df62",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_ba7c85918a7a49f5a6a06dca541fd070",
       "value": 1
      }
     },
     "d827078b46f744258498f9fdd24eddee": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "d90b4f8f00ec46bcb907296109621967": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "dd17ebdda8ca447eaaf2a9377fa0e8d2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_e2e32ae4bb364268be9d2bf4067e433a",
       "placeholder": "​",
       "style": "IPY_MODEL_4e8b287eed4343ec96952ac5d3800e03",
       "value": "0/|/| 0/? [00:00&lt;?, ? examples/s]"
      }
     },
     "e0a46145714b4562abb386821ee51655": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_e428e592e9614cd48ed1631b54abbc13",
        "IPY_MODEL_85d134417b7842c995e6588f53ba9bfa"
       ],
       "layout": "IPY_MODEL_d827078b46f744258498f9fdd24eddee"
      }
     },
     "e25f29522bd14be0b4ce740d0c5c36f5": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_0aba56b1892b41abad9d5006e3459a41",
        "IPY_MODEL_a9a8ccb99bb149f2b5e233da599b7531"
       ],
       "layout": "IPY_MODEL_85cce9a9248b4bf2aab8690d9f3800e3"
      }
     },
     "e2e32ae4bb364268be9d2bf4067e433a": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "e428e592e9614cd48ed1631b54abbc13": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "",
       "description": "Shuffling...",
       "description_tooltip": null,
       "layout": "IPY_MODEL_73374448986441d2801acf8aba0fc764",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_ce8dec3295d84a7fba99b8597723e68b",
       "value": 1
      }
     },
     "e5f3a7624c38420d9c141533269a1568": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "e86f3d20721f42d4872e61eee254a20f": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "e8eeb63c176b44a2835417e2bdf91801": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "IntProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "",
       "description": "Shuffling...",
       "description_tooltip": null,
       "layout": "IPY_MODEL_13ccc0ff408441cbbfdd212672564a80",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_7aa8b03fe2b047b7b8cc1b8a06a12ea7",
       "value": 1
      }
     },
     "e9d2bfec8cad47a38250b408b11e5955": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_5c8ff3746b884122bd3a988a0b6300ac",
        "IPY_MODEL_dd17ebdda8ca447eaaf2a9377fa0e8d2"
       ],
       "layout": "IPY_MODEL_3d7636b5eefc475db1e1ecf759cab5c3"
      }
     },
     "eaaace21f00448a4add8049ba15a5d9c": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "ee2dd761dc514e4e86f2c45ec6106f8f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_143e18c17fe24615a8e19f3143c5be21",
       "placeholder": "​",
       "style": "IPY_MODEL_0f83553dc1834e738c9b97d4c15f68d1",
       "value": "1/|/100% 1/1 [00:14&lt;00:00, 14.25s/ file]"
      }
     },
     "f038a2caea44466e96ae8e27bbdbe205": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "f3d672b66f0d4b479bbbc43f59e785bc": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "f4efab5407684405b05766532960f620": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "f5aa95bd91464a7cbfd90d923d214829": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "f63a45d0ad38403dae52cab2f8a03dfb": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "f941837557684ad3b22e2ce2de5495a1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "fa0e541b862a4ec08fb1f53ff7ba8913": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "ff19821b27164ffebd2d259d090a95aa": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}