internlm2_5-from-scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Internlm2.5 implemented from scratch with kvcache\n",
    "in this file, i implemented intern2.5-7B from scratch, one tensor and matrix multiplication at a time.\n",
    "<br><br>\n",
    "the weights is huggingface format, suffix with safetensors.\n",
    "<br><br>\n",
    "use \"If I have seen further, it is by standing\" as prompt and generate result \"on the shoulders of giants.\" in an auto-regressive manner.\n",
    "<br><br>\n",
    "Cache the vector of k and v during the prefill phase. and load it in decoder phase.\n",
    "\n",
    "从头开始实现internlm2.5，支持huggingface格式，以safetensors为后缀的权重，并支持kvcache，进行自回归推理。\n",
    "\n",
    "以 \"If I have seen further, it is by standing\" 作为输入的prompt，先进行prefill的并行运算，生成并缓存 k 和 v，然后进行自回归生成，将前一次输出作为下一次输入，同时加载缓存的kvcache，进行推理，逐个token生成\"on the shoulders of giants.\"。\n",
    "\n",
    "前面的推理流程和原工程是一样的，只是在迭代layer生成第一个token时，缓存了k和v。生成第一个token之后，将其作为输入，是新增部分。\n",
    "\n",
    "这里以internlm2_5-7B为例，其他不同大小的模型需要修改个别参数适配。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1597,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'hello world!'"
      ]
     },
     "execution_count": 1597,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tokenizers import Tokenizer as TokenizerFast\n",
    "import os\n",
    "\n",
    "model_path = \"./Shanghai_AI_Laboratory/internlm2_5-7b\"\n",
    "\n",
    "tokenizer = TokenizerFast.from_file(os.path.join(model_path, 'tokenizer.json'))\n",
    "tokenizer.decode(tokenizer.encode(\"hello world!\").ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### reading the model file\n",
    "normally, reading this depends on how the model classes are written and the variable names inside them.\n",
    "<br>\n",
    "but since we are implementing internlm2.5 from scratch we will read the file one tensor at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1598,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model.layers.0.attention.wo.weight torch.Size([4096, 4096])\n",
      "model.layers.0.attention.wqkv.weight torch.Size([6144, 4096])\n",
      "model.layers.0.attention_norm.weight torch.Size([4096])\n",
      "model.layers.0.feed_forward.w1.weight torch.Size([14336, 4096])\n",
      "model.layers.0.feed_forward.w2.weight torch.Size([4096, 14336])\n",
      "model.layers.0.feed_forward.w3.weight torch.Size([14336, 4096])\n",
      "model.layers.0.ffn_norm.weight torch.Size([4096])\n",
      "model.layers.1.attention.wo.weight torch.Size([4096, 4096])\n",
      "model.layers.1.attention.wqkv.weight torch.Size([6144, 4096])\n",
      "model.layers.1.attention_norm.weight torch.Size([4096])\n",
      "model.layers.1.feed_forward.w1.weight torch.Size([14336, 4096])\n",
      "model.layers.1.feed_forward.w2.weight torch.Size([4096, 14336])\n",
      "model.layers.1.feed_forward.w3.weight torch.Size([14336, 4096])\n",
      "model.layers.1.ffn_norm.weight torch.Size([4096])\n",
      "model.layers.10.attention.wo.weight torch.Size([4096, 4096])\n",
      "model.layers.10.attention.wqkv.weight torch.Size([6144, 4096])\n",
      "model.layers.10.attention_norm.weight torch.Size([4096])\n",
      "model.layers.10.feed_forward.w1.weight torch.Size([14336, 4096])\n"
     ]
    }
   ],
   "source": [
    "from safetensors import safe_open\n",
    "model = {}\n",
    "for i in range(1,3):  # 遍历加载权重文件，不同大小的模型需要针对性修改\n",
    "    with safe_open(os.path.join(model_path, f\"model-0000{i}-of-00002.safetensors\"), framework=\"pt\", device=\"cpu\") as f:\n",
    "        count = 1\n",
    "        for k in f.keys():\n",
    "            model[k] = f.get_tensor(k)\n",
    "            count += 1\n",
    "            if count < 20 and i == 1:   # 仅显示部分权重及其shape，作为示例\n",
    "                print(k, model[k].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1599,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bias': False,\n",
       " 'bos_token_id': 1,\n",
       " 'eos_token_id': 2,\n",
       " 'hidden_act': 'silu',\n",
       " 'hidden_size': 4096,\n",
       " 'initializer_range': 0.02,\n",
       " 'intermediate_size': 14336,\n",
       " 'max_position_embeddings': 262144,\n",
       " 'model_type': 'internlm2',\n",
       " 'num_attention_heads': 32,\n",
       " 'num_hidden_layers': 32,\n",
       " 'num_key_value_heads': 8,\n",
       " 'pad_token_id': 2,\n",
       " 'rms_norm_eps': 1e-05,\n",
       " 'rope_scaling': {'factor': 2.0, 'type': 'dynamic'},\n",
       " 'rope_theta': 50000000,\n",
       " 'tie_word_embeddings': False,\n",
       " 'torch_dtype': 'bfloat16',\n",
       " 'transformers_version': '4.41.0',\n",
       " 'use_cache': True,\n",
       " 'vocab_size': 92544}"
      ]
     },
     "execution_count": 1599,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "with open(os.path.join(model_path, 'config.json'), \"r\") as f:\n",
    "    config = json.load(f)\n",
    "dict(zip(list(config.keys())[3:], list(config.values())[3:]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### we use this config to infer details about the model like\n",
    "1. the model has 32 transformer layers\n",
    "2. each multi-head attention block has 32 heads\n",
    "3. the vocab size and so on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1600,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "dim = config[\"hidden_size\"]\n",
    "n_layers = config[\"num_hidden_layers\"]\n",
    "n_heads = config[\"num_attention_heads\"]\n",
    "n_kv_heads = config[\"num_key_value_heads\"]\n",
    "vocab_size = config[\"vocab_size\"]\n",
    "ffn_dim_multiplier = config[\"intermediate_size\"]\n",
    "norm_eps = config[\"rms_norm_eps\"]\n",
    "rope_theta = torch.tensor(config[\"rope_theta\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### converting text to tokens\n",
    "here we use tiktoken (i think an openai library) as the tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1601,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 2841, 489, 746, 4040, 4784, 328, 563, 505, 684, 11410]\n",
      "['If', ' I', ' have', ' seen', ' further', ',', ' it', ' is', ' by', ' standing']\n"
     ]
    }
   ],
   "source": [
    "# prompt = \"the answer to the ultimate question of life, the universe, and everything is \"\n",
    "# prompt = \"Genius is one percent inspiration and\"   # generate “ninety-nine percent perspiration.”\n",
    "prompt = \"If I have seen further, it is by standing\"   #  generate \"on the shoulders of giants.\"\n",
    "tokens = tokenizer.encode(prompt).ids\n",
    "print(tokens)\n",
    "tokens = torch.tensor(tokens[1:])   # tokenizer默认会加上'bos_token_id'，所以tokens第一个id是1，这里切片将其去掉。要不要都行\n",
    "prompt_split_as_tokens = [tokenizer.decode([token.item()]) for token in tokens]\n",
    "print(prompt_split_as_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### converting tokens to their embedding\n",
    "IM SORRY but this is the only part of the codebase where i use an inbuilt neural network module\n",
    "<br>\n",
    "anyway, so our [10x1] tokens are now [4096], i.e. 10 embeddings (one for each token) of length 4096\n",
    "<br>\n",
    "<br>\n",
    "note: keep track of the shapes, it makes it much easier to understand everything"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1602,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 4096])"
      ]
     },
     "execution_count": 1602,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_layer = torch.nn.Embedding(vocab_size, dim)\n",
    "embedding_layer.weight.data.copy_(model[\"model.tok_embeddings.weight\"])\n",
    "token_embeddings_unnormalized = embedding_layer(tokens).to(torch.bfloat16)\n",
    "token_embeddings_unnormalized.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### we then normalize the embedding using rms normalization\n",
    "please, note after this step the shapes dont change, the values are just normalized\n",
    "<br>\n",
    "things to keep in mind, we need a norm_eps (from config) because we dont want to accidently set rms to 0 and divide by 0\n",
    "<br>\n",
    "here is the formula:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1603,
   "metadata": {},
   "outputs": [],
   "source": [
    "# def rms_norm(tensor, norm_weights):\n",
    "#     rms = (tensor.pow(2).mean(-1, keepdim=True) + norm_eps)**0.5\n",
    "#     return tensor * (norm_weights / rms)\n",
    "def rms_norm(tensor, norm_weights):\n",
    "    return (tensor * torch.rsqrt(tensor.pow(2).mean(-1, keepdim=True) + norm_eps)) * norm_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### building the first layer of the transformer\n",
    "\n",
    "### normalization\n",
    "you will see me accessing layer.0 from the model dict (this is the first layer)\n",
    "<br>\n",
    "anyway, so after normalizing our shapes are still [10x4096] same as embedding but normalized "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1604,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 4096])"
      ]
     },
     "execution_count": 1604,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_embeddings = rms_norm(token_embeddings_unnormalized, model[\"model.layers.0.attention_norm.weight\"])\n",
    "token_embeddings.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### attention implemented from scratch\n",
    "let's load the attention heads of the first layer of the transformer\n",
    "\n",
    "&gt; when we load the weight of query/key/value and weights of output vectors from the model we notice the shapes to be [6144x4096] , [4096x4096]\n",
    "<br>\n",
    "&gt; at first glance this is weird because ideally we want each q,k,v and o for each head individually\n",
    "<br>\n",
    "&gt; the authors of the code bundled them togeather because its easy it helps parallize attention head multiplication.\n",
    "<br>\n",
    "&gt; im going to unwrap everything... \n",
    "<br>\n",
    "&gt; Note: 6144 = (n_heads + 2 * n_kv_heads) * head_dim = (32 + 2 * 8) * 128， 2 means k and v. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1605,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([6144, 4096]) torch.Size([4096, 4096])\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    model[\"model.layers.0.attention.wqkv.weight\"].shape,\n",
    "    model[\"model.layers.0.attention.wo.weight\"].shape,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### unwrapping query\n",
    "in the next section we will unwrap the queries from multiple attention heads, the resulting shape is [32x128x4096]\n",
    "<br><br>\n",
    "here, 32 is the number of attention heads in interlm2.5-7B, 128 is the size of the query vector and 4096 is the size of the token embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1606,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([32, 128, 4096])\n"
     ]
    }
   ],
   "source": [
    "qkv_layer0 = model[\"model.layers.0.attention.wqkv.weight\"]\n",
    "head_dim = dim // n_heads\n",
    "\n",
    "q_layer0, k_layer0, v_layer0 = qkv_layer0.reshape(n_kv_heads, 2+n_heads//n_kv_heads, head_dim, dim).split([n_heads//n_kv_heads, 1, 1], dim=1)\n",
    "q_layer0, k_layer0, v_layer0 = q_layer0.reshape(-1, dim), k_layer0.reshape(-1, dim), v_layer0.reshape(-1, dim)\n",
    "\n",
    "q_layer0 = q_layer0.reshape(n_heads, head_dim, dim)\n",
    "print(q_layer0.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### im going to implement the first head of the first layer\n",
    "here i access the query weight matrix first head of the first layer, the size of this query weight matrix is [128x4096]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1607,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([128, 4096])"
      ]
     },
     "execution_count": 1607,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q_layer0_head0 = q_layer0[0]\n",
    "q_layer0_head0.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### we now multiply the query weights with the token embedding, to recive a query for the token\n",
    "here you can see the resulting shape is [10x128], this is because we have 10 tokens and for each token there is a 128 length query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1608,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 128])"
      ]
     },
     "execution_count": 1608,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q_per_token = torch.matmul(token_embeddings, q_layer0_head0.T)\n",
    "q_per_token.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### keys (almost the same as queries)\n",
    "im lazy, so im not going to go through the math for keys, the only things you need to keep in mind are:\n",
    "<br>\n",
    "&gt; keys generate key vectors also of dimention 128\n",
    "<br>\n",
    "&gt; keys have only 1/4th the number of the weights as queries, because the weights for keys are shared across 4 heads at a time, to reduce the number of computations need\n",
    "<br>\n",
    "&gt; keys are also rotated to add positional info, just like queries because of the same reasons "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1609,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([8, 128, 4096])\n"
     ]
    }
   ],
   "source": [
    "k_layer0 = k_layer0.reshape(n_kv_heads, head_dim, dim)\n",
    "print(k_layer0.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1610,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([128, 4096])"
      ]
     },
     "execution_count": 1610,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "k_layer0_head0 = k_layer0[0]\n",
    "k_layer0_head0.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1611,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 128])"
      ]
     },
     "execution_count": 1611,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "k_per_token = torch.matmul(token_embeddings, k_layer0_head0.T)\n",
    "k_per_token.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rope  positioning encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1612,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rotate_half(x):\n",
    "    x1 = x[..., : x.shape[-1] // 2]\n",
    "    x2 = x[..., x.shape[-1] // 2 :]\n",
    "    return torch.cat((-x2, x1), dim=-1)\n",
    "\n",
    "def apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):\n",
    "    q_embed = (q * cos) + (rotate_half(q) * sin)\n",
    "    k_embed = (k * cos) + (rotate_half(k) * sin)\n",
    "    return q_embed, k_embed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1613,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rotary_emb(position_ids):\n",
    "    inv_freq = 1.0 / (rope_theta ** (torch.arange(0, head_dim, 2, dtype=torch.int64).float() / head_dim))\n",
    "    # t = torch.arange(len(position_ids), device='cpu', dtype=torch.float32)\n",
    "    freqs = torch.outer(position_ids, inv_freq)\n",
    "\n",
    "    emb = torch.cat((freqs, freqs), dim=-1)\n",
    "    cos_cached = emb.cos().to(dtype=torch.bfloat16)\n",
    "    sin_cached = emb.sin().to(dtype=torch.bfloat16)\n",
    "    return cos_cached, sin_cached"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1614,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([10, 128]), torch.Size([10, 128]))"
      ]
     },
     "execution_count": 1614,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "position_ids = torch.arange(len(tokens))\n",
    "cos, sin = rotary_emb(position_ids)\n",
    "q_per_token_rotated, k_per_token_rotated = apply_rotary_pos_emb(q_per_token, k_per_token, cos, sin, position_ids)\n",
    "q_per_token_rotated.shape, k_per_token_rotated.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### at this stage now have both the rotated values of queries and keys, for each token. \n",
    "each of the queries and keys are now of shape [10x128]. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### in the next step we will multiply the queries and key matrices\n",
    "doing this will give us a score mapping each token with one another\n",
    "<br>\n",
    "this score describes how well each token's query relates to the each tokens's key. \n",
    "THIS IS SELF ATTENTION :)\n",
    "<br>\n",
    "the shape of the attention score matrix (qk_per_token) is [10x10] where 10 is the number of tokens in the prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1615,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 10])"
      ]
     },
     "execution_count": 1615,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qk_per_token = torch.matmul(q_per_token_rotated, k_per_token_rotated.T)/(head_dim)**0.5\n",
    "qk_per_token.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### we now have to mask query key scores\n",
    "during the training process of internlm, the future token qk scores are masked.\n",
    "<br>\n",
    "why? because during training we only learn to predict tokens using past tokens.\n",
    "<br>\n",
    "as a result, during inference we set the future tokens to zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1616,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiYAAAGdCAYAAAAmK7htAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAAA+WElEQVR4nO3de3QU9f3/8dfmttmQGwgmASMBBBMhAkLl5k9AQ4kiX1FUBASiXGq/RAREhdbKTZtqtUJFikIlYkX5VtRWqaAg0SMqN0UBMUi4JNYAipAYkE2yO78/MCvLLhCyCTPZPB/nzDnsznxm3rtkN++835+ZsRmGYQgAAMACQswOAAAAoAqJCQAAsAwSEwAAYBkkJgAAwDJITAAAgGWQmAAAAMsgMQEAAJZBYgIAACwjzOwAgonb7da3336rmJgY2Ww2s8MBAJwjwzD0448/qnnz5goJqbu/3Y8fP67y8vKA9xMREaHIyMhaiMg6SExq0bfffqvk5GSzwwAABKioqEgXXXRRnez7+PHjatUyWvsPugLeV2Jiovbs2RNUyQmJSS2KiYmRJPVcNkZhUREmR/OLPTuTzA7BR6tXnWaH4GPPLXazQ/ARv9163daoWvgyrW32HyrMDsFH+OGfzA7BR0lqnNkh+CiPsdbPuKv8uLa/MtvzfV4XysvLtf+gS3s2t1RsAK+/9Ee3WnXZp/LychIT+FfVvgmLilBYI+v8kgtxWO8HNizMeq2uEId1/s+qhEZY60tbksLCrZeYhIWFmh2Cj7BQt9kh+AgLt953gcuCP+OSzks7PjYmJKDEJFiRmAAAYAKX4ZYrgNvougzrJb+1gcQEAAATuGXIrZpnJoGMtTISEwAATOCWW4HUPAIbbV00twAAgGVQMQEAwAQuw5DLqHk7JpCxVkZiAgCACZhj4h+tHAAAYBlUTAAAMIFbhlxUTHyQmAAAYAJaOf7RygEAAJZBYnIahmFo3LhxatKkiWw2m7Zs2WJ2SACAIFJ1Vk4gSzCilXOSrKwsHTlyRG+88YZWrlyp3Nxc5eXlqXXr1mratKnZ4QEAgoj75yWQ8cGIxOQ0CgoKlJSUpJ49e5odCgAADQaJiR9ZWVl64YUXJJ24w2TLli21d+9ec4MCAAQVV4Bn5QQy1spITPyYO3eu2rRpo+eee04bN25UaKj/W6o7nU45nU7P49LS0vMVIgCgnnMZCvDuwrUXi5Uw+dWPuLg4xcTEKDQ0VImJiWrWrJnf7XJychQXF+dZkpOTz3OkAID6yl0LSzAiMQnAtGnTVFJS4lmKiorMDgkAgHqNVk4A7Ha77Ha72WEAAOoht2xyyRbQ+GBEYgIAgAncxoklkPHBiFYOAACwDComAACYwBVgKyeQsVZGYnKS3Nxcz78nTpyoiRMnmhYLACC4kZj4RysHAABYBhUTAABM4DZschsBnJUTwFgrIzEBAMAEtHL8o5UDAAAsg4oJAAAmcClErgDqA65ajMVKSEwAADCBEeAcE4M5JgAAoLYwx8Q/5pgAAADLoGICAIAJXEaIXEYAc0yC9F45JCYAAJjALZvcATQu3ArOzIRWDgAAsAwqJnWgTcz3ioiOMDsMj8LDF5kdgo/Q45Vmh+DDCLXO/1mVsGNmR+Ar6r8/mR2Cj9BvvjM7BB/u0h/NDsGHo1kjs0PwYYRY63NXWXH+qhBMfvWPxAQAABMEPseEVg4AAECdomICAIAJTkx+DeAmfrRyAABAbXEHeEl6zsoBAACoY1RMAAAwAZNf/SMxAQDABG6FcIE1P0hMAAAwgcuwyRXAHYIDGWtlzDEBAACWQcUEAAATuAI8K8dFKwcAANQWtxEidwCTX91BOvmVVg4AALAMKiYAAJiAVo5/JCYAAJjArcDOrHHXXiiWQivnDPLy8mSz2XTkyBGzQwEAoEGgYgIAgAkCv8BacNYWgvNVAQBgcVWXpA9kqYlnnnlGKSkpioyMVLdu3bRhw4bTbpubmyubzea1REZG1vQlVwsVkwA4nU45nU7P49LSUhOjAQDgzJYtW6bJkydrwYIF6tatm+bMmaP+/fsrPz9fF154od8xsbGxys/P9zy22er2irNUTAKQk5OjuLg4z5KcnGx2SACAesItW8DLufrLX/6isWPH6s4779Rll12mBQsWKCoqSs8///xpx9hsNiUmJnqWhISEQF72WZGYBGDatGkqKSnxLEVFRWaHBACoJ2qrlVNaWuq1nFzJP1l5ebk2b96sjIwMz3MhISHKyMjQxx9/fNo4y8rK1LJlSyUnJ+vGG2/U9u3ba/eNOAWJSQDsdrtiY2O9FgAAqqPqOiaBLJKUnJzsVb3Pycnxe7zvv/9eLpfLp+KRkJCg/fv3+x1z6aWX6vnnn9e//vUv/eMf/5Db7VbPnj31zTff1O6bcRLmmAAAUI8VFRV5/WFst9trbd89evRQjx49PI979uyptLQ0Pfvss5o9e3atHedkJCYAAJjAbdjkDuQCaz+PrW7FvmnTpgoNDdWBAwe8nj9w4IASExOrdczw8HB17txZu3btOveAq4lWDgAAJnAH2MY51+uYREREqEuXLlqzZs0vMbjdWrNmjVdV5ExcLpe2bt2qpKSkczr2uaBicgZ9+vSREaR3bwQANDyTJ0/WqFGj1LVrV1155ZWaM2eOjh49qjvvvFOSNHLkSLVo0cIzT2XWrFnq3r27LrnkEh05ckR//vOftW/fPo0ZM6bOYiQxAQDABG4jRO4aXiStavy5GjJkiL777js9/PDD2r9/vzp16qSVK1d6JsQWFhYqJOSX/R4+fFhjx47V/v371bhxY3Xp0kUfffSRLrvsshrHfTYkJgAAmMAlm1w1uBbJyeNrIjs7W9nZ2X7X5eXleT1+6qmn9NRTT9XoODXFHBMAAGAZVEwAADCBGa2c+oDEBAAAE7hU83ZM1fhgFJzpFgAAqJeomAAAYAJaOf6RmAAAYIKTb8RX0/HBiMQEAAATGLLJHcAcEyOAsVYWnOkWAACol6iYAABgAlo5/pGY1IFEe4ki7eFmh+FRkew0OwQfRy9uZHYIPsJirfc+ORtHmR2Cj/LGtXdL9VoT18LsCHw4vj5odgg+jl9gne+lKpUOa7UjXKHnL57aurtwsAnOdAsAANRLVEwAADCBSyFyBVAfCGSslZGYAABgAlo5/gVnugUAAOolKiYAAJjArRC5A6gPBDLWykhMAAAwgcuwyRVAOyaQsVYWnOkWAACol6iYAABgAia/+kdiAgCACYwA7y5scOVXAABQW1yyyRXAjfgCGWtlwZluAQCAeomKCQAAJnAbgc0TcRu1GIyFkJgAAGACd4BzTAIZa2X14lXl5uYqPj7e7DAAAEAdo2ICAIAJ3LLJHcAE1kDGWhmJCQAAJuDKr/7Vi1ZOlVWrViktLU3R0dHKzMxUcXGxZ93GjRvVr18/NW3aVHFxcerdu7c+/fRTz/phw4ZpyJAhXvurqKhQ06ZNtWTJEkmS2+1WTk6OWrVqJYfDoY4dO+rVV189Py8OAADUn8Tk2LFjeuKJJ/Tiiy/qgw8+UGFhoaZMmeJZ/+OPP2rUqFH68MMP9cknn6ht27a6/vrr9eOPP0qShg8frjfffFNlZWWeMatWrdKxY8d00003SZJycnK0ZMkSLViwQNu3b9ekSZN0xx136P333/cbk9PpVGlpqdcCAEB1VE1+DWQJRvWmlVNRUaEFCxaoTZs2kqTs7GzNmjXLs/6aa67x2v65555TfHy83n//fd1www3q37+/GjVqpNdff10jRoyQJC1dulT/8z//o5iYGDmdTv3xj3/U6tWr1aNHD0lS69at9eGHH+rZZ59V7969fWLKycnRzJkz6+olAwCCmFsBXpI+SOeY1Jt0KyoqypOUSFJSUpIOHjzoeXzgwAGNHTtWbdu2VVxcnGJjY1VWVqbCwkJJUlhYmG677Ta99NJLkqSjR4/qX//6l4YPHy5J2rVrl44dO6Z+/fopOjrasyxZskQFBQV+Y5o2bZpKSko8S1FRUV29fAAAGoR6UzEJDw/3emyz2WQYv1xdZtSoUTp06JDmzp2rli1bym63q0ePHiovL/dsM3z4cPXu3VsHDx7Uu+++K4fDoczMTEnytHhWrFihFi1aeB3Lbrf7jclut592HQAAZ2IEeFaOEaQVk3qTmJzNunXrNH/+fF1//fWSpKKiIn3//fde2/Ts2VPJyclatmyZ3n77bd16662ehOeyyy6T3W5XYWGh37YNAAC1ibsL+xc0iUnbtm314osvqmvXriotLdX9998vh8Phs92wYcO0YMEC7dy5U2vXrvU8HxMToylTpmjSpElyu9266qqrVFJSonXr1ik2NlajRo06ny8HABDkuPKrf0Hzqv7+97/r8OHDuuKKKzRixAhNmDBBF154oc92w4cP15dffqkWLVqoV69eXutmz56tP/zhD8rJyVFaWpoyMzO1YsUKtWrV6ny9DAAAGrR6UTHJyspSVlaW13ODBg3ymmPSuXNnbdy40WubW265xWdfaWlpXuNOZrPZdO+99+ree+8NPGgAAM6AVo5/9SIxAQAg2HBJev+CppUDAADqPyomAACYgFaOfyQmAACYgMTEP1o5AADAMqiYAABgAiom/pGYAABgAhIT/2jlAAAAy6BiAgCACQwFdi0S/5cKrf9ITAAAMAGtHP9ITAAAMAGJiX8kJnVg3fdtFPaT3ewwPOLij5kdgo/Ig5Fmh+DD9YN1/s+qWPF7x/5tmdkh+Pjp4hizQ/BRua/I7BB8xEaEmx2Cjx/Tm5kdgpfKCrfZITR4JCYAAJiAiol/JCYAAJiAxMQ/ThcGAACWQcUEAAATGIZNRgBVj0DGWhmJCQAAJnDLFtB1TAIZa2W0cgAAgGVQMQEAwARMfvWPxAQAABMwx8Q/WjkAAMAyqJgAAGACWjn+kZgAAGACWjn+kZgAAGACI8CKSbAmJswxAQAAlkHFBAAAExiSDCOw8cGIxAQAABO4ZZONK7/6oJUDAAAso14lJvv27dPAgQPVuHFjNWrUSO3bt9d//vMfz/pt27bpuuuuU3R0tBISEjRixAh9//33nvVut1s5OTlq1aqVHA6HOnbsqFdffdWzPi8vTzabTWvWrFHXrl0VFRWlnj17Kj8//7y+TgBA8Ks6KyeQJRjVq8Rk/Pjxcjqd+uCDD7R161Y99thjio6OliQdOXJE11xzjTp37qxNmzZp5cqVOnDggG677TbP+JycHC1ZskQLFizQ9u3bNWnSJN1xxx16//33vY7z+9//Xk8++aQ2bdqksLAw3XXXXX7jcTqdKi0t9VoAAKiOquuYBLIEo3o1x6SwsFCDBw9Wenq6JKl169aedfPmzVPnzp31xz/+0fPc888/r+TkZO3cuVMtW7bUH//4R61evVo9evTwjP/www/17LPPqnfv3p5xjz76qOfx1KlTNWDAAB0/flyRkZFe8eTk5GjmzJl19noBAGho6lViMmHCBP32t7/VO++8o4yMDA0ePFiXX365JOnzzz/X2rVrPRWUkxUUFKiiokLHjh1Tv379vNaVl5erc+fOXs9V7VOSkpKSJEkHDx7UxRdf7LXdtGnTNHnyZM/j0tJSJScnB/YiAQANgmEEeFZOkJ6WU69aOWPGjNHu3bs1YsQIbd26VV27dtXTTz8tSSorK9PAgQO1ZcsWr+Xrr7/W1VdfrbKyMknSihUrvNZ/+eWXXvNMJCk8PNzzb5vtRKnM7Xb7xGO32xUbG+u1AABQHWbNMXnmmWeUkpKiyMhIdevWTRs2bDjj9v/85z+VmpqqyMhIpaene83trAv1KjGRpOTkZN1999167bXXdN9992nhwoWSpCuuuELbt29XSkqKLrnkEq+lUaNGuuyyy2S321VYWOiznioHAKAhWLZsmSZPnqzp06fr008/VceOHdW/f38dPHjQ7/YfffSRhg4dqtGjR+uzzz7ToEGDNGjQIG3btq3OYqxXicnEiRO1atUq7dmzR59++qnWrl2rtLQ0SScmxv7www8aOnSoNm7cqIKCAq1atUp33nmnXC6XYmJiNGXKFE2aNEkvvPCCCgoK9Omnn+rpp5/WCy+8YPIrAwA0NGZUTP7yl79o7NixuvPOO3XZZZdpwYIFioqK0vPPP+93+7lz5yozM1P333+/0tLSNHv2bF1xxRWaN29eoC//tOpVYuJyuTR+/HilpaUpMzNT7dq10/z58yVJzZs317p16+RyufTrX/9a6enpmjhxouLj4xUScuJlzp49W3/4wx+Uk5Pj2ceKFSvUqlUrM18WAKABOt9n5ZSXl2vz5s3KyMjwPBcSEqKMjAx9/PHHfsd8/PHHXttLUv/+/U+7fW2oV5Nfq+aTnE7btm312muvnXa9zWbTvffeq3vvvdfv+j59+sg4ZTZRp06dfJ4DACBQtTX59dRLVdjtdtntdp/tv//+e7lcLiUkJHg9n5CQoK+++srvMfbv3+93+/3799c88LOoVxUTAADgLTk5WXFxcZ4lJyfH7JACUq8qJgAABIsTFZOaXyStqmJSVFTkdVaov2qJJDVt2lShoaE6cOCA1/MHDhxQYmKi3zGJiYnntH1toGICAIAJamvy66mXrThdYhIREaEuXbpozZo1nufcbrfWrFnjufDoqXr06OG1vSS9++67p92+NlAxAQCggZg8ebJGjRqlrl276sorr9ScOXN09OhR3XnnnZKkkSNHqkWLFp520L333qvevXvrySef1IABA/TKK69o06ZNeu655+osRhITAABMYPy8BDL+XA0ZMkTfffedHn74Ye3fv1+dOnXSypUrPRNcCwsLPWeySlLPnj21dOlSPfTQQ/rd736ntm3b6o033lCHDh0CiPzMSEwAADBBoHcIrunY7OxsZWdn+12Xl5fn89ytt96qW2+9tUbHqgnmmAAAAMugYgIAgBnM6OXUAyQmAACYIcBWjgIZa2EkJgAAmKC2rvwabJhjAgAALIOKSR1wVobJVWmdt7a01GF2CD4Sfio3OwRfbv8XJTKTzW12BL5sFvwzLaKkwuwQ6gVbRaXZIfj4qYm1/j52lZ+/eMw6K8fqrPPbEwCAhsSwBTZPJEgTE2ulqgAAoEGjYgIAgAmY/OofiQkAAGbgOiZ+0coBAACWQcUEAAATcFaOfyQmAACYJUjbMYGglQMAACyDigkAACagleMfiQkAAGbgrBy/SEwAADCF7eclkPHBhzkmAADAMqiYAABgBlo5fp33iskbb7yhSy65RKGhoZo4cWKt7z83N1fx8fG1vl8AAGqVUQtLEDrviclvfvMb3XLLLSoqKtLs2bMD2ldKSormzJlTO4EBAADTnddWTllZmQ4ePKj+/furefPmNd5PeXm5IiIiajGys6uoqFB4ePh5PSYAIIgZthNLIOOD0HmrmOTl5SkmJkaSdM0118hmsykvL08zZsxQp06dvLadM2eOUlJSPI+zsrI0aNAgPfroo2revLkuvfRS9enTR/v27dOkSZNks9lks3n/B61atUppaWmKjo5WZmamiouLvdYvWrRIaWlpioyMVGpqqubPn+9Zt3fvXtlsNi1btky9e/dWZGSkXnrppdp9QwAADVrV3YUDWYLReauY9OzZU/n5+br00ku1fPly9ezZU02aNFFeXl61xq9Zs0axsbF69913JUlJSUnq2LGjxo0bp7Fjx3pte+zYMT3xxBN68cUXFRISojvuuENTpkzxJBcvvfSSHn74Yc2bN0+dO3fWZ599prFjx6pRo0YaNWqUZz9Tp07Vk08+qc6dOysyMtInJqfTKafT6XlcWlp6rm8LAAA4yXlLTCIiInThhRdKkpo0aaLExMRzGt+oUSMtWrTIq4UTGhqqmJgYn31VVFRowYIFatOmjSQpOztbs2bN8qyfPn26nnzySd18882SpFatWunLL7/Us88+65WYTJw40bONPzk5OZo5c+Y5vQ4AACRxVs5p1JvThdPT06s9ryQqKsqTlEgnqisHDx6UJB09elQFBQUaPXq0V6WlsrJScXFxXvvp2rXrGY8zbdo0TZ482fO4tLRUycnJ1YoRANDAMcfEL9MTk5CQEBmnNMoqKip8tmvUqFG193nqJFWbzeY5RllZmSRp4cKF6tatm9d2oaGh53RMu90uu91e7bgAAMCZmZ6YNGvWTPv375dhGJ4JrFu2bKnW2IiICLlcrnM6XkJCgpo3b67du3dr+PDh5xouAAC1wmacWAIZH4xMT0z69Omj7777To8//rhuueUWrVy5Um+//bZiY2PPOjYlJUUffPCBbr/9dtntdjVt2rRax5w5c6YmTJiguLg4ZWZmyul0atOmTTp8+LBXawYAgDrDHBO/TL9XTlpamubPn69nnnlGHTt21IYNGzRlypRqjZ01a5b27t2rNm3aqFmzZtU+5pgxY7Ro0SItXrxY6enp6t27t3Jzc9WqVauavgwAAM5N1RyTQJYgZDNOneCBGistLVVcXJy6vT5BYY2sM/fkvwfjzQ7BR9s55WaH4GPnqGizQ/AR+3Xo2Tc6z5q/d8jsEHxUxjvMDsGHbd0Ws0PwEZZysdkh+DiQ0cLsELy4yo/rixd+r5KSkmpV7mui6ndF8lOzFeLwvRRFdbl/Oq6iSX+o01jNYHorBwCABolWjl8kJgAAmIHExC/T55gAAABUoWICAIAZqJj4RWICAIAZuPKrX7RyAACAZVAxAQDABFz51T8SEwAAzMAcE79o5QAAAMsgMQEAAJZBKwcAABPYFOAck1qLxFpITOpAdIRTYRFmR/EL99Fws0Pw4WxqvXvANCqyXkxR37nNDsHHsaecZofgI2Km9e6VU3lNF7ND8HH4Ygt9Mf0sYc23ZofgpdJ9Hn++OV3YL1o5AADAMqiYAABgBs7K8YvEBAAAM5CY+EUrBwAAWAYVEwAATMCVX/0jMQEAwAy0cvyilQMAACyDigkAAGagYuIXiQkAACZgjol/tHIAAIBlUDEBAMAMXJLeLxITAADMwBwTv0hMAAAwAXNM/GOOCQAAsAwqJgAAmIFWjl8kJgAAmCHAVk6wJia0cgAAgGVQMQmA0+mU0+n0PC4tLTUxGgBAvUIrxy8qJgHIyclRXFycZ0lOTjY7JABAfWHUwhKESEwCMG3aNJWUlHiWoqIis0MCAKBeo5UTALvdLrvdbnYYAIB6iOuY+EfFBAAAWAaJCQAAsAwSkzPIzc2VzRacN0kCAJiMya9+McfkDPbs2aPevXubHQYAIAgxx8Q/KiZn8Pbbb+vxxx83OwwAQLCycLXkhx9+0PDhwxUbG6v4+HiNHj1aZWVlZxzTp08f2Ww2r+Xuu+8+p+NSMTmDDRs2mB0CAACmGD58uIqLi/Xuu++qoqJCd955p8aNG6elS5eecdzYsWM1a9Ysz+OoqKhzOi6JCQAAZrDwlV937NihlStXauPGjeratask6emnn9b111+vJ554Qs2bNz/t2KioKCUmJtb42LRyAAAwQdUck0AW6cTtUE5eTr5VSk19/PHHio+P9yQlkpSRkaGQkBCtX7/+jGNfeuklNW3aVB06dNC0adN07Nixczo2FRMAAOqxU2+HMn36dM2YMSOgfe7fv18XXnih13NhYWFq0qSJ9u/ff9pxw4YNU8uWLdW8eXN98cUXevDBB5Wfn6/XXnut2scmMQEAwAy11MopKipSbGys5+kzXZF86tSpeuyxx8642x07dtQ4pHHjxnn+nZ6erqSkJF177bUqKChQmzZtqrUPEhMAAExQW6cLx8bGeiUmZ3LfffcpKyvrjNu0bt1aiYmJOnjwoNfzlZWV+uGHH85p/ki3bt0kSbt27SIxAQAA3po1a6ZmzZqddbsePXroyJEj2rx5s7p06SJJeu+99+R2uz3JRnVs2bJFkpSUlFTtMUx+BQDADBa+8mtaWpoyMzM1duxYbdiwQevWrVN2drZuv/12zxk5//3vf5Wamuq5tEZBQYFmz56tzZs3a+/evfr3v/+tkSNH6uqrr9bll19e7WNTMQEAwAwWPl1YOnF2TXZ2tq699lqFhIRo8ODB+utf/+pZX1FRofz8fM9ZNxEREVq9erXmzJmjo0ePKjk5WYMHD9ZDDz10TsclMQEAAD6aNGlyxouppaSkyDB+yY6Sk5P1/vvvB3xcEpM6cFncftmjw80Ow+Mr58Vmh+DDGW+9LmJZarnZIfgwQiLMDsHXe6e/sJJZLnQEft2G2hZ2rNLsEHyEVFrne6nKd72t9fPkKj8u7T0/x+JeOf6RmAAAYAaLt3LMQmICAIAZSEz8sl49HQAANFhUTAAAMAFzTPwjMQEAwAy0cvyilQMAACyDigkAACagleMfiQkAAGagleMXrRwAAGAZVEwAADADFRO/SEwAADCB7eclkPHBiFYOAACwDComAACYgVaOX1RMfpabm6v4+HizwwAANBBVpwsHsgQjEpOfDRkyRDt37vQ8njFjhjp16mReQACA4GbUwhKEaOX8zOFwyOFwmB0GAAANGhWTn53cysnNzdXMmTP1+eefy2azyWazKTc319T4AABBiGqJDyomfgwZMkTbtm3TypUrtXr1aklSXFycz3ZOp1NOp9PzuLS09LzFCACo37gkvX9UTPxwOByKjo5WWFiYEhMTlZiY6LfNk5OTo7i4OM+SnJxsQrQAAAQPEpMATJs2TSUlJZ6lqKjI7JAAAPUFk1/9opUTALvdLrvdbnYYAIB6iFaOf1RMTiMiIkIul8vsMAAAaFBITE4jJSVFe/bs0ZYtW/T99997TXIFACBgtHL8IjE5jcGDByszM1N9+/ZVs2bN9PLLL5sdEgAgiHDlV/+YY/KzrKwsZWVleR7b7Xa9+uqr5gUEAEADRGICAIAZuImfXyQmAACYgcTELxITAABMwOnC/jH5FQAAWAYVEwAAzEArxy8SEwAATGAzDNmMmmcXgYy1Mlo5AADAMqiYAABgBlo5fpGYAABgAs7K8Y9WDgAAsAwqJgAAmIFWjl8kJnVg1oVbFRtjnWLUh5e2NjsEH4e/TzQ7BB+XpBwwOwQfu39sYXYIPtwOl9kh+Dj63wizQ/ARUhludgg+jje2zvdSlbDj1vrt6j6P4dDK8c96P6UAAKDBomICAIAZaOX4RWICAIAJaOX4R2ICAIAZqJj4xRwTAABgGVRMAAAwSbC2YwJBYgIAgBkM48QSyPggRCsHAABYBhUTAABMwFk5/pGYAABgBs7K8YtWDgAAsAwqJgAAmMDmPrEEMj4YkZgAAGAGWjl+NfhWTm5uruLj480OAwAAiMREQ4YM0c6dO80OAwDQwFSdlRPIEowafCvH4XDI4XCYHQYAoKHhAmt+NfiKyamtnM8//1x9+/ZVTEyMYmNj1aVLF23atMm8AAEAQYmKiX8NvmJyquHDh6tz587629/+ptDQUG3ZskXh4eF+t3U6nXI6nZ7HpaWl5ytMAACCEonJKQoLC3X//fcrNTVVktS2bdvTbpuTk6OZM2eer9AAAMGEs3L8avCtnFNNnjxZY8aMUUZGhv70pz+poKDgtNtOmzZNJSUlnqWoqOg8RgoAqM9o5fhHYnKKGTNmaPv27RowYIDee+89XXbZZXr99df9bmu32xUbG+u1AACAmiMx8aNdu3aaNGmS3nnnHd18881avHix2SEBAIJN1Vk5gSxBiMTkJD/99JOys7OVl5enffv2ad26ddq4caPS0tLMDg0AEGRo5fjH5NeThIaG6tChQxo5cqQOHDigpk2b6uabb2aCKwAA50mDT0yysrKUlZUlSYqIiNDLL79sbkAAgIaBs3L8avCJCQAAZgi0HROsrRzmmAAAAMugYgIAgBncxoklkPFBiMQEAAAzMMfEL1o5AACYwKYATxeu4/geffRR9ezZU1FRUV43uz0TwzD08MMPKykpSQ6HQxkZGfr666/P6bgkJgAAwEd5ebluvfVW/fa3v632mMcff1x//etftWDBAq1fv16NGjVS//79dfz48Wrvg1YOAABmCPTqrXV85deqa3jl5uZWa3vDMDRnzhw99NBDuvHGGyVJS5YsUUJCgt544w3dfvvt1doPFRMAAExQW1d+LS0t9VqcTqcpr2fPnj3av3+/MjIyPM/FxcWpW7du+vjjj6u9HxITAADqseTkZMXFxXmWnJwcU+LYv3+/JCkhIcHr+YSEBM+66iAxAQDADEYtLJKKiopUUlLiWaZNm3baQ06dOlU2m+2My1dffVVHL7h6mGMCAIAJbIYhWwDzRKrGxsbGKjY2tlpj7rvvPs9tWE6ndevWNYonMTFRknTgwAElJSV5nj9w4IA6depU7f2QmNSBI65jcrmsU4yKCHWZHYIPW6XZEfgKseBFAYww68VkK7fOz3aVyiizI/DlDrfe+1QRbXYEvspj6/qk13PjclorntrWrFkzNWvWrE723apVKyUmJmrNmjWeRKS0tFTr168/pzN7rPfJAQCgIXDXwlKHCgsLtWXLFhUWFsrlcmnLli3asmWLysrKPNukpqbq9ddflyTZbDZNnDhRjzzyiP79739r69atGjlypJo3b65BgwZV+7hUTAAAMEFttXLqysMPP6wXXnjB87hz586SpLVr16pPnz6SpPz8fJWUlHi2eeCBB3T06FGNGzdOR44c0VVXXaWVK1cqMjKy2sclMQEAAD5yc3PPeg0T45TkyGazadasWZo1a1aNj0tiAgCAGbhXjl8kJgAAmMHiV341C4kJAAAmOPnqrTUdH4w4KwcAAFgGFRMAAMxAK8cvEhMAAExgc59YAhkfjGjlAAAAy6BiAgCAGWjl+EViAgCAGbiOiV+0cgAAgGU0mMQkNzdX8fHxZocBAICkX+6VE8gSjGjlAABgBuaY+NVgKiYAAMD6Glxi8sYbb6ht27aKjIxU//79VVRUJEnau3evQkJCtGnTJq/t58yZo5YtW8rtDtITxgEA5jAkuQNYgrNg0rASk2PHjunRRx/VkiVLtG7dOh05ckS33367JCklJUUZGRlavHix15jFixcrKytLISG+b5XT6VRpaanXAgBAdTDHxL8GlZhUVFRo3rx56tGjh7p06aIXXnhBH330kTZs2CBJGjNmjF5++WU5nU5J0qeffqqtW7fqzjvv9Lu/nJwcxcXFeZbk5OTz9loAAPWcoV/mmdRoMfsF1I0GlZiEhYXpV7/6ledxamqq4uPjtWPHDknSoEGDFBoaqtdff13SiTN5+vbtq5SUFL/7mzZtmkpKSjxLVVsIAADUTINKTM4mIiJCI0eO1OLFi1VeXq6lS5fqrrvuOu32drtdsbGxXgsAANUSULUkwDN6LKxBJSaVlZVek1vz8/N15MgRpaWleZ4bM2aMVq9erfnz56uyslI333yzGaECAIJdIBNfq5Yg1KASk/DwcN1zzz1av369Nm/erKysLHXv3l1XXnmlZ5u0tDR1795dDz74oIYOHSqHw2FixAAANCwNKjGJiorSgw8+qGHDhqlXr16Kjo7WsmXLfLYbPXq0ysvLz9jGAQAgEJyV41+DufJrVlaWsrKyJOms7Zn//ve/Sk9P95ooCwBAreLKr341qIrJ2ZSVlWnbtm2aN2+e7rnnHrPDAQCgwSExOUl2dra6dOmiPn360MYBANQtzsrxq8G0cqojNzdXubm5ZocBAGgIaOX4RcUEAABYBhUTAADM4JZkC3B8ECIxAQDABIGe8svpwgAAoPYwx8Qv5pgAAADLoGICAIAZ3IZkC6Dq4Q7OigmJCQAAZqCV4xetHAAAYBlUTOrAEbdbLgudxpXT5jWzQ/Ax+j3rXfL/okZHzA7Bx05Hotkh+IjZHmF2CD5+bGW9vxwdB8yOwJfLYb33yWU3OwJv7uPn8z0K9Oqt1vv/rA0kJgAAmIFWjl+0cgAAgGVQMQEAwAxuQwG1YzgrBwAA1BrDfWIJZHwQopUDAAAsg4oJAABmYPKrXyQmAACYgTkmfpGYAABgBiomfjHHBAAAWAYVEwAAzGAowIpJrUViKSQmAACYgVaOX7RyAACAZVAxAQDADG63pAAukubmAmuWlpubq/j4eM/jGTNmqFOnTqbFAwDAGVW1cgJZglCdJSZ5eXmy2Ww6cuRIXR3ijKZMmaI1a9aYcmwAAFAzQdvKiY6OVnR0tNlhAADgH5Nf/QqoYrJv3z4NHDhQjRs3VqNGjdS+fXv95z//0d69e9W3b19JUuPGjWWz2ZSVlSVJWrlypa666irFx8frggsu0A033KCCggLPPvfu3SubzabXXntNffv2VVRUlDp27KiPP/7Y69i5ubm6+OKLFRUVpZtuukmHDh3yWn9qKycrK0uDBg3SE088oaSkJF1wwQUaP368KioqPNsUFxdrwIABcjgcatWqlZYuXaqUlBTNmTMnkLcJAABfbiPwJQgFlJiMHz9eTqdTH3zwgbZu3arHHntM0dHRSk5O1vLlyyVJ+fn5Ki4u1ty5cyVJR48e1eTJk7Vp0yatWbNGISEhuummm+Q+ZRLP73//e02ZMkVbtmxRu3btNHToUFVWVkqS1q9fr9GjRys7O1tbtmxR37599cgjj5w13rVr16qgoEBr167VCy+8oNzcXOXm5nrWjxw5Ut9++63y8vK0fPlyPffcczp48OBp9+d0OlVaWuq1AACAmguolVNYWKjBgwcrPT1dktS6dWvPuiZNmkiSLrzwQq9JqYMHD/bax/PPP69mzZrpyy+/VIcOHTzPT5kyRQMGDJAkzZw5U+3bt9euXbuUmpqquXPnKjMzUw888IAkqV27dvroo4+0cuXKM8bbuHFjzZs3T6GhoUpNTdWAAQO0Zs0ajR07Vl999ZVWr16tjRs3qmvXrpKkRYsWqW3btqfdX05OjmbOnHm2twkAAB+G4ZZh1PzMmkDGWllAFZMJEybokUceUa9evTR9+nR98cUXZx3z9ddfa+jQoWrdurViY2OVkpIi6USSc7LLL7/c8++kpCRJ8lQvduzYoW7dunlt36NHj7Meu3379goNDfXab9U+8/PzFRYWpiuuuMKz/pJLLlHjxo1Pu79p06appKTEsxQVFZ01BgAAJJ2YIxJIG4c5Jr7GjBmj3bt3a8SIEdq6dau6du2qp59++oxjBg4cqB9++EELFy7U+vXrtX79eklSeXm513bh4eGef9tsNknyafecq5P3WbXfQPZpt9sVGxvrtQAAUC2cLuxXwKcLJycn6+6779Zrr72m++67TwsXLpQkRURESJJcLpdn20OHDik/P18PPfSQrr32WqWlpenw4cPnfMy0tDRPQlPlk08+CeBVSJdeeqkqKyv12WefeZ7btWtXjeIDAAA1E9Ack4kTJ+q6665Tu3btdPjwYa1du1ZpaWmSpJYtW8pms+mtt97S9ddfL4fDocaNG+uCCy7Qc889p6SkJBUWFmrq1KnnfNwJEyaoV69eeuKJJ3TjjTdq1apVZ51fcjapqanKyMjQuHHj9Le//U3h4eG677775HA4PBUbAABqjdst2QLoBDDHxJfL5dL48eOVlpamzMxMtWvXTvPnz5cktWjRQjNnztTUqVOVkJCg7OxshYSE6JVXXtHmzZvVoUMHTZo0SX/+85/P+bjdu3fXwoULNXfuXHXs2FHvvPOOHnrooUBeiiRpyZIlSkhI0NVXX62bbrpJY8eOVUxMjCIjIwPeNwAAXmjl+GUzjCB9ZbXgm2++UXJyslavXq1rr732rNuXlpYqLi5On22/UDEx1rna/wGXw+wQfIxeeI/ZIfjofuPZJ2+fb+9tTzU7BB8x2yPMDsHH8Qut9zXmOGC9SquzsfXeJ5fd7Ai8uY8f196Hf6+SkpI6mzdY9bvi2uhhCrPV/PNUaZRrTdnSOo3VDEF75deaeO+991RWVqb09HQVFxfrgQceUEpKiq6++mqzQwMABBnD7ZYRQCsnWE8XJjE5SUVFhX73u99p9+7diomJUc+ePfXSSy/5nM0DAEDADEMSl6Q/FYnJSfr376/+/fubHQYAAA0WiQkAAGZwG5KNismpSEwAADCDYUgK5HTh4ExMrHPqCAAAaPComAAAYALDbcgIoJUTrFf7IDEBAMAMhluBtXI4XRgAANQSKib+MccEAABYBhWTWlSVvZaVWau8dtRlrXgkyeU8bnYIPsrLys0OwYf7J+u9Ty6n9X6e3Met95ejy2m9S9K7LPg+uS0Wkvv4ic/c+ahGVBrOgNoxlaqoxWisg3vl1KKqe+sAAOq3oqIiXXTRRXWy7+PHj6tVq1bav39/wPtKTEzUnj17gupmsyQmtcjtduvbb79VTEyMbLbA/loqLS1VcnKyioqKLHNzJmKqHmKqHmKqHmKqntqKyTAM/fjjj2revLlCQuputsPx48dVXh54lTYiIiKokhKJVk6tCgkJqfUMOzY21jIf/CrEVD3EVD3EVD3EVD21EVNcXFwtRXN6kZGRQZdQ1BYmvwIAAMsgMQEAAJZBYmJRdrtd06dPl91uNzsUD2KqHmKqHmKqHmKqHivGhJph8isAALAMKiYAAMAySEwAAIBlkJgAAADLIDGxMMMwNG7cODVp0kQ2m01btmwxO6Q6lZubq/j4eLPDsIw33nhDl1xyiUJDQzVx4sRa3z/vtzervR9WiccqcQTq1NcxY8YMderUybR4cHokJhaTlZWlQYMGSZJWrlyp3NxcvfXWWyouLlaHDh3MDe5neXl5stlsOnLkiNmhBLXf/OY3uuWWW1RUVKTZs2cHtK+UlBTNmTOndgILUkOGDNHOnTs9j83+xXVqPMHG7O+RKVOmaM2aNaYcG2fGlV8trKCgQElJSerZs6fZoeA8Kysr08GDB9W/f381b968xvspLy9XRERELUZ2dhUVFQoPDz+vx6wNDodDDofD7DA8rBZPsImOjlZ0dLTZYcAPKiYWlZWVpXvuuUeFhYWy2WxKSUkxO6TzZtWqVUpLS1N0dLQyMzNVXFzsWbdx40b169dPTZs2VVxcnHr37q1PP/3Us37YsGEaMmSI1/4qKirUtGlTLVmyRNKJexrl5OSoVatWcjgc6tixo1599VW/sezbt08DBw5U48aN1ahRI7Vv317/+c9/POu3bdum6667TtHR0UpISNCIESP0/fffe9af7VhVfzWuWbNGXbt2VVRUlNq3b6+YmBhJ0jXXXCObzaa8vDy/f8HPmTPH62ejquL26KOPqnnz5rr00kvVp08f7du3T5MmTZLNZvO5j9OZ3m9JWrRokdLS0hQZGanU1FTNnz/fs27v3r2y2WxatmyZevfurcjISL300kt+30urO7nUn5ubq5kzZ+rzzz/3vGe5ubmmxSNJn3/+ufr27auYmBjFxsaqS5cu2rRp03mL54033lDbtm0VGRmp/v37q6ioSNKJn4GQkBCfWObMmaMWLVrohhtu8Pn87N27V3379pUkNW7cWDabTVlZWZJOVIqvuuoqxcfH64ILLtANN9yggoICz36rfuZee+019e3bV1FRUerYsaM+/vhjr+Pn5ubq4osvVlRUlG666SYdOnTIa/2pn6eqz84TTzyhpKQkXXDBBRo/frwqKn65g29xcbEGDBggh8OhVq1aaenSpVQj64IBSxk1apRx4403GkeOHDFmzZplXHTRRUZxcbFx8OBBs0PzWLt2rSHJOHz4cK3ud/HixUZ4eLiRkZFhbNy40di8ebORlpZmDBs2zLPNmjVrjBdffNHYsWOH8eWXXxqjR482EhISjNLSUsMwDOOtt94yHA6H8eOPP3rGvPnmm4bD4fBs88gjjxipqanGypUrjYKCAmPx4sWG3W438vLyfGIaMGCA0a9fP+OLL74wCgoKjDfffNN4//33DcMwjMOHDxvNmjUzpk2bZuzYscP49NNPjX79+hl9+/b1jD/bsarey27duhl5eXnG9u3bjV69ehmdO3c2JBnLly83iouLDafTaUyfPt3o2LGjV3xPPfWU0bJlS8/jUaNGGdHR0caIESOMbdu2Gdu2bTMOHTpkXHTRRcasWbOM4uJio7i4uNrv9z/+8Q8jKSnJWL58ubF7925j+fLlRpMmTYzc3FzDMAxjz549hiQjJSXFs8233357zv/3VrB48WIjLi7OMAzDOHbsmHHfffcZ7du397xnx44dMy0ewzCM9u3bG3fccYexY8cOY+fOncb//d//GVu2bDkvcYSHhxtdu3Y1PvroI2PTpk3GlVdeafTs2dOzTb9+/Yz//d//9Rp3+eWXG23btvX7+amsrDSWL19uSDLy8/ON4uJi48iRI4ZhGMarr75qLF++3Pj666+Nzz77zBg4cKCRnp5uuFwuwzB++ZlLTU013nrrLSM/P9+45ZZbjJYtWxoVFRWGYRjGJ598YoSEhBiPPfaYkZ+fb8ydO9eIj4/3ej9P/TyNGjXKiI2NNe6++25jx44dxptvvmlERUUZzz33nGebjIwMo1OnTsYnn3xibN682ejdu7fhcDiMp556qpbf9YaNxMRiqhITw/D9pWMVdZmYSDJ27drlee6ZZ54xEhISTjvG5XIZMTExxptvvmkYhmFUVFQYTZs2NZYsWeLZZujQocaQIUMMwzCM48ePG1FRUcZHH33ktZ/Ro0cbQ4cO9dl/enq6MWPGDL/Hnj17tvHrX//a67mioiLPl211jlX1Xq5evdqzfsWKFYYkQ5Kxdu1az/PVTUwSEhIMp9PptV3Lli19vjyr8363adPGWLp0qc/r7tGjh2EYv/ySmDNnjp93qH45NRHw936bGU9MTIwnITzfcUgyPvnkE89zO3bsMCQZ69evNwzDMJYtW2Y0btzYOH78uGEYhrF582bDZrMZl1566Wk/P9X9Hvnuu+8MScbWrVsNw/jlZ27RokWebbZv325IMnbs2GEYxonP/PXXX++1nyFDhpw1MWnZsqVRWVnpee7WW2/1fHdUveaNGzd61n/99deGJBKTWkYrB5YSFRWlNm3aeB4nJSXp4MGDnscHDhzQ2LFj1bZtW8XFxSk2NlZlZWUqLCyUJIWFhem2227ztBOOHj2qf/3rXxo+fLgkadeuXTp27Jj69evn6TFHR0dryZIlXuXiKhMmTNAjjzyiXr16afr06friiy886z7//HOtXbvWaz+pqamSTswPOpdjXX755V6vORDp6enVnldypvf76NGjKigo0OjRo73if+SRR3zi79q1a0Ax4+wmT56sMWPGKCMjQ3/605/8/rzWlbCwMP3qV7/yPE5NTVV8fLx27NghSRo0aJBCQ0P1+uuvSzrRRunbt6+mTJly2s/P6Xz99dcaOnSoWrdurdjYWE+rsuozXsXfZ6bqZ3fHjh3q1q2b1/Y9evQ467Hbt2+v0NBQr/1W7TM/P19hYWG64oorPOsvueQSNW7c+Kz7xblh8iss5dRJkzabTcZJd00YNWqUDh06pLlz56ply5ay2+3q0aOHysvLPdsMHz5cvXv31sGDB/Xuu+/K4XAoMzNT0olJpZK0YsUKtWjRwutY/u6xMWbMGPXv318rVqzQO++8o5ycHD355JO65557VFZWpoEDB+qxxx7zGZeUlKRt27ZV+1gnv+5T54BUCQkJ8XovJHn1v6s0atTI73h/zvR+V71XCxcu9PmSP/nL+1yPiZqZMWOGhg0bphUrVujtt9/W9OnT9corr+imm24yOzRFRERo5MiRWrx4sW6++WYtXbpUc+fO1fDhw0/7+TmdgQMHqmXLllq4cKGaN28ut9utDh06eH3GJf+fGbfbHdDr8Pd5CHSfOHckJqhX1q1bp/nz5+v666+XJBUVFXlNNpWknj17Kjk5WcuWLdPbb7+tW2+91fOFc9lll8lut6uwsFC9e/eu1jGTk5N199136+6779a0adO0cOFC3XPPPbriiiu0fPlypaSkKCzM96NUk2OdSbNmzbR//34ZhuH5Iq7utW0iIiLkcrnO6XgJCQlq3ry5du/e7ak4NSQ1ec/qWrt27dSuXTtNmjRJQ4cO1eLFi89LYlJZWalNmzbpyiuvlHSienDkyBGlpaV5thkzZow6dOig+fPnq7KyUjfffLOk039+qqp6J7/Hhw4dUn5+vhYuXKj/9//+nyTpww8/POd409LStH79eq/nPvnkk3Pez8kuvfRSVVZW6rPPPlOXLl0knajAHj58OKD9wheJCeqVtm3b6sUXX1TXrl1VWlqq+++/3+8plcOGDdOCBQu0c+dOrV271vN8TEyMpkyZokmTJsntduuqq65SSUmJ1q1bp9jYWI0aNcprPxMnTtR1112ndu3a6fDhw1q7dq3ny3j8+PFauHChhg4dqgceeEBNmjTRrl279Morr2jRokXnfKyz6dOnj7777js9/vjjuuWWW7Ry5Uq9/fbbio2NPevYlJQUffDBB7r99ttlt9vVtGnTah1z5syZmjBhguLi4pSZmSmn06lNmzbp8OHDmjx58jnFX9+kpKRoz5492rJliy666CLFxMSYdufan376Sffff79uueUWtWrVSt988402btyowYMHn5fjh4eH65577tFf//pXhYWFKTs7W927d/ckKtKJZKB79+568MEHddddd8nhcJzx89OyZUvZbDa99dZbuv766+VwONS4cWNdcMEFeu6555SUlKTCwkJNnTr1nOOdMGGCevXqpSeeeEI33nijVq1apZUrVwb0HqSmpiojI0Pjxo3T3/72N4WHh+u+++6Tw+E4bZUTNcMcE9Qrf//733X48GFdccUVGjFihCZMmKALL7zQZ7vhw4fryy+/VIsWLdSrVy+vdbNnz9Yf/vAH5eTkKC0tTZmZmVqxYoVatWrlsx+Xy6Xx48d7tmvXrp3ndNnmzZtr3bp1crlc+vWvf6309HRNnDhR8fHxCgkJOedjnU1aWprmz5+vZ555Rh07dtSGDRs0ZcqUao2dNWuW9u7dqzZt2qhZs2bVPuaYMWO0aNEiLV68WOnp6erdu7dyc3NrFP+pcnNzLf2FPnjwYGVmZqpv375q1qyZXn75ZdNiCQ0N1aFDhzRy5Ei1a9dOt912m6677jrNnDnzvBw/KipKDz74oIYNG6ZevXopOjpay5Yt89lu9OjRKi8v11133SXpzJ+fFi1aaObMmZo6daoSEhKUnZ2tkJAQvfLKK9q8ebM6dOigSZMm6c9//vM5x9u9e3ctXLhQc+fOVceOHfXOO+/ooYceCuxNkLRkyRIlJCTo6quv1k033aSxY8cqJiZGkZGRAe8bv7AZpzatAeA8mD59ut5//33l5eWZHQpqyezZs/XPf/6zWpNcg8E333yj5ORkrV69Wtdee63Z4QQNWjkATPH2229r3rx5ZoeBWlBWVqa9e/dq3rx5euSRR8wOp8689957KisrU3p6uoqLi/XAAw8oJSVFV199tdmhBRUSEwCm2LBhg9khoJZkZ2fr5Zdf1qBBgzxtnGBUUVGh3/3ud9q9e7diYmLUs2dPvfTSS/XyFgxWRisHAABYBpNfAQCAZZCYAAAAyyAxAQAAlkFiAgAALIPEBAAAWAaJCQAAsAwSEwAAYBkkJgAAwDJITAAAgGX8f/Hn0IIkz6atAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "def display_qk_heatmap(qk_per_token):\n",
    "    _, ax = plt.subplots()\n",
    "    im = ax.imshow(qk_per_token.to(float).detach(), cmap='viridis')\n",
    "    ax.set_xticks(range(len(prompt_split_as_tokens)))\n",
    "    ax.set_yticks(range(len(prompt_split_as_tokens)))\n",
    "    ax.set_xticklabels(prompt_split_as_tokens)\n",
    "    ax.set_yticklabels(prompt_split_as_tokens)\n",
    "    ax.figure.colorbar(im, ax=ax)\n",
    "    \n",
    "display_qk_heatmap(qk_per_token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1617,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf],\n",
       "        [0., 0., -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf],\n",
       "        [0., 0., 0., -inf, -inf, -inf, -inf, -inf, -inf, -inf],\n",
       "        [0., 0., 0., 0., -inf, -inf, -inf, -inf, -inf, -inf],\n",
       "        [0., 0., 0., 0., 0., -inf, -inf, -inf, -inf, -inf],\n",
       "        [0., 0., 0., 0., 0., 0., -inf, -inf, -inf, -inf],\n",
       "        [0., 0., 0., 0., 0., 0., 0., -inf, -inf, -inf],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., -inf, -inf],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., -inf],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])"
      ]
     },
     "execution_count": 1617,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mask = torch.full((len(tokens), len(tokens)), float(\"-inf\"), device=tokens.device)\n",
    "mask = torch.triu(mask, diagonal=1)\n",
    "mask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1618,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi8AAAGdCAYAAADaPpOnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAABLh0lEQVR4nO3deVxU9f4/8NcMyzAIwxLIYgTuoCJuXxH1l5ooqHm1rFxIxBSvJZriStfcUMmyco/SFOlqesvyZhYuJPnIXDErjShNBZMlF5iA6wAz5/eHcXIcwGEGOIzzej4en8d1zvl8Pud9zmXg3efzOefIBEEQQERERGQh5FIHQERERFQXTF6IiIjIojB5ISIiIovC5IWIiIgsCpMXIiIisihMXoiIiMiiMHkhIiIii8LkhYiIiCyKrdQBPEx0Oh2uX78OZ2dnyGQyqcMhIqI6EgQBf/75J3x9fSGXN9x/39+5cwfl5eVm92Nvbw8HB4d6iMiyMHmpR9evX4efn5/UYRARkZlyc3Px6KOPNkjfd+7cQUt/J+QXas3uy9vbG5cvX7a6BIbJSz1ydnYGALR7/2XYOCokjuZvJ4b+S+oQiIgsglqthp+fn/j7vCGUl5cjv1CLy5n+UDmbPrqj/lOHlt2vory8nMkLma5qqsjGUdGkkheVSiV1CEREFqUxpv5VznKzkhdrxuSFiIhIAlpBB60Zr0bWCrr6C8bCMHkhIiKSgA4CdDA9ezGnraVj8kJERCQBHXQwZ+zEvNaWjZNtREREZFE48kJERCQBrSBAK5g+9WNOW0vH5IWIiEgCXPNiOk4bERERkUXhyAsREZEEdBCg5ciLSZi8EBERSYDTRqbjtBERERFZFCYvNRAEAVOmTIG7uztkMhnOnTsndUhERPQQqbrbyJxirThtdI+YmBgUFRVh7969SEtLQ0pKCjIyMtCqVSt4eHhIHR4RET1EdH8Vc9pbKyYvNbh06RJ8fHzQu3dvqUMhIiKiezB5qUZMTAy2b98O4O6bRf39/XHlyhVpgyIiooeK1sy7jcxpa+m45qUaa9euxbJly/Doo48iLy8Pp0+frraeRqOBWq3WK0RERMbQCuYXU2zcuBEBAQFwcHBAaGgoTp06VWPd/v37QyaTGZRhw4aJdWJiYgz2R0ZGmhackTjyUg0XFxc4OzvDxsYG3t7eNdZLSkrC0qVLGzEyIiJ6WEix5mX37t2Ij49HcnIyQkNDsWbNGkRERCA7OxvNmzc3qP/JJ5+gvLxc/Hzz5k2EhITg2Wef1asXGRmJbdu2iZ8VCoUJ0RmPIy9mSEhIQHFxsVhyc3OlDomIiKhGb731FmJjYzFx4kR06NABycnJcHR0xNatW6ut7+7uDm9vb7EcOnQIjo6OBsmLQqHQq+fm5tag58HkxQwKhQIqlUqvEBERGUMHGbRmFB1kAGCwfEGj0VR7vPLycmRmZiI8PFzcJpfLER4ejuPHjxsV8/vvv48xY8agWbNmetszMjLQvHlztG/fHi+++CJu3rxp4lUxDpMXIiIiCegE8wsA+Pn5wcXFRSxJSUnVHu/GjRvQarXw8vLS2+7l5YX8/PwHxnvq1CmcP38ekydP1tseGRmJ1NRUpKenY9WqVfj6668xZMgQaLVa0y6MEbjmhYiIyILl5ubqjfw31HqT999/H8HBwejZs6fe9jFjxoj/Dg4ORufOndG6dWtkZGRg4MCBDRILR16IiIgkYM6UUVUBYLB8oabkxcPDAzY2NigoKNDbXlBQUOvNKQBQWlqKXbt2YdKkSQ88r6oHu168eNHIK1F3TF7ukZKSgr179wIAZs6cyWe7EBFRg6mv5MVY9vb26N69O9LT08VtOp0O6enpCAsLq7XtRx99BI1Gg+eff/6Bx7l27Rpu3rwJHx+fOsVXF0xeiIiIrER8fDw2b96M7du3IysrCy+++CJKS0sxceJEAEB0dDQSEhIM2r3//vsYOXIkHnnkEb3tJSUlmDt3Lk6cOIErV64gPT0dI0aMQJs2bRAREdFg58E1L0RERBLQCTLohLqNntzfvq5Gjx6NP/74A4sWLUJ+fj66dOmCtLQ0cRFvTk4O5HL9cY3s7Gx88803OHjwoEF/NjY2+OGHH7B9+3YUFRXB19cXgwcPRmJiYoM+64XJCxERkQRMmfq5v70p4uLiEBcXV+2+jIwMg23t27eHUMMbrJVKJQ4cOGBSHObgtBERERFZFI68EBERSUALObRmjCE03FNUmj4mL0RERBIQzFzzIpjR1tIxeSEiIpKAVGteHgZc80JEREQWhSMvREREEtAKcmgFM9a8VH8DkFVg8kJERCQBHWTQmTEBooP1Zi+cNiIiIiKLwpGXBtDa/QbsmtlLHYYoZMbbUodg4Pt1s6QOgYhIUlywazomL0RERBIwf80Lp42IiIiILAJHXoiIiCRwd8GuGS9m5LQRERERNSadma8H4N1GRERERBaCIy9EREQS4IJd0zF5ISIikoAOcj6kzkRMXoiIiCSgFWTQmvFmaHPaWjqueSEiIiKLwpEXIiIiCWjNvNtIy2kjIiIiakw6QQ6dGQt2dVa8YJfTRkRERGRROPJCREQkAU4bmY7JCxERkQR0MO+OIV39hWJxOG1Ui4yMDMhkMhQVFUkdChEREf2FIy9EREQSMP8hddY7/sDkhYiISALmvx6AyQuZQKPRQKPRiJ/VarWE0RAREVkH603b6kFSUhJcXFzE4ufnJ3VIRERkIXSQmV2sFZMXMyQkJKC4uFgsubm5UodEREQWomrayJxirThtZAaFQgGFQiF1GEREZIHMf86L9SYv1nvmREREZJE48kJERCQBnSCDzpyH1JnR1tIxeSEiIpKAzsxpIz7nharVv39/CFb81k4iIqKmiMkLERGRBHSCHDoz7hgyp62ls94zJyIikpAWMrOLKTZu3IiAgAA4ODggNDQUp06dqrFuSkoKZDKZXnFwcNCrIwgCFi1aBB8fHyiVSoSHh+PXX381KTZjMXkhIiKyErt370Z8fDwWL16Ms2fPIiQkBBERESgsLKyxjUqlQl5enliuXr2qt//111/HunXrkJycjJMnT6JZs2aIiIjAnTt3Guw8mLwQERFJoGrayJxSV2+99RZiY2MxceJEdOjQAcnJyXB0dMTWrVtrbCOTyeDt7S0WLy8vcZ8gCFizZg0WLlyIESNGoHPnzkhNTcX169exd+9eUy6LUZi8EBERSUALc6eO6qa8vByZmZkIDw8Xt8nlcoSHh+P48eM1tispKYG/vz/8/PwwYsQIXLhwQdx3+fJl5Ofn6/Xp4uKC0NDQWvs0F5MXIiIiC6ZWq/XKvS8MvteNGzeg1Wr1Rk4AwMvLC/n5+dW2ad++PbZu3Yr//ve/+Pe//w2dTofevXvj2rVrACC2q0uf9YHJCxERkQTqa9rIz89P7yXBSUlJ9RZjWFgYoqOj0aVLF/Tr1w+ffPIJPD098e6779bbMUzBW6WJiIgkYO7LFava5ubmQqVSidtreueeh4cHbGxsUFBQoLe9oKAA3t7eRh3Tzs4OXbt2xcWLFwFAbFdQUAAfHx+9Prt06WL0udQVR16IiIgkIEAGnRlF+OtWaZVKpVdqSl7s7e3RvXt3pKeni9t0Oh3S09MRFhZmVMxarRY//vijmKi0bNkS3t7een2q1WqcPHnS6D5NwZEXIiIiKxEfH48JEyagR48e6NmzJ9asWYPS0lJMnDgRABAdHY0WLVqIU0/Lli1Dr1690KZNGxQVFeGNN97A1atXMXnyZAB370SaOXMmli9fjrZt26Jly5Z49dVX4evri5EjRzbYeTB5ISIikkB9TRvVxejRo/HHH39g0aJFyM/PR5cuXZCWliYuuM3JyYFc/ne/t2/fRmxsLPLz8+Hm5obu3bvj22+/RYcOHcQ68+bNQ2lpKaZMmYKioiL07dsXaWlpBg+zq08ygS/vqTdqtRouLi7459FRsHeykzoc0b6ve0gdgoH2W25JHYKBtPMrpA6BiCRW9Xu8uLhYbx1JQxxj9rEnoTDjb4WmpAJv9vm8QWNtqrjmhYiIiCwKp42IiIgkoIUcWjPGEMxpa+mYvBAREUlAJ8igE0x7uWJVe2tlvWkbERERWSSOvBAREUlABzl0ZowhmNPW0jF5ISIikoBWkEFrxtSPOW0tnfWmbURERGSROPJCREQkAS7YNR2TFyIiIgkI97wZ2tT21orJCxERkQS0kEELM9a8mNHW0llv2kZEREQWiSMvREREEtAJ5q1b0VnxmwmZvBAREUlAZ+aaF3PaWjqLOPOUlBS4urpKHQYRERE1ARx5ISIikoAOMujMWHRrTltLx+SFiIhIAnzCruksYtqoyoEDBxAUFAQnJydERkYiLy9P3Hf69GkMGjQIHh4ecHFxQb9+/XD27Flx/7hx4zB69Gi9/ioqKuDh4YHU1FQAgE6nQ1JSElq2bAmlUomQkBB8/PHHjXNyREREZBSLSV7KysqwevVqfPDBBzh69ChycnIwZ84ccf+ff/6JCRMm4JtvvsGJEyfQtm1bDB06FH/++ScAICoqCvv27UNJSYnY5sCBAygrK8NTTz0FAEhKSkJqaiqSk5Nx4cIFzJo1C88//zy+/vrramPSaDRQq9V6hYiIyBhVC3bNKdbKYqaNKioqkJycjNatWwMA4uLisGzZMnH/E088oVf/vffeg6urK77++ms8+eSTiIiIQLNmzfDpp59i/PjxAICdO3fiH//4B5ydnaHRaLBy5UocPnwYYWFhAIBWrVrhm2++wbvvvot+/foZxJSUlISlS5c21CkTEdFDTAczXw9gxWteLCZtc3R0FBMXAPDx8UFhYaH4uaCgALGxsWjbti1cXFygUqlQUlKCnJwcAICtrS2ee+457NixAwBQWlqK//73v4iKigIAXLx4EWVlZRg0aBCcnJzEkpqaikuXLlUbU0JCAoqLi8WSm5vbUKdPREREf7GYkRc7Ozu9zzKZDILw9xN6JkyYgJs3b2Lt2rXw9/eHQqFAWFgYysvLxTpRUVHo168fCgsLcejQISiVSkRGRgKAOJ20f/9+tGjRQu9YCoWi2pgUCkWN+4iIiGojmHm3kWDFIy8Wk7w8yLFjx7Bp0yYMHToUAJCbm4sbN27o1enduzf8/Pywe/dufPnll3j22WfFpKhDhw5QKBTIycmpdoqIiIioPvGt0qZ7aJKXtm3b4oMPPkCPHj2gVqsxd+5cKJVKg3rjxo1DcnIyfvnlFxw5ckTc7uzsjDlz5mDWrFnQ6XTo27cviouLcezYMahUKkyYMKExT4eIiB5yfMKu6R6aM3///fdx+/ZtdOvWDePHj8eMGTPQvHlzg3pRUVH46aef0KJFC/Tp00dvX2JiIl599VUkJSUhKCgIkZGR2L9/P1q2bNlYp0FEREQPIBPuXThCZlGr1XBxccE/j46CvZPdgxs0kn1f95A6BAPtt9ySOgQDaedXSB0CEUms6vd4cXExVCpVgx5jxMEXYNfM3uR+KkrL8d/BWxs01qbqoZk2IiIisiR8PYDpHpppIyIiIrIOHHkhIiKSAO82Mh2TFyIiIgkweTEdp42IiIjIonDkhYiISAIceTEdkxciIiIJMHkxHaeNiIiIyKJw5IWIiEgCAsx7Vos1P2GWyQsREZEEOG1kOiYvREREEmDyYjomLw3g6O+tYeOokDqMv3lqpI7AgJCbJ3UIBtove1vqEAxkL5oldQhE9JDZuHEj3njjDeTn5yMkJATr169Hz549q627efNmpKam4vz58wCA7t27Y+XKlXr1Y2JisH37dr12ERERSEtLa7Bz4IJdIiIiCVSNvJhT6mr37t2Ij4/H4sWLcfbsWYSEhCAiIgKFhYXV1s/IyMDYsWNx5MgRHD9+HH5+fhg8eDB+//13vXqRkZHIy8sTy4cffmjSNTEWkxciIiIJSJG8vPXWW4iNjcXEiRPRoUMHJCcnw9HREVu3bq22/o4dO/DSSy+hS5cuCAwMxJYtW6DT6ZCenq5XT6FQwNvbWyxubm4mXRNjMXkhIiKyYGq1Wq9oNNUvFSgvL0dmZibCw8PFbXK5HOHh4Th+/LhRxyorK0NFRQXc3d31tmdkZKB58+Zo3749XnzxRdy8edP0EzICkxciIiIJCILM7AIAfn5+cHFxEUtSUlK1x7tx4wa0Wi28vLz0tnt5eSE/P9+omOfPnw9fX1+9BCgyMhKpqalIT0/HqlWr8PXXX2PIkCHQarUmXpkH44JdIiIiCeggM+s5L1Vtc3NzoVKpxO0KRcPcMPLaa69h165dyMjIgIODg7h9zJgx4r+Dg4PRuXNntG7dGhkZGRg4cGCDxMKRFyIiIgumUqn0Sk3Ji4eHB2xsbFBQUKC3vaCgAN7e3rUeY/Xq1Xjttddw8OBBdO7cuda6rVq1goeHBy5evFi3E6kDJi9EREQSaOwFu/b29ujevbveYtuqxbdhYWE1tnv99deRmJiItLQ09OjR44HHuXbtGm7evAkfH586xVcXTF6IiIgkUF9rXuoiPj4emzdvxvbt25GVlYUXX3wRpaWlmDhxIgAgOjoaCQkJYv1Vq1bh1VdfxdatWxEQEID8/Hzk5+ejpKQEAFBSUoK5c+fixIkTuHLlCtLT0zFixAi0adMGERER9XOhqsE1L0RERFZi9OjR+OOPP7Bo0SLk5+ejS5cuSEtLExfx5uTkQC7/e1zjnXfeQXl5OZ555hm9fhYvXowlS5bAxsYGP/zwA7Zv346ioiL4+vpi8ODBSExMbLC1NwCTFyIiIklI9XqAuLg4xMXFVbsvIyND7/OVK1dq7UupVOLAgQMmxWEOJi9EREQSMHXq59721orJCxERkQQEM0derDl54YJdIiIisigceSEiIpKAAEAQzGtvrZi8EBERSUAHGWT18IRda8RpIyIiIrIoFpW8XL16FcOHD4ebmxuaNWuGjh074osvvhD3nz9/HkOGDIGTkxO8vLwwfvx43LhxQ9yv0+mQlJSEli1bQqlUIiQkBB9//LG4PyMjAzKZDOnp6ejRowccHR3Ru3dvZGdnN+p5EhHRw0+Kh9Q9LCwqeZk2bRo0Gg2OHj2KH3/8EatWrYKTkxMAoKioCE888QS6du2KM2fOIC0tDQUFBXjuuefE9klJSUhNTUVycjIuXLiAWbNm4fnnn8fXX3+td5x//etfePPNN3HmzBnY2trihRdeqDYejUZj8CpyIiIiYzT26wEeJha15iUnJwejRo1CcHAwgLsvf6qyYcMGdO3aFStXrhS3bd26FX5+fvjll1/g7++PlStX4vDhw+I7HFq1aoVvvvkG7777Lvr16ye2W7Fihfh5wYIFGDZsGO7cuaP3Fk3gbjK0dOnSBjtfIiIiMmRRycuMGTPw4osv4uDBgwgPD8eoUaPEt1t+//33OHLkiDgSc69Lly6hoqICZWVlGDRokN6+8vJydO3aVW/bvW/MrHqxVGFhIR577DG9egkJCYiPjxc/q9Vq+Pn5mXeSRERkFQTBzLuNrPh2I4tKXiZPnoyIiAjs378fBw8eRFJSEt58801Mnz4dJSUlGD58OFatWmXQzsfHB+fPnwcA7N+/Hy1atNDbf//7F+zs7MR/y2R3h+V0Op1BvwqFokHf3UBERA8vPmHXdBaVvACAn58fpk6diqlTpyIhIQGbN2/G9OnT0a1bN+zZswcBAQGwtTU8rQ4dOkChUCAnJ0dvioiIiIgsi0Ut2J05cyYOHDiAy5cv4+zZszhy5AiCgoIA3F3Me+vWLYwdOxanT5/GpUuXcODAAUycOBFarRbOzs6YM2cOZs2ahe3bt+PSpUs4e/Ys1q9fj+3bt0t8ZkREZG14t5HpLGrkRavVYtq0abh27RpUKhUiIyPx9ttvAwB8fX1x7NgxzJ8/H4MHD4ZGo4G/vz8iIyPF13snJibC09MTSUlJ+O233+Dq6opu3brhlVdekfK0iIjICukEGWQSvFX6YWBRycv69etr3d+2bVt88sknNe6XyWR4+eWX8fLLL1e7v3///hDuWwHVpUsXg21ERETm4oJd01nUtBERERGRRY28EBERPSzujryYc7dRPQZjYZi8EBERSYC3SpuO00ZERERkUTjyQkREJAHhr2JOe2vF5IWIiEgCnDYyHaeNiIiIyKJw5IWIiEgKnDcyGZMXIiIiKZj7iH8rnjZi8kJERCQBPmHXdFzzQkRERBaFIy8NoFIrh6BtOnmhcEshdQgGdH/+KXUIBmRaqSMw1PfpN6QOwcA3n8yVOgSihwLvNjIdkxciIiIpCDLz1q1YcfLSdIYHiIiIiIzAkRciIiIJcMGu6Zi8EBERSYHPeTEZp42IiIjIonDkhYiISAK828h0TF6IiIikYsVTP+bgtBERERFZFCYvREREEqiaNjKnmGLjxo0ICAiAg4MDQkNDcerUqVrrf/TRRwgMDISDgwOCg4PxxRdf3HceAhYtWgQfHx8olUqEh4fj119/NSk2YzF5ISIikoJQD6WOdu/ejfj4eCxevBhnz55FSEgIIiIiUFhYWG39b7/9FmPHjsWkSZPw3XffYeTIkRg5ciTOnz8v1nn99dexbt06JCcn4+TJk2jWrBkiIiJw586dugdoJCYvREREkpDVQ6mbt956C7GxsZg4cSI6dOiA5ORkODo6YuvWrdXWX7t2LSIjIzF37lwEBQUhMTER3bp1w4YNGwDcHXVZs2YNFi5ciBEjRqBz585ITU3F9evXsXfv3jrHZywmL0RERBZMrVbrFY1GU2298vJyZGZmIjw8XNwml8sRHh6O48ePV9vm+PHjevUBICIiQqx/+fJl5Ofn69VxcXFBaGhojX3WByYvREREUqinaSM/Pz+4uLiIJSkpqdrD3bhxA1qtFl5eXnrbvby8kJ+fX22b/Pz8WutX/W9d+qwPjZ687N27F23atIGNjQ1mzpxZ7/2npKTA1dW13vslIiKqV/WUvOTm5qK4uFgsCQkJjXseEmj05OWf//wnnnnmGeTm5iIxMdGsvgICArBmzZr6CYyIiMgCqVQqvaJQKKqt5+HhARsbGxQUFOhtLygogLe3d7VtvL29a61f9b916bM+NGryUlJSgsLCQkRERMDX1xfOzs4m9VNeXl7PkT1YRUVFox+TiIgeYoLM/FIH9vb26N69O9LT08VtOp0O6enpCAsLq7ZNWFiYXn0AOHTokFi/ZcuW8Pb21qujVqtx8uTJGvusD42WvGRkZIjJyhNPPAGZTIaMjAwsWbIEXbp00au7Zs0aBAQEiJ9jYmIwcuRIrFixAr6+vmjfvj369++Pq1evYtasWZDJZJDJ9P9PPHDgAIKCguDk5ITIyEjk5eXp7d+yZQuCgoLg4OCAwMBAbNq0Sdx35coVyGQy7N69G/369YODgwN27NhRvxeEiIisWtVbpc0pdRUfH4/Nmzdj+/btyMrKwosvvojS0lJMnDgRABAdHa037fTyyy8jLS0Nb775Jn7++WcsWbIEZ86cQVxcHABAJpNh5syZWL58OT777DP8+OOPiI6Ohq+vL0aOHFkfl6lajfZ6gN69eyM7Oxvt27fHnj170Lt3b7i7uyMjI8Oo9unp6VCpVDh06BAAwMfHByEhIZgyZQpiY2P16paVlWH16tX44IMPIJfL8fzzz2POnDliArJjxw4sWrQIGzZsQNeuXfHdd98hNjYWzZo1w4QJE8R+FixYgDfffBNdu3aFg4ODQUwajUZvVbdara7rZSEiImo0o0ePxh9//IFFixYhPz8fXbp0QVpamrjgNicnB3L53+MavXv3xs6dO7Fw4UK88soraNu2Lfbu3YtOnTqJdebNm4fS0lJMmTIFRUVF6Nu3L9LS0qr9u1lfGi15sbe3R/PmzQEA7u7udZ4La9asGbZs2QJ7e3txm42NDZydnQ36qqioQHJyMlq3bg0AiIuLw7Jly8T9ixcvxptvvomnn34awN1hr59++gnvvvuuXvIyc+ZMsU51kpKSsHTp0jqdBxEREQCTHzSn194EcXFx4sjJ/aobUHj22Wfx7LPP1tifTCbDsmXL9P7ONjSLeTFjcHCwXuJSG0dHRzFxAe6O0lQ9PbC0tBSXLl3CpEmT9EZsKisr4eLiotdPjx49aj1OQkIC4uPjxc9qtRp+fn5GxUhERFbOhHUrBu2tlOTJi1wuh3DfxF11i2ObNWtmdJ92dnZ6n2UymXiMkpISAMDmzZsRGhqqV8/GxqZOx1QoFDWu6iYiIqKGIXny4unpifz8fAiCIC66PXfunFFt7e3todVq63Q8Ly8v+Pr64rfffkNUVFRdwyUiIqoXMuFuMae9tZI8eenfvz/++OMPvP7663jmmWeQlpaGL7/8EiqV6oFtAwICcPToUYwZMwYKhQIeHh5GHXPp0qWYMWMGXFxcEBkZCY1GgzNnzuD27dt600BEREQNRqI1Lw8DyV8PEBQUhE2bNmHjxo0ICQnBqVOnMGfOHKPaLlu2DFeuXEHr1q3h6elp9DEnT56MLVu2YNu2bQgODka/fv2QkpKCli1bmnoaREREddPIz3l5mMiE+xeckMnUajVcXFzQ9t8LYOPYdNbC/O+aaQ8DbEhtXz4hdQgGcl/tLXUIBjx+rJQ6BAPffDJX6hCIGkzV7/Hi4mKjZgDMOYbf24mQK02/nVj3vzvInfVqg8baVEk+bURERGSVOG1kMiYvREREUmDyYjLJ17wQERER1QVHXoiIiKTAkReTMXkhIiKSAp+wazJOGxEREZFF4cgLERGRBPiEXdMxeSEiIpIC17yYjNNGREREZFGYvBAREZFF4bQRERGRBGQwc81LvUVieZi8NABHRQVsFE1nUKtS3fR+xG1b+EodggGXSzqpQzDgmFsqdQgGOuxdInUIBn4auUTqEIjqjrdKm6zp/IUlIiIiMgJHXoiIiKTAu41MxuSFiIhICkxeTMZpIyIiIrIoHHkhIiKSAJ+wazomL0RERFLgtJHJOG1EREREFoUjL0RERFLgyIvJmLwQERFJgGteTMdpIyIiIrIoHHkhIiKSAl8PYDImL0RERFLgmheTMXkhIiKSANe8mI5rXoiIiMiicOSFiIhICpw2MhmTFyIiIimYOW1kzckLp42IiIjIojB5MYNGo4FardYrRERERhHqoTSgW7duISoqCiqVCq6urpg0aRJKSkpqrT99+nS0b98eSqUSjz32GGbMmIHi4mK9ejKZzKDs2rWrTrFx2sgMSUlJWLp0qdRhEBGRJWria16ioqKQl5eHQ4cOoaKiAhMnTsSUKVOwc+fOautfv34d169fx+rVq9GhQwdcvXoVU6dOxfXr1/Hxxx/r1d22bRsiIyPFz66urnWKjcmLGRISEhAfHy9+VqvV8PPzkzAiIiIi82VlZSEtLQ2nT59Gjx49AADr16/H0KFDsXr1avj6+hq06dSpE/bs2SN+bt26NVasWIHnn38elZWVsLX9O+VwdXWFt7e3yfFx2sgMCoUCKpVKrxARERmj6jkv5hQABssXNBqN2bEdP34crq6uYuICAOHh4ZDL5Th58qTR/RQXF0OlUuklLgAwbdo0eHh4oGfPnti6dSsEoW7DSExeiIiILJifnx9cXFzEkpSUZHaf+fn5aN68ud42W1tbuLu7Iz8/36g+bty4gcTEREyZMkVv+7Jly/Cf//wHhw4dwqhRo/DSSy9h/fr1dYqP00ZEREQWLDc3V2/kX6FQ1Fh3wYIFWLVqVa39ZWVlmR2TWq3GsGHD0KFDByxZskRv36uvvir+u2vXrigtLcUbb7yBGTNmGN0/k5dapKSkYOLEiXUeziIiInqgelqwW5dlC7Nnz0ZMTEytdVq1agVvb28UFhbqba+srMStW7ceuFblzz//RGRkJJydnfHpp5/Czs6u1vqhoaFITEyERqOpNfG6F5OXWly+fBn9+vWTOgwiInoISfFuI09PT3h6ej6wXlhYGIqKipCZmYnu3bsDAL766ivodDqEhobW2E6tViMiIgIKhQKfffYZHBwcHnisc+fOwc3NzejEBWDyUqsvv/wSGzZskDoMIiJ6WDXRgf2goCBERkYiNjYWycnJqKioQFxcHMaMGSPeafT7779j4MCBSE1NRc+ePaFWqzF48GCUlZXh3//+t97zzzw9PWFjY4N9+/ahoKAAvXr1goODAw4dOoSVK1dizpw5dYqPyUstTp06JXUIREREktixYwfi4uIwcOBAyOVyjBo1CuvWrRP3V1RUIDs7G2VlZQCAs2fPincitWnTRq+vy5cvIyAgAHZ2dti4cSNmzZoFQRDQpk0bvPXWW4iNja1TbExeiIiIpNDEH1Ln7u5e4wPpACAgIEBvTWj//v0fuEY0MjJS7+F0pmLyQkREJAEp1rw8LPicFyIiIrIoHHkhIiKSQhOfNmrKmLwQERFJgNNGpuO0EREREVkUjrwQERFJgdNGJmPyQkREJAUmLybjtBERERFZFI68NIAg93zYO9lLHYbopMZD6hAMaJu7SR2CgRtdZFKHYECQO0sdgoFmnza96zTkpTipQzDw5XW+WoRqxwW7pmPyQkREJAVOG5mMyQsREZEUmLyYjGteiIiIyKJw5IWIiEgCXPNiOiYvREREUuC0kck4bUREREQWhSMvREREEuC0kemYvBAREUmB00Ym47QRERERWRSOvBAREUmBIy8mY/JCREQkAdlfxZz21orTRkRERGRROPJCREQkBU4bmYwjL39JSUmBq6ur1GEQEZGVqLpV2pxirZi8/GX06NH45ZdfxM9LlixBly5dpAuIiIgebkI9FCvFaaO/KJVKKJVKqcMgIiKiB+DIy1/unTZKSUnB0qVL8f3330Mmk0EmkyElJUXS+IiI6CHEUReTcOSlGqNHj8b58+eRlpaGw4cPAwBcXFwM6mk0Gmg0GvGzWq1utBiJiMiy8fUApuPISzWUSiWcnJxga2sLb29veHt7VzullJSUBBcXF7H4+flJEC0REZF1YfJihoSEBBQXF4slNzdX6pCIiMhScMGuyThtZAaFQgGFQiF1GEREZIE4bWQ6jrzUwN7eHlqtVuowiIiI6D5MXmoQEBCAy5cv49y5c7hx44bewlwiIiKzcdrIZExeajBq1ChERkZiwIAB8PT0xIcffih1SERE9BDhE3ZNxzUvf4mJiUFMTIz4WaFQ4OOPP5YuICIiIqoWkxciIiIp8MWMJuO0ERERkRSa+JqXW7duISoqCiqVCq6urpg0aRJKSkpqbdO/f3/xyfRVZerUqXp1cnJyMGzYMDg6OqJ58+aYO3cuKisr6xQbR16IiIgk0NRvlY6KikJeXh4OHTqEiooKTJw4EVOmTMHOnTtrbRcbG4tly5aJnx0dHcV/a7VaDBs2DN7e3vj222+Rl5eH6Oho2NnZYeXKlUbHxuSFiIiI9GRlZSEtLQ2nT59Gjx49AADr16/H0KFDsXr1avj6+tbY1tHREd7e3tXuO3jwIH766SccPnwYXl5e6NKlCxITEzF//nwsWbIE9vb2RsXHaSMiIiIp1NO0kVqt1iv18WiP48ePw9XVVUxcACA8PBxyuRwnT56ste2OHTvg4eGBTp06ISEhAWVlZXr9BgcHw8vLS9wWEREBtVqNCxcuGB0fR16IiIgkIBMEyATT536q2t7/Xr3FixdjyZIl5oSG/Px8NG/eXG+bra0t3N3dkZ+fX2O7cePGwd/fH76+vvjhhx8wf/58ZGdn45NPPhH7vTdxASB+rq3f+zF5ISIismC5ublQqVTi59peW7NgwQKsWrWq1v6ysrJMjmXKlCniv4ODg+Hj44OBAwfi0qVLaN26tcn93o/JCxERkRTq6VZplUqll7zUZvbs2XrPNKtOq1at4O3tjcLCQr3tlZWVuHXrVo3rWaoTGhoKALh48SJat24Nb29vnDp1Sq9OQUEBANSpXyYvREREEpDibiNPT094eno+sF5YWBiKioqQmZmJ7t27AwC++uor6HQ6MSExxrlz5wAAPj4+Yr8rVqxAYWGhOC116NAhqFQqdOjQweh+uWCXiIiI9AQFBSEyMhKxsbE4deoUjh07hri4OIwZM0a80+j3339HYGCgOJJy6dIlJCYmIjMzE1euXMFnn32G6OhoPP744+jcuTMAYPDgwejQoQPGjx+P77//HgcOHMDChQsxbdq0Wqe77sfkhYiISApN/CF1O3bsQGBgIAYOHIihQ4eib9++eO+998T9FRUVyM7OFu8msre3x+HDhzF48GAEBgZi9uzZGDVqFPbt2ye2sbGxweeffw4bGxuEhYXh+eefR3R0tN5zYYwhEwQzljqTHrVaDRcXF9z+pRVUzk0nLwz8ZrzUIRhwOugkdQgGKp4skjoEAxWn3aQOwUClU9P7lRGwr+zBlRqZvFwrdQgGDp5aLHUITV7V7/Hi4mKj15GYeoxuY1fAxt7B5H605Xdw9sN/NWisTVXT+QtLREREZAQu2CUiIpICX8xoMiYvREREEmjq7zZqypi8EBERSYEjLybjmhciIiKyKBx5ISIikog1T/2Yg8kLERGRFAThbjGnvZXitBERERFZFI68EBERSYB3G5mOyQsREZEUeLeRyThtRERERBaFIy9EREQSkOnuFnPaWysmL0RERFLgtJHJrH7aKCUlBa6urlKHQUREREay+uRl9OjR+OWXX6QOg4iIrEzV3UbmFGtl9dNGSqUSSqVS6jCIiMja8CF1JrP6kZf7p42+//57DBgwAM7OzlCpVOjevTvOnDkjXYBERPRQ4siL6ax+5OV+UVFR6Nq1K9555x3Y2Njg3LlzsLOzq7auRqOBRqMRP6vV6sYKk4iIyGoxeblPTk4O5s6di8DAQABA27Zta6yblJSEpUuXNlZoRET0MOHdRiaz+mmj+8XHx2Py5MkIDw/Ha6+9hkuXLtVYNyEhAcXFxWLJzc1txEiJiMiScdrIdExe7rNkyRJcuHABw4YNw1dffYUOHTrg008/rbauQqGASqXSK0RERNSwmLxUo127dpg1axYOHjyIp59+Gtu2bZM6JCIiethU3W1kTrFSTF7u8b///Q9xcXHIyMjA1atXcezYMZw+fRpBQUFSh0ZERA8ZThuZjgt272FjY4ObN28iOjoaBQUF8PDwwNNPP81FuURERE2I1ScvMTExiImJAQDY29vjww8/lDYgIiKyDrzbyGRWn7wQERFJwdypH2ueNuKaFyIiIrIoHHkhIiKSgk64W8xpb6WYvBAREUmBa15MxuSFiIhIAjKYueal3iKxPFzzQkRERBaFIy9ERERSMPcpuVb8hF0mL0RERBLgrdKm47QRERERWRSOvBAREUmBdxuZjCMvREREEpAJgtmlId26dQtRUVFQqVRwdXXFpEmTUFJSUmP9K1euQCaTVVs++uijv8+7mv27du2qU2wceWkAN7WlKNc2nbzQzk4rdQgG5JVSR2DIRq6TOgQD5U3wG2pb1vRu0KxQ2UsdggGtouldpx4vvCV1CAbObI2XOgSqQVRUFPLy8nDo0CFUVFRg4sSJmDJlCnbu3FltfT8/P+Tl5elte++99/DGG29gyJAhetu3bduGyMhI8bOrq2udYmuCvxqJiIisgO6vYk77BpKVlYW0tDScPn0aPXr0AACsX78eQ4cOxerVq+Hr62vQxsbGBt7e3nrbPv30Uzz33HNwcnLS2+7q6mpQty6azvAAERGRFWnK00bHjx+Hq6urmLgAQHh4OORyOU6ePGlUH5mZmTh37hwmTZpksG/atGnw8PBAz549sXXrVgh1PBeOvBAREVkwtVqt91mhUEChUJjVZ35+Ppo3b663zdbWFu7u7sjPzzeqj/fffx9BQUHo3bu33vZly5bhiSeegKOjIw4ePIiXXnoJJSUlmDFjhtHxceSFiIhICkI9FNxda+Li4iKWpKSkGg+5YMGCGhfVVpWff/7Z7FP73//+h507d1Y76vLqq6+iT58+6Nq1K+bPn4958+bhjTfeqFP/HHkhIiKSQj09YTc3NxcqlUrcXNuoy+zZsxETE1Nrt61atYK3tzcKCwv1tldWVuLWrVtGrVX5+OOPUVZWhujo6AfWDQ0NRWJiIjQajdEjRkxeiIiIJFBfT9hVqVR6yUttPD094enp+cB6YWFhKCoqQmZmJrp37w4A+Oqrr6DT6RAaGvrA9u+//z7+8Y9/GHWsc+fOwc3NrU5TXUxeiIiISE9QUBAiIyMRGxuL5ORkVFRUIC4uDmPGjBHvNPr9998xcOBApKamomfPnmLbixcv4ujRo/jiiy8M+t23bx8KCgrQq1cvODg44NChQ1i5ciXmzJlTp/iYvBAREUmhib+YcceOHYiLi8PAgQMhl8sxatQorFu3TtxfUVGB7OxslJWV6bXbunUrHn30UQwePNigTzs7O2zcuBGzZs2CIAho06YN3nrrLcTGxtYpNiYvREREEpDp7hZz2jckd3f3Gh9IBwABAQHV3uK8cuVKrFy5sto2kZGReg+nMxXvNiIiIiKLwpEXIiIiKTTxaaOmjMkLERGRFPhWaZNx2oiIiIgsitUkLykpKXV+ayUREVFDacrvNmrqOG1EREQkBa55MZnVjLwQERHRw8Hqkpe9e/eibdu2cHBwQEREBHJzcwEAV65cgVwux5kzZ/Tqr1mzBv7+/tDpGviGeiIisi4CAJ0ZxXoHXqwreSkrK8OKFSuQmpqKY8eOoaioCGPGjAFw92E74eHh2LZtm16bbdu2ISYmBnK54aXSaDRQq9V6hYiIyBhc82I6q0peKioqsGHDBoSFhaF79+7Yvn07vv32W5w6dQoAMHnyZHz44YfQaDQAgLNnz+LHH3/ExIkTq+0vKSlJ7zXkfn5+jXYuRERk4QT8ve7FpCL1CUjHqpIXW1tb/N///Z/4OTAwEK6ursjKygIAjBw5EjY2Nvj0008B3L1DacCAAQgICKi2v4SEBBQXF4ulagqKiIiIGo5VJS8PYm9vj+joaGzbtg3l5eXYuXMnXnjhhRrrKxQK8VXkdXklORERkXmjLmbeqWThrCp5qays1FuQm52djaKiIgQFBYnbJk+ejMOHD2PTpk2orKzE008/LUWoRET0sDNnsW5VsVJWlbzY2dlh+vTpOHnyJDIzMxETE4NevXqhZ8+eYp2goCD06tUL8+fPx9ixY6FUKiWMmIiIiO5nVcmLo6Mj5s+fj3HjxqFPnz5wcnLC7t27DepNmjQJ5eXltU4ZERERmYN3G5nOap6wGxMTg5iYGAB44FTQ77//juDgYL3FvURERPWKT9g1mVWNvDxISUkJzp8/jw0bNmD69OlSh0NERETVYPJyj7i4OHTv3h39+/fnlBERETUs3m1kMquZNjJGSkoKUlJSpA6DiIisAaeNTMaRFyIiIrIoHHkhIiKSgg6AzMz2VorJCxERkQTMvd2Zt0oTERFR4+KaF5NxzQsRERFZFI68EBERSUEnADIzRk901jvywuSFiIhICpw2MhmnjYiIiMiicOSlAdzWARVN6Ba2NZ0NXz4ptYS9U6QOwYCvSi11CAaym7lLHYKB5qeb0A/3X24F2UkdggGna1qpQzBQ4WjOfbkNo+O8t6UOQY9Wc6cRj2buU3Ktd+SFyQsREZEUOG1kMk4bERERkUXhyAsREZEUdALMmvrh3UZERETUqATd3WJOeyvFaSMiIiKyKBx5ISIikgIX7JqMyQsREZEUuObFZExeiIiIpMCRF5NxzQsREREZWLFiBXr37g1HR0e4uroa1UYQBCxatAg+Pj5QKpUIDw/Hr7/+qlfn1q1biIqKgkqlgqurKyZNmoSSkpI6xcbkhYiISAoC/h59Mak0bHjl5eV49tln8eKLLxrd5vXXX8e6deuQnJyMkydPolmzZoiIiMCdO38/uTgqKgoXLlzAoUOH8Pnnn+Po0aOYMqVuT13ntBEREZEUmvi00dKlSwEAKSkpRtUXBAFr1qzBwoULMWLECABAamoqvLy8sHfvXowZMwZZWVlIS0vD6dOn0aNHDwDA+vXrMXToUKxevRq+vr5GHYsjL0RERBZMrVbrFY1GI0kcly9fRn5+PsLDw8VtLi4uCA0NxfHjxwEAx48fh6urq5i4AEB4eDjkcjlOnjxp9LGYvBAREUlBpzO/APDz84OLi4tYkpKSJDmd/Px8AICXl5fedi8vL3Fffn4+mjdvrrff1tYW7u7uYh1jPDTJS0pKit6CoiVLlqBLly6SxUNERFQrs9a7/D3llJubi+LiYrEkJCTUeMgFCxZAJpPVWn7++efGugIma7A1LxkZGRgwYABu375t9Crl+jRnzhxMnz690Y9LRETUmFQqFVQqlVF1Z8+ejZiYmFrrtGrVyqQ4vL29AQAFBQXw8fERtxcUFIiDCd7e3igsLNRrV1lZiVu3bontjfHQLth1cnKCk5OT1GEQERFVT4IFu56envD09DT9mLVo2bIlvL29kZ6eLiYrarUaJ0+eFO9YCgsLQ1FRETIzM9G9e3cAwFdffQWdTofQ0FCjj2XWtNHVq1cxfPhwuLm5oVmzZujYsSO++OILXLlyBQMGDAAAuLm5QSaTiZleWloa+vbtC1dXVzzyyCN48skncenSJbHPK1euQCaT4ZNPPsGAAQPg6OiIkJAQcbFPlZSUFDz22GNwdHTEU089hZs3b+rtv3/aKCYmBiNHjsTq1avh4+ODRx55BNOmTUNFRYVYJy8vD8OGDYNSqUTLli2xc+dOBAQEYM2aNeZcJiIiIkM6wfzSgHJycnDu3Dnk5ORAq9Xi3LlzOHfunN4zWQIDA/Hpp58CAGQyGWbOnInly5fjs88+w48//ojo6Gj4+vpi5MiRAICgoCBERkYiNjYWp06dwrFjxxAXF4cxY8YYfacRYObIy7Rp01BeXo6jR4+iWbNm+Omnn+Dk5AQ/Pz/s2bMHo0aNQnZ2NlQqFZRKJQCgtLQU8fHx6Ny5M0pKSrBo0SI89dRTOHfuHOTyv3Opf/3rX1i9ejXatm2Lf/3rXxg7diwuXrwIW1tbnDx5EpMmTUJSUhJGjhyJtLQ0LF68+IHxHjlyBD4+Pjhy5AguXryI0aNHo0uXLoiNjQUAREdH48aNG8jIyICdnR3i4+MNhrfupdFo9FZ1q9VqUy8lERFRk7Jo0SJs375d/Ny1a1cAd/+W9u/fHwCQnZ2N4uJisc68efNQWlqKKVOmoKioCH379kVaWhocHBzEOjt27EBcXBwGDhwIuVyOUaNGYd26dXWKzazkJScnB6NGjUJwcDAA/Xkyd3d3AEDz5s311ryMGjVKr4+tW7fC09MTP/30Ezp16iRunzNnDoYNGwbg7r3mHTt2xMWLFxEYGIi1a9ciMjIS8+bNAwC0a9cO3377LdLS0mqN183NDRs2bICNjQ0CAwMxbNgwpKenIzY2Fj///DMOHz6sd+/5li1b0LZt2xr7S0pKEu+DJyIiqgtB0EEQdGa1b0gpKSkPfMaLcN/UlUwmw7Jly7Bs2bIa27i7u2Pnzp1mxWbWtNGMGTOwfPly9OnTB4sXL8YPP/zwwDa//vorxo4di1atWkGlUiEgIADA3UToXp07dxb/XbXwp2oUJCsry2BuLCws7IHH7tixI2xsbPT6reozOzsbtra26Natm7i/TZs2cHNzq7G/hIQEvRXeubm5D4yBiIgIwN01K+ZMGfHdRqaZPHkyfvvtN4wfPx4//vgjevTogfXr19faZvjw4bh16xY2b96MkydPig+lKS8v16tnZ2cn/lsmkwEAdDrzssx7+6zq15w+FQqFuMq7Lqu9iYiI6utWaWtk9nNe/Pz8MHXqVHzyySeYPXs2Nm/eDACwt7cHAGi1WrHuzZs3kZ2djYULF2LgwIEICgrC7du363zMoKAggyfxnThxwoyzANq3b4/Kykp899134raLFy+aFB8RERE1HLPWvMycORNDhgxBu3btcPv2bRw5cgRBQUEAAH9/f8hkMnz++ecYOnQolEol3Nzc8Mgjj+C9996Dj48PcnJysGDBgjofd8aMGejTpw9Wr16NESNG4MCBAw9c7/IggYGBCA8Px5QpU/DOO+/Azs4Os2fPhlKpFEd+iIiI6o1OB8jMmFFo4DUvTZlZIy9arRbTpk0Tb31q164dNm3aBABo0aIFli5digULFsDLywtxcXGQy+XYtWsXMjMz0alTJ8yaNQtvvPFGnY/bq1cvbN68GWvXrkVISAgOHjyIhQsXmnMqAP5+gdTjjz+Op556CrGxsXB2dtZbJU1ERFQvOG1kMplw/1JhEl27dg1+fn44fPgwBg4c+MD6arUaLi4uOH3BC07OTefNC7mVTW8tTsKyur3+vDE0n3hF6hAMZJ8KkDoEA81PN73/2it51ObBlRqZ0zXtgys1sjLPpnedKh2ljkCfVnMH2eteQXFxcYOtY6z6WzHQaRxsZfYm91MplCO9ZGeDxtpUPbRP2DXFV199hZKSEgQHByMvLw/z5s1DQEAAHn/8calDIyKih4yg00EwY9qooW+VbsqYvNyjoqICr7zyCn777Tc4Ozujd+/e2LFjh8FdSkRERGYTBACN+3qAhwWTl3tEREQgIiJC6jCIiIioFkxeiIiIpKATABlHXkzB5IWIiEgKggDAnFulrTd5aTq3xBAREREZgSMvREREEhB0AgQzpo2s+UknTF6IiIikIOhg3rQRb5UmIiKiRsSRF9NxzQsRERFZFI681KOqLLikpGkN5ZVWNr3HlGvL70gdgoGK0nKpQzCgu9P0rlNlRdP6+QYAraYJPva+oil+75reddI2sZC0mrvfucYY1agUNGZN/VSioh6jsSx8t1E9qnoXEhERWbbc3Fw8+uijDdL3nTt30LJlS+Tn55vdl7e3Ny5fvmx1LxBm8lKPdDodrl+/DmdnZ8hkMrP6UqvV8PPzQ25ubpN54RZjMg5jMg5jMg5jMk59xSQIAv7880/4+vpCLm+4lRV37txBebn5o7329vZWl7gAnDaqV3K5vN4zdZVK1WR+OVRhTMZhTMZhTMZhTMapj5hcXFzqKZqaOTg4WGXSUV+4YJeIiIgsCpMXIiIisihMXpoohUKBxYsXQ6FQSB2KiDEZhzEZhzEZhzEZpynGRA2HC3aJiIjIonDkhYiIiCwKkxciIiKyKExeiIiIyKIweWnCBEHAlClT4O7uDplMhnPnzkkdUoNKSUmBq6ur1GE0GXv37kWbNm1gY2ODmTNn1nv/vN76mtr1aCrxNJU4zHX/eSxZsgRdunSRLB4yD5OXJiYmJgYjR44EAKSlpSElJQWff/458vLy0KlTJ2mD+0tGRgZkMhmKioqkDuWh9s9//hPPPPMMcnNzkZiYaFZfAQEBWLNmTf0E9pAaPXo0fvnlF/Gz1H/c7o/nYSP175E5c+YgPT1dkmOT+fiE3Sbs0qVL8PHxQe/evaUOhRpZSUkJCgsLERERAV9fX5P7KS8vh729fT1G9mAVFRWws7Nr1GPWB6VSCaVSKXUYoqYWz8PGyckJTk5OUodBJuLISxMVExOD6dOnIycnBzKZDAEBAVKH1GgOHDiAoKAgODk5ITIyEnl5eeK+06dPY9CgQfDw8ICLiwv69euHs2fPivvHjRuH0aNH6/VXUVEBDw8PpKamArj7DqqkpCS0bNkSSqUSISEh+Pjjj6uN5erVqxg+fDjc3NzQrFkzdOzYEV988YW4//z58xgyZAicnJzg5eWF8ePH48aNG+L+Bx2r6r8+09PT0aNHDzg6OqJjx45wdnYGADzxxBOQyWTIyMiodiRgzZo1ej8bVSN3K1asgK+vL9q3b4/+/fvj6tWrmDVrFmQymcF7t2q73gCwZcsWBAUFwcHBAYGBgdi0aZO478qVK5DJZNi9ezf69esHBwcH7Nixo9pr2dTdO62QkpKCpUuX4vvvvxevWUpKimTxAMD333+PAQMGwNnZGSqVCt27d8eZM2caLZ69e/eibdu2cHBwQEREBHJzcwHc/RmQy+UGsaxZswYtWrTAk08+afD9uXLlCgYMGAAAcHNzg0wmQ0xMDIC7I859+/aFq6srHnnkETz55JO4dOmS2G/Vz9wnn3yCAQMGwNHRESEhITh+/Lje8VNSUvDYY4/B0dERTz31FG7evKm3//7vU9V3Z/Xq1fDx8cEjjzyCadOmoaLi7zc35+XlYdiwYVAqlWjZsiV27tzJUU2pCNSkTJgwQRgxYoRQVFQkLFu2THj00UeFvLw8obCwUOrQREeOHBEACLdv367Xfrdt2ybY2dkJ4eHhwunTp4XMzEwhKChIGDdunFgnPT1d+OCDD4SsrCzhp59+EiZNmiR4eXkJarVaEARB+PzzzwWlUin8+eefYpt9+/YJSqVSrLN8+XIhMDBQSEtLEy5duiRs27ZNUCgUQkZGhkFMw4YNEwYNGiT88MMPwqVLl4R9+/YJX3/9tSAIgnD79m3B09NTSEhIELKysoSzZ88KgwYNEgYMGCC2f9Cxqq5laGiokJGRIVy4cEHo06eP0LVrVwGAsGfPHiEvL0/QaDTC4sWLhZCQEL343n77bcHf31/8PGHCBMHJyUkYP368cP78eeH8+fPCzZs3hUcffVRYtmyZkJeXJ+Tl5Rl9vf/9738LPj4+wp49e4TffvtN2LNnj+Du7i6kpKQIgiAIly9fFgAIAQEBYp3r16/X+f/7pmDbtm2Ci4uLIAiCUFZWJsyePVvo2LGjeM3Kysoki0cQBKFjx47C888/L2RlZQm//PKL8J///Ec4d+5co8RhZ2cn9OjRQ/j222+FM2fOCD179hR69+4t1hk0aJDw0ksv6bXr3Lmz0LZt22q/P5WVlcKePXsEAEJ2draQl5cnFBUVCYIgCB9//LGwZ88e4ddffxW+++47Yfjw4UJwcLCg1WoFQfj7Zy4wMFD4/PPPhezsbOGZZ54R/P39hYqKCkEQBOHEiROCXC4XVq1aJWRnZwtr164VXF1d9a7n/d+nCRMmCCqVSpg6daqQlZUl7Nu3T3B0dBTee+89sU54eLjQpUsX4cSJE0JmZqbQr18/QalUCm+//XY9X3V6ECYvTUxV8iIIhn+YmoqGTF4ACBcvXhS3bdy4UfDy8qqxjVarFZydnYV9+/YJgiAIFRUVgoeHh5CamirWGTt2rDB69GhBEAThzp07gqOjo/Dtt9/q9TNp0iRh7NixBv0HBwcLS5YsqfbYiYmJwuDBg/W25ebmir+QjTlW1bU8fPiwuH///v0CAAGAcOTIEXG7scmLl5eXoNFo9Or5+/sb/II15nq3bt1a2Llzp8F5h4WFCYLw9x+SNWvWVHOFLMv9yUJ111vKeJydncWksbHjACCcOHFC3JaVlSUAEE6ePCkIgiDs3r1bcHNzE+7cuSMIgiBkZmYKMplMaN++fY3fH2N/j/zxxx8CAOHHH38UBOHvn7ktW7aIdS5cuCAAELKysgRBuPudHzp0qF4/o0ePfmDy4u/vL1RWVorbnn32WfF3R9U5nz59Wtz/66+/CgCYvEiA00bUpDg6OqJ169biZx8fHxQWFoqfCwoKEBsbi7Zt28LFxQUqlQolJSXIyckBANja2uK5554Tpy5KS0vx3//+F1FRUQCAixcvoqysDIMGDRLnvJ2cnJCamqo3NF1lxowZWL58Ofr06YPFixfjhx9+EPd9//33OHLkiF4/gYGBAO6uV6rLsTp37qx3zuYIDg42ep1Lbde7tLQUly5dwqRJk/TiX758uUH8PXr0MCtmerD4+HhMnjwZ4eHheO2116r9eW0otra2+L//+z/xc2BgIFxdXZGVlQUAGDlyJGxsbPDpp58CuDtlM2DAAMyZM6fG709Nfv31V4wdOxatWrWCSqUSp0WrvuNVqvvOVP3sZmVlITQ0VK9+WFjYA4/dsWNH2NjY6PVb1Wd2djZsbW3RrVs3cX+bNm3g5ub2wH6p/nHBLjUp9y/0lMlkEO55g8WECRNw8+ZNrF27Fv7+/lAoFAgLC0N5eblYJyoqCv369UNhYSEOHToEpVKJyMhIAHcXwgLA/v370aJFC71jVfdOlMmTJyMiIgL79+/HwYMHkZSUhDfffBPTp09HSUkJhg8fjlWrVhm08/Hxwfnz540+1r3nff+alCpyuVzvWgDQm4+v0qxZs2rbV6e26111rTZv3mzwh+DeX/B1PSaZZsmSJRg3bhz279+PL7/8EosXL8auXbvw1FNPSR0a7O3tER0djW3btuHpp5/Gzp07sXbtWkRFRdX4/anJ8OHD4e/vj82bN8PX1xc6nQ6dOnXS+44D1X9ndDqdWedR3ffB3D6pYTB5IYty7NgxbNq0CUOHDgUA5Obm6i2QBYDevXvDz88Pu3fvxpdffolnn31W/KXUoUMHKBQK5OTkoF+/fkYd08/PD1OnTsXUqVORkJCAzZs3Y/r06ejWrRv27NmDgIAA2NoafpVMOVZtPD09kZ+fD0EQxF/Wxj77x97eHlqttk7H8/Lygq+vL3777Tdx5MqamHLNGlq7du3Qrl07zJo1C2PHjsW2bdsaJXmprKzEmTNn0LNnTwB3RyGKiooQFBQk1pk8eTI6deqETZs2obKyEk8//TSAmr8/VaOD917jmzdvIjs7G5s3b8b/+3//DwDwzTff1DneoKAgnDx5Um/biRMn6tzPvdq3b4/Kykp899136N69O4C7I7m3b982q18yDZMXsiht27bFBx98gB49ekCtVmPu3LnV3k46btw4JCcn45dffsGRI0fE7c7OzpgzZw5mzZoFnU6Hvn37ori4GMeOHYNKpcKECRP0+pk5cyaGDBmCdu3a4fbt2zhy5Ij4C3vatGnYvHkzxo4di3nz5sHd3R0XL17Erl27sGXLljof60H69++PP/74A6+//jqeeeYZpKWl4csvv4RKpXpg24CAABw9ehRjxoyBQqGAh4eHUcdcunQpZsyYARcXF0RGRkKj0eDMmTO4ffs24uPj6xS/pQkICMDly5dx7tw5PProo3B2dpbsjcX/+9//MHfuXDzzzDNo2bIlrl27htOnT2PUqFGNcnw7OztMnz4d69atg62tLeLi4tCrVy8xmQHuJgy9evXC/Pnz8cILL0CpVNb6/fH394dMJsPnn3+OoUOHQqlUws3NDY888gjee+89+Pj4ICcnBwsWLKhzvDNmzECfPn2wevVqjBgxAgcOHEBaWppZ1yAwMBDh4eGYMmUK3nnnHdjZ2WH27NlQKpU1jpZSw+GaF7Io77//Pm7fvo1u3bph/PjxmDFjBpo3b25QLyoqCj/99BNatGiBPn366O1LTEzEq6++iqSkJAQFBSEyMhL79+9Hy5YtDfrRarWYNm2aWK9du3bircK+vr44duwYtFotBg8ejODgYMycOROurq6Qy+V1PtaDBAUFYdOmTdi4cSNCQkJw6tQpzJkzx6i2y5Ytw5UrV9C6dWt4enoafczJkydjy5Yt2LZtG4KDg9GvXz+kpKSYFP/9UlJSmvQv/VGjRiEyMhIDBgyAp6cnPvzwQ8lisbGxwc2bNxEdHY127drhueeew5AhQ7B06dJGOb6joyPmz5+PcePGoU+fPnBycsLu3bsN6k2aNAnl5eV44YUXANT+/WnRogWWLl2KBQsWwMvLC3FxcZDL5di1axcyMzPRqVMnzJo1C2+88Uad4+3Vqxc2b96MtWvXIiQkBAcPHsTChQvNuwgAUlNT4eXlhccffxxPPfUUYmNj4ezsDAcHB7P7prqRCfdPohMRNYLFixfj66+/RkZGhtShUD1JTEzERx99ZNTC3IfBtWvX4Ofnh8OHD2PgwIFSh2NVOG1ERJL48ssvsWHDBqnDoHpQUlKCK1euYMOGDVi+fLnU4TSYr776CiUlJQgODkZeXh7mzZuHgIAAPP7441KHZnWYvBCRJE6dOiV1CFRP4uLi8OGHH2LkyJHilNHDqKKiAq+88gp+++03ODs7o3fv3tixY4dFvg7D0nHaiIiIiCwKF+wSERGRRWHyQkRERBaFyQsRERFZFCYvREREZFGYvBAREZFFYfJCREREFoXJCxEREVkUJi9ERERkUZi8EBERkUX5/3LOfh591yetAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "qk_per_token_after_masking = qk_per_token + mask\n",
    "display_qk_heatmap(qk_per_token_after_masking)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1619,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhsAAAGiCAYAAABOCgSdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAABAaUlEQVR4nO3dfVyUVf7/8fdwN4AwoKGIxoqZJpnh3Wpom9pqUOauVpupqZjatutNQpbSVnhT0f3qrpqrpmSb5W5u9SsNS1bqm1net1mmqSlUgGkKocXNzPX7w5icQAWH4WKc1/PxOI+Hc8051/nMyDAfzjnXuSyGYRgCAADwED+zAwAAABc2kg0AAOBRJBsAAMCjSDYAAIBHkWwAAACPItkAAAAeRbIBAAA8imQDAAB4FMkGAADwKJINAADgUSQbAAD4iPfee0+DBw9Wq1atZLFY9Nprr52zTW5urrp16yar1apLL71UWVlZde6XZAMAAB9x4sQJJSQkaMGCBbWq/+WXX2rQoEHq37+/du7cqalTp2r8+PFat25dnfq1cCM2AAB8j8Vi0auvvqohQ4acsc706dO1Zs0a7dq1y3nstttu0/Hjx5WdnV3rvgLcCRSuHA6HvvnmG4WHh8tisZgdDgCgjgzD0Pfff69WrVrJz89zg/8//vijysvL3T6PYRjVvm+sVqusVqvb55akTZs2acCAAS7HkpKSNHXq1Dqdh2SjHn3zzTeKjY01OwwAgJvy8/N18cUXe+TcP/74o9q2CVPhYbvb5woLC1NpaanLsYyMDM2cOdPtc0tSYWGhoqOjXY5FR0erpKREP/zwg0JCQmp1HpKNehQeHi5JOrQ9TrawxrMcZmiHzmaHAABeoVIVel9rnb/PPaG8vFyFh+36clsb2cLP/7ui5HuH2nY/pPz8fNlsNufx+hrVqE8kG/WoaijLFubn1g9QfQuwBJodAgB4h59WMTbEVLgtvH6+K2w2m0uyUZ9atmypoqIil2NFRUWy2Wy1HtWQSDYAADCF3XDI7sYlGnbDUX/BnEFiYqLWrl3rcuydd95RYmJinc7TeP78BgDAhzhkuF3qqrS0VDt37tTOnTslnbq0defOncrLy5Mkpaena/To0c76d911lw4cOKD77rtPn3/+uRYuXKh//etfSk1NrVO/jGwAAGAChxxyZ2zifFpv3bpV/fv3dz5OS0uTJI0ZM0ZZWVkqKChwJh6S1LZtW61Zs0apqamaN2+eLr74Yi1dulRJSUl16pdkAwAAH9GvXz+dbXutmnYH7devn3bs2OFWvyQbAACYwG4Ysruxr6Y7bRsayQYAACY433UXp7f3FiwQBQAAHsXIBgAAJnDIkN1HRjZINgAAMAHTKAAAAPWEZOMMDMPQnXfeqWbNmslisTg3QAEAoD5UXY3iTvEWTKOcJiUlRcePH9drr72m7OxsZWVlKTc3V5dccomioqLMDg8AcAFx/FTcae8tSDbOYP/+/YqJiVHv3r3NDgUAAK9GslGDlJQUPf/885JO3fmvTZs2OnjwoLlBAQAuKHY3r0Zxp21DI9mowbx589SuXTstXrxYW7Zskb+/f431ysrKVFZW5nxcUlLSUCECALyc3ZCbd32tv1g8jQWiNYiIiFB4eLj8/f3VsmVLNW/evMZ6mZmZioiIcJbY2NgGjhQA4K0c9VC8BcmGG9LT01VcXOws+fn5ZocEAECjwzSKG6xWq6xWq9lhAAC8kEMW2WVxq723INkAAMAEDuNUcae9t2AaBQAAeBQjGwAAmMDu5jSKO20bGsnGabKyspz/njp1qqZOnWpaLACAC5svJRtMowAAAI9iZAMAABM4DIschhtXo7jRtqGRbAAAYAKmUQAAAOoJIxsAAJjALj/Z3fib316PsXgayQYAACYw3FyzYbBmAwAAnA1rNgAAAOoJIxsAAJjAbvjJbrixZsOL7o1CsgEAgAkcssjhxgSDQ96TbTCNAgAAPIqRDQ9I/vRGBTSxmh2G08lxLc0OoZqLnttkdggAYCpfWiBKsgEAgAncX7PBNAoAAIAkRjYAADDFqQWibtyIjWkUAABwNg43tyvnahQAAICfMLIBAIAJfGmBKMkGAAAmcMjPZzb1ItkAAMAEdsMiuxt3bnWnbUNjzQYAAPAoRjYAADCB3c2rUexMowAAgLNxGH5yuLFA1OFFC0SZRgEAAB7FyAYAACZgGgUAAHiUQ+5dUeKov1A8jmmUs8jNzZXFYtHx48fNDgUAAK/FyAYAACZwf1Mv7xkvINkAAMAE7m9XTrLhE8rKylRWVuZ8XFJSYmI0AAA0Tt6TFjVCmZmZioiIcJbY2FizQwIAeAmHLG4Xb0Gy4Yb09HQVFxc7S35+vtkhAQC8RNU0ijvFWzCN4gar1Sqr1Wp2GAAAL+T+Phvek2x4T6QAAMArMbIBAIAJHIZFDnc29fKiW8yTbAAAYAKHm9Mo7LNxgejXr58ML7qrHgAAjRHJBgAAJnD/FvOMbAAAgLOwyyK7G3tluNO2oXlPWgQAALwSIxsAAJiAaRQAAOBRdrk3FWKvv1A8znvSIgAA4JUY2QAAwARMowAAAI9y92Zq3nQjNu+JFACAC4jh5u3ljfNc77FgwQLFxcUpODhYvXr10ubNm89af+7cubrssssUEhKi2NhYpaam6scff6xTnyQbAAD4iFWrViktLU0ZGRnavn27EhISlJSUpMOHD9dYf+XKlZoxY4YyMjK0e/duPffcc1q1apXuv//+OvVLsgEAgAmqplHcKXX1zDPPaMKECRo7dqwuv/xyLVq0SKGhoVq2bFmN9T/44AP16dNHI0aMUFxcnK677joNHz78nKMhv8SaDQ/4Ou8i+YUEmx2GU/NGeH2UpWsns0OoxtjxqdkhAPAh9XXX15KSEpfjVqtVVqu1Wv3y8nJt27ZN6enpzmN+fn4aMGCANm3aVGMfvXv31j//+U9t3rxZPXv21IEDB7R27VqNGjWqTrEysgEAgBeLjY1VRESEs2RmZtZY78iRI7Lb7YqOjnY5Hh0drcLCwhrbjBgxQrNnz9bVV1+twMBAtWvXTv369avzNAojGwAAmMDu5i3mq9rm5+fLZrM5j9c0qnG+cnNz9eijj2rhwoXq1auX9u3bp7vvvltz5szRgw8+WOvzkGwAAGCC+ppGsdlsLsnGmURFRcnf319FRUUux4uKitSyZcsa2zz44IMaNWqUxo8fL0nq3LmzTpw4oTvvvFN/+ctf5OdXu2SJaRQAAHxAUFCQunfvrpycHOcxh8OhnJwcJSYm1tjm5MmT1RIKf39/SZJhGLXum5ENAABM4JCfHG78zX8+bdPS0jRmzBj16NFDPXv21Ny5c3XixAmNHTtWkjR69Gi1bt3aue5j8ODBeuaZZ9S1a1fnNMqDDz6owYMHO5OO2iDZAADABHbDIrsb0yjn03bYsGH69ttv9dBDD6mwsFBdunRRdna2c9FoXl6ey0jGAw88IIvFogceeEBff/21mjdvrsGDB+uRRx6pU78Woy7jIDirkpISRURE6OJ5sxrXpa+bGl9O2ezjknNXamBc+gqg0qhQrl5XcXFxrdZBnI+q74o//d9NsoYFnvd5ykor9Oxv/uPRWOtL4/sWAgDAB9TXAlFvQLIBAIAJDDfv+mp40Y3YSDYAADCBXRbZz/NmalXtvYX3pEUAAMArMbIBAIAJHIZ76y4cXnR5B8kGAAAmcLi5ZsOdtg3NKyLNyspSZGSk2WEAAIDzwMgGAAAmcMgihxuLPN1p29BINgAAMIEZO4iaxSumUaqsW7dO8fHxCgsLU3JysgoKCpzPbdmyRQMHDlRUVJQiIiLUt29fbd++3fn8iBEjNGzYMJfzVVRUKCoqSitWrJB06oY0mZmZatu2rUJCQpSQkKBXXnmlYV4cAAAXKK9JNk6ePKmnnnpKL7zwgt577z3l5eVp2rRpzue///57jRkzRu+//74+/PBDtW/fXjfccIO+//57SdLIkSP1xhtvqLS01Nlm3bp1OnnypIYOHSpJyszM1IoVK7Ro0SJ9+umnSk1N1e2336533323xpjKyspUUlLiUgAAqI2qBaLuFG/hNdMoFRUVWrRokdq1aydJmjRpkmbPnu18/tprr3Wpv3jxYkVGRurdd9/VjTfeqKSkJDVp0kSvvvqqRo0aJUlauXKlfve73yk8PFxlZWV69NFHtX79euetdi+55BK9//77+sc//qG+fftWiykzM1OzZs3y1EsGAFzAHHJzu3IvWrPhNWlRaGioM9GQpJiYGB0+fNj5uKioSBMmTFD79u0VEREhm82m0tJS5eXlSZICAgJ066236sUXX5QknThxQq+//rpGjhwpSdq3b59OnjypgQMHKiwszFlWrFih/fv31xhTenq6iouLnSU/P99TLx8AAK/lNSMbgYGud8azWCw6/Ya1Y8aM0dGjRzVv3jy1adNGVqtViYmJKi8vd9YZOXKk+vbtq8OHD+udd95RSEiIkpOTJck5vbJmzRq1bt3apS+r1VpjTFar9YzPAQBwNoabV6MYXjSy4TXJxrls3LhRCxcu1A033CBJys/P15EjR1zq9O7dW7GxsVq1apXeeust/eEPf3AmMZdffrmsVqvy8vJqnDIBAKA+cddXL9S+fXu98MIL6tGjh0pKSnTvvfcqJCSkWr0RI0Zo0aJF2rt3rzZs2OA8Hh4ermnTpik1NVUOh0NXX321iouLtXHjRtlsNo0ZM6YhXw4A4ALHDqJe6LnnntOxY8fUrVs3jRo1SlOmTFGLFi2q1Rs5cqQ+++wztW7dWn369HF5bs6cOXrwwQeVmZmp+Ph4JScna82aNWrbtm1DvQwAAC44FuP0hQ9wS0lJiSIiInTxvFnyCwk2Oxyn5psa3wBWs48b32XCxo5PzQ4BgMkqjQrl6nUVFxfLZrN5pI+q74rfv32HApsEnfd5Kk6U6/Xrlnk01vrS+L6FAADwAb60XfkFM40CAAAaJ0Y2AAAwAVejAAAAj/KlZINpFAAA4FGMbAAAYAJfGtkg2QAAwAS+lGwwjQIAADyKkQ0AAExgyL29MrxpR06SDQAATOBL0ygkGwAAmIBkA26J+DRQ/kGBZodxmsY32Ob3/UmzQ6jm5KBfmx1CNdY1W8wOAQDcRrIBAIAJGNkAAAAe5UvJBpe+AgAAj2JkAwAAExiGRYYboxPutG1oJBsAAJjAIYtb+2y407ahMY0CAAA8ipENAABM4EsLREk2AAAwgS+t2WAaBQAAeBQjGwAAmIBpFAAA4FG+NI1CsgEAgAkMN0c2vCnZYM0GAADwKEY2AAAwgSHJcOOm3I3vft5nRrIBAIAJHLLIwg6iAAAA7vOqZOPQoUMaPHiwmjZtqiZNmqhTp05au3at8/ldu3bp+uuvV1hYmKKjozVq1CgdOXLE+bzD4VBmZqbatm2rkJAQJSQk6JVXXnE+n5ubK4vFopycHPXo0UOhoaHq3bu39uzZ06CvEwBw4au6GsWd4i28KtmYOHGiysrK9N577+mTTz7R448/rrCwMEnS8ePHde2116pr167aunWrsrOzVVRUpFtvvdXZPjMzUytWrNCiRYv06aefKjU1Vbfffrveffddl37+8pe/6Omnn9bWrVsVEBCgO+64o8Z4ysrKVFJS4lIAAKiNqn023CnewqvWbOTl5enmm29W586dJUmXXHKJ87n58+era9euevTRR53Hli1bptjYWO3du1dt2rTRo48+qvXr1ysxMdHZ/v3339c//vEP9e3b19nukUcecT6eMWOGBg0apB9//FHBwcEu8WRmZmrWrFkee70AAFwIvCrZmDJliv70pz/p7bff1oABA3TzzTfryiuvlCR9/PHH2rBhg3Ok43T79+9XRUWFTp48qYEDB7o8V15erq5du7ocqzqnJMXExEiSDh8+rF/96lcu9dLT05WWluZ8XFJSotjYWPdeJADAJxiGm1ejeNHlKF6VbIwfP15JSUlas2aN3n77bWVmZurpp5/W5MmTVVpaqsGDB+vxxx+v1i4mJka7du2SJK1Zs0atW7d2ed5qtbo8DgwMdP7bYjk1TOVwOKqd12q1VmsLAEBtsINoIxYbG6u77rpLd911l9LT07VkyRJNnjxZ3bp10+rVqxUXF6eAgOov6/LLL5fValVeXp7LlAkAAPAsr1ogOnXqVK1bt05ffvmltm/frg0bNig+Pl7SqcWj3333nYYPH64tW7Zo//79WrduncaOHSu73a7w8HBNmzZNqampev7557V//35t375df//73/X888+b/MoAAL7Gl65G8aqRDbvdrokTJ+qrr76SzWZTcnKy/vrXv0qSWrVqpY0bN2r69Om67rrrVFZWpjZt2ig5OVl+fqdyqjlz5qh58+bKzMzUgQMHFBkZqW7duun+++8382UBAHyQw7DI4iN3fbUYhjctMWncSkpKFBERoU53Pir/oOBzN2gggSca339x1PuFZodQzcnLoswOoRrrmi1mhwD4lEqjQrl6XcXFxbLZbB7po+q7osOLM+Qfev7r/uwny7R35GMejbW+eNU0CgAA8D5eNY0CAMCF4tSlr+5cjVKPwXgYyQYAACbwpUtfmUYBAAAexcgGAAAmMH4q7rT3FiQbAACYgGkUAACAesLIBgAAZvCheRRGNgAAMIO7W5Wf5zTKggULFBcXp+DgYPXq1UubN28+a/3jx49r4sSJiomJkdVqVYcOHbR27do69cnIBgAAJjDjFvOrVq1SWlqaFi1apF69emnu3LlKSkrSnj171KJFi2r1y8vLNXDgQLVo0UKvvPKKWrdurUOHDikyMrJO/ZJsAADgI5555hlNmDBBY8eOlSQtWrRIa9as0bJlyzRjxoxq9ZctW6bvvvtOH3zwgQIDAyVJcXFxde6XZMMDAk8Y8q9oPJNpwcftZodQjaWi0uwQqvErc5gdQjWWrp3MDqEaY8enZocAXBDq62qUkpISl+NWq1VWa/V7rpSXl2vbtm1KT093HvPz89OAAQO0adOmGvv4f//v/ykxMVETJ07U66+/rubNm2vEiBGaPn26/P39ax0razYAADBD1boLd4qk2NhYRUREOEtmZmaN3R05ckR2u13R0dEux6Ojo1VYWPPNMQ8cOKBXXnlFdrtda9eu1YMPPqinn35aDz/8cJ1eKiMbAAB4sfz8fJe7vtY0qnG+HA6HWrRoocWLF8vf31/du3fX119/rSeffFIZGRm1Pg/JBgAAJqivBaI2m61Wt5iPioqSv7+/ioqKXI4XFRWpZcuWNbaJiYlRYGCgy5RJfHy8CgsLVV5erqCgoFrFyjQKAABmMOqh1EFQUJC6d++unJwc5zGHw6GcnBwlJibW2KZPnz7at2+fHI6f17Tt3btXMTExtU40JJINAAB8RlpampYsWaLnn39eu3fv1p/+9CedOHHCeXXK6NGjXRaQ/ulPf9J3332nu+++W3v37tWaNWv06KOPauLEiXXql2kUAABMYMa9UYYNG6Zvv/1WDz30kAoLC9WlSxdlZ2c7F43m5eXJz+/ncYjY2FitW7dOqampuvLKK9W6dWvdfffdmj59ep36JdkAAMAsJuySMGnSJE2aNKnG53Jzc6sdS0xM1IcffuhWn0yjAAAAj2JkAwAAE/jSLeZJNgAAMIMP3fWVZAMAAFNYfirutPcOrNkAAAAexcgGAABm8KFplAYf2Xjttdd06aWXyt/fX1OnTq3382dlZSkyMrLezwsAQL1q4B1EzdTgycYf//hH3XLLLcrPz9ecOXPcOldcXJzmzp1bP4EBAACPaNBplNLSUh0+fFhJSUlq1arVeZ+nLjd/qS8VFRUKDAxs0D4BABew024Tf97tvUSDjWzk5uYqPDxcknTttdfKYrEoNzdXM2fOVJcuXVzqzp07V3Fxcc7HKSkpGjJkiB555BG1atVKl112mfr166dDhw4pNTVVFotFFovrm75u3TrFx8crLCxMycnJKigocHl+6dKlio+PV3BwsDp27KiFCxc6nzt48KAsFotWrVqlvn37Kjg4WC+++GL9viEAAJ9WdddXd4q3aLCRjd69e2vPnj267LLLtHr1avXu3VvNmjWrcWvUmuTk5Mhms+mdd96RdOq2twkJCbrzzjs1YcIEl7onT57UU089pRdeeEF+fn66/fbbNW3aNGfC8OKLL+qhhx7S/Pnz1bVrV+3YsUMTJkxQkyZNNGbMGOd5ZsyYoaefflpdu3ZVcHBwtZjKyspUVlbmfFxSUlLXtwUAgAtegyUbQUFBatGihSSpWbNmatmyZZ3aN2nSREuXLnWZPvH391d4eHi1c1VUVGjRokVq166dpFP7wM+ePdv5fEZGhp5++mnddNNNkqS2bdvqs88+0z/+8Q+XZGPq1KnOOjXJzMzUrFmz6vQ6AACQ5FNXo3jNpa+dO3eu9TqN0NBQZ6IhnRoFOXz4sCTpxIkT2r9/v8aNG+cyIlJZWamIiAiX8/To0eOs/aSnpystLc35uKSkRLGxsbWKEQDg43xozYbpyYafn5+MX0w8VVRUVKvXpEmTWp/zlws5LRaLs4/S0lJJ0pIlS9SrVy+Xev7+/nXq02q1ymq11jouAAB8kenJRvPmzVVYWCjDMJyLPHfu3FmrtkFBQbLb7XXqLzo6Wq1atdKBAwc0cuTIuoYLAEC9sBinijvtvYXpyUa/fv307bff6oknntAtt9yi7OxsvfXWW7LZbOdsGxcXp/fee0+33XabrFaroqKiatXnrFmzNGXKFEVERCg5OVllZWXaunWrjh075jItAgCAx/jQmg3T740SHx+vhQsXasGCBUpISNDmzZs1bdq0WrWdPXu2Dh48qHbt2ql58+a17nP8+PFaunSpli9frs6dO6tv377KyspS27Ztz/dlAABQN1VrNtwpXsJi/HLBBM5bSUmJIiIi1GXkI/IPqn6prFmCj9dtqqkhhO8oOHelBvZD+xZmh1BN0NEfzQ6hGmPHp2aHAHhMpVGhXL2u4uLiWo2wn4+q74rYv86RX8j5f1c4fvhR+akPejTW+mL6NAoAAD7Jh6ZRSDYAADCDDyUbpq/ZAAAAFzZGNgAAMIMPjWyQbAAAYAYf2kGUaRQAAOBRjGwAAGACdhAFAACe5UNrNphGAQAAHkWyAQAAPIppFAAATGCRm2s26i0SzyPZ8ICAkw4FVDjMDsMp+Ntys0OoztH4JhuDC0rNDqEay8nGd2+UkqG9zA6hmtBXPzI7BKDuuPQVAACgfjCyAQCAGXzoahSSDQAAzOBDyQbTKAAAwKMY2QAAwATsIAoAADyLaRQAAID6wcgGAABm8KGRDZINAABM4EtrNphGAQAAHsXIBgAAZvCh7cpJNgAAMANrNgAAgCexZgMAAKCeMLIBAIAZmEYBAAAe5eY0ijclG0yjAAAAj2Jkww1lZWUqKytzPi4pKTExGgCAV/GhaRRGNtyQmZmpiIgIZ4mNjTU7JACAtzDqoXgJkg03pKenq7i42Fny8/PNDgkAgEaHaRQ3WK1WWa1Ws8MAAHgh9tkAAACoJyQbAADAo0g2ziIrK0sWi/fc6AYA4EV8aIEoazbO4ssvv1Tfvn3NDgMAcAHypTUbJBtn8dZbb2n+/PlmhwEAuFB5UcLgDpKNs9i8ebPZIQAA4PVINgAAMIMP7SBKsgEAgAl8ac0GV6MAAACPYmQDAAAzMI0CAAA8iWkUAACAekKyAQCAGUzaQXTBggWKi4tTcHCwevXqVettHl5++WVZLBYNGTKkzn2SbAAAYAYTko1Vq1YpLS1NGRkZ2r59uxISEpSUlKTDhw+ftd3Bgwc1bdo0/eY3v6l7pyLZAADAq5WUlLiUsrKyM9Z95plnNGHCBI0dO1aXX365Fi1apNDQUC1btuyMbex2u0aOHKlZs2bpkksuOa8YWSDqAY5Ai+xBjecGbn4/VpgdQnV+jef9qVJpCzY7hGoCKu1mh1BN+GdHzQ6hGktMS7NDqKayoNDsENDI1dcC0djYWJfjGRkZmjlzZrX65eXl2rZtm9LT053H/Pz8NGDAAG3atOmM/cyePVstWrTQuHHj9H//93/nFSvJBgAAZqinS1/z8/Nls9mch61Wa43Vjxw5IrvdrujoaJfj0dHR+vzzz2ts8/777+u5557Tzp073QiUZAMAAHPUU7Jhs9lcko368v3332vUqFFasmSJoqKi3DoXyQYAAD4gKipK/v7+KioqcjleVFSkli2rT0Xu379fBw8e1ODBg53HHA6HJCkgIEB79uxRu3btatU3C0QBADBB1ZoNd0pdBAUFqXv37srJyXEeczgcysnJUWJiYrX6HTt21CeffKKdO3c6y+9+9zv1799fO3furLZW5GwY2QAAwAwmbFeelpamMWPGqEePHurZs6fmzp2rEydOaOzYsZKk0aNHq3Xr1srMzFRwcLCuuOIKl/aRkZGSVO34uZBsAADgI4YNG6Zvv/1WDz30kAoLC9WlSxdlZ2c7F43m5eXJz6/+Jz1INgAAMIFZ90aZNGmSJk2aVONzubm5Z22blZV1Xn2SbAAAYAYfuusrC0QBAIBHMbIBAIAZfGhkg2QDAAATWH4q7rT3FkyjAAAAj2JkAwAAM/jQNAojGz/JyspyblYCAICnNfQOomYi2fjJsGHDtHfvXufjmTNnqkuXLuYFBAC4sBn1ULwE0yg/CQkJUUhIiNlhAABwwWFk4yenT6NkZWVp1qxZ+vjjj2WxWGSxWM571zQAAM7IB0Y1JEY2ajRs2DDt2rVL2dnZWr9+vSQpIiKiWr2ysjKVlZU5H5eUlDRYjAAA72bWduVmYGSjBiEhIQoLC1NAQIBatmypli1b1jjFkpmZqYiICGepy+12AQDwFSQbbkhPT1dxcbGz5Ofnmx0SAMBbsEAUtWG1WmW1Ws0OAwDghZhGgYKCgmS3280OAwAAr0eycQZxcXH68ssvtXPnTh05csRlISgAAG7zoWkUko0zuPnmm5WcnKz+/furefPmeumll8wOCQBwAfGlHURZs/GTlJQUpaSkOB9brVa98sor5gUEAMAFgmQDAAAz+NCN2Eg2AAAwA8kGAADwJC59BQAAqCeMbAAAYAamUQAAgCdZDEMW4/wzBnfaNjSmUQAAgEcxsgEAgBmYRgEAAJ7E1SgAAAD1hJENAADMwDQK3OEItMgSaDE7DCeLvRH+RNodZkdQjV95pdkhVGMprzA7hGqMAH+zQ6jGqLSbHUI1/lEXmR1CNfYjR80OAadhGgUAAKCeMLIBAIAZmEYBAACe5EvTKCQbAACYwYdGNlizAQAAPIqRDQAATOJNUyHuINkAAMAMhnGquNPeSzCNAgAAPIqRDQAATMDVKAAAwLO4GgUAAKB+MLIBAIAJLI5TxZ323oJkAwAAMzCN4juysrIUGRlpdhgAAFywfD7ZGDZsmPbu3Wt2GAAAH1N1NYo7xVv4/DRKSEiIQkJCzA4DAOBr2NTLd/xyGuXjjz9W//79FR4eLpvNpu7du2vr1q3mBQgAuCAxsuHDRo4cqa5du+rZZ5+Vv7+/du7cqcDAwBrrlpWVqayszPm4pKSkocIEAMBrkGz8Ql5enu6991517NhRktS+ffsz1s3MzNSsWbMaKjQAwIWEq1F8V1pamsaPH68BAwboscce0/79+89YNz09XcXFxc6Sn5/fgJECALyZL02jkGz8wsyZM/Xpp59q0KBB+u9//6vLL79cr776ao11rVarbDabSwEAAK5INmrQoUMHpaam6u2339ZNN92k5cuXmx0SAOBCU3U1ijvFS5BsnOaHH37QpEmTlJubq0OHDmnjxo3asmWL4uPjzQ4NAHCB8aVpFBaInsbf319Hjx7V6NGjVVRUpKioKN10000sAgUAwA0+n2ykpKQoJSVFkhQUFKSXXnrJ3IAAAL7Bh65G8flkAwAAM7g7FeJN0yis2QAAAB7FyAYAAGZwGKeKO+29BMkGAABmYM0GAADwJIvcXLNRb5F4Hms2AACARzGyAQCAGdzdBdSLdhAl2QAAwARc+goAAC5ICxYsUFxcnIKDg9WrVy9t3rz5jHWXLFmi3/zmN2ratKmaNm2qAQMGnLX+mZBsAABgBqMeSh2tWrVKaWlpysjI0Pbt25WQkKCkpCQdPny4xvq5ubkaPny4NmzYoE2bNik2NlbXXXedvv766zr1S7IBAIAJLIbhdpGkkpISl1JWVnbGPp955hlNmDBBY8eO1eWXX65FixYpNDRUy5Ytq7H+iy++qD//+c/q0qWLOnbsqKVLl8rhcCgnJ6dOr5U1Gx4QcNKhgAqH2WH8rDEuIrLbzY6gGou9Eb5PjfD/zlJRaXYI1TS+d0mSv7/ZEVQTENPS7BCqqSwoNDsErxcbG+vyOCMjQzNnzqxWr7y8XNu2bVN6errzmJ+fnwYMGKBNmzbVqq+TJ0+qoqJCzZo1q1OMJBsAAJjB8VNxp72k/Px82Ww252Gr1Vpj9SNHjshutys6OtrleHR0tD7//PNadTl9+nS1atVKAwYMqFOoJBsAAJjg9KmQ820vSTabzSXZ8JTHHntML7/8snJzcxUcHFyntiQbAAD4gKioKPn7+6uoqMjleFFRkVq2PPsU21NPPaXHHntM69ev15VXXlnnvlkgCgCAGRr4apSgoCB1797dZXFn1WLPxMTEM7Z74oknNGfOHGVnZ6tHjx516/QnjGwAAGAGE3YQTUtL05gxY9SjRw/17NlTc+fO1YkTJzR27FhJ0ujRo9W6dWtlZmZKkh5//HE99NBDWrlypeLi4lRYeGpBb1hYmMLCwmrdL8kGAAAmMGMH0WHDhunbb7/VQw89pMLCQnXp0kXZ2dnORaN5eXny8/t50uPZZ59VeXm5brnlFpfznOmKlzMh2QAAwIdMmjRJkyZNqvG53Nxcl8cHDx6slz5JNgAAMAM3YgMAAJ5kcZwq7rT3FlyNAgAAPIqRDQAAzMA0CgAA8KjzvHOrS3svwTQKAADwKJ9JNrKyshQZGWl2GAAASKq/W8x7A6ZRAAAwgw+t2fCZkQ0AAGAOn0s2XnvtNbVv317BwcFKSkpSfn6+pFO7pPn5+Wnr1q0u9efOnas2bdrI4fCiC5oBAI2fIcnhRvGegQ3fSjZOnjypRx55RCtWrNDGjRt1/Phx3XbbbZKkuLg4DRgwQMuXL3dps3z5cqWkpLjsFV+lrKxMJSUlLgUAgNrwpTUbPpVsVFRUaP78+UpMTFT37t31/PPP64MPPtDmzZslSePHj9dLL72ksrIySdL27dv1ySefOO+G90uZmZmKiIhwltjY2AZ7LQAAL2fo53Ub51XMfgG151PJRkBAgH796187H3fs2FGRkZHavXu3JGnIkCHy9/fXq6++KunUFSz9+/dXXFxcjedLT09XcXGxs1RNyQAAgJ/5VLJxLkFBQRo9erSWL1+u8vJyrVy5UnfccccZ61utVtlsNpcCAECtuDWq4eaVLA3Mp5KNyspKlwWge/bs0fHjxxUfH+88Nn78eK1fv14LFy5UZWWlbrrpJjNCBQBc6NxZHFpVvIRPJRuBgYGaPHmyPvroI23btk0pKSm66qqr1LNnT2ed+Ph4XXXVVZo+fbqGDx+ukJAQEyMGAMD7+VSyERoaqunTp2vEiBHq06ePwsLCtGrVqmr1xo0bp/Ly8rNOoQAA4A5fuhrFZ3YQTUlJUUpKiiSdc2rk66+/VufOnV0WkwIAUK/YQdQ3lZaWateuXZo/f74mT55sdjgAAFwQSDZOM2nSJHXv3l39+vVjCgUA4Fk+dDWKz0yj1EZWVpaysrLMDgMA4AuYRgEAAKgfjGwAAGAGhySLm+29BMkGAAAmcPfyVS59BQAAZ8eaDQAAgPrByAYAAGZwGJLFjdEJh/eMbJBsAABgBqZRAAAA6gcjGx5gLbErIMBudhhOFU0b351rg745YnYI1Tka4XVk9sYXk/HDD2aHUI0loPH9KjMqK80OoZrG+HdwQExLs0Nw5SiXChuqM3d3AW2M/6M1a3yfUAAAfAHTKAAAAPWDkQ0AAMzgMOTWVAhXowAAgLMyHKeKO+29BNMoAADAoxjZAADADD60QJRkAwAAM7BmAwAAeJQPjWywZgMAAHgUIxsAAJjBkJsjG/UWiceRbAAAYAamUQAAAOoHIxsAAJjB4ZDkxsZcjfHmkWdwwYxsZGVlKTIy0vl45syZ6tKli2nxAABwVlXTKO4UL+GxZCM3N1cWi0XHjx/3VBdnNW3aNOXk5JjSNwAA+NkFO40SFhamsLAws8MAAKBmLBCtnUOHDmnw4MFq2rSpmjRpok6dOmnt2rU6ePCg+vfvL0lq2rSpLBaLUlJSJEnZ2dm6+uqrFRkZqYsuukg33nij9u/f7zznwYMHZbFY9J///Ef9+/dXaGioEhIStGnTJpe+s7Ky9Ktf/UqhoaEaOnSojh496vL8L6dRUlJSNGTIED311FOKiYnRRRddpIkTJ6qiosJZp6CgQIMGDVJISIjatm2rlStXKi4uTnPnznXnbQIAoDqH4X7xEm4lGxMnTlRZWZnee+89ffLJJ3r88ccVFham2NhYrV69WpK0Z88eFRQUaN68eZKkEydOKC0tTVu3blVOTo78/Pw0dOhQOX6x0OUvf/mLpk2bpp07d6pDhw4aPny4KisrJUkfffSRxo0bp0mTJmnnzp3q37+/Hn744XPGu2HDBu3fv18bNmzQ888/r6ysLGVlZTmfHz16tL755hvl5uZq9erVWrx4sQ4fPnzG85WVlamkpMSlAAAAV25No+Tl5enmm29W586dJUmXXHKJ87lmzZpJklq0aOGycPPmm292OceyZcvUvHlzffbZZ7riiiucx6dNm6ZBgwZJkmbNmqVOnTpp37596tixo+bNm6fk5GTdd999kqQOHTrogw8+UHZ29lnjbdq0qebPny9/f3917NhRgwYNUk5OjiZMmKDPP/9c69ev15YtW9SjRw9J0tKlS9W+ffszni8zM1OzZs0619sEAEA1huGQ4cZt4t1p29DcGtmYMmWKHn74YfXp00cZGRn63//+d842X3zxhYYPH65LLrlENptNcXFxkk4lLqe78sornf+OiYmRJOcow+7du9WrVy+X+omJiefsu1OnTvL393c5b9U59+zZo4CAAHXr1s35/KWXXqqmTZue8Xzp6ekqLi52lvz8/HPGAACApFNrLtyZQvGVNRvjx4/XgQMHNGrUKH3yySfq0aOH/v73v5+1zeDBg/Xdd99pyZIl+uijj/TRRx9JksrLy13qBQYGOv9tsVgkqdpUS12dfs6q87pzTqvVKpvN5lIAAKgVLn2tvdjYWN111136z3/+o3vuuUdLliyRJAUFBUmS7Ha7s+7Ro0e1Z88ePfDAA/rtb3+r+Ph4HTt2rM59xsfHO5OUKh9++KEbr0K67LLLVFlZqR07djiP7du377ziAwAAP3NrzcbUqVN1/fXXq0OHDjp27Jg2bNig+Ph4SVKbNm1ksVj05ptv6oYbblBISIiaNm2qiy66SIsXL1ZMTIzy8vI0Y8aMOvc7ZcoU9enTR0899ZR+//vfa926dedcr3EuHTt21IABA3TnnXfq2WefVWBgoO655x6FhIQ4R1YAAKg3DodkcWPE3lfWbNjtdk2cOFHx8fFKTk5Whw4dtHDhQklS69atNWvWLM2YMUPR0dGaNGmS/Pz89PLLL2vbtm264oorlJqaqieffLLO/V511VVasmSJ5s2bp4SEBL399tt64IEH3HkpkqQVK1YoOjpa11xzjYYOHaoJEyYoPDxcwcHBbp8bAAAXPjSNYjEML4q2gX311VeKjY3V+vXr9dvf/vac9UtKShQREaE+v52pgIDGk6D4lTe+7Dfo08a3mNaIucjsEKrxO1ZqdgjVGD/8YHYI1VgCGt/+hMZPl+o3KqctkG8sLH6N664ZlY5yrS9crOLiYo+tw6v6rvht2AgFWILO+zyVRrlySld6NNb60vg+oSb673//q9LSUnXu3FkFBQW67777FBcXp2uuucbs0AAAFxjD4ZDhxjSKN136SrJxmoqKCt1///06cOCAwsPD1bt3b7344ovVrmIBAMBthiHJN7YrJ9k4TVJSkpKSkswOAwCACwrJBgAAZnAYkoWRDQAA4CmGIcmdS1+9J9loXMuAAQDABYeRDQAATGA4DBluTKN4084VJBsAAJjBcMi9aRTvufSVaRQAAExgOAy3y/lYsGCB4uLiFBwcrF69emnz5s1nrf/vf/9bHTt2VHBwsDp37qy1a9fWuU+SDQAAfMSqVauUlpamjIwMbd++XQkJCUpKStLhw4drrP/BBx9o+PDhGjdunHbs2KEhQ4ZoyJAh2rVrV536ZbvyelRcXKzIyEj16juD7crPIWj312aHUI3RspnZIVTjd7wxblf+o9khVMN25bXEduXnVOkoV+7hLB0/flwREREe6aNqu/KrdYMCdP6bRlaqQu9rrfLz8122K7darbJarTW26dWrl379619r/vz5kiSHw6HY2FhNnjy5xhujDhs2TCdOnNCbb77pPHbVVVepS5cuWrRoUe2DNVBv8vPzq7aDo1AoFIoXl/z8fI99V/zwww9Gy5Yt6yXOsLCwascyMjJq7LesrMzw9/c3Xn31VZfjo0ePNn73u9/V2CY2Ntb461//6nLsoYceMq688so6vebG9+eAF2vVqpXy8/MVHh7u9m3pS0pKFBsbWy1jNRMx1Q4x1Q4x1Q4x1U59xWQYhr7//nu1atWqHqNzFRwcrC+//FLl5eVun8swjGrfN2ca1Thy5Ijsdruio6NdjkdHR+vzzz+vsU1hYWGN9QsLC+sUJ8lGPfLz89PFF19cr+e02WyN5sNchZhqh5hqh5hqh5hqpz5i8tT0yemCg4MVHNx4pts9rXFNlgEAAI+IioqSv7+/ioqKXI4XFRWpZcuWNbZp2bJlneqfCckGAAA+ICgoSN27d1dOTo7zmMPhUE5OjhITE2tsk5iY6FJfkt55550z1j8TplEaKavVqoyMjDPOvZmBmGqHmGqHmGqHmGqnMcbUGKWlpWnMmDHq0aOHevbsqblz5+rEiRMaO3asJGn06NFq3bq1MjMzJUl33323+vbtq6efflqDBg3Syy+/rK1bt2rx4sV16pdLXwEA8CHz58/Xk08+qcLCQnXp0kV/+9vf1KtXL0lSv379FBcXp6ysLGf9f//733rggQd08OBBtW/fXk888YRuuOGGOvVJsgEAADyKNRsAAMCjSDYAAIBHkWwAAACPItloxAzD0J133qlmzZrJYrFo586dZofkUVlZWYqMjDQ7jEbjtdde06WXXip/f39NnTq13s/P++2qsb0fjSWexhKHu375OmbOnKkuXbqYFo+vIdloZFJSUjRkyBBJUnZ2trKysvTmm2+qoKBAV1xxhbnB/SQ3N1cWi0XHjx83O5QL2h//+Efdcsstys/P15w5c9w6V1xcnObOnVs/gV2ghg0bpr179zofm/1l9Mt4LjRm/x6ZNm1atf0j4Dnss9GI7d+/XzExMerdu7fZoaCBlZaW6vDhw0pKSnLrHg3l5eUKCgqqx8jOraKiQoGB538nS7OEhIQoJCTE7DCcGls8F5qwsDCFhYWZHYbPYGSjkUpJSdHkyZOVl5cni8WiuLg4s0NqMOvWrVN8fLzCwsKUnJysgoIC53NbtmzRwIEDFRUVpYiICPXt21fbt293Pj9ixAgNGzbM5XwVFRWKiorSihUrJJ3aMS8zM1Nt27ZVSEiIEhIS9Morr9QYy6FDhzR48GA1bdpUTZo0UadOnbR27Vrn87t27dL111+vsLAwRUdHa9SoUTpy5Ijz+XP1VfXXXU5Ojnr06KHQ0FB16tRJ4eHhkqRrr71WFotFubm5Nf6lPXfuXJefjaqRsUceeUStWrXSZZddpn79+unQoUNKTU2VxWKpdtOms73fkrR06VLFx8crODhYHTt21MKFC53PHTx4UBaLRatWrVLfvn0VHBysF198scb3srE7fZg9KytLs2bN0scff+x8z07fd6Ch45Gkjz/+WP3791d4eLhsNpu6d++urVu3Nlg8r732mtq3b6/g4GAlJSUpPz9f0qmfAT8/v2qxzJ07V61bt9aNN95Y7fNz8OBB9e/fX5LUtGlTWSwWpaSkSDo1onv11VcrMjJSF110kW688Ubt37/fed6qn7n//Oc/6t+/v0JDQ5WQkKBNmza59J+VlaVf/epXCg0N1dChQ3X06FGX53/5ear67Dz11FOKiYnRRRddpIkTJ6qiosJZp6CgQIMGDVJISIjatm2rlStXMmpYW3W6Ryw8bsyYMcbvf/974/jx48bs2bONiy++2CgoKDAOHz5sdmhOGzZsMCQZx44dq9fzLl++3AgMDDQGDBhgbNmyxdi2bZsRHx9vjBgxwlknJyfHeOGFF4zdu3cbn332mTFu3DgjOjraKCkpMQzDMN58800jJCTE+P77751t3njjDSMkJMRZ5+GHHzY6duxoZGdnG/v37zeWL19uWK1WIzc3t1pMgwYNMgYOHGj873//M/bv32+88cYbxrvvvmsYhmEcO3bMaN68uZGenm7s3r3b2L59uzFw4ECjf//+zvbn6qvqvezVq5eRm5trfPrpp0afPn2Mrl27GpKM1atXGwUFBUZZWZmRkZFhJCQkuMT317/+1WjTpo3z8ZgxY4ywsDBj1KhRxq5du4xdu3YZR48eNS6++GJj9uzZRkFBgVFQUFDr9/uf//ynERMTY6xevdo4cOCAsXr1aqNZs2ZGVlaWYRiG8eWXXxqSjLi4OGedb775ps7/943B8uXLjYiICMMwDOPkyZPGPffcY3Tq1Mn5np08edK0eAzDMDp16mTcfvvtxu7du429e/ca//rXv4ydO3c2SByBgYFGjx49jA8++MDYunWr0bNnT6N3797OOgMHDjT+/Oc/u7S78sorjfbt29f4+amsrDRWr15tSDL27NljFBQUGMePHzcMwzBeeeUVY/Xq1cYXX3xh7Nixwxg8eLDRuXNnw263G4bx889cx44djTfffNPYs2ePccsttxht2rQxKioqDMMwjA8//NDw8/MzHn/8cWPPnj3GvHnzjMjISJf385efpzFjxhg2m8246667jN27dxtvvPGGERoaaixevNhZZ8CAAUaXLl2MDz/80Ni2bZvRt29fIyQkpNot2FEdyUYjU5VsGEb1L5LGwpPJhiRj3759zmMLFiwwoqOjz9jGbrcb4eHhxhtvvGEYhmFUVFQYUVFRxooVK5x1hg8fbgwbNswwDMP48ccfjdDQUOODDz5wOc+4ceOM4cOHVzt/586djZkzZ9bY95w5c4zrrrvO5Vh+fr7zF2ht+qp6L9evX+98fs2aNYYkQ5KxYcMG5/HaJhvR0dFGWVmZS702bdpU+4VYm/e7Xbt2xsqVK6u97sTERMMwfv7FP3fu3BreIe/yyy/3mt5vM+MJDw93JnkNHYck48MPP3Qe2717tyHJ+OijjwzDMIxVq1YZTZs2NX788UfDMAxj27ZthsViMS677LIzfn5q+3vk22+/NSQZn3zyiWEYP//MLV261Fnn008/NSQZu3fvNgzj1Gf+hhtucDnPsGHDzplstGnTxqisrHQe+8Mf/uD83VH1mrds2eJ8/osvvjAkkWzUAtMoaFRCQ0PVrl075+OYmBgdPnzY+bioqEgTJkxQ+/btFRERIZvNptLSUuXl5UmSAgICdOuttzqH8k+cOKHXX39dI0eOlCTt27dPJ0+e1MCBA51ztmFhYVqxYoXLUG2VKVOm6OGHH1afPn2UkZGh//3vf87nPv74Y23YsMHlPB07dpR0ar1NXfq68sorXV6zOzp37lzrdRpne79PnDih/fv3a9y4cS7xP/zww9Xi79Gjh1sx49zS0tI0fvx4DRgwQI899liNP6+eEhAQoF//+tfOxx07dlRkZKR2794tSRoyZIj8/f316quvSjo1hdG/f39NmzbtjJ+fM/niiy80fPhwXXLJJbLZbM5pwqrPeJWaPjNVP7u7d+92br9dpTY3DuvUqZP8/f1dzlt1zj179iggIEDdunVzPn/ppZeqadOm5zwvWCCKRuaXCwstFouM03bUHzNmjI4ePap58+apTZs2slqtSkxMVHl5ubPOyJEj1bdvXx0+fFjvvPOOQkJClJycLOnUwktJWrNmjVq3bu3SV003cBo/frySkpK0Zs0avf3228rMzNTTTz+tyZMnq7S0VIMHD9bjjz9erV1MTIx27dpV675Of92/XFNRxc/Pz+W9kOQyn1ylSZMmNbavydne76r3asmSJdV+cZ/+C7mufeL8zJw5UyNGjNCaNWv01ltvKSMjQy+//LKGDh1qdmgKCgrS6NGjtXz5ct10001auXKl5s2bp5EjR57x83MmgwcPVps2bbRkyRK1atVKDodDV1xxhctnXKr5M+NwONx6HTV9Htw9J04h2YBX2bhxoxYuXOi8CVB+fr7LgkxJ6t27t2JjY7Vq1Sq99dZb+sMf/uD8JXL55ZfLarUqLy9Pffv2rVWfsbGxuuuuu3TXXXcpPT1dS5Ys0eTJk9WtWzetXr1acXFxCgio/lE6n77Opnnz5iosLJRhGM5frrXdeyUoKEh2u71O/UVHR6tVq1Y6cOCAc2TIl5zPe+ZpHTp0UIcOHZSamqrhw4dr+fLlDZJsVFZWauvWrerZs6ekU3/lHz9+XPHx8c4648eP1xVXXKGFCxeqsrJSN910k6Qzf36qRt9Of4+PHj2qPXv2aMmSJfrNb34jSXr//ffrHG98fLw++ugjl2Mffvhhnc9zussuu0yVlZXasWOHunfvLunUSOmxY8fcOq+vINmAV2nfvr1eeOEF9ejRQyUlJbr33ntrvDxwxIgRWrRokfbu3asNGzY4j4eHh2vatGlKTU2Vw+HQ1VdfreLiYm3cuFE2m01jxoxxOc/UqVN1/fXXq0OHDjp27Jg2bNjg/AU7ceJELVmyRMOHD9d9992nZs2aad++fXr55Ze1dOnSOvd1Lv369dO3336rJ554Qrfccouys7P11ltvyWaznbNtXFyc3nvvPd12222yWq2KioqqVZ+zZs3SlClTFBERoeTkZJWVlWnr1q06duyY0tLS6hS/t4mLi9OXX36pnTt36uKLL1Z4eLhpty//4YcfdO+99+qWW25R27Zt9dVXX2nLli26+eabG6T/wMBATZ48WX/7298UEBCgSZMm6aqrrnImH9KpL/irrrpK06dP1x133KGQkJCzfn7atGkji8WiN998UzfccINCQkLUtGlTXXTRRVq8eLFiYmKUl5enGTNm1DneKVOmqE+fPnrqqaf0+9//XuvWrVN2drZb70HHjh01YMAA3XnnnXr22WcVGBioe+65RyEhIWccjcTPWLMBr/Lcc8/p2LFj6tatm0aNGqUpU6aoRYsW1eqNHDlSn332mVq3bq0+ffq4PDdnzhw9+OCDyszMVHx8vJKTk7VmzRq1bdu22nnsdrsmTpzorNehQwfnpZ+tWrXSxo0bZbfbdd1116lz586aOnWqIiMj5efnV+e+ziU+Pl4LFy7UggULlJCQoM2bN2vatGm1ajt79mwdPHhQ7dq1U/PmzWvd5/jx47V06VItX75cnTt3Vt++fZWVlXVe8f9SVlZWo/4lffPNNys5OVn9+/dX8+bN9dJLL5kWi7+/v44eParRo0erQ4cOuvXWW3X99ddr1qxZDdJ/aGiopk+frhEjRqhPnz4KCwvTqlWrqtUbN26cysvLdccdd0g6++endevWmjVrlmbMmKHo6GhNmjRJfn5+evnll7Vt2zZdccUVSk1N1ZNPPlnneK+66iotWbJE8+bNU0JCgt5++2098MAD7r0JklasWKHo6Ghdc801Gjp0qCZMmKDw8HAFBwe7fe4LHbeYB2CKjIwMvfvuu8rNzTU7FNSTOXPm6N///netFoJeCL766ivFxsZq/fr1+u1vf2t2OI0a0ygATPHWW29p/vz5ZoeBelBaWqqDBw9q/vz5evjhh80Ox2P++9//qrS0VJ07d1ZBQYHuu+8+xcXF6ZprrjE7tEaPZAOAKTZv3mx2CKgnkyZN0ksvvaQhQ4Y4p1AuRBUVFbr//vt14MABhYeHq3fv3nrxxRe9cnv+hsY0CgAA8CgWiAIAAI8i2QAAAB5FsgEAADyKZAMAAHgUyQYAAPAokg0AAOBRJBsAAMCjSDYAAIBH/X89nEoFndDSswAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "qk_per_token_after_masking_after_softmax = torch.nn.functional.softmax(qk_per_token_after_masking, dim=1).to(torch.bfloat16)\n",
    "display_qk_heatmap(qk_per_token_after_masking_after_softmax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### values (almost the end of attention)\n",
    "these scores (0-1) are used to determine how much of value matrix is used per token\n",
    "<br>\n",
    "&gt; just like keys, value weights are also shared acorss every 4 attention heads (to save computation)\n",
    "<br>\n",
    "&gt; as a result, the shape of the value weight matrix below is [8x128x4096]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1620,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([8, 128, 4096])\n"
     ]
    }
   ],
   "source": [
    "v_layer0 = v_layer0.view(n_kv_heads, head_dim, dim)\n",
    "print(v_layer0.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the first layer, first head value weight matrix is given below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1621,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([128, 4096])"
      ]
     },
     "execution_count": 1621,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v_layer0_head0 = v_layer0[0]\n",
    "v_layer0_head0.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### value vectors\n",
    "we now use the value weghts to get the attention values per token, this is of size [10x128] where 10 is the number of tokens in the prompt and 128 is the dim of the value vector per token"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1622,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 128])"
      ]
     },
     "execution_count": 1622,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v_per_token = torch.matmul(token_embeddings, v_layer0_head0.T)\n",
    "v_per_token.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### attention\n",
    "the resultant attention vector after multipying with the values per token is of shape [10*128]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1623,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 128])"
      ]
     },
     "execution_count": 1623,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qkv_attention = torch.matmul(qk_per_token_after_masking_after_softmax, v_per_token)\n",
    "qkv_attention.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### multi head attention\n",
    "We now have the attention value of the first layer and first head.\n",
    "<br>\n",
    "now im going to run a loop and perform the exact same math as the cells above but for every head in the first layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1624,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 1624,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qkv_attention_store = []\n",
    "GQA_num = n_heads // n_kv_heads # \n",
    "for head in range(n_heads):\n",
    "    q_layer0_head = q_layer0[head]\n",
    "    k_layer0_head = k_layer0[head//GQA_num] \n",
    "    v_layer0_head = v_layer0[head//GQA_num]\n",
    "    q_per_token = torch.matmul(token_embeddings, q_layer0_head.T)\n",
    "    k_per_token = torch.matmul(token_embeddings, k_layer0_head.T)\n",
    "    v_per_token = torch.matmul(token_embeddings, v_layer0_head.T)\n",
    "\n",
    "    position_ids = torch.arange(len(tokens))\n",
    "    cos, sin = rotary_emb(position_ids)\n",
    "    q_per_token_rotated, k_per_token_rotated = apply_rotary_pos_emb(q_per_token, k_per_token, cos, sin, position_ids)  \n",
    "\n",
    "    qk_per_token = torch.matmul(q_per_token_rotated, k_per_token_rotated.T)/(128)**0.5\n",
    "    mask = torch.full((len(tokens), len(tokens)), float(\"-inf\"), device=tokens.device)\n",
    "    mask = torch.triu(mask, diagonal=1)\n",
    "    qk_per_token_after_masking = qk_per_token + mask\n",
    "    qk_per_token_after_masking_after_softmax = torch.nn.functional.softmax(qk_per_token_after_masking, dim=1).to(torch.bfloat16)\n",
    "    qkv_attention = torch.matmul(qk_per_token_after_masking_after_softmax, v_per_token)\n",
    "    qkv_attention_store.append(qkv_attention)\n",
    "\n",
    "len(qkv_attention_store)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we now have a the qkv_attention matrix for all 32 heads on the first layer, next im going to merge all attention scores into one large matrix of size [10x4096]\n",
    "<br>\n",
    "we are almost at the end :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1625,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 4096])"
      ]
     },
     "execution_count": 1625,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacked_qkv_attention = torch.cat(qkv_attention_store, dim=-1)\n",
    "stacked_qkv_attention.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### weight matrix, one of the final steps\n",
    "one of the last things to do for a layer 0 attention is, is to multiply the weight matrix of the "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1626,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4096, 4096])"
      ]
     },
     "execution_count": 1626,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w_layer0 = model[\"model.layers.0.attention.wo.weight\"]\n",
    "w_layer0.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### this is a simple linear layer, so we just matmul"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1627,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 4096])"
      ]
     },
     "execution_count": 1627,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_delta = torch.matmul(stacked_qkv_attention, w_layer0.T)\n",
    "embedding_delta.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we now have the change in the embedding value after attention, that should be adding to the original token embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1628,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 4096])"
      ]
     },
     "execution_count": 1628,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_after_edit = token_embeddings_unnormalized + embedding_delta\n",
    "embedding_after_edit.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### we normalize and then run a feed forward neural network through the embedding delta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1629,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 4096])"
      ]
     },
     "execution_count": 1629,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_after_edit_normalized = rms_norm(embedding_after_edit, model[\"model.layers.0.ffn_norm.weight\"])\n",
    "embedding_after_edit_normalized.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### loading the ff weights and implementing the feed forward network\n",
    "in internlm2.5, they used a SwiGLU feedforward network, this network architecture is really good at adding non linearity when needed by the model.\n",
    "<br>\n",
    "its pretty standard to use this feed forward network architecture in llms these days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1630,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 4096])"
      ]
     },
     "execution_count": 1630,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w1 = model[\"model.layers.0.feed_forward.w1.weight\"]\n",
    "w2 = model[\"model.layers.0.feed_forward.w2.weight\"]\n",
    "w3 = model[\"model.layers.0.feed_forward.w3.weight\"]\n",
    "output_after_feedforward = torch.matmul(torch.functional.F.silu(torch.matmul(embedding_after_edit_normalized, w1.T)) * torch.matmul(embedding_after_edit_normalized, w3.T), w2.T)\n",
    "output_after_feedforward.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We finally have new edited embeddings for each token after the first layer.\n",
    "just 31 more layers to go before we are done (one for loop away)\n",
    "<br>\n",
    "you can imagine this edited embedding as having information about all queries asked on the first layer\n",
    "<br>\n",
    "now each layer will encode more and more complex queries on the quesions asked, until we have an embedding that knows everything about the next token that we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1631,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 4096])"
      ]
     },
     "execution_count": 1631,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer_0_embedding = embedding_after_edit+output_after_feedforward\n",
    "layer_0_embedding.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### god, everything all at once\n",
    "yep, this is it. everything we did before, all at once, for every single layer.\n",
    "<br>\n",
    "\n",
    "### have fun reading :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1632,
   "metadata": {},
   "outputs": [],
   "source": [
    "k_cache, v_cache = [], []   # init k v cache"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1633,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_embedding = token_embeddings_unnormalized\n",
    "GQA_num = n_heads // n_kv_heads\n",
    "for layer in range(n_layers):\n",
    "    k_cache.append([])\n",
    "    v_cache.append([])\n",
    "    qkv_attention_store = []\n",
    "    layer_embedding_norm = rms_norm(final_embedding, model[f\"model.layers.{layer}.attention_norm.weight\"])\n",
    "\n",
    "    qkv_layer = model[f\"model.layers.{layer}.attention.wqkv.weight\"]\n",
    "    head_dim = dim // n_heads\n",
    "    q_layer, k_layer, v_layer = qkv_layer.reshape(n_kv_heads, 2+n_heads//n_kv_heads, head_dim, dim).split([n_heads//n_kv_heads, 1, 1], dim=1)\n",
    "    q_layer, k_layer, v_layer = q_layer.reshape(-1, dim), k_layer.reshape(-1, dim), v_layer.reshape(-1, dim)\n",
    "    \n",
    "    q_layer = q_layer.reshape(n_heads, head_dim, dim)\n",
    "    k_layer = k_layer.reshape(n_kv_heads, head_dim, dim)\n",
    "    v_layer = v_layer.reshape(n_kv_heads, head_dim, dim)\n",
    "    for head in range(n_heads):\n",
    "        q_layer_head = q_layer[head]\n",
    "        k_layer_head = k_layer[head//GQA_num]\n",
    "        v_layer_head = v_layer[head//GQA_num]\n",
    "        q_per_token = torch.matmul(layer_embedding_norm, q_layer_head.T)\n",
    "        k_per_token = torch.matmul(layer_embedding_norm, k_layer_head.T)\n",
    "        v_per_token = torch.matmul(layer_embedding_norm, v_layer_head.T)\n",
    "        if head % GQA_num == 0:   # \n",
    "            v_cache[-1].append(v_per_token)   # cache v vector\n",
    "            \n",
    "        position_ids = torch.arange(len(tokens))\n",
    "        cos, sin = rotary_emb(position_ids)\n",
    "        q_per_token_rotated, k_per_token_rotated = apply_rotary_pos_emb(q_per_token, k_per_token, cos, sin, position_ids)\n",
    "\n",
    "        if head % GQA_num == 0:   # \n",
    "            k_cache[-1].append(k_per_token_rotated)  # cache k vector\n",
    "        qk_per_token = torch.matmul(q_per_token_rotated, k_per_token_rotated.T)/(128)**0.5\n",
    "        mask = torch.full((len(token_embeddings_unnormalized), len(token_embeddings_unnormalized)), float(\"-inf\"))\n",
    "        mask = torch.triu(mask, diagonal=1)\n",
    "        qk_per_token_after_masking = qk_per_token + mask\n",
    "        qk_per_token_after_masking_after_softmax = torch.nn.functional.softmax(qk_per_token_after_masking, dim=1).to(torch.bfloat16)\n",
    "        qkv_attention = torch.matmul(qk_per_token_after_masking_after_softmax, v_per_token)\n",
    "        qkv_attention_store.append(qkv_attention)\n",
    "\n",
    "    stacked_qkv_attention = torch.cat(qkv_attention_store, dim=-1)\n",
    "    w_layer = model[f\"model.layers.{layer}.attention.wo.weight\"]\n",
    "    embedding_delta = torch.matmul(stacked_qkv_attention, w_layer.T)\n",
    "    embedding_after_edit = final_embedding + embedding_delta\n",
    "    embedding_after_edit_normalized = rms_norm(embedding_after_edit, model[f\"model.layers.{layer}.ffn_norm.weight\"])\n",
    "    w1 = model[f\"model.layers.{layer}.feed_forward.w1.weight\"] \n",
    "    w2 = model[f\"model.layers.{layer}.feed_forward.w2.weight\"]\n",
    "    w3 = model[f\"model.layers.{layer}.feed_forward.w3.weight\"]\n",
    "    output_after_feedforward = torch.matmul(torch.functional.F.silu(torch.matmul(embedding_after_edit_normalized, w1.T)) * torch.matmul(embedding_after_edit_normalized, w3.T), w2.T)\n",
    "    final_embedding = embedding_after_edit+output_after_feedforward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1634,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([10, 128])\n",
      "torch.Size([10, 128])\n",
      "32\n",
      "8\n",
      "torch.Size([10, 128])\n",
      "torch.Size([10, 128])\n",
      "32\n",
      "8\n"
     ]
    }
   ],
   "source": [
    "# k v cache info\n",
    "print(k_cache[0][0].shape)\n",
    "print(k_cache[0][-1].shape)\n",
    "print(len(k_cache))\n",
    "print(len(k_cache[0]))\n",
    "print(v_cache[0][0].shape)\n",
    "print(v_cache[0][-1].shape)\n",
    "print(len(v_cache))\n",
    "print(len(v_cache[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### we now have the final embedding, the best guess the model could make about the next token\n",
    "the shape of the embedding is the same as regular token embeddings [10x3854] where 10 is the number of tokens and 3584 is the embedding dim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1635,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 4096])"
      ]
     },
     "execution_count": 1635,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_embedding = rms_norm(final_embedding, model[\"model.norm.weight\"])\n",
    "final_embedding.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### finally, lets decode the embedding into the token value\n",
    "we will use the output decoder to convert the final embedding into a token"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1636,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([92544, 4096])"
      ]
     },
     "execution_count": 1636,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model[\"output.weight\"].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### we use the embedding of the last token to predict the next value\n",
    "hopefully in our case: on "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1637,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([92544])"
      ]
     },
     "execution_count": 1637,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logits = torch.matmul(final_embedding[-1], model[\"output.weight\"].T)\n",
    "logits.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### the model predicted token number 519 as the next token, is this the token number for on?\n",
    "this is the last cell of code, hopefully you had fun :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1638,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(519)"
      ]
     },
     "execution_count": 1638,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next_token = torch.argmax(logits, dim=-1)\n",
    "next_token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### decode output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1639,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' on'"
      ]
     },
     "execution_count": 1639,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.decode([next_token.item()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prefill stage is finish, we get the k_cache and v_cache for input_token.\n",
    "we also get a output(next token), and we can use it as a input token for generat next token. This is the feature for Decoder only model.\n",
    "</br>预填充阶段推理完成，生成第一个token且缓存了prompt阶段过程中生成的k和v。下一阶段进行decoder推理，加载前面生成的k和v，进行加速推理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1640,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_new_len = 6\n",
    "seq_len = len(tokens)\n",
    "GQA_num = n_heads // n_kv_heads "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1641,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "519\n",
      "token decode:   the\n",
      "410\n",
      "token decode:   shoulders\n",
      "27315\n",
      "token decode:   of\n",
      "446\n",
      "token decode:   giants\n",
      "44722\n",
      "token decode:  .\n"
     ]
    }
   ],
   "source": [
    "next_token = torch.tensor([next_token.item()])\n",
    "for _ in range(max_new_len-1):\n",
    "    print(next_token[-1].item())\n",
    "    if next_token.item() == 2:    # 2 is \"eos_token_id\"\n",
    "        break\n",
    "    next_token = next_token[-1:]\n",
    "    next_token_embeddings_unnormalized = embedding_layer(next_token).to(torch.bfloat16)\n",
    "\n",
    "    final_embedding = next_token_embeddings_unnormalized\n",
    "    for layer in range(n_layers):\n",
    "        qkv_attention_store = []\n",
    "        layer_embedding_norm = rms_norm(final_embedding, model[f\"model.layers.{layer}.attention_norm.weight\"])\n",
    "        \n",
    "        qkv_layer = model[f\"model.layers.{layer}.attention.wqkv.weight\"]\n",
    "        q_layer, k_layer, v_layer = qkv_layer.reshape(n_kv_heads, 2+n_heads//n_kv_heads, head_dim, dim).split([n_heads//n_kv_heads, 1, 1], dim=1)\n",
    "        q_layer, k_layer, v_layer = q_layer.reshape(-1, dim), k_layer.reshape(-1, dim), v_layer.reshape(-1, dim)\n",
    "        \n",
    "        q_layer = q_layer.reshape(n_heads, head_dim, dim)\n",
    "        k_layer = k_layer.reshape(n_kv_heads, head_dim, dim)\n",
    "        v_layer = v_layer.reshape(n_kv_heads, head_dim, dim)\n",
    "        \n",
    "        for head in range(n_heads):\n",
    "            q_layer_head = q_layer[head]\n",
    "            k_layer_head = k_layer[head//GQA_num]\n",
    "            v_layer_head = v_layer[head//GQA_num]\n",
    "            q_per_token = torch.matmul(layer_embedding_norm, q_layer_head.T)\n",
    "\n",
    "            cos, sin = rotary_emb(torch.tensor([seq_len]))\n",
    "            q_per_token_rotated, k_per_token_rotated = apply_rotary_pos_emb(q_per_token, k_per_token, cos, sin)\n",
    "\n",
    "            if head % GQA_num == 0:\n",
    "                v_per_token = torch.matmul(layer_embedding_norm, v_layer_head.T)\n",
    "                v_cache[layer][head//GQA_num] = torch.cat([v_cache[layer][head//GQA_num], v_per_token], dim=0)    # update v_cache\n",
    "\n",
    "                k_per_token = torch.matmul(layer_embedding_norm, k_layer_head.T)   \n",
    "                q_per_token_rotated, k_per_token_rotated = apply_rotary_pos_emb(q_per_token, k_per_token, cos, sin)\n",
    "                k_cache[layer][head//GQA_num] = torch.cat([k_cache[layer][head//GQA_num], k_per_token_rotated], dim=0)    # update k_cache\n",
    "                \n",
    "            qk_per_token = torch.matmul(q_per_token_rotated, k_cache[layer][head//GQA_num].T)/(128)**0.5\n",
    "            #mask = torch.full((len(token_embeddings_unnormalized), len(token_embeddings_unnormalized)), float(\"-inf\"))   # don't need mask\n",
    "            #mask = torch.triu(mask, diagonal=1)\n",
    "            qk_per_token_after_masking = qk_per_token  # + mask\n",
    "            qk_per_token_after_masking_after_softmax = torch.nn.functional.softmax(qk_per_token_after_masking, dim=1).to(torch.bfloat16)\n",
    "            qkv_attention = torch.matmul(qk_per_token_after_masking_after_softmax, v_cache[layer][head//GQA_num])\n",
    "            qkv_attention_store.append(qkv_attention)\n",
    "\n",
    "        stacked_qkv_attention = torch.cat(qkv_attention_store, dim=-1)\n",
    "        w_layer = model[f\"model.layers.{layer}.attention.wo.weight\"]\n",
    "        embedding_delta = torch.matmul(stacked_qkv_attention, w_layer.T)\n",
    "        embedding_after_edit = final_embedding + embedding_delta\n",
    "        embedding_after_edit_normalized = rms_norm(embedding_after_edit, model[f\"model.layers.{layer}.ffn_norm.weight\"])\n",
    "        w1 = model[f\"model.layers.{layer}.feed_forward.w1.weight\"]\n",
    "        w2 = model[f\"model.layers.{layer}.feed_forward.w2.weight\"]\n",
    "        w3 = model[f\"model.layers.{layer}.feed_forward.w3.weight\"]\n",
    "        output_after_feedforward = torch.matmul(torch.functional.F.silu(torch.matmul(embedding_after_edit_normalized, w1.T)) * torch.matmul(embedding_after_edit_normalized, w3.T), w2.T)\n",
    "        final_embedding = embedding_after_edit+output_after_feedforward\n",
    "    \n",
    "    final_embedding = rms_norm(final_embedding, model[\"model.norm.weight\"])\n",
    "    logits = torch.matmul(final_embedding, model[\"output.weight\"].T)\n",
    "    next_token = torch.argmax(logits, dim=-1)\n",
    "    print(\"token decode: \", tokenizer.decode([next_token.item()]))\n",
    "    seq_len += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "thank you, i love you :)\n",
    "\n",
    "This is the end. Hopefully you enjoyed reading it!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}