word_embedding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 使用数字代替文本 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## one-hot向量"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用one-hot向量十分低效，one-hot向量十分稀疏，大部分都是0\n",
    "并且随着词汇表大小的增加，one-hot向量的维度也会不断变大"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用单独的数字进行编码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们对每一个单词进行编码，每一个单词对应一个数字，这样虽然十分高效，但是：\n",
    "1. 使用一个数字编码一个单词过于绝对，这样不能获得到单词间的联系\n",
    "2. 使用一个数字是的模型对于该单词解释过于困难。如线性分类器为每个特征学习单个权重。因为两个单词之间的相似性与其编码之间的相似性没有任何关系，这种特征权重的组合是没有意义的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## word embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "词嵌入为我们提供了一种使用高效、密集表示的方法，其中相似的单词具有相似的编码。重要的是，我们不必手工指定这种编码。嵌入是浮点值的密集向量（向量的长度是您指定的参数）。它们不是手动指定嵌入的值，而是可训练的参数（训练期间模型学习的权重，与模型学习稠密层权重的方式相同）。在处理大型数据集时，常见的词嵌入是8维的（对于小型数据集），最多可达1024维。更高维度的嵌入可以捕获单词之间的细粒度关系，但需要更多的数据来学习。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow.keras as keras\n",
    "import tensorflow.keras.layers as layers\n",
    "import tensorflow_datasets as tfds\n",
    "tfds.disable_progress_bar()\n",
    "\n",
    "\n",
    "import os\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '1'\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'\n",
    "os.environ[\"KMP_DUPLICATE_LIB_OK\"]=\"TRUE\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于此数据集，进行一点说明，只是我自己的想法，此文本编码器将可逆地对任何字符串进行编码，必要时返回字节编码。  \n",
    "有些词汇并不在我们最终训练的词汇表中。\n",
    "最终的词汇表里面有写并不是一个完整的单词。\n",
    "可能是 因为词汇表只有8185.可能有些词语没有收录，就只能对该单词部分字节编码了（词汇表中包含了用于构建一些大单词的部分词根，所以可以构建任何单词。）  \n",
    "比如词汇表中没有tensorflow，就会将其拆分为单词表中有的'ten'+'sor'+'fl'+'ow'这四部分词根"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "(train_data, test_data), info = tfds.load(\n",
    "    'imdb_reviews/subwords8k', \n",
    "    split = (tfds.Split.TRAIN, tfds.Split.TEST), \n",
    "    with_info=True, as_supervised=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"_\" in the vocabulary represent spaces. Note how the vocabulary includes whole words (ending with \"_\") and partial words which it can use to build larger words:\n",
    "单词后面的_代表的是空格，词汇表中还会看到一些部分单词，这代表的是用于构建一些单词的部分词根，如‘br’，‘er’等"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['the_',\n",
       " ', ',\n",
       " '. ',\n",
       " 'a_',\n",
       " 'and_',\n",
       " 'of_',\n",
       " 'to_',\n",
       " 's_',\n",
       " 'is_',\n",
       " 'br',\n",
       " 'in_',\n",
       " 'I_',\n",
       " 'that_',\n",
       " 'this_',\n",
       " 'it_',\n",
       " ' /><',\n",
       " ' />',\n",
       " 'was_',\n",
       " 'The_',\n",
       " 'as_']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoder=info.features['text'].encoder\n",
    "encoder.subwords[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(<tf.Tensor: shape=(855,), dtype=int64, numpy=\n",
      "array([ 249,    4,  277,  309,  560,    6, 6639, 4574,    2,   12,   31,\n",
      "       7759, 3525, 2128,   93, 2306,   43, 2312, 2527,    6,   30, 1334,\n",
      "       8044,   24,   10,   16,   10,   17,  977,   30,  815, 3339,   41,\n",
      "        841, 7911,  376, 7974, 1923,    6,  607,  219,   44,  242, 1603,\n",
      "         11, 4329,  102, 2821, 1139,    2,  969,  161,   51,   18,    4,\n",
      "       5844, 2820,  123, 4493,   40,    6, 4571,   13,  117,   35,  289,\n",
      "        850,  455,   50,  460, 6359, 1069,  343,   20,    1, 3733, 3511,\n",
      "       7670,    3,  147,    4,  336,    2,   42,   18,    4, 3422,  409,\n",
      "       3533,  871, 2836,  311,    5, 5080, 1209,    3,  183,  117,   35,\n",
      "       1187,    5, 1955,   11,    1,  226, 7745,    3,  183, 1466, 7359,\n",
      "       7961, 1466,  665,    2, 6854, 3178, 1377, 6266, 1447,  297,    2,\n",
      "       5797,   36, 4740,  847, 8050,    2,    5, 1929, 1631, 5986,   22,\n",
      "       5541, 5688,    5, 1036, 3746, 8050,    3,   69,  264,   35,    4,\n",
      "       4224,    2,   26,   42,   18,    4,  474, 7968,    8, 1626,   24,\n",
      "         10,   16,   10,   17,  134,   15,    9,    1,  435,   13,    9,\n",
      "         55,  598, 2357,   48,   30, 6611, 8044,    3,  500,    1,  101,\n",
      "          6, 2793,    2, 6548, 7961, 5861,  311,    9,  139, 5398, 2270,\n",
      "         11, 3674, 1642,   25, 7796, 7961, 5012,    3,  404,    6,    1,\n",
      "        495, 2632,    5, 5550,  353,    9,  181,  741,    2,   26, 7297,\n",
      "          6,    1, 7890, 8043,    2, 6452, 8050,    2,    5, 6082,  626,\n",
      "       6854, 3178, 1064, 8050,    3,   19, 2227,    9,   56,    4, 1593,\n",
      "          2, 5366,  194, 2808,  106,    1,   98, 1451, 1377,   43,    4,\n",
      "       7757, 1203,  316,   89,    4, 1334, 1209,    3,  135, 7968,    8,\n",
      "         33,   81, 6639, 7837,   44,  208,   24,   10,   16,   10,   17,\n",
      "       7968, 6548, 7962, 1251,   25,    1,  108,  778,    2,    9,   97,\n",
      "       7269, 7961,    7,    1, 1961,    3,   19, 7483, 2632,    6, 7255,\n",
      "       5365,    9, 2125,  652, 4831,  104,    2,   48,    1, 2880, 1477,\n",
      "         34, 1992, 1577, 7961,    7,    1, 6064,  360, 2387, 1209, 2356,\n",
      "          3,   19,  724, 1882, 7974,  177,  256,    1, 1992, 1577, 7961,\n",
      "          5, 4022, 5062,  863,    2,   11,   73,    1, 5758,    9, 6460,\n",
      "        453,  130,    4, 1668,    2,    4, 2403,  120,  538, 7968,   21,\n",
      "         31,   93,  193, 1257,    3, 3371,  713, 6626,   49,    9,   41,\n",
      "       5234, 7961, 3864,   24,   10,   16,   10,   17,  128,    1, 3246,\n",
      "        858, 5550,  353,    9,  114,  224,    2,    1, 2398,  339, 2493,\n",
      "       3553, 2975,    6,    1, 2696,    2,    1, 1183, 8029,    2, 4967,\n",
      "       1274,    2, 1036, 3746,   47,    5, 5541, 5688,   29,   51, 3339,\n",
      "         56,   20, 3608,   20,   19, 5057, 7961,   70,   31, 5324, 8029,\n",
      "         24,   10,   16,   10,   17,  134,  144,   91,    9, 3498,    3,\n",
      "       3669, 7961, 1325, 1093,   47,    9,    4, 7046, 7961,   20,    1,\n",
      "       5224,   47, 4296,  967,  633, 1074,    2, 6279, 6548, 7961, 4438,\n",
      "         40,   57, 2408, 2811, 5848,    7, 1506,  334,   24,   10,   16,\n",
      "         10,   17, 2130,   33, 6241, 6007,  124,    2, 5863, 4685, 1397,\n",
      "       8045,   95,  546,    4, 4296, 7961,   48, 7449, 7968,    8, 2236,\n",
      "       8045, 7974,  272,   95,  379,    4,  763, 3347,   47,   22, 1379,\n",
      "       1004, 7961,   43,  376, 7974,  652, 4831,   34,  138,   20,    4,\n",
      "         97,   68, 2902,  899,  102, 3595,   89,   18, 6639,  413, 7968,\n",
      "          8,  787,    3,  147,    1, 1334,    2,   42,   18, 2577, 7878,\n",
      "       6455, 1344,    2, 5782,   34,   63,    6,   30,  285, 2754,    7,\n",
      "          1, 3130,  863,    8,   11, 1147,    7, 3563, 7961, 1431,   28,\n",
      "        271,   42,   18,  114, 2135,    5, 6161, 8029,  158, 3056, 3950,\n",
      "          8,   11,    1,   65,    2,   42,    9, 3339,   20,  195,    6,\n",
      "          4,  336, 5803,  810, 7974, 6770, 8030,    2,    4,  599,    6,\n",
      "       7028, 2684,  352, 2040, 6491,  316,   89,    4, 5147,  125,    6,\n",
      "       1515,    2,    1, 4812,  139,    4,  933, 7974, 5568, 1339, 7974,\n",
      "       5299,  812, 1621,  166,  475,    3, 4976, 1858,    2,   67,  214,\n",
      "        763, 3347,  440,   29,  949,   22, 6639, 4574, 7968,    8,  787,\n",
      "          3,  404,    6,    1,  107, 2140, 1588,   29, 5760, 2073,    3,\n",
      "        768,  302, 6054,   49,    5, 1665,    9, 2322, 1139,   36,    7,\n",
      "       4746,  182, 3733, 1463, 4662,  353, 6279,   44,    1,  226,  101,\n",
      "       7033, 2117,   23, 1515,  120, 4735, 5210, 2701,  190,    3, 6071,\n",
      "       7961,    9,  176, 2145,  360,    7, 1537,    2,  894, 1383, 6708,\n",
      "         49, 5788, 1990,   64, 2736,    8, 1808, 2153,  100,    2,    5,\n",
      "       2408, 2811, 5848,  265,    4, 6720,   64, 5764, 1868, 7974, 7689,\n",
      "       2842,   24,   10,   16,   10,   17, 2478, 1739,    9, 2097,    2,\n",
      "        271,   53,  110, 1336,  557, 1692, 1370, 2089, 3793,    3, 4312,\n",
      "       2407,  895, 2026, 6924,   30, 3870,  124,    3, 3266,   30,    5,\n",
      "         82, 4370,  211,    9,    1,   45,   11,   73,    1,  386,  327,\n",
      "       6905,  254,  130,   46,   96, 3972, 1581,  135, 7968,    8, 1370,\n",
      "       2089, 3793, 8050,    2,   26,   39, 7968,    8, 6639, 7837,  192,\n",
      "          5, 1884,   24,   10,   16,   10,   17,   12,  117,  186,  136,\n",
      "          3,   12,  117, 2215,  228,    1,  650, 1771, 7974, 3642, 8044,\n",
      "          2,    1, 4129, 7961,    6,    1, 2186,    5, 2234,    3,  173,\n",
      "          9, 1382, 7974,  887,  188,   48,   68, 1127, 4388, 3305,    8,\n",
      "         89,   37,  117, 3977,   49,    4, 2797,  565,   24,   10,   16,\n",
      "         10,   17,   12, 3784,   21,    7,  257,   15,  467, 4854,    2,\n",
      "         26, 8002, 7968,  111,  861,   68,   52, 6690,  766,  837, 1076,\n",
      "          3,  699, 3942,   15,   76,   22,  155, 2400, 3045, 1064,    3,\n",
      "        684,  314,   57,   93,  193, 5418, 8029, 7975])>, <tf.Tensor: shape=(), dtype=int64, numpy=1>)\n",
      "(<tf.Tensor: shape=(194,), dtype=int64, numpy=\n",
      "array([2080, 4956,   90, 7174,    4, 4669,  190,   25,  162,   15,  377,\n",
      "         90, 6989, 7961, 6278,   90,   62,   27, 3931, 8044,    3,   69,\n",
      "         18,   45,    6,    1, 7574,  171,  131,  284,    6,  310,    3,\n",
      "        134,  131,   31,    7,  202,  131,  304,   11,  174,   22,  310,\n",
      "          2,   87,   18,  122,   11,   14, 6972,   24,   10,   16,   10,\n",
      "         17, 1321,  390,   18, 4449,    5,   71,  131,  580,    1,  107,\n",
      "         18,    1, 5353,    5,    1, 3267, 4908,   49,   15,   18, 1006,\n",
      "         37,   83,  509,   13,   53,  356,    4,  238,   94,    1,   27,\n",
      "          1, 1109, 4223, 7961,   85, 2266,   24,   10,   16,   10,   17,\n",
      "         12,  710,   14, 2059, 3646,    5,  955,   83, 1508,    1,   65,\n",
      "          2,    1, 2155,    9, 4627,    9,   67, 2035,    5, 8034, 7968,\n",
      "        111,  399,  955,   29,  215,    7,   52,   39,    3, 7174,    4,\n",
      "       5557,   40,  199,  507,   68,    5,   84, 1037,    1,   27, 2490,\n",
      "         55,  143, 4073, 8044,    3, 1321, 2156, 7974,   76,    5, 3267,\n",
      "       4908,   49,    9,   67,   67, 4004,    5,    1, 1109,   25, 1109,\n",
      "        211,    9, 2266,    3, 2591,   64,    1,   45,  145,   87,  354,\n",
      "         52,   41, 7362,    3,  135, 7968,    8,    4,  279,   86,    5,\n",
      "        131,  710,  955, 3576,   82, 2213, 8044])>, <tf.Tensor: shape=(), dtype=int64, numpy=1>)\n",
      "(<tf.Tensor: shape=(408,), dtype=int64, numpy=\n",
      "array([  12,  284,   14,   32,   25, 1975,   49, 7079,   28,   73, 3632,\n",
      "       3979, 3346,  126,  105, 8044,  297, 3768, 1440,   58,   26, 6137,\n",
      "        124,   95,   15,   18,  248,    3, 6224,   88,  407,   99,    6,\n",
      "          1, 1006,  184, 3976,   50, 2039,  898, 1527,   28,    6,   73,\n",
      "         91,    9, 4549,   58,   26,   99, 6137,  124,    2,  971,   71,\n",
      "        121,  202,   12,  270,    1,   32,   18, 7911,   11,   39, 7968,\n",
      "          8, 6216, 7961,    6, 6561, 7961, 5024,   22, 6970, 2437,   24,\n",
      "         10,   16,   10,   17,  135, 7968,    8, 1597, 5217,  515, 7968,\n",
      "          8, 2589, 3108,    2,  466, 7968,  182,   33, 3030,  826,    2,\n",
      "       1690, 5270, 7961,  170,   46, 2144, 3095,   49,  192, 6647, 7961,\n",
      "         22, 3850,  200,  662, 7145,    3, 5434, 7968,  182,  671,    5,\n",
      "         53,  964,   52,  671, 2815,  221,   10,   16,   10,   17, 5434,\n",
      "       7968,  182, 2880,  475,    2, 7161,    5, 3208,   64, 4862,   60,\n",
      "       3836,    3, 1071,    2,   14,  117,   35,   43, 3047, 7961,    5,\n",
      "         53,  117,   56,   35,  132,  263,   26,   12,  109, 7968,   21,\n",
      "        129,  325,    3, 3431, 4321, 7961, 1946, 8037, 2267, 1098, 5822,\n",
      "        190,   28,   72,   33,  337,    7,   35, 2724,   58,    4,   67,\n",
      "       2507,  390,    5,   12,  674,  138,   23,   39,    3, 2114, 1408,\n",
      "       1166,  758, 1660, 5673, 3559, 1723, 1361,    5, 3208,    2,   26,\n",
      "         87,   18, 6970,   21,    5,    4, 6561,   88,   19, 6728,    6,\n",
      "          1,  146,  260, 7968,   21,  984, 1003, 1182, 1493,   25,  155,\n",
      "       3190,  863,    3,   19,  184,   18, 7620, 2343,    5,   26,   55,\n",
      "        747,   88,   12, 6870, 7961,  295,    2,   30, 1544,  709,    5,\n",
      "        770, 6900,    7, 1671, 8030,   95,   59, 5640,    2,  419, 2044,\n",
      "        495, 5542,   40, 1881,    7, 1118,  161, 7269, 7961, 1176,    3,\n",
      "         12,  129,    1,  400,    2, 2399,   43,  406, 1019,    2, 3098,\n",
      "       4949,  797, 3136,   31,   93, 1517,    5, 5672,   26,   53, 2191,\n",
      "       7968, 8045,    3,   69, 3219,    8,    1, 5013,  472,    6,    1,\n",
      "       2583,  466, 7968,  182,  179,    2,   13,  227, 7968,    8, 2598,\n",
      "         11, 1538,    5,   87, 6237,   25, 1618,  199,   22,    1, 6069,\n",
      "          3,   12,  524, 6166,   44,   59,   23,  142,    2,   12,  524,\n",
      "       6166,   44,  138,   23,  139,  178,    4, 1234,    5,   23,  886,\n",
      "       2261,  310,    3,   12,  524,   15,  208,   24,   10,   16,   10,\n",
      "         17,  147,    1,  272,    2,   12,  674,   15,    5,   70,  505,\n",
      "         39,   24,   10,   16,   10,   17, 2561,   61,   23,    1,  211,\n",
      "        145, 3431, 4321, 7961, 1946, 8037, 2267, 1098, 2070,   48,    1,\n",
      "       2462, 5837,  199, 2110, 4123, 7887, 6044, 3105,   95, 1744, 1142,\n",
      "       7975])>, <tf.Tensor: shape=(), dtype=int64, numpy=1>)\n",
      "(<tf.Tensor: shape=(248,), dtype=int64, numpy=\n",
      "array([  62,   18,    4, 4518, 2920,    5,  966,   27,   13,   12,  767,\n",
      "        243,  140, 6118, 7961,    6,  218, 6091,    2,  143,  603,    3,\n",
      "         19, 2222,   18, 5095, 7961,   11, 6660,   34,   76,    1,  266,\n",
      "         22,    1, 1823,  400,    3,  173,   29,  198,    6,  199,   61,\n",
      "        307,   46,   81,  110, 2284,   74,  263,    5,   41, 2818,   98,\n",
      "       6093,    3,  249,   23, 2635,  131, 2082,  504,    2,   87,    9,\n",
      "        486,    5,    4, 5574, 7961,    7,  119,  312,    6,  651,   11,\n",
      "         73,   87, 1837,    3,   12,  278,  115,    4,  344,    7,   86,\n",
      "       2635,  131, 2082,  591,   11,   51,   59, 1715,    3,  329,    9,\n",
      "          4, 2402, 1215,    5,    4, 6323,    7,  163,   20,  340, 4281,\n",
      "        452,    6,   59,  168,  327,    7,  277,    3,   12,   83,   77,\n",
      "         35, 3424, 6333, 7961,   72,   12,   86,  178,   41, 4449, 1050,\n",
      "         23,  107,    6,    1, 3701, 5141,  126,   68, 2523,  832,   74,\n",
      "        400,    2,   74,  602,    8,    5,   74,  443,    3,   19,   27,\n",
      "       2856, 1220, 1872,  190,    2,   33, 4648, 1067, 4594, 5814,    2,\n",
      "         46,  832, 4478,  708,  472,    5, 2036,    3,  693,  650,    7,\n",
      "         86,  212, 4271,    8,   52, 2407, 2249, 1137,  136,    2, 3393,\n",
      "       2106, 5632, 7961, 6504, 7617,    8,    5, 4954,  222,  687,  736,\n",
      "          3,   12,   70,   52,    7,   86,   14,   27, 2274,    1, 4073,\n",
      "          8,   15, 7071, 8044,    3, 7076,   37,  477,   23,    4, 5805,\n",
      "       7884,    8,  702,    6, 2891,    3,   12, 1131, 3424, 4211,   49,\n",
      "          1, 1626,    2,  575,    2, 6087,    2,    5,   51,  198,   46,\n",
      "        162,  178,    4,  822, 1457, 7975])>, <tf.Tensor: shape=(), dtype=int64, numpy=1>)\n",
      "(<tf.Tensor: shape=(193,), dtype=int64, numpy=\n",
      "array([  12,   31,   84,  480,   71,    1,  108, 7748,    9, 1130,  776,\n",
      "        424,   14,   18,    4,  650,   65,    2,    5, 2411,    4,  471,\n",
      "          6,    1, 3330,   13, 1200,   52,  101, 3007,    3,   19,  170,\n",
      "         29,  485, 2554,    2, 2647,    9,    4, 1000, 3043,   23, 7995,\n",
      "       2962,  136,    2, 7455,   49,    9, 1804, 1182, 2860,    5,    4,\n",
      "        610,   40, 3208,    2, 5615, 5608,  131,    9,    4, 2865, 2145,\n",
      "        413, 7974,  267, 1943, 1671, 2898, 8045,  230,   51,   29, 2055,\n",
      "          2, 5958,  449,    2,    5, 2292,  475,  230,   33,    1, 6810,\n",
      "       7961, 7182,    8,   53,   29,   11, 5083, 4426,   24,   10,   16,\n",
      "         10,   17, 5083, 4426, 7961,  815,   20,   41, 4858,  612,    6,\n",
      "       5115,   66,    2,    5,   12,  674,   15,   23,  402, 5785,    2,\n",
      "         26,   14,  815,   52,    4, 2654, 5992,   21,    5, 8002, 4115,\n",
      "         47,   66,    2,    5,   18,   48,   82, 1767, 2309,    1,  175,\n",
      "       4974,    6,   71,    1, 7697,  279,   31,  524,  463,    3, 4108,\n",
      "       1540,    2, 1551,    2,   26,  446,  376, 2723, 1595,   82, 1303,\n",
      "         11,    1,  919,    6,  756, 1161,   47,    5, 7995, 2962,   25,\n",
      "        242,  101,   12,  129,   60,    1,   66,    3,  373,    6,   82,\n",
      "        208, 7974,  101, 4271, 8044, 7975])>, <tf.Tensor: shape=(), dtype=int64, numpy=1>)\n"
     ]
    }
   ],
   "source": [
    "for i in train_data.take(5):\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用padded_batch方法来标准化评论的长度\n",
    "padded_shapes=([None],())\n",
    "train_batchs=train_data.shuffle(1000).padded_batch(10,padded_shapes=padded_shapes)\n",
    "test_batchs=test_data.shuffle(1000).padded_batch(10,padded_shapes=padded_shapes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[1233, 4438, 2184, ...,    0,    0,    0],\n",
       "        [ 173,    9,   45, ...,    0,    0,    0],\n",
       "        [6172,    8,    7, ...,    0,    0,    0],\n",
       "        ...,\n",
       "        [  12,  456,  247, ...,    0,    0,    0],\n",
       "        [  12,  988,   52, ...,    0,    0,    0],\n",
       "        [  19, 6636, 5119, ...,    0,    0,    0]]), (10, 968))"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 样本都被padding到了每一批中最长的样本长度\n",
    "train_batch,train_label=next(iter(train_batchs))\n",
    "train_batch.numpy(),train_batch.numpy().shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 创建连续词袋模型 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 接下来，嵌入层采用整数编码的词汇表，并查找每个单词索引的嵌入向量。这些向量作为模型训练来学习。嵌入向量向输出数组添加一个维度。生成的维度是：（批处理、序列、嵌入）。 \n",
    "2. 接下来，GlobalAveragePooling1D层通过在序列维度上求平均值来为每个示例返回固定长度的输出向量。这允许模型以最简单的方式处理可变长度的输入。\n",
    "3. 这个固定长度的输出向量通过一个具有16个隐藏单元的完全连接（密集）层进行管道传输。\n",
    "4. 最后一层与单个输出节点紧密连接。使用sigmoid激活函数，这个值是介于0和1之间的一个浮点数，表示评审为正的概率（或置信水平）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, None, 256)         2095360   \n",
      "_________________________________________________________________\n",
      "global_average_pooling1d_1 ( (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 257       \n",
      "=================================================================\n",
      "Total params: 2,095,617\n",
      "Trainable params: 2,095,617\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "embedding_dim=256\n",
    "model=keras.Sequential([\n",
    "    layers.Embedding(encoder.vocab_size,embedding_dim),\n",
    "    layers.GlobalAveragePooling1D(),\n",
    "    # 删除这一层，模型更加简单，获得的词向量更具有解释性\n",
    "#     layers.Dense(16,activation='relu'),\n",
    "    layers.Dense(1,activation='sigmoid')\n",
    "])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint_save_path = \"./word_embedding/word_embedding.ckpt\"\n",
    "\n",
    "# 用于加载模型，checkpoint中保存好了已经训练好了的模型权重，加载之后可以不用训练了\n",
    "# if os.path.exists(checkpoint_save_path + '.index'):\n",
    "#     print('-------------load the model-----------------')\n",
    "#     model.load_weights(checkpoint_save_path)\n",
    "\n",
    "cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_save_path,\n",
    "                                                 save_weights_only=True,\n",
    "                                                 # monitor='loss',\n",
    "                                                 # save_best_only=True,\n",
    "                                                 verbose=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/8\n",
      "   2500/Unknown - 62s 25ms/step - loss: 0.5001 - accuracy: 0.7710- 61s 25ms/step - loss: \n",
      "Epoch 00001: saving model to ./word_embedding/word_embedding.ckpt\n",
      "2500/2500 [==============================] - 69s 28ms/step - loss: 0.5001 - accuracy: 0.7710 - val_loss: 0.3575 - val_accuracy: 0.8699\n",
      "Epoch 2/8\n",
      "2499/2500 [============================>.] - ETA: 0s - loss: 0.2939 - accuracy: 0.8966\n",
      "Epoch 00002: saving model to ./word_embedding/word_embedding.ckpt\n",
      "2500/2500 [==============================] - 68s 27ms/step - loss: 0.2939 - accuracy: 0.8966 - val_loss: 0.3183 - val_accuracy: 0.8787\n",
      "Epoch 3/8\n",
      "2497/2500 [============================>.] - ETA: 0s - loss: 0.2354 - accuracy: 0.9175\n",
      "Epoch 00003: saving model to ./word_embedding/word_embedding.ckpt\n",
      "2500/2500 [==============================] - 68s 27ms/step - loss: 0.2353 - accuracy: 0.9175 - val_loss: 0.3139 - val_accuracy: 0.8802\n",
      "Epoch 4/8\n",
      "2498/2500 [============================>.] - ETA: 0s - loss: 0.2027 - accuracy: 0.9302\n",
      "Epoch 00004: saving model to ./word_embedding/word_embedding.ckpt\n",
      "2500/2500 [==============================] - 69s 28ms/step - loss: 0.2026 - accuracy: 0.9302 - val_loss: 0.3330 - val_accuracy: 0.8779\n",
      "Epoch 5/8\n",
      "2499/2500 [============================>.] - ETA: 0s - loss: 0.1795 - accuracy: 0.9369\n",
      "Epoch 00005: saving model to ./word_embedding/word_embedding.ckpt\n",
      "2500/2500 [==============================] - 67s 27ms/step - loss: 0.1794 - accuracy: 0.9369 - val_loss: 0.3513 - val_accuracy: 0.8750\n",
      "Epoch 6/8\n",
      "2498/2500 [============================>.] - ETA: 0s - loss: 0.1609 - accuracy: 0.9453\n",
      "Epoch 00006: saving model to ./word_embedding/word_embedding.ckpt\n",
      "2500/2500 [==============================] - 66s 26ms/step - loss: 0.1609 - accuracy: 0.9453 - val_loss: 0.3849 - val_accuracy: 0.8680\n",
      "Epoch 7/8\n",
      "2499/2500 [============================>.] - ETA: 0s - loss: 0.1486 - accuracy: 0.9495\n",
      "Epoch 00007: saving model to ./word_embedding/word_embedding.ckpt\n",
      "2500/2500 [==============================] - 66s 27ms/step - loss: 0.1485 - accuracy: 0.9496 - val_loss: 0.4082 - val_accuracy: 0.8671\n",
      "Epoch 8/8\n",
      "2497/2500 [============================>.] - ETA: 0s - loss: 0.1336 - accuracy: 0.9537\n",
      "Epoch 00008: saving model to ./word_embedding/word_embedding.ckpt\n",
      "2500/2500 [==============================] - 67s 27ms/step - loss: 0.1338 - accuracy: 0.9536 - val_loss: 0.4669 - val_accuracy: 0.8518\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer='adam',\n",
    "             loss='binary_crossentropy',\n",
    "             metrics=['accuracy'])\n",
    "history=model.fit(train_batchs,validation_data=test_batchs,epochs=8,callbacks=[cp_callback])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>val_loss</th>\n",
       "      <th>val_accuracy</th>\n",
       "      <th>epoch</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.202594</td>\n",
       "      <td>0.93024</td>\n",
       "      <td>0.332956</td>\n",
       "      <td>0.87792</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.179438</td>\n",
       "      <td>0.93688</td>\n",
       "      <td>0.351303</td>\n",
       "      <td>0.87504</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.160939</td>\n",
       "      <td>0.94532</td>\n",
       "      <td>0.384939</td>\n",
       "      <td>0.86796</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.148546</td>\n",
       "      <td>0.94956</td>\n",
       "      <td>0.408163</td>\n",
       "      <td>0.86712</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.133788</td>\n",
       "      <td>0.95364</td>\n",
       "      <td>0.466915</td>\n",
       "      <td>0.85176</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       loss  accuracy  val_loss  val_accuracy  epoch\n",
       "3  0.202594   0.93024  0.332956       0.87792      3\n",
       "4  0.179438   0.93688  0.351303       0.87504      4\n",
       "5  0.160939   0.94532  0.384939       0.86796      5\n",
       "6  0.148546   0.94956  0.408163       0.86712      6\n",
       "7  0.133788   0.95364  0.466915       0.85176      7"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "hist=pd.DataFrame(history.history)\n",
    "hist['epoch']=history.epoch\n",
    "hist.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtcAAAIWCAYAAACcOqFMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nOzdeXjU1d3+8feZyb6HJEAgCUnYV1kCYq0oWkWrxQWFTG2tPm1t3ZfHtWpr3Zc+avvUbk9/trW1AorWtWpVFK0bCTsIyBrCmoQsZE9mzu+PSWNUlgAz+c5M7td15Qoz853kDnrh7eHzPcdYaxERERERkaPncjqAiIiIiEikULkWEREREQkQlWsRERERkQBRuRYRERERCRCVaxERERGRAFG5FhEREREJkCinAwRKZmamzc/PdzqGiIiIiES40tLSSmtt1v5ei5hynZ+fT0lJidMxRERERCTCGWO2Hug1jYWIiIiIiASIyrWIiIiISICoXIuIiIiIBEjEzFyLiIiISPe0tbVRXl5Oc3Oz01FCWlxcHDk5OURHR3f7PSrXIiIiIr1MeXk5ycnJ5OfnY4xxOk5IstZSVVVFeXk5BQUF3X6fxkJEREREepnm5mYyMjJUrA/CGENGRsZhr+6rXIuIiIj0QirWh3Ykv0cq1yIiIiLS45KSkpyOEBQq1yIiIiIiAaJyLSIiIiKOsdZy4403MmbMGMaOHcu8efMA2LlzJ9OmTWP8+PGMGTOG9957D6/Xy8UXX9x57aOPPupw+q/SbiEiIiIivdi118KyZYH9muPHw2OPde/a5557jmXLlrF8+XIqKyuZPHky06ZN4+9//zszZszgtttuw+v10tjYyLJly9i+fTurVq0CoKamJrDBA0Ar1yIiIiLimPfffx+Px4Pb7aZfv36ceOKJLF68mMmTJ/OnP/2JO++8k5UrV5KcnExhYSGbNm3iqquu4rXXXiMlJcXp+F+hlWsRERGRXqy7K8zBYq3d7/PTpk1j0aJFvPLKK3z3u9/lxhtv5KKLLmL58uW8/vrrPP7448yfP58nnniihxMfnFauRURERMQx06ZNY968eXi9XioqKli0aBFTpkxh69at9O3blx/+8Id8//vfZ8mSJVRWVuLz+Zg1axZ33303S5YscTr+V2jlWkREREQcc+655/Lhhx9yzDHHYIzhoYceon///vzlL3/h4YcfJjo6mqSkJJ588km2b9/OJZdcgs/nA+D+++93OP1XmQMtxYeboqIiW1JS4nQMERERkZD36aefMnLkSKdjhIX9/V4ZY0qttUX7u15jISIiIiIiARLUcm2MOd0Ys84Ys8EYc8t+Xr/YGFNhjFnW8fGDLq99zxjzWcfH94KZU0REREQkEII2c22McQOPA6cC5cBiY8yL1to1X7p0nrX2yi+9tw/wM6AIsEBpx3urg5X3SFkLTU2QkOB0EhERERFxWjBXrqcAG6y1m6y1rcBc4OxuvncG8C9r7d6OQv0v4PQg5TxiPh+MGwc33+x0EhEREREJBcEs1wOBbV0el3c892WzjDErjDHPGmNyD+e9xphLjTElxpiSioqKQOXuNpcLRo6E+fOhvb3Hv72IiIiIhJhglmuzn+e+vDXJS0C+tXYc8Cbwl8N4L9baP1hri6y1RVlZWUcV9kgVF8OePbBwoSPfXkRERERCSDDLdTmQ2+VxDrCj6wXW2iprbUvHw/8DJnX3vaHim9+ElBR4+mmnk4iIiIiI04JZrhcDQ40xBcaYGKAYeLHrBcaY7C4PZwKfdvz6deA0Y0y6MSYdOK3juZATFwfnngvPPQctLYe+XkREREQOT1JS0gFf27JlC2PGjOnBNAcXtHJtrW0HrsRfij8F5ltrVxtj7jLGzOy47GpjzGpjzHLgauDijvfuBe7GX9AXA3d1PBeSPB6orYV//tPpJCIiIiLipKAef26tfRV49UvP/bTLr28Fbj3Ae58AnghmvkA55RTIyvKPhpxzjtNpRERERLrvs8+upb5+WUC/ZlLSeIYOfeyAr998880MGjSIyy+/HIA777wTYwyLFi2iurqatrY27rnnHs4+u7sbzfk1Nzdz2WWXUVJSQlRUFI888gjTp09n9erVXHLJJbS2tuLz+ViwYAEDBgxg9uzZlJeX4/V6ueOOO5gzZ85R/dwQ5HLdW0RFwQUXwJ/+BPX1cJC/uRARERHp9YqLi7n22ms7y/X8+fN57bXXuO6660hJSaGyspKpU6cyc+ZMjNnfPhf79/jjjwOwcuVK1q5dy2mnncb69ev53e9+xzXXXMOFF15Ia2srXq+XV199lQEDBvDKK68AUFtbG5CfTeU6QDwe+M1v4IUX4MILnU4jIiIi0j0HW2EOlgkTJrBnzx527NhBRUUF6enpZGdnc91117Fo0SJcLhfbt29n9+7d9O/fv9tf9/333+eqq64CYMSIEQwaNIj169dz3HHHce+991JeXs55553H0KFDGTt2LDfccAM333wzZ511FieccEJAfragHn/em3zta5Cbq11DRERERLrj/PPP59lnn2XevHkUFxfz1FNPUVFRQWlpKcuWLaNfv340Nzcf1te09is7NwPw7W9/mxdffJH4+HhmzJjB22+/zbBhwygtLWXs2LHceuut3HXXXYH4sVSuA8Xl8u95/frrsDdkb70UERERCQ3FxcXMnTuXZ599lvPPP5/a2lr69u1LdHQ0CxcuZOvWrYf9NadNm8ZTTz0FwPr16ykrK2P48OFs2rSJwsJCrr76ambOnMmKFSvYsWMHCQkJfOc73+GGG25gyZIlAfm5VK4DyOPxn9S4YIHTSURERERC2+jRo9m3bx8DBw4kOzubCy+8kJKSEoqKinjqqacYMWLEYX/Nyy+/HK/Xy9ixY5kzZw5//vOfiY2NZd68eYwZM4bx48ezdu1aLrroIlauXMmUKVMYP3489957L7fffntAfi5zoOXzcFNUVGRLSkoczWCt/zj0AQPg7bcdjSIiIiJyQJ9++ikjR450OkZY2N/vlTGm1FpbtL/rtXIdQMb4V6/feQd2hOR5kiIiIiISTCrXAVZc7F/Bnj/f6SQiIiIikWPlypWMHz/+Cx/HHnus07G+QlvxBdjw4TBhgn/XkGuvdTqNiIiIyP5Zaw9rD2mnjR07lmXLAnvYzaEcyfi0Vq6DwOOBTz6BjRudTiIiIiLyVXFxcVRVVR1ReewtrLVUVVURFxd3WO/TynUQzJkDN90Ec+fCbbc5nUZERETki3JycigvL6eiosLpKCEtLi6OnJycw3qPdgsJkhNOgOpqWLXK6SQiIiIiEkjaLcQBHg+sXg0rVzqdRERERER6isp1kFxwAbjdOg5dREREpDdRuQ6SrCz4xjf8c9cRMnkjIiIiIoegch1EHg9s3uzfOUREREREIp/KdRCdey7Exmo0RERERKS3ULkOopQUOPNMmDcPvF6n04iIiIhIsKlcB5nHA7t2wbvvOp1ERERERIJN5TrIzjwTkpI0GiIiIiLSG6hcB1l8PJxzDixYAK2tTqcRERERkWBSue4BHo//tMbXX3c6iYiIiIgEk8p1Dzj1VMjI0GiIiIiISKRTue4B0dFw/vnwwgvQ0OB0GhEREREJFpXrHuLxQGMjvPSS00lEREREJFhUrnvICSfAwIEaDRERERGJZCrXPcTlgjlz4J//9N/cKCIiIiKRR+W6B3k80NYGzz/vdBIRERERCQaV6x40aRIMGaLREBEREZFIpXLdg4zxr16//bb/SHQRERERiSwq1z2suBh8PnjmGaeTiIiIiEigqVz3sFGjYNw4jYaIiIiIRCKVawd4PPDhh7Bli9NJRERERCSQVK4dUFzs/zx3rrM5RERERMJRa2sFK1eeTWPjOqejfIXKtQPy8+G44zQaIiIiInK46utXUFo6merqN1Su5XMeD6xYAWvWOJ1EREREJDxUVDzHkiVfw9p2xo9/j8zMmU5H+gqVa4fMnu0/tVGr1yIiIiIHZ62PLVvuYvXqWSQmjmHSpMWkpBQ5HWu/VK4d0q8fnHyyv1xb63QaERERkdDk9TawZs0ctmz5Gf36XcT48e8QG5vtdKwDUrl2kMcDGzdCaanTSURERERCT3PzVpYsOZ6KiucYPPh/GDHiz7jdcU7HOiiVaweddx7ExGg0REREROTLamrep7R0Ms3NWxg79hVyc6/HGON0rENSuXZQWhqccQbMm+c/tVFEREREYMeOP7J8+clERaUzadLHZGSc7nSkblO5dpjHA9u3w3vvOZ1ERERExFk+XzuffXY169f/kLS0k5k48SMSEoY7HeuwqFw77KyzICFBoyEiIiLSu7W1VbFixels3/6/5ORcz9ixLxMdne50rMOmcu2wxEQ4+2x49lloa3M6jYiIiEjPa2hYTWnpsdTWvsfw4X9iyJD/weWKcjrWEVG5DgEeD1RVwb/+5XQSERERkZ5VWfkSS5Ych8/XwPjx75CdfbHTkY6KynUImDED0tM1GiIiIiK9h7WWrVsfYNWqs4mPH8bEiYtJTT3O6VhHLTzX2yNMTAzMmgVz50Jjo38GW0RERCRSeb1NrFv3ffbseZq+fT0MH/7/cLvjnY4VEFq5DhEeD9TXwyuvOJ1EREREJHiam8tZuvQE9uyZS0HB/Ywc+VTEFGtQuQ4ZJ54I2dkaDREREZHIVVv7EUuWTKapaR1jxrzAoEG3hMXBMIdD5TpEuN0weza8+irU1jqdRkRERCSwdu36C8uWnYjLlcDEiR+RmfktpyMFhcp1CPF4oKUF/vEPp5OIiIiIBIa1XjZsuIG1ay8mNfXrTJr0CYmJo52OFTRBLdfGmNONMeuMMRuMMbcc5LrzjTHWGFPU8TjfGNNkjFnW8fG7YOYMFVOmQGGhRkNEREQkMrS11bBy5VmUl/8PAwdexbhxrxEdneF0rKAK2m4hxhg38DhwKlAOLDbGvGitXfOl65KBq4GPv/QlNlprxwcrXygyBoqL4cEHoaICsrKcTiQiIiJyZBob17Fy5UyamzczbNgfGDDgh05H6hHBXLmeAmyw1m6y1rYCc4Gz93Pd3cBDQHMQs4QNjwe8XnjmGaeTiIiIiByZqqrXKC09lvb2vRxzzFu9plhDcMv1QGBbl8flHc91MsZMAHKttS/v5/0Fxpilxph3jTEnBDFnSBkzBkaP1miIiIiIhB9rLdu2/Q8rV55JXNwgJk0qIS2t19Q4ILjlen/7qtjOF41xAY8C/72f63YCedbaCcD1wN+NMSlf+QbGXGqMKTHGlFRUVAQotvM8Hnj/fSgrczqJiIiISPd4vc2sXXsJGzfeQGbmuUyY8G/i4gY5HavHBbNclwO5XR7nADu6PE4GxgDvGGO2AFOBF40xRdbaFmttFYC1thTYCAz78jew1v7BWltkrS3KiqAB5eJi/+d585zNISIiItIdLS07Wb58Ort3/4X8/DsZPXo+UVFJTsdyRDDL9WJgqDGmwBgTAxQDL/7nRWttrbU201qbb63NBz4CZlprS4wxWR03RGKMKQSGApuCmDWkDB7s3zlEoyEiIiIS6urqSigtnUx9/QpGj15Afv7P8A8o9E5B+8mtte3AlcDrwKfAfGvtamPMXcaYmYd4+zRghTFmOfAs8GNr7d5gZQ1FHg8sXQrr1jmdRERERGT/du9+mmXLTsCYKCZO/ICsrPOcjuQ4Y6099FVhoKioyJaUlDgdI2B27ICcHPjpT+HOO51OIyIiIvI5a31s3nw7ZWX3k5o6jdGjnyUmJnJGdA/FGFNqrS3a32u9d80+xA0YACed5B8NiZD//xEREZEI0N5ex6pV51BWdj/Z2ZdyzDH/6lXF+lBUrkOYxwPr18OyZU4nEREREYGmpo0sWXIcVVWvMnTo4wwb9jtcrhinY4UUlesQNmsWREfrxkYRERFxXnX1W5SWTqa1dRfHHPMGAwdejjH723m5d1O5DmF9+sCMGTB3Lvh8TqcRERGR3shaS3n5/7J8+QxiYgYwadJi0tNPdjpWyFK5DnEeD2zbBh984HQSERER6W18vlbWr7+UDRuuJiPjTCZO/JD4+EKnY4U0lesQN3MmxMdrNERERER6VmvrHpYvP4WdO/9IXt5tjBnzPFFRyU7HCnkq1yEuKQm+9S145hlob3c6jYiIiPQG+/Yto7R0Mvv2lTBy5NMUFt7Tqw+GORz6XQoDHg9UVMBbbzmdRERERCLdnj3PsnTp8VjrZcKE9+nXr9jpSGFF5ToMnHEGpKZqNERERESCx38wzM9Ys+YCkpLGMWlSCcnJk5yOFXZUrsNAbCycdx48/zw0NzudRkRERCJNe3s9q1dfwNatd9G//8WMH/8OsbH9nY4VllSuw4THA3V18OqrTicRERGRSNLUtIWlS4+nsvIfDB78KMOHP4HLFet0rLClch0mpk+Hvn01GiIiIiKBU1OziCVLJtPSUsa4cf8kN/daHQxzlFSuw0RUFMyeDS+/7F/BFhERETkaO3b8geXLTyEqKoOJEz+mT5/TnI4UEVSuw4jH45+5fvFFp5OIiIhIuPL52li//grWr/8R6emnMmnSxyQkDHM6VsRQuQ4jxx0HgwZpNERERESOTGtrJStWnMaOHb8hN/dGxo59iaioVKdjRRSV6zBiDBQXwxtvQFWV02lEREQknNTXr2LJkinU1n7IiBFPMnjwQxjjdjpWxFG5DjMej/+kxmefdTqJiIiIhIvKyhdYuvQ4fL5mJkx4l/79v+t0pIilch1mxo2DkSM1GiIiIiKHZq1l69Z7WbXqHBISRjBp0mJSUo51OlZEU7kOM/8ZDVm0CLZvdzqNiIiIhCqvt5E1azxs3nw7fft+m/HjFxEbO9DpWBFP5ToMeTxgLcyb53QSERERCUXNzdtYuvQEKirmU1j4ACNH/g23O97pWL2CynUYGjoUJk3SaIiIiIh8VW3tB5SWTqap6TPGjHmRvLybdTBMD1K5DlMeD5SUwGefOZ1EREREQsXOnX9i2bLpuN1JTJz4EZmZZzkdqddRuQ5Tc+b456/nznU6iYiIiDjN52tnw4brWbfuv0hLm8akSZ+QmDjK6Vi9ksp1mMrJgRNO8I+GWOt0GhEREXFKW1s1K1eeSXn5owwceA1jx/6T6Og+TsfqtVSuw5jHA59+CitXOp1EREREnNDQ8ClLlhxLTc1Chg//I0OHPobLFeV0rF5N5TqMnX8+REXpxkYREZHeqKrqVZYsmUp7ey3jxy8kO/v7TkcSVK7DWmYmnHqqf+5aoyEiIiK9g7WWsrKHWbnyLOLjC5k0aTGpqcc7HUs6qFyHOY8HtmyBjz5yOomIiIgEm9fbzNq1F7Fp001kZZ3PhAnvExeX53Qs6ULlOsydfTbExWk0REREJNK1tOxg2bIT2b37b+Tn382oUfNwuxOdjiVfonId5lJS4MwzYf58aG93Oo2IiIgEQ13dJ5SWFtHQsJrRo58jP/92HQwTolSuI4DHA7t3wzvvOJ1EREREAm3Xrr+xdOk0XK5YJk78kKysc52OJAehch0BvvlNSE7WaIiIiEgksdbLxo03s3btd0lJmcrEiYtJShrrdCw5BJXrCBAfD+eeCwsWQEuL02lERETkaLW317Jy5Uy2bXuIAQN+zDHH/IuYmEynY0k3qFxHCI8HamvhtdecTiIiIiJHo7HxM5YsmUp19RsMHfobhg37LS5XtNOxpJtUriPEKaf4973WaIiIiEj42rv3XyxZMoXW1grGjfsXAwde5nQkOUwq1xEiOhouuABefBHq651OIyIiIofDWkt5+S9ZseJ0YmNzmDRpMenpJzkdS46AynUE8XigqQleesnpJCIiItJdPl8L69b9gA0briUzcyYTJnxAfHyB07HkCKlcR5Djj4ecHI2GiIiIhIvW1t0sW3Yyu3Y9waBBdzB69AKiopKdjiVHQeU6grhcUFzsv6lx716n04iIiMjB7Nu3hNLSIurrlzJq1DwKCu7CGFWzcKd/ghHG44G2NnjuOaeTiIiIyIHs2TOfpUu/DhgmTPg3ffvOdjqSBIjKdYSZMAGGDdNoiIiISCiy1sfmzXewZs0ckpImMGnSYpKTJzgdSwJI5TrCGOMfDVm4EHbudDqNiIiI/Ed7+z5WrTqPrVvvoX///2L8+LeJienndCwJMJXrCOTxgLUwf77TSURERASgqWkzS5d+jaqqlxgy5JcMH/5HXK5Yp2NJEKhcR6ARI2D8eI2GiIiIhILq6ncoLZ1MS0s548a9Rk7O1RhjnI4lQaJyHaE8Hvj4Y9i0yekkIiIivZPX20x5+a9YseJUYmKymDjxE/r0OdXpWBJkKtcRqrjY/3nuXGdziIiI9Cbt7fXs2fMMq1cX88EHWWzYcA3p6acxceJHJCQMdTqe9IAopwNIcOTl+Q+Vefpp+MlPnE4jIiISudrba6msfInKygXs3fsaPl8z0dFZ9O3rIStrFunpp2r/6l5E5TqCeTxw5ZWwahWMGeN0GhERkcjR2lpJVdULVFQsoLr6TaxtIyZmANnZPyAzcxZpaSdgjNvpmOIAlesIdsEFcM01/tGQe+5xOo2IiEh4a2nZSWXl81RULKCm5l3AS1xcPgMHXk1W1ixSUo7VCrWoXEeyvn3hlFP8oyF33+3fA1tERES6r7l5KxUVz1FRsYC6ug8AS3z8cPLybiYraxZJSRO084d8gcp1hPN44JJLYPFimDLF6TQiIiKhr7HxMyoqFlBZuYB9+0oASEwcR37+nWRlzSIhYZQKtRyQynWEO/dc+PGP/avXKtciIiJfZa2loWE1lZULqKhYQEPDSgCSkydTWPgAmZmzSEgY4nBKCRdBLdfGmNOBXwJu4I/W2gcOcN35wDPAZGttScdztwLfB7zA1dba14OZNVKlpsI3vwnz5sEvfgFu3VshIiKCtZb6+iVUVPgLdVPTesCQmno8gwc/SlbWecTF5TkdU8JQ0Mq18d8i+zhwKlAOLDbGvGitXfOl65KBq4GPuzw3CigGRgMDgDeNMcOstd5g5Y1kxcXw/POwaBFMn+50GhEREWdY66Ou7qOOkY/naG7eArhJSzuJnJxrycw8h9jYbKdjSpgL5sr1FGCDtXYTgDFmLnA2sOZL190NPATc0OW5s4G51toWYLMxZkPH1/swiHkj1llnQVKSfzRE5VpERHoTn6+d2tr3Ogr187S27sCYaNLTT2XQoDvIyJhJTEym0zElggSzXA8EtnV5XA4c2/UCY8wEINda+7Ix5oYvvfejL7134Je/gTHmUuBSgLw8/dXNgSQkwNlnw7PPwq9/DTExTicSEREJHp+vlerqt6msXEBl5T9oa6vE5YqnT5/TycqaRUbGWURFpTodUyJUMMv1/m6jtZ0v+jeCfBS4+HDf2/mEtX8A/gBQVFT0ldflcx4PPPUUvPGGfyVbREQkkni9TVRXv0FFxQKqql6ivb0GtzuJjIyzyMycRUbGGbjdiU7HlF4gmOW6HMjt8jgH2NHlcTIwBninYzub/sCLxpiZ3XivHKZTT4U+ffyjISrXIiISCdrb69m799WOQv0KPl8DUVHpZGSc3XnsuNsd53RM6WWCWa4XA0ONMQXAdvw3KH77Py9aa2uBziEnY8w7wA3W2hJjTBPwd2PMI/hvaBwKfBLErBEvJgbOP9+/et3Y6B8VERERCTdtbTVUVb3Ucez46/h8zURHZ9Gv34VkZc0iLW06Lle00zGlFwtaubbWthtjrgRex78V3xPW2tXGmLuAEmvtiwd572pjzHz8Nz+2A1dop5Cj5/HAH/4AL78Ms2c7nUZERKR7Wlsrqaz8B5WVC6iufgtr24iJGUh29g/JyppFaurX8W9SJuI8Y21kjCoXFRXZkpISp2OENK8X8vL8h8k8/7zTaURERA6spWUHlZXPU1GxgJqadwEfcXEFZGXNIjNzFikpU/DfviXS84wxpdbaov29phMaexG3G+bMgccfh5oaSEtzOpGIiMjnmpu3dh7qUlf3IWBJSBhBXt6tZGXNIilpvI4dl5Cnct3LeDzw6KP+letLLnE6jYiI9HaNjes7C3V9fSkAiYnHkJ//c7KyZpGYOMrhhCKHR+W6lykqgsGD/buGqFyLiEhPs9bS0LCaiopnqaxcQEPDKgCSk6dQWPggmZnnkZAwxOGUIkdO5bqXMca/en3ffbB7N/Tr53QiERGJdNZa9u0rpbLyOSoqFtDUtB4wpKZ+nSFDHiMz8zzi4nIP+XVEwoHKdS9UXAz33APPPANXXul0GhERiUTW+qir+7Bj5OM5Wlq2Am7S06eTk3MdmZnnEBvb3+mYIgGnct0LjR4NY8f6R0NUrkVEJFB8vnZqaxdRUbGAysrnaW3diTExpKefSn7+z8jMnEl0dIbTMUWCSuW6l/J44Cc/ga1bYdAgp9OIiEi48vlaqa5+q+OUxBdoa6vE5YqnT5/Tyco6n4yMM4mKSnU6pkiPUbnupYqL/eV67ly4+Wan04iISDjxepuorn6j46bEl/B6a3G7k8jIOIvMzFlkZJyB253odEwRR6hc91IFBTB1qn80ROVaREQOpb29nr17X6Gi4jmqql7B52sgKiqdzMxzyMqaRXr6qbjdcU7HFHGcynUv5vHANdfAp5/CyJFOpxERkVDT1lZDVdVLVFQ8y969r2NtC9HRfenX70KysmaRljYdlyva6ZgiIUXluhebPRuuu84/GvLznzudRkREQkFrawWVlS9QWbmA6uq3sLaNmJiBDBhwKVlZs0hN/TrGuJ2OKRKyVK57sf79Yfp0/2jInXf698AWEZHep6VlZ+ce1DU17wI+4uIKyMm5hszMWaSkTMEYl9MxRcKCynUv5/HAD34AS5bApElOpxERkZ5UX7+KbdseZPfupwEvCQkjyMu7laysWSQljcdo1UXksKlc93LnnQeXXeZfvVa5FhHpHWprP6Cs7H6qql7G5UokJ+cqsrN/SGLiKKejiYQ9leteLj0dzjgD5s2Dhx4Cl/7WT0QkIllr2bv3n5SVPUBt7XtERWWQn/9zBg68Qge7iASQyrVQXAwvvgjvvw/TpjmdRkREAsnna6ei4hnKyh6goWEFsbG5DBnyS7Kzv6+9qEWCQOVamDkTEhL8oyEq1yIikcHrbWLXrj+zbWxux+oAACAASURBVNvDNDdvJiFhJCNG/Jm+fT24XDFOxxOJWCrXQmKiv2A/8wz86lcQrS1LRUTCVltbDTt2/Jby8sdoa9tDcvKxDBnyKBkZ39KOHyI9QOVaAP+uIXPnwptv+mewRUQkvLS07KK8/DF27PgtXm8d6ekzyMu7hbS0E7Xrh0gPUrkWAGbMgLQ0/2iIyrWISPhoatpIWdnD7Nr1Z6xtIyvrfPLybiE5eYLT0UR6JZVrASA2FmbN8u8a0tQE8fFOJxIRkYPZt28Z27Y9yJ498zEmiv79LyY39wYSEoY6HU2kV9PwlXTyeKC+Hl591ekkIiKyP9ZaamreZcWKMygtnUBV1Svk5t7A1KlbGD789yrWIiFAK9fS6aST/EeiP/20fxVbRERCg7U+qqpepqzsAerqPiQ6OouCgnsZMOByoqPTnI4nIl2oXEsntxtmz4bf/x7q6iAlxelEIiK9m8/Xxp49T1NW9iCNjWuIi8tn6NDH6d//Etxuze+JhCKNhcgXeDzQ0gL/+IfTSUREei+vt5Hy8v/l44+HsHbt9zDGxciRf2PKlM8YOPByFWuREKaVa/mCY4+FggL/aMhFFzmdRkSkd2lrq2b79l+zffuvaGurJCXleIYN+w19+nxT2+mJhAmVa/kCY/zHoT/0EFRUQFaW04lERCJfS8t2tm17lJ07f4/XW0+fPmd27FH9daejichh0liIfEVxMXi98OyzTicREYlsjY3rWLv2B3z0UQHl5Y+RkXE2RUXLGTfuZRVrkTCllWv5irFjYdQo/2jIZZc5nUZEJPLU1ZVQVvYAlZXP4XLFkp39Q3JzbyA+vsDpaCJylFSu5SuM8d/YeMcdsG0b5OY6nUhEJPz596h+m7KyB6iufhO3O5W8vFvJybmamJh+TscTkQDRWIjsV3Gx//O8ec7mEBEJd9b6qKh4jiVLjmX58m/Q0LCKwsIHOe64MgoL71WxFokwWrmW/RoyBCZP9o+G3HCD02lERMKPz9fK7t1/o6zsIZqa1hEXN5hhw35Pv34X4XbHOR1PRIJE5VoOyOOB66+H9eth2DCn04iIhIf29np27vw/tm37H1pbt5OUNJ5Ro+aSlXU+xridjiciQaaxEDmgOXP889dz5zqdREQk9LW2VrJ588/46KM8Nm68noSEoYwb9xqTJi2hb985KtYivYRWruWABgyAE0/0j4bccYe/aIuIyBc1N5exbdsj7Nz5f/h8jWRmnkNu7s2kpk51OpqIOEDlWg7K44Ef/QiWL4fx451OIyISOhoa1lBW9hB79jwFQN++F5KXdxOJiaMcTiYiTlK5loOaNQuuuMK/eq1yLSICtbUfsW3bg1RW/gOXK4EBA64gN/d64uLynI4mIiFA5VoOKiMDZszwz13ffz+4NKUvIr2QtZbq6jcoK3uAmpp3iIpKZ9CgnzJw4FXExGQ6HU9EQojKtRySxwOvvAIffgjHH+90GhGRnmOtl4qKZykre4D6+mXExAxk8OBHyM7+IVFRSU7HE5EQpHIthzRzJsTF+UdDVK5FpDfwepvZvftJysoeorl5I/Hxwxg+/P/Rr9+FuFyxTscTkRCmci2HlJwM3/oWPPMMPPYYROnfGhGJUO3tdezY8TvKyx+ltXUXyclFDB68gMzMs7WVnoh0i2qSdIvH4y/Xb78Np53mdBoRkcBqbd1Defkv2b79cbzeWtLTv8HIkU+RljYdo31IReQwqFxLt5xxBqSk+EdDVK5FJFI0NW1m27ZfsGvXE/h8LWRlzSI392ZSUoqcjiYiYUrlWrolLg7OOw+eew5++1v/YxGRcFVfv5KysgfZs2cuxrjo1+8i8vJuIiFhmNPRRCTMaWO1o2StdTpCj/F4oK4O/vlPp5OIiByZ2tp/s2LFWZSUjKOy8h/k5FzL1KmbGTHijyrWIhIQWrk+SmvWzCYubjB5ebcQHZ3mdJygOvlk6NvXv+f1uec6nUZEpHustezd+yplZQ9QW/s+UVEZ5OffxcCBVxAd3cfpeCISYVSuj4LP14rLFc+2bQ+yc+f/MWjQHQwceFnEbtMUFQUXXABPPAH19ZCkLV5FJIT5fO1UVMyjrOxBGhpWEhubx5AhvyI7+79wuxOdjiciEUpjIUfB5Yph5MgnmTRpCcnJE9m48To++WQku3fPxVqf0/GCwuOBpiZ44QWnk4iI7J/X28T27b/hk0+G8umn38FaLyNGPMmxx24gJ+cqFWsRCSoTKTPDRUVFtqSkxNEMe/e+wcaNN9LQsILk5CIKCx8mPf0kRzMFms8HBQUwdiy8/LLTaUREPtfWVsOOHb+hvPwx2toqSEmZSl7erWRknIUxWksSkcAxxpRaa/e7rZD+tAmgPn1Oo6hoCSNG/IXW1t0sXz6dFSvOor5+ldPRAsblguJieP11qKpyOo2ICLS07GTjxpv56KM8Nm++jeTkIsaPf5cJEz4gM3OmirWI9Cj9iRNgxrjp3/8ipkxZR2Hhg9TWvk9JyTGsXfsDWlq2Ox0vIDweaG+HBQucTiIivVlj4wbWrfsRH32Uz7ZtvyAj40wmTVrKuHGvkpY2TYe/iIgjNBYSZG1tVWzdei/bt/8aY6LIybmevLybiIpKcTraEbMWRo6E7GxYuNDpNCLS2+zbt5SysgepqHgGY6LJzr6E3NwbiI8f7HQ0EeklHBsLMcacboxZZ4zZYIy5ZT+v/9gYs9IYs8wY874xZlTH8/nGmKaO55cZY34XzJzBFB2dwZAhjzBlyjoyM8+hrOxePv54MOXlv8bna3U63hExxr96/e67sD0yFuNFJMT5fC3s3fsGy5efTmnpRPbufZXc3BuZOnUzw4b9VsVaREJG0FaujTFuYD1wKlAOLAY81to1Xa5JsdbWdfx6JnC5tfZ0Y0w+8LK1dkx3v1+orlx/WV1dCZs23UhNzTvExw+hoOA+srLOD7u/vly/HoYPh0cegeuuczqNiEQan6+dfftKqKl5m5qahdTW/hufr4no6L7k5FzHgAE/jvizBUQkdB1s5TqY+1xPATZYazd1hJgLnA10luv/FOsOiUBkzKgcREpKEccc8zZ79/6TjRtvYs2a2SQnH8vgwQ+TlnaC0/G6bdgwmDgRnn5a5VpEjp61Xurrl1NTs5Dq6reprX0Pr3cfAImJ48jOvpT09Omkp5+G2x3vcFoRkQMLZrkeCGzr8rgcOPbLFxljrgCuB2KAk7u8VGCMWQrUAbdba9/bz3svBS4FyMvLC1zyIDPGkJHxTfr0mcGuXU+yefPtLFs2jYyMmRQWPkBi4kinI3aLxwM33ggbNsCQIU6nEZFwYq2lsXEN1dVvd6xOv0t7ezUACQkj6NfvO6SlnUxa2onExGQ5nFZEpPuCORZyATDDWvuDjsffBaZYa686wPXf7rj+e8aYWCDJWltljJkE/AMY/aWV7i8Il7GQ/fF6Gykvf4yysgfwehvIzv4B+fl3Ehub7XS0g9q2DfLy4J574LbbnE4jIqHMWktT0wZqat6munohNTULaWvbA0BcXAFpaSeTnj6dtLTpxMYOcDitiMjBHWwsJJjl+jjgTmvtjI7HtwJYa+8/wPUuoNpam7qf194BbrDWHrA9h3O5/o/W1gq2br2bHTt+izEx5ObeQG7uDURFJTsd7YCmTYO9e2FV5GzlLSIB0ty8taNIv0119du0tvrvgI6JGdhRpE8mLW068fH5zgYVETlMTs1cLwaGGmMKgO1AMfDtLwUbaq39rOPhmcBnHc9nAXuttV5jTCEwFNgUxKwhISYmi6FDf8XAgVezefNtbN16Fzt2/I78/DvJzv4BLle00xG/wuOByy+HlSv9pzaKSO/V0rKzc2a6pmYhzc3+P7ajo7NIS5tOevp/yvTQsLuJW0Sku4JWrq217caYK4HXATfwhLV2tTHmLqDEWvsicKUx5htAG1ANfK/j7dOAu4wx7YAX+LG1dm+wsoaahIQhjB49j7q669m48UY+++xyyssfo7DwATIzzwmp/yidfz5cdZX/xkaVa5HepbW1kpqad6ip8a9ONzauBSAqKo20tJPIybmGtLSTSUwcHVJ/bomIBJMOkQlx1lqqql5m06abaWz8lJSUrzF48MOkpn7N6WidzjgD1q2DjRv9e2CLSGRqa6uhtnZR5+p0Q8MKANzuJFJTp3WuTiclHYN/N1YRkcjk1FiIBIAxhszMb9Gnzxns2vUntmz5KUuXHk9m5nkUFt5PQsIwpyPi8cD3vgcffwxTpzqdRkQCxettoLb2/c4xj337SgEfLlccKSnHU1BwL2lp00lOLgrJsTURESdo5TrMeL0NbNv2CNu2PYTX28SAAT8iP/+nxMT0cyxTXR307Qs/+hH88peOxRCRo+T1NlNX92Hnjh779n2Mte0YE01KytTOlemUlKm4XLFOxxURcYwju4X0tN5Srv+jtXU3W7bcxY4dv8ftjic39yZyc6/H7U50JM+sWfDBB1BeDm79bbBIWPD52ti375POHT1qaz/A2hbARXLy5M4dPVJTv+bYny0iIqFI5TqCNTauY9Omn1BZ+RwxMf3Jz7+L/v0vweXq2YmfZ5+FCy6AN9+EU07p0W8tIt1krZd9+5Z2bo1XW/s+Pl8DYEhKOqZza7y0tBOIivrKrqgiItJB5boXqK39gI0bb6Su7gMSEkZSWPgAGRnf6rE79JuaoF8/mD0b/vjHHvmWInII1vpoaFjVOTNdU/MuXm8tAAkJo7psj3ci0dEZDqcVEQkfKte9hLWWysp/sGnTLTQ1rSc19QQGD36YlJSvnDofFBddBC+9BLt2QazGMUV6nP9I8XWdW+PV1LxDW1slAHFxgzuK9MmkpZ1EbGx/h9OKiIQv7RbSSxhjyMo6l4yMs9i5849s2XInS5ZMJSvrAgoK7iMhYUhQv7/HA3/9K7z+OsycGdRvJSIdmpo2d4551NQspLV1JwCxsbn06XNm55HicXF5DicVEekdtHIdwdrb97Ft2y/Ytu0XWNvGgAGXMWjQ7cTEZAXl+7W1QXY2nHYa/P3vQfkWIr1ec3N5x8q0f6/plpatAERH9+u8ATE9/WTi4gp1cIuISJBoLKSXa2nZyZYtP2fnzj/idieQl3cLOTnX4nYnBPx7XXYZPPkk7NkDidpcQOSotbbu6SjS/lGPpqbPAIiK6kNa2kmdR4onJIxUmRYR6SEq1wJAQ8OnbNp0K1VVLxATM4CCgrvp3/97AT1JbdEiOPFE/3HoxcUB+7IivUZbWzU1Ne92zEwvpKFhFQBudzJpaSd27OZxMklJ4zDG5XBaEZHeSeVavqCm5j02bryRffs+JjFxDIWFD9KnzxkBWfXy+WDQIJg4EV54IQBhRSJce/s+amvf6xzzqK9fClhcrnhSU0/ocqT4xB7fYlNERPZPNzTKF6SlncDEiR9SUbGATZtuYeXKM0lLm05h4UOkpOz335Nuc7lgzhz41a+guhrS0wMUWiRCeL2N1NZ+0LmjR13dYsCLMTGkpBxHfv6dpKVNJyVlik5BFBEJQ1q57uV8vlZ27PgDW7f+nLa2Svr29VBQcC/x8QVH/DVLS6GoyL/f9fe/H8CwImHI52ulru7jziPF6+o+xNpWwE1KypQuR4p/Dbc73um4IiLSDRoLkUNqb6+jrOwhyssfwdp2Bg68kkGDbjuigyWshWHD/OMhb74ZhLAiIczna6e+vrRzazz/KYhN+E9BnNjlSPGvExWV7HRcERE5AirX0m0tLdvZvPln7Nr1J9zuZAYN+gkDB1512CtqP/0p3HsvbN8O/XVWhUQw/8Eta6mufpPq6jepqXkHr7cOgMTEMV2OFD+R6GjNSYmIRAKVazls9fWr2LTpFvbufYXY2FwKCu6hX78Lu72zyKefwqhR8MtfwtVXBzmsSA9radlBdfVbnYW6tXUHAHFxhaSnn0J6+jdISzuJmJi+DicVEZFgULmWI1ZdvZBNm25i374SEhOPYfDgh+jT57RuvXf8eIiPhw8/DHJIkSBrb6+jpuadjjL9Fo2NawCIjs4kLc1fptPTTzmqexVERCR8aLcQOWLp6dOZOPFj9uyZz+bNP2HFihmkp59KYeGDJCdPOOh7PR645RbYvBkK1DkkjPhvQvyoc2W6ru4TwNuxPd40+ve/hPT0b2ivaRER+QqtXEu3+XwtbN/+W7ZuvZv29mr69fsOBQV3Exc3aL/Xb90K+flw331w6609m1XkcFjro6FhVZe56UX4fA2Ai+TkyR0r098gNfU4bY8nIiIaC5HAamuroazsAcrLHwMgJ+dq8vJu3e/NWscfD/X1sHx5T6cUObjm5rLOMl1d/RZtbXsAiI8f3lmm09JOIjo6zeGkIiISalSuJSiam8vYvPmn7N79JFFRaQwadBsDBlyB2x3Xec2vfw1XXQWrV/tvcBRxiv9Y8YWdhbqp6TMAoqP7dZbp9PRTiIvLdTipiIiEOpVrCar6+hVs2nQze/e+RmzsIAoL76VvXw/GuNi9GwYMgJ/8BO6+2+mk0pt4vc3U1X3QWab37SsFfLjdSaSlndSxMn0KiYmjMcY4HVdERMKIyrX0iL1732TTppuor19KUtIEBg9+mPT0UzjtNNi0CT77DNRhJFis9VFfv6yzTNfWvofP14wxUaSkTO1cnU5OnoLLFe10XBERCWMq19JjrPWxZ8/TbNr0E1payujT53RWrHiQ7353HJ98ApMnO51QIklT06YvzE23t+8F/Ie3fH4T4jSdhCgiIgGlci09zuttZseOx9m69R7a22t5443vAXdx332aZ5Uj19paQU3N252Furl5CwCxsTldbkI8mdjYbGeDiohIRFO5Fse0te2lrOx+Nm/+FT6fi8LCa8nPv4WoqFSno0kY8Hobqa19r7NM19cvA8DtTiU9fXpnoY6PH6a5aRER6TFHfYiMMWYwUG6tbTHGnASMA5601tYELqZEoujoPgwe/DBLllzB0qV3EBPzALt2/R/5+XcwYMBluFwxTkeUEOLztVNfX9plbvoDrG3FmBhSU4+noODejsNbJuJy6QwsEREJPd1auTbGLAOKgHzgdeBFYLi19ptBTXcYtHId2hoboW9fuPLKJRQX30RNzVvExRVSWHgfWVmzterYS1lraWpa32VueiFeby0ASUkTusxNfx23O8HhtCIiIn5HPRZijFlirZ1ojLkRaLbW/q8xZqm19uDnX/cglevQd+GF8NprsGOHpaHhDTZuvImGhhUkJ0+msPAh0tNPcjqi9ICWll3U1LzVWahbWsoBiIvLJz391I656enExGQ5nFRERGT/jnosBGgzxniA7wHf6nhOe1nJYfF44O9/hzffNJx55gzS07/B7t1/Y/Pm21m+fDoZGWdRWPgAiYmjnY4qAdTevo+amnc7jhV/i4aGVQBERfUhPf2ULnPThQ4nFREROXrdXbkeBfwY+NBa+7QxpgCYY619INgBu0sr16GvtRX694czz4S//vXz573eJrZv/xVbt96H11tPdvZ/kZ//c2JjBzgXVo6Yz9dGXd3HXQ5v+Rhr23G54khNPaGzTCcljccYl9NxRUREDltAdwsxxqQDudbaFYEIFygq1+Hh0kvh6adh925I+NIIbWtrJWVl97J9++MYE0VOzvX07XsBUVFpuN2pREWlqIyFIGstDQ2ru9yE+C5ebz1gSE4u6izTKSlfw+2OczquiIjIUQvEzPU7wEz8YyTLgArgXWvt9QHMeVRUrsPDwoVw8skwfz5ccMH+r2lq2sTmzbezZ8/TX3rF4HanEBWVSlRUWpfPX/y1v4jv/3XtThIYzc3bqK5+q3PUo7V1FwDx8cM6yvQppKVNJzo63eGkIiIigReIcr3UWjvBGPMD/KvWPzPGrLDWjgt02COlch0evF7IzYWpU+G55w5+bUPDahob19LeXkt7e02Xz5//2uv94nNw8H+fXa64bpXwrr/+/Lo03O7EXrmzSVtbDTU173SuTjc1rQMgOrpv58p0evopxMXlOZxUREQk+AJxQ2OUMSYbmA3cFrBk0uu43TBnDvz2t1BbC6kHOUsmMXH0Yd3caK0Pr7f+S0X88/L9eRHvWsiraW7e3PFcNda2Huon6CjdBy/hB1pdd7tTw2J/Zp+vhdraD7vMTS8GfLhciaSlnciAAT8iPf0bJCaO6ZX/syEiInIg3f2v/F3497f+t7V2sTGmEPgseLEkknk88Nhj8PzzcPHFgfu6xriIikohKirliL+G19t8gBJ+4LLe1LSxy3P7Dvk93O6kQ6yaH3zExeWKC3ihtdZHff3yzlGP2tpF+HxNgJuUlGMZNOj2jrnpYzVaIyIichA6/lx6nLUwZIj/4/XXnU4TWNZ6aW+v68aK+YFHXKxtP+j3MCb6iFbNv3xjaFPT5s6V6Zqat2lrqwQgIWFU56hHWtqJR/U/KyIiIpEoEMef5wD/CxyPf6j1feAaa215wFJKr2EMFBfDgw/Cnj3+kxsjhTFuoqPTj/hGPmstPl/jYc2Zt7fX0NKyo/M5n6/xUClxuxM7dvSAmJgB9Onzzc65aW2BKCIicuS6OxbyJ+DvwH/2d/hOx3OnBiOURD6PB+67D555Bq64wuk0ocMYf/F1uxOPuOT6fG20t9fut4R3/RwfP5j09FNJSBiuuWkREZEA6e5uIcusteMP9ZyTNBYSfsaO9d/Q+P77TicRERER6b6DjYV090SOSmPMd4wx7o6P7wBVgYsovZHHA//+N5SVOZ1EREREJDC6W67/C/82fLuAncD5wCXBCiW9Q3Gx//Pcuc7mEBEREQmUbpVra22ZtXamtTbLWtvXWnsOcF6Qs0mEKyyEY49VuRYREZHI0d2V6/0JmaPPJXx5PLB0Kaxb53QSERERkaN3NOVa2wvIUZs9G1wuePppp5OIiIiIHL2jKdeRcfqMOCo7G046yV+uI+Q8IxEREenFDlqujTH7jDF1+/nYB+ikCQkIjwfWr/ePh4iIiIiEs4OWa2ttsrU2ZT8fydba7h5AI3JQs2ZBdLRGQ0RERCT8Hc1YiEhApKfD6af7dw3x+ZxOIyIiInLkVK4lJBQXQ3m5/1AZERERkXClci0hYeZMiI/XaIiIiIiEt6CWa2PM6caYdcaYDcaYW/bz+o+NMSuNMcuMMe8bY0Z1ee3WjvetM8bMCGZOcV5Skr9gP/MMtLU5nUZERETkyAStXBtj3MDjwBnAKMDTtTx3+Lu1dqy1djzwEPBIx3tHAcXAaOB04DcdX08imMcDlZXw1ltOJxERERE5MsFcuZ4CbLDWbrLWtgJzgbO7XmCtrevyMJHP984+G5hrrW2x1m4GNnR8PYlgp58OaWkaDREREZHwFcxyPRDY1uVxecdzX2CMucIYsxH/yvXVh/NeiSyxsXDeefD889Dc7HQaERERkcMXzHK9v+PRv3IGn7X2cWvtYOBm4PbDea8x5lJjTIkxpqSiouKowkpo8Hhg3z549VWnk4iIiIgcvmCW63Igt8vjHGDHQa6fC5xzOO+11v7BWltkrS3Kyso6yrgSCqZPh379NBoiIiIi4SmY5XoxMNQYU2CMicF/g+KLXS8wxgzt8vBM4LOOX78IFBtjYo0xBcBQ4JMgZpUQ4XbD7Nnw8stQV3fo60VERERCSdDKtbW2HbgSeB34FJhvrV1tjLnLGDOz47IrjTGrjTHLgOuB73W8dzUwH1gDvAZcYa31BiurhBaPxz9z/cILTicREREROTzG2q+MMoeloqIiW1JS4nQMCQBrobAQRo7U7LWIiIiEHmNMqbW2aH+v6YRGCTnG+I9D/9e//Ptei4iIiIQLlWsJScXF0N4Of/2r00lEREREuk/lWkLSuHFwwglwww3wyCP+URERERGRUKdyLSHJGP+89bnnwn//N1xyiQ6Wkf/f3p1HSV1e+R9/XxBkVVTQKKsIagCNaAOKgmsmxAW3JGoWjZqY32Qbs4yacbJokkmiJtEkniRGnbgbR1wYE3FDE1fsBhUFXBA1IO6iBhQVeH5/PNVT3YBGoIpvV/X7dU6d7vpWNdyuk3g+Pt7vvZIktX2Ga7VZPXrAVVfBD34AF10Ee+0Fzz1XdFWSJEnvzXCtNq1DB/j+92HSJHjkEWhogMbGoquSJElaPcO1asJhh8E990DnzrkX+7LLiq5IkiRpVYZr1Ywdd8yn1rvuCp/9LJx8Mix3tZAkSWpDDNeqKb175/nXX/4ynHEGTJwIr79edFWSJEmZ4Vo1p1MnOPdc+N3v4OabYcwYePzxoquSJEkyXKuGfelLcNtt8MorMHo03HRT0RVJkqT2znCtmjZ+fO7DHjgQ9t/fhTOSJKlYhmvVvEGD8iSR5oUzn/+8C2ckSVIxDNeqC92754Uzp50GF1+cF84sXFh0VZIkqb0xXKtudOgA3/seXHNNXjgzapQLZyRJ0vpluFbdOfRQuPfe8sKZSy8tuiJJktReGK5Vl3bYIZ9a77YbfO5zcNJJLpyRJEnVZ7hW3erdO8/B/vKX4cwz4aCD4LXXiq5KkiTVM8O16lrzwpnf/z5vdtx1VxfOSJKk6jFcq1044YTWC2emTCm6IkmSVI8M12o3xo+HpqY8F/uAA+DnP3fhjCRJqizDtdqVgQPh7rvhsMPg29+GY45x4YwkSaocw7XaneaFM6efDpdcAnvu6cIZSZJUGYZrtUsR8N3v5oUzs2ZBQwPcf3/RVUmSpFpnuFa71rxwpkuX3JN9ySVFVyRJkmqZ4Vrt3g475FPrsWPh6KPh3//dhTOSJGntGK4l8sKZm26Cr3wFzjoLDjzQhTOSJGnNGa6lkk6d4De/yQtnbr0VxoyBxx4ruipJklRLDNfSSk44AaZOhUWL8sKZG28suiJJklQrDNfSaowbB42NMHhwXjhz5pkunJEkSf+c4Vp6DwMHwl13wSc+ASedlG92fOutoquSJEltmeFaeh/du8Of/gQ//CFcemleOPPss0VXJUmS2irDtfRPRMB//idcYiuLkgAAH+dJREFUey3MmQOjRsG0aUVXJUmS2iLDtfQBHXJIeeHMnnvCxRcXXZEkSWprDNfSGhgxIt/oOHYsHHMMfOtbsGxZ0VVJkqS2wnAtraHNNssLZ776VfjFL/LCmUWLiq5KkiS1BYZraS106gS//jWcd16eiT1mDDz6aNFVSZKkohmupXXwxS/mcP3aazlg/+UvRVckSZKKZLiW1tEee0BTU144c+CBcMYZLpyRJKm9MlxLFTBgQHnhzMknw+c+58IZSZLaI8O1VCHNC2d+9CO47DIYP96FM5IktTeGa6mCIuDUU+G66/INjg0NcN99RVclSZLWF8O1VAUHH5wXznTrlhfOXHRR0RVJkqT1wXAtVcmIEXD//fmGx89/3oUzkiS1B4ZrqYo22wymTIGvfS0vnDngABfOSJJUzwzXUpV16gS/+hX84Q9w++15HvacOUVXJUmSqsFwLa0nX/hCXjjz+uuw664unJEkqR4ZrqX1aI89oLERttkmL5z52c9cOCNJUj0xXEvrWfPCmU9+Ek45BT77WRfOSJJULwzXUgG6dYMrr4Qf/xguvzwvnFmwoOiqJEnSujJcSwWJgP/4D7j++rxwZtQoF85IklTrDNdSwSZOzKG6eeHMH/9YdEWSJGltVTVcR8SEiHgsIuZGxCmref2bETE7ImZGxG0RMbDFa8sj4sHSY3I165SKNnx4eeHMscfCN7/pwhlJkmpR1cJ1RHQEzgU+DgwDjoqIYSu97QGgIaW0I3A1cEaL195KKe1UekysVp1SW7HZZnDTTfD1r8Mvfwn77+/CGUmSak01T65HA3NTSvNSSu8AVwIHt3xDSun2lNKbpaf3Af2qWI/U5m2wAZxzDpx/PtxxB4we7cIZSZJqSTXDdV9gfovnC0rX3svxwI0tnneJiKaIuC8iDlndD0TECaX3NL300kvrXrHURhx/fN7m+MYbeaPjn/9cdEWSJOmDqGa4jtVcW+26jIj4LNAAnNni8oCUUgPwaeDsiNhmlT8spfNSSg0ppYY+ffpUomapzdh997xwZsgQOOggF85IklQLqhmuFwD9WzzvByxc+U0RsR9wKjAxpfR28/WU0sLS13nAHcDIKtYqtUnNC2c+9SkXzkiSVAuqGa4bgaERsXVEdAaOBFpN/YiIkcDvycH6xRbXN4mIDUvf9wZ2B2ZXsVapzerWDa64Ii+cueIKGDfOhTOSJLVVVQvXKaVlwFeBm4A5wFUppVkRcXpENE//OBPoAfzPSiP3Pgw0RcRDwO3AT1NKhmu1Wy0Xzjz2GDQ0wL33Fl2VJElaWaQ6aeJsaGhITU1NRZchVd2sWXDwwTB/Pvzud3kutiRJWn8iYnrp3sBVuKFRqjHNC2fGjYPjjoNvfMOFM5IktRWGa6kGbbopTJmSF86cfXZeOPPqq0VXJUmSDNdSjWpeOHPBBXnhzJgxLpyRJKlohmupxh13XOuFMzfcUHRFkiS1X4ZrqQ7svjs0NcHQoTBxIvz0py6ckSSpCIZrqU707w933glHHAHf+Q585jPw5ptFVyVJUvtiuJbqSLducPnl8JOfwJVXwvjxLpyRJGl9MlxLdSYir0qfPBkefzwvnLnnnqKrkiSpfTBcS3XqwAPhvvugZ0/Ye2+48MKiK5Ikqf4ZrqU6NmwYTJuW20OOPx5OPNGFM5IkVZPhWqpzm24KN96Yg/U558DHP+7CGUmSqsVwLbUDG2wAv/xlbg35299g9Gi4996iq5Ikqf4YrqV25Nhj8zbHJUtg7NjcLjJ5MqxYUXRlkiTVB8O11M7stlueIvLLX8Izz8DBB8Pw4XD++bB0adHVSZJU2wzXUjvUs2fuwZ47Fy67DLp0gS9+EQYNgv/6L1i0qOgKJUmqTYZrqR3r1Ak+/WmYMQNuvRV22glOPTVvezzxxHyyLUmSPjjDtSQiYN99YcoUeOghOOwwOPdc2GabHL4feKDoCiVJqg2Ga0mt7LgjXHwxzJuXT69vuAF23hn22w9uuglSKrpCSZLaLsO1pNXq3x/OOgv+/nf42c9gzhyYMCG3jlxyCbz7btEVSpLU9hiuJb2vXr3gpJPgqafgv/8bli+Ho4+GwYPh5z+HN94oukJJktoOw7WkD6RzZ/j85+Hhh+HPf4YhQ+Db384n3CefDAsXFl2hJEnFM1xLWiMRsP/+cPvt0NiYW0XOOiuP8Tv2WJg1q+gKJUkqjuFa0lpraIA//QmeeAK+9CW46ioYMaIcvr35UZLU3hiuJa2zwYPh17/ONz+efjo0NcE++8Do0TlwL1tWdIWSJK0fhmtJFbPZZvDd7+blM7/7Hbz+OhxxBGy7bQ7fS5YUXaEkSdVluJZUcV275jaRRx+Fa6+FLbeEr38dBgzI4fvFF4uuUJKk6jBcS6qaDh3gkEPg7rvhrrtg3Dj48Y9zyP7Sl+Dxx4uuUJKkyjJcS1ovdt8drrsuL6M55hi46CLYfns49FC4556iq5MkqTIM15LWq+22g9//Pvdln3oq/PWvOXg3h+8VK4quUJKktWe4llSILbaAH/4Q5s+HX/0qL6E59FD48IfhvPNg6dKiK5Qkac0ZriUVqnt3+NrX8qzsK6+Enj1zP/bAgfCjH8ErrxRdoSRJH5zhWlKbsMEGeWxfYyNMnQq77JIniwwYkCeNPPVU0RVKkvTPGa4ltSkRsPfe8Je/wMMPwyc/mWdmDxkCRx6ZF9RIktRWGa4ltVkjRsAf/5hPrb/1LbjxRhg1qhy+Xa8uSWprDNeS2ry+feGMM/J69TPPzP3ZBxwAO+yQw/c77xRdoSRJmeFaUs3YeGP49rdh3rw8J7tDBzj2WNh66xy+X3+96AolSe2d4VpSzencGY4+Gh56KLeKbL89nHwy9O+fw/eCBUVXKElqrwzXkmpWBEyYALfdBtOnw4EHwtln55Pso4+GmTOLrlCS1N4YriXVhZ13hssvh7lz4StfgWuugY98pBy+vflRkrQ+GK4l1ZVBg/Lp9d//npfQPPgg7Ldfnpt9xRWwbFnRFUqS6pnhWlJd2nRTOPVUePpp+MMf4M034dOfzvOyzzkHFi8uukJJUj0yXEuqa126wBe+ALNnw/XX55seTzwxb3489VR4/vmiK5Qk1RPDtaR2oUMHmDgR7rwT7r03L6L5yU9g4ED44hfh0UeLrlCSVA8M15LanV13hUmT4LHH4Ljj4NJL4cMfhoMPhrvu8uZHSdLaM1xLareGDoXf/haeeQa+9z24+24YNw7Gjs3he/nyoiuUJNUaw7Wkdm/zzeG00/KEkd/8Bl58ET7xibyc5re/hbfeKrpCSVKtMFxLUkm3bnlG9uOPw1VXwSabwJe/nG9+PO00ePnloiuUJLV1hmtJWknHjvDJT8K0aXDHHTBmDPzgBzlkf+Ur8OSTRVcoSWqrDNeS9B4iYM894YYbYNYsOPLIPDN7221z+L7//qIrlCS1NYZrSfoAhg2DCy/MS2lOOgluuSWfaDeH7xUriq5QktQWGK4laQ1stVWejz1/PvziF/DUU3DQQTBiRA7fb79ddIWSpCIZriVpLfTsCd/4Ru6/vvRS6NwZjj8eBg2CU06BxkbnZUtSe1TVcB0REyLisYiYGxGnrOb1b0bE7IiYGRG3RcTAFq8dExFPlB7HVLNOSVpbnTrBZz4DDzwAN98MI0fCz38Oo0fn7Y8nnpi3QjozW5Lah0hVOlqJiI7A48BHgQVAI3BUSml2i/fsDUxLKb0ZEf8K7JVSOiIiNgWagAYgAdOBXVJKi97r72toaEhNTU1V+V0kaU28+ir87//mRTQ335xbRbbYAg49FA4/PPdpd+pUdJWSpLUVEdNTSg2re62aJ9ejgbkppXkppXeAK4GDW74hpXR7SunN0tP7gH6l7z8G3JJSerUUqG8BJlSxVkmqmE03hWOOgcmT4aWX4IorYPx4uPhi+OhH4UMfgmOPzTdC2qMtSfWlmuG6LzC/xfMFpWvv5XjgxjX52Yg4ISKaIqLppZdeWsdyJanyevbMI/yuuiovobn2Wth///z1oIOgTx846ii4+mpYsqToaiVJ66qa4TpWc221PSgR8VlyC8iZa/KzKaXzUkoNKaWGPn36rHWhkrQ+dO0KhxwCl1ySV6zfeCMccQTcemuem92nDxx2WL5B8vXXi65WkrQ2qhmuFwD9WzzvByxc+U0RsR9wKjAxpfT2mvysJNWqzp1hwoS8lOa552Dq1DxtZNo0+NznctDef3+44ILcWiJJqg3VvKFxA/INjfsCz5JvaPx0SmlWi/eMBK4GJqSUnmhxfVPyTYw7ly7NIN/Q+Op7/X3e0CipHqxYkQP2pEn58fTT0KFDvgny8MPzTZFbbVV0lZLUvhVyQ2NKaRnwVeAmYA5wVUppVkScHhETS287E+gB/E9EPBgRk0s/+yrwQ3IgbwROf79gLUn1okMH2G03OOssmDcPZsyA73wnn25/9avQty+MHZvH/T39dNHVSpJWVrWT6/XNk2tJ9W7OnPKJ9oMP5ms775xPtA87DLbfvtj6JKm9eL+Ta8O1JNWgJ5+Ea67Jj/vuy9eGDctB+/DDYccdIVZ3a7gkaZ0ZriWpji1YkEf7TZqUt0GuWAHbbJNPsw8/HEaNyu0mkqTKMFxLUjvx4otw/fU5aN92GyxbBv36lbdD7rEHdOxYdJWSVNsM15LUDi1alLdATpoEN90ES5fC5pvnWduHHQb77OMadklaG4ZrSWrnFi+Gv/wlB+0//zlvg+zVCyZOzCfa//Iv0KVL0VVKUm0wXEuS/s9bb8Ett+SgPXkyvPYa9OiRl9Ycfnj+2qNH0VVKUttluJYkrdY778Add+Sgfd11uWe7Sxf42Mdy68hBB8EmmxRdpSS1LYZrSdI/tXw53HVXecTfggWwwQaw7775RPvgg3PPtiS1d4ZrSdIaWbECGhvLS2vmzcvj/MaNK69h79ev6ColqRiGa0nSWksJZs4sB+3Zs/P1MWPKS2sGDy62RklanwzXkqSKefTR3DYyaRLMmJGv7bRTeWnNsGHF1idJ1Wa4liRVxVNPlXu077knX9t++/KJ9k47uYZdUv0xXEuSqm7hwvIa9r/+Nfdtb711+UR7zBjXsEuqD4ZrSdJ69fLL5TXst94K774LW21VXsM+blyeRCJJtchwLUkqzGuv5a2QkybBlCl5iU3v3uU17PvuC507F12lJH1whmtJUpuwZAnceGPu0b7hBvjHP2DjjfOymsMPz8trunYtukpJen+Ga0lSm7N0aW4ZmTQpt5AsWgTdupXXsB9wAPTsWXSVkrQqw7UkqU179918E+SkSfmmyBdegA03hI9+NAftiRNh002LrlKSMsO1JKlmLF+ex/o1z9KePz/f/Lj33vDxj+ce7REjnDwiqTiGa0lSTUoJmprKJ9qPP56v9+6dw/Y+++TH0KHO05a0/hiuJUl1Yf58mDo1P267DZ59Nl/v27cctPfZBwYMKLZOSfXNcC1Jqjspwdy55bA9dWqerw0wZEg5aO+9N2y+ebG1SqovhmtJUt1bsQJmzcon2lOn5hsk33gjvzZiRDls77kn9OpVbK2SapvhWpLU7ixbBjNmlE+177orL7Dp0AF22aUctnffHbp3L7paSbXEcC1JavfefhumTSv3a993Xw7gnTrBrruWw/aYMXkMoCS9F8O1JEkrWbIkn2Y3n2xPn577uLt2hT32yCP/9tkHdt4ZOnYsulpJbYnhWpKkf2LRIvjb38ph+5FH8vWNN8592s0n28OHO2Nbau8M15IkraEXXoDbby+H7SefzNf79Gk99m+bbZyxLbU3hmtJktbRM8+Uw/Ztt8HChfl6//6tw3a/fsXWKan6DNeSJFVQSnlbZPOp9u23wyuv5NeGDm09Y7tPn2JrlVR5hmtJkqpoxQp4+OFy2P7rX+Ef/8iv7bhjOWyPH597uCXVNsO1JEnr0bJlefpIcwvJ3XfD0qX5RsiGhtYztrt1K7paSWvKcC1JUoGWLs1ztZtPtqdNywG8c2fYbbdy2B49Ol+T1LYZriVJakMWL84ztptXtT/wQO7j7tYNxo0rh+2RI52xLbVFhmtJktqwV1/NfdrNJ9uzZ+frvXrBXnuVw/awYY79k9oCw7UkSTXk+edbz9ieNy9f33zz1mP/Bg82bEtFMFxLklTDnn66HLSnToXnnsvXBwwor2nfe2/o27fQMqV2w3AtSVKdSAkee6z1jO1XX82vbbdd+VR7r72gd+9CS5XqluFakqQ6tWIFPPRQOWz/7W/5hkmAnXYqh+1x42CjjYqtVaoXhmtJktqJd9+FpqZy2L77bnj77Tx1ZNSoctgeOxa6di26Wqk2Ga4lSWqnli6Fe+8tj/27/35YvjzP0x47tvXYPxfaSB+M4VqSJAF5Lfudd5ZPth98MPdxR8CQITBiRH7ssEP+OnQobLBB0VVLbcv7hWv/7yJJUjvSsyfsv39+ALzySu7TnjkTHn4YHnkErr8+93JDPuHefvty2G4O3gMGOAZQWh1PriVJUitLl8KcOTloNz8efhjmzy+/p2dPGD689Sn3iBF5FrdU72wLkSRJ6+z112HWrPIJd3PofuWV8ns237wctJuD9/DhOYxL9cK2EEmStM423jjfBDl2bPlaSvDCC63D9iOPwAUXwJIl5fcNHLhqa8l228GGG67/30OqJsO1JElaaxHwoQ/lx377la+vWJE3S67cWjJlCixblt/TsSNsu+2qrSWDB+fXpFpkuJYkSRXXoUMOyYMHw8SJ5evvvANPPNG6tWTGDLj66nwKDnn+9rBhq04u2Worb6JU22e4liRJ603nzrkHe/jw1teXLIHZs1ufct98M1x0Ufk9m2zSup+7+bHppuv3d5Dej+FakiQVrnv3vEFy1KjW1195ZdXWkssvzzdXNttqq1VbS4YNcymOimG4liRJbdZmm8Gee+ZHs5Tg2Wdb30D5yCNw7rl5jCDk9pFttln9UpxOnYr5XdQ+GK4lSVJNiYB+/fJjwoTy9eXL4cknV51cMnnyqktxVh4XOGBA7hOX1lVV51xHxATgHKAjcH5K6acrvT4eOBvYETgypXR1i9eWAw+Xnv49pTSR9+Gca0mStDpLl8Kjj67aXvL3v5ff06NH7gNfeVygS3G0OoUskYmIjsDjwEeBBUAjcFRKaXaL9wwCNgK+DUxeKVwvTin1+KB/n+FakiStiddfzzdRrrwU5+WXy+/p02fV1pLhw2GjjYqrW8UraonMaGBuSmleqYgrgYOB/wvXKaWnS6+tqGIdkiRJq9h4Y9htt/xolhK8+OKqrSUXXrjqUpyVW0u2396lOKpuuO4LzG/xfAEwZg1+vktENAHLgJ+mlK5b+Q0RcQJwAsCAAQPWoVRJkqTcz73FFvmx777l6ytWwDPPrNpacvPN8O67+T0dO+YbJlu2lowYAUOG2M/dnlQzXK9uzPua9KAMSCktjIjBwNSIeDil9GSrPyyl84DzILeFrH2pkiRJ761DB9h66/w46KDy9XffzUtxWp5yP/BA66U4PXvCLruURw2OGpVPvl2IU5+qGa4XAP1bPO8HLPygP5xSWlj6Oi8i7gBGAk++7w9JkiStR5065Znaw4bBpz5Vvr5kCcyZAzNnwvTp0NgI55yTN1QC9O4NDQ3lsN3QAFtuWczvoMqqZrhuBIZGxNbAs8CRwKc/yA9GxCbAmymltyOiN7A7cEbVKpUkSaqg7t1zYG5ogOOOy9feeSefbjc2lh8331weE9i3b+uw3dDg9slaVO1RfPuTR+11BC5MKf04Ik4HmlJKkyNiFHAtsAmwFHg+pTQ8IsYCvwdWAB2As1NKF7zf3+W0EEmSVGvefDO3kbQM3E88UX59m23KYXvUKNh55zw2UMUqZBTf+ma4liRJ9eC118qtJE1N+WvzTO4OHeDDH27dUvKRjzilZH0zXEuSJNWwF17IQbs5bDc25pGBkPu+d9ih9Q2Tw4bBBu7hrhrDtSRJUh1JCebPb3263dSUF+MAdO0KI0e27uEeOtSRgJViuJYkSapzK1bA3LmtT7dnzIC33sqvb7xxeSRgc1vJgAGOBFwbhmtJkqR2aNmyPBKw5Q2TM2eWF9/06dP6dHvUqLxAR+/PcC1JkiQA3n47B+zmsN3UBLNnl0cC9u/fOmw3NECvXsXW3Na8X7i21V2SJKkd2XDD8ml1s8WLyyMBm9tKrrmm/PqQIa1vmBw5Ms/y1qoM15IkSe1cjx4wblx+NFu0qPWEkjvvhCuuyK916JAnkrQM3Dvs4EhAsC1EkiRJH9Dzz7c+3W5shJdfzq917gw77ti6h3vYMOjYsdiaq8Gea0mSJFVcSvDMM63D9vTp8MYb+fVu3fJWyZY93EOG1P6EEnuuJUmSVHERMGhQfnziE/naihV5hXvLCSW//S0sXZpf79UrB+2WWyb79av9wN3Mk2tJkiRV1bJlMGtW6wklM2fm65DH/7UM26NG5TGBbZVtIZIkSWpTli6Fhx5q3cM9Z05uNYG84KZl2N5ll7wIpy2wLUSSJEltSpcuMGZMfjT7xz/yVsmWgXvSpPLr227b+obJkSNzX3db4sm1JEmS2qxXXmk9ErCxERYuzK/94Q/whS+s/5o8uZYkSVJN2mwz+NjH8qPZwoU5bO+yS3F1vRfDtSRJkmrKVlvBxIlFV7F6HYouQJIkSaoXhmtJkiSpQgzXkiRJUoUYriVJkqQKMVxLkiRJFWK4liRJkirEcC1JkiRViOFakiRJqhDDtSRJklQhhmtJkiSpQgzXkiRJUoUYriVJkqQKMVxLkiRJFWK4liRJkirEcC1JkiRViOFakiRJqhDDtSRJklQhhmtJkiSpQgzXkiRJUoUYriVJkqQKiZRS0TVURES8BDxT0F/fG3i5oL+73vnZVo+fbfX42VaPn231+NlWj59t9RT12Q5MKfVZ3Qt1E66LFBFNKaWGouuoR3621eNnWz1+ttXjZ1s9frbV42dbPW3xs7UtRJIkSaoQw7UkSZJUIYbryjiv6ALqmJ9t9fjZVo+fbfX42VaPn231+NlWT5v7bO25liRJkirEk2tJkiSpQgzX6yAiJkTEYxExNyJOKbqeehIRF0bEixHxSNG11JOI6B8Rt0fEnIiYFRH/VnRN9SIiukTE/RHxUOmzPa3omupNRHSMiAci4oaia6knEfF0RDwcEQ9GRFPR9dSTiOgVEVdHxKOlf+7uVnRN9SAitiv977X58UZEnFh0Xc1sC1lLEdEReBz4KLAAaASOSinNLrSwOhER44HFwMUppRFF11MvImJLYMuU0oyI6AlMBw7xf7frLiIC6J5SWhwRnYC7gH9LKd1XcGl1IyK+CTQAG6WUDiy6nnoREU8DDSkl5zBXWERcBNyZUjo/IjoD3VJKrxVdVz0p5bFngTEppaL2nbTiyfXaGw3MTSnNSym9A1wJHFxwTXUjpfQ34NWi66g3KaXnUkozSt//A5gD9C22qvqQssWlp51KD08vKiQi+gEHAOcXXYv0QUTERsB44AKAlNI7Buuq2Bd4sq0EazBcr4u+wPwWzxdgSFENiYhBwEhgWrGV1I9S28KDwIvALSklP9vKORs4CVhRdCF1KAE3R8T0iDih6GLqyGDgJeC/S+1M50dE96KLqkNHAlcUXURLhuu1F6u55imVakJE9AAmASemlN4oup56kVJanlLaCegHjI4IW5oqICIOBF5MKU0vupY6tXtKaWfg48BXSm15WncbADsDv00pjQSWAN6fVUGlVpuJwP8UXUtLhuu1twDo3+J5P2BhQbVIH1ipH3gScFlK6Zqi66lHpf/0ewcwoeBS6sXuwMRSb/CVwD4RcWmxJdWPlNLC0tcXgWvJbY9adwuABS3+C9bV5LCtyvk4MCOl9ELRhbRkuF57jcDQiNi69G9ORwKTC65Jel+lm+4uAOaklH5RdD31JCL6RESv0vddgf2AR4utqj6klL6TUuqXUhpE/mft1JTSZwsuqy5ERPfSzc2UWhb+BXBKUwWklJ4H5kfEdqVL+wLePF5ZR9HGWkIg/ycLrYWU0rKI+CpwE9ARuDClNKvgsupGRFwB7AX0jogFwPdTShcUW1Vd2B34HPBwqTcY4D9SSn8psKZ6sSVwUenO9Q7AVSklR8aprdsCuDb/ezcbAJenlKYUW1Jd+RpwWekQbh5wbMH11I2I6Eae2PalomtZmaP4JEmSpAqxLUSSJEmqEMO1JEmSVCGGa0mSJKlCDNeSJElShRiuJUmSpAoxXEtSjYqI5RHxYItHxba/RcSgiHDesSStIedcS1Ltequ0bl2S1EZ4ci1JdSYino6In0XE/aXHkNL1gRFxW0TMLH0dULq+RURcGxEPlR5jS39Ux4j4Q0TMioibS5sniYivR8Ts0p9zZUG/piS1SYZrSapdXVdqCzmixWtvpJRGA78Bzi5d+w1wcUppR+Ay4Fel678C/ppS+giwM9C8bXYocG5KaTjwGnB46fopwMjSn/P/qvXLSVItckOjJNWoiFicUuqxmutPA/uklOZFRCfg+ZTSZhHxMrBlSund0vXnUkq9I+IloF9K6e0Wf8Yg4JaU0tDS85OBTimlH0XEFGAxcB1wXUppcZV/VUmqGZ5cS1J9Su/x/Xu9Z3XebvH9csr36RwAnAvsAkyPCO/fkaQSw7Uk1acjWny9t/T9PcCRpe8/A9xV+v424F8BIqJjRGz0Xn9oRHQA+qeUbgdOAnoBq5yeS1J75WmDJNWurhHxYIvnU1JKzeP4NoyIaeRDlKNK174OXBgR/w68BBxbuv5vwHkRcTz5hPpfgefe4+/sCFwaERsDAfwypfRaxX4jSapx9lxLUp0p9Vw3pJReLroWSWpvbAuRJEmSKsSTa0mSJKlCPLmWJEmSKsRwLUmSJFWI4VqSJEmqEMO1JEmSVCGGa0mSJKlCDNeSJElShfx/Q8q8M5TYneoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtAAAAIaCAYAAAAJJHpOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deZhcZZn38e/dW5YO2UNICJBogiCQCLSCbLKMqDNIxA0QUDIigywKuCCOSkZxGfcFxMmgyGpEEIdhfFEQEEVQE0VZAsIADoFAQjbI2tvz/lHVnU4vSR3SlVNd/f1cV119zqlTp+4uQufXT+7zPJFSQpIkSVJpavIuQJIkSRpIDNCSJElSBgZoSZIkKQMDtCRJkpSBAVqSJEnKwAAtSZIkZVC2AB0RP4iIpRHxYB/PR0R8OyIej4i/RsR+5apFkiRJ6i/lHIH+IfDmLTz/FmBG8XE6cFkZa5EkSZL6RdkCdErpbmDFFk6ZDVyVCu4DRkfEpHLVI0mSJPWHPHugdwae7rK/uHhMkiRJqlh1Ob539HKs13XFI+J0Cm0eNDY27r/HHnuUsy5JkiSJhQsXvpBSmtD9eJ4BejGwS5f9KcCzvZ2YUpoHzANoampKCxYsKH91kiRJGtQi4u+9Hc+zheNm4L3F2TgOBFanlJbkWI8kSZK0VWUbgY6IHwGHA+MjYjFwEVAPkFL6HvBz4B+Bx4F1wJxy1SJJkiT1l7IF6JTSiVt5PgFnlev9JUmSpHLIswdakiRpUGppaWHx4sVs2LAh71IEDB06lClTplBfX1/S+QZoSZKk7Wzx4sXssMMOTJ06lYjeJibT9pJSYvny5SxevJhp06aV9Jo8byKUJEkalDZs2MC4ceMMzxUgIhg3blymfw0wQEuSJOXA8Fw5sv63MEBLkiRJGRigJUmSVBatra15l1AWBmhJkqRB6G1vexv7778/e+21F/PmzQPg1ltvZb/99mPWrFkcddRRAKxZs4Y5c+awzz77MHPmTG688UYARowY0XmtG264gVNPPRWAU089lfPPP58jjjiCCy64gD/84Q8cdNBB7Lvvvhx00EE8+uijALS1tfHRj36087rf+c53+NWvfsVxxx3Xed3bbruNt7/97dvj48jEWTgkSZJydO65cP/9/XvN17wGvvnNLZ/zgx/8gLFjx7J+/Xpe+9rXMnv2bD7wgQ9w9913M23aNFasWAHA5z73OUaNGsUDDzwAwMqVK7f6/n/729+4/fbbqa2t5cUXX+Tuu++mrq6O22+/nU9+8pPceOONzJs3jyeffJI///nP1NXVsWLFCsaMGcNZZ53FsmXLmDBhAldccQVz5lTeWnsGaEmSpEHo29/+NjfddBMATz/9NPPmzeOwww7rnMpt7NixANx+++3Mnz+/83VjxozZ6rXf9a53UVtbC8Dq1at53/vex2OPPUZE0NLS0nndM844g7q6us3e75RTTuGaa65hzpw53HvvvVx11VX99B33HwO0JElSjrY2UlwOd911F7fffjv33nsvw4cP5/DDD2fWrFmd7RVdpZR6naWi67HuU8A1NjZ2bn/605/miCOO4KabbuKpp57i8MMP3+J158yZw1vf+laGDh3Ku971rs6AXUnsgZYkSRpkVq9ezZgxYxg+fDiPPPII9913Hxs3buTXv/41Tz75JEBnC8fRRx/NJZdc0vnajhaOiRMnsmjRItrb2ztHsvt6r5133hmAH/7wh53Hjz76aL73ve913mjY8X6TJ09m8uTJXHzxxZ191ZXGAC1JkjTIvPnNb6a1tZWZM2fy6U9/mgMPPJAJEyYwb9483v72tzNr1iyOP/54AD71qU+xcuVK9t57b2bNmsWdd94JwJe+9CWOOeYYjjzySCZNmtTne3384x/nwgsv5OCDD6atra3z+Gmnncauu+7KzJkzmTVrFtddd13ncyeddBK77LILr371q8v0CWybSCnlXUMmTU1NacGCBXmXIUmS9LItWrSIPffcM+8yKtbZZ5/Nvvvuy/vf//7t9p69/TeJiIUppabu51ZeU4kkSZIGrf3335/Gxka+9rWv5V1KnwzQkiRJqhgLFy7Mu4StsgdakiRJysAALUmSJGVggJYkSZIyMEBLkiRJGRigJUmSpAwM0JIkSdqiESNG5F1CRTFAS5IkaUDoWPY7b84DLUmSlKPHHjuXNWvu79drjhjxGmbM+Gafz19wwQXstttunHnmmQDMnTuXiODuu+9m5cqVtLS0cPHFFzN79uytvteaNWuYPXt2r6+76qqr+OpXv0pEMHPmTK6++mqef/55zjjjDJ544gkALrvsMiZPnswxxxzDgw8+CMBXv/pV1qxZw9y5czn88MM56KCDuOeeezj22GPZfffdufjii2lubmbcuHFce+21TJw4kTVr1nDOOeewYMECIoKLLrqIVatW8eCDD/KNb3wDgP/8z/9k0aJFfP3rX9+mz9cALUmSNMiccMIJnHvuuZ0B+vrrr+fWW2/lvPPOY+TIkbzwwgsceOCBHHvssUTEFq81dOhQbrrpph6ve/jhh/n85z/PPffcw/jx41mxYgUAH/rQh3jDG97ATTfdRFtbG2vWrGHlypVbfI9Vq1bx61//GoCVK1dy3333ERFcfvnlfPnLX+ZrX/san/vc5xg1ahQPPPBA53kNDQ3MnDmTL3/5y9TX13PFFVfwH//xH9v68RmgJUmS8rSlkeJy2XfffVm6dCnPPvssy5YtY8yYMUyaNInzzjuPu+++m5qaGp555hmef/55dtpppy1eK6XEJz/5yR6vu+OOO3jnO9/J+PHjARg7diwAd9xxB1dddRUAtbW1jBo1aqsB+vjjj+/cXrx4MccffzxLliyhubmZadOmAXD77bczf/78zvPGjBkDwJFHHsktt9zCnnvuSUtLC/vss0/GT6snA7QkSdIg9M53vpMbbriB5557jhNOOIFrr72WZcuWsXDhQurr65k6dSobNmzY6nX6el1Kaauj1x3q6upob2/v3O/+vo2NjZ3b55xzDueffz7HHnssd911F3PnzgXo8/1OO+00vvCFL7DHHnswZ86ckurZGm8ilCRJGoROOOEE5s+fzw033MA73/lOVq9ezY477kh9fT133nknf//730u6Tl+vO+qoo7j++utZvnw5QGcLx1FHHcVll10GQFtbGy+++CITJ05k6dKlLF++nI0bN3LLLbds8f123nlnAK688srO40cffTSXXHJJ537HqPYBBxzA008/zXXXXceJJ55Y6sezRQZoSZKkQWivvfbipZdeYuedd2bSpEmcdNJJLFiwgKamJq699lr22GOPkq7T1+v22msv/vVf/5U3vOENzJo1i/PPPx+Ab33rW9x5553ss88+7L///jz00EPU19fzmc98hgMOOIBjjjlmi+89d+5c3vWud3HooYd2tocAfOpTn2LlypXsvffezJo1izvvvLPzuXe/+90cfPDBnW0d2ypSSv1yoe2lqakpLViwIO8yJEmSXrZFixax55575l3GoHHMMcdw3nnncdRRR/V5Tm//TSJiYUqpqfu5jkBLkiSpKq1atYrdd9+dYcOGbTE8Z+VNhJIkSdqqBx54gFNOOWWzY0OGDOH3v/99ThVt3ejRo/nb3/7W79c1QEuSJOUgyywVlWCfffbh/vv7d8GXSpG1pdkWDkmSpO1s6NChLF++PHNwU/9LKbF8+XKGDh1a8mscgZYkSdrOpkyZwuLFi1m2bFnepYjCLzRTpkwp+XwDtCRJ0nZWX1/fuYKeBh5bOCRJkqQMDNCSJElSBgZoSZIkKQMDtCRJkpSBNxFKkiSp37W2wpo1sHZt4WvHI+t+Swv85S95fzebM0BLkiQNYm1tsG7dyw+5fT23cWPpNdTWwg47QGMjjBix6TFxYuFrezvUVFDfhAFakiRpAEhpU9Dd1lHdrvvr15deQ8TmAbcj8I4dC7vu2jMAl7rf0FC49kBhgJYkSepHKcGGDf0bctesKYTnLAsXNjb2DKw77ACTJmUPuB2PoUMHVtAtFwO0JEkatFIqjMB2hNSXXto8tHbf39I5XQNve3vpNQwd2ntYHT8+W7jtemz48Mpqeag2BmhJkjQgdIzsZgm1pZxT6qhuTc2mUdyuoXXy5J5BttTR3cZGqDONDTj+J5MkSf2uI+z2V8jNOrLbEXa7ti6MGLF5+0L3INzbsa77ti+ogwFakqRBLqXCjAkvN9T2daytrbT377gxrXt4nTgRpk/fcsDt69iwYYZdlY8BWpKkKrJ2LSxZAs89V3h03V62rO8wnDXsdg+wEyfCK1/58kZ2DbsaaAzQkiRVuLa2QvjtKxh33V+zpufra2sLAXfCBBg5svD1Fa/IFnI7HsOHG3YlA7QkSTlIqRB2txSGO7aXLeu993fUKNhpp8KjqWnT9qRJm2+PG+eMDFJ/MkBLktSPWlth6dK+w3DX7XXrer6+rm5T+J0yBV772t6D8U47FVofJG1/BmhJkrYipUKv8JbCcNc+496mRRs9elMAft3reh8p3mmnwopujhZLlc0ALUkatFpaCqPFpfQW97bccX39pgA8dSoceGDvwXjixMIUaJKqgwFaklRVUoLVq0vrLX7hhd6vMXbspgB80EF9t1CMHesNddJgZICWJA0Izc3w/PNbb6F47rnCAh7dNTRsCsCveAUcfPDmYbjjuYkTYciQ7f/9SRo4DNCSpFy0t8OqVbB8eWEkePnyTdu9BePly3u/zrhxmwLwIYf03Vs8erSjxZL6hwFakrTNNm7cFID7enQNycuXw8qVfS/LPGTIpuC7++5w2GG9t1BMnFgYWZak7ckALUnq1DHbRCkBuOujt8U7OgwbVhglHjcOxo+HXXbZtN/9MX584euoUY4WS6pcBmhJqlKtrYVR3lJDcMejpaXva44evSnk7rQT7LVX7wG468O5iiVVGwO0JA0A69eXFoK7PrdqVd/Xq6/fPOTuvvuWQ/C4cTBmTGGRD0ka7PxRKEnbUXt7YYq1LL3Cy5f3PgdxhxEjNg+6r3xl3y0SHQF5xAhbJCTp5TJAS9LL1NKSLQQvXw4rVkBbW+/Xq6kpjPJ2jADvuivsu++We4XHjnXKNUna3gzQktSHlArTpz3+ODz2WOHRsf3kk/Dii32/dsiQzVsh9t57673Co0e7hLMkDQQGaEmDWkqFxTm6B+SO7bVrN51bV1dYgGP6dDj0UNhxx77bJIYPt0VCkqqVAVpS1esakrsH5Mcf33wKtro6mDYNZsyAww8vhOUZMwqPXXf1JjpJkgFaUpXoCMkdAbl7UO4tJE+fXligY8aMTUF5t90MyZKkLfOvCUkDRkqwdGnfI8kvvbTp3NraTSPJhx3WcyS5vj6/70OSNLAZoCVVlI6Q3NdIcl8h+ZBDeo4kG5IlSeVggJa03aUEy5b1HpAfe6z3kDx9eiEkdx1JNiRLkvJggJZUFh0hubeA/Pjjm08BV1sLU6cWQvFBB20KyNOnF44bkiVJlcQALellS6mwYEhfI8m9heTp0wshuetIsiFZkjSQGKAlbVFHSO5rJHn16k3n1tRsGkk+8MCeI8kNDXl9F5Ik9R8DtCRSKiwz3VtAfuyx3kPy9OmFkNx9JNmQLEmqdgZoaZDoCMm9jST3FpJ3260Qik86afOR5GnTDMmSpMHNAC1VoVWr4He/gz/8Af72t01BedWqTed0hOTp0zeF5I7R5KlTYciQ3MqXJKmiGaClKvD00/Db3256PPBAYcS5pqawaMiMGfCe92wKyB0jyYZkSZKyM0BLA0x7Ozz88OaB+e9/Lzw3YgS8/vXwjnfAoYfC614HjY351itJUrUxQEsVbuNGWLgQfvObQli+5x5YubLw3E47FRYXOf/8wteZM6HO/6slSSor/6qVKkxH/3LH6PIf/lAI0QCvehW8/e2F0eVDDoFXvAIi8q1XkqTBxgAt5Wzx4kJQ7hhh7uhfrquD/faDs84qhOWDD4Ydd8y7WkmSZICWtqP2dli0aPPA3Fv/8iGHwAEH2L8sSVIlMkBLZdTRv9wRmLv2L0+cWGjFOO+8QmCeNcv+ZUmSBgL/upb60apVcO+9mwJzb/3LhxxSeLzylfYvS5I0EBmgpW3Q0b/cEZjtX5YkqfoZoKUSde1f7gjMHf3LjY1w0EH2L0uSNBgYoKU+dO1f7ph/ecWKwnP2L0uSNHj5V75UtHp1z/mXN2woPLf77nDccfYvS5IkA7QGsWee2TSV3G9/C3/9a6F/uba20L985pn2L0uSpJ4M0BoU2tvhkUc2D8xPPVV4rrGxMP/y3Ln2L0uSpK0ra4COiDcD3wJqgctTSl/q9vxuwA+ACcAK4OSU0uJy1qTBobm50L/cEZi79y8fcgice679y5IkKbuyxYaIqAUuBd4ILAb+GBE3p5Qe7nLaV4GrUkpXRsSRwBeBU8pVk6rX6tWF+Zc7AnP3/uW3va1w05/9y5IkaVuVc9ztdcDjKaUnACJiPjAb6BqgXw2cV9y+E/hZGetRFXnmmc2Xw+7ev/zBDxYCs/3LkiSpv5UzQO8MPN1lfzFwQLdz/gK8g0Kbx3HADhExLqW0vOtJEXE6cDrArrvuWraCVZk6+pe7zr/cvX/5oosKgdn+ZUmSVG7lDNC9/SN56rb/UeCSiDgVuBt4Bmjt8aKU5gHzAJqamrpfQ1Wmo3+5Iyz31r/84Q8XArP9y5IkaXsrZ/RYDOzSZX8K8GzXE1JKzwJvB4iIEcA7Ukqry1iTKlBH/3JHYO6tf/mQQwqB2f5lSZKUt3IG6D8CMyJiGoWR5ROA93Q9ISLGAytSSu3AhRRm5NAgcuutMHt2YdS5a/9yx/zLEyfmXaEkSdLmyhagU0qtEXE28AsK09j9IKX0UER8FliQUroZOBz4YkQkCi0cZ5WrHlWepUvhfe8rjDJ/85uF/uURI/KuSpIkacvK2j2aUvo58PNuxz7TZfsG4IZy1qDKlBK8//2F9o3bb4d99sm7IkmSpNJ4+5Vy8R//AbfcUhh5NjxLkqSBpCbvAjT4PPIInH8+HH00nHNO3tVIkiRlY4DWdtXcDCedBMOHww9/CDX+CZQkSQOMLRzarj7zGfjTn+Cmm2DSpLyrkSRJys7xP203d90FX/4ynHZaYW5nSZKkgcgAre1i5Up473sLC6F84xt5VyNJkvTy2cKhsksJzjwTliyB3/3OuZ4lSdLAZoBW2V17LcyfDxdfDK99bd7VSJIkbRtbOFRWTz5ZGH0+5BD4xCfyrkaSJGnbGaBVNq2tcMopEAFXXw21tXlXJEmStO1s4VDZfOlLcM89hfA8dWre1UiSJPUPR6BVFn/4A8ydCyecUFg4RZIkqVoYoNXv1qwphObJk+GyywotHNr+UkqklPIuQ5KkqmMLh/rdeefB//4v3HknjB6ddzV9S6mNlFppb28hpVZS6v3rQH6+oJaamiHU1AwhouFlb2/r67tvR9QR/nYlSRqADNDqVzfdBJdfXphx4w1v6Pl8Sm00Ny+juflZmpuXsHHjElpaltLevrHEcNh/ARO2/+hsITTWEVHf69eamr6ODyViRJ/P9/V6qCGlluLn20x7+8Y+t1tbV5PSRtrbm4vHu29v7OfPLHoJ1kOoqWnYjtuF/c23Gwz2kqQtMkCrX7S3t/L008/zxS8u4eSTl/DBDz7Lk08u2SwoF7aXAm19XKU2c0CMqC+GoRFbDKBbC6jb5/naAR/M2ttbtxKy+w7f7e3NL2u7rW0t7e0ruxzv+QtA4Rei/lMI9d2DdW/bQ6mpGVb8OpTa2k3b3Z/re7v764YV/7wM7D8rklTNDNDaovb2Fpqbn6O5+dliCF7S63ZLy1Ig8eUvF173xBOFr/X1O9LQMIkhQyYxYsSszu2Ghsmd2/X1E4sh2MBQ6Wpq6oA6amsb8y5lMym1FQP3lkfZt7Sd/TUbaG1dTXv7Btrb1xe/bujc37bR+ug1aPcd1ksL7qUEfFtrJGnrDNCDVHv7xi0G4o7tlpYXenl1DQ0NOxZD8M7ssEMTf/7zZK67bhLvfvckjj22EI4bGiZSU1O/3b83DT4RtdTWDgOG5V0K0HEDZ0uPcN3W1jNo9769pdetp719Pa2tK/u8xrap2eoIeSnB/eUE/MIvaJJU+fxpVWXa2tYXWyaeLYbg3rdbW1f08upaGhp2YsiQSQwdOpWRI1/fY7S4oWEy9fUTNvuL7oEH4NRT4eijC7NvOHilwS4iOttAYOR2fe9CeG/uI3hnDes9X9fWtoaWlhd6fV1KG7ep9o5+/5qa4dTXjy8+JnRuNzRM6HGsvn5C8ZcnSdp+DNADRGvrmi0G4o7ttrbVPV4bUU9Dw040NExm+PAZjB59GA0Nk3tppRhPRLblAjdsgPe8pzDbxuWXG56lvBXCe6FHu65u1HZ975Tai73vpY+09xbyCyF9OS0tL7Bu3SJaWl4o/mtYe6/vuylwT+gWtLuG7a7Hxmb+WSdJXRmgc5RSoq3tpZJGjNvaXurx+oghxRA8ieHDX82YMUcVWycmdx4vjBiPJaI8U35feCE8+CD8/Oew445leQtJA0REDbW1Q6mtHdrv106pndbWVcUwvYyWlhdobl7WGa47jrW0LGP9+r/R0vJCrz83i5VSVze2hKC9abu2ttHecEmdDNBlkFKitXVVSSPG7e3rery+pmZYZwgu3Hj3ls0Cccd2Xd2YXH+g//KX8M1vwtlnw1veklsZkgaBiBrq68dSXz8W2L2k17S1baC1dXkfQXvT9vr1j/Pii/fR0vJCnzO61NQM3ULA7tlmUlc3zp5uqYrFQFuprKmpKS1YsCCX9y4E4xVdpmRb0m17Uzju7UaemppGhgzpaJmY3Oto8ZAhk6itHVnxIx0vvAD77ANjx8KCBTDMFkRJA1zhZ/zqPoN2byPfvbXNdairG7PVoN31eG3tDhX/s18abCJiYUqpqftxfz0uwerV9/HwwyfQ3LyElJp7PF9bO7IzEBduvOsIx5M2266r2yGH6vtfSvCBD8Dy5fD//p/hWVJ1iAjq60dTXz8amF7Sa9rbm4v92r2PcneE7Q0bnuKllxbQ0rKsyyqh3d+/oUeo3nKbybjijaqStjcDdAkaGiYUb7yb1Gs4rq0dnneJ29X3vw8/+xl85SvwmtfkXY0k5aempoEhQwo3Ypei496XvoJ212Nr1vyJlpYXaG1d2ef1amtHbXWWkq5f6+pGOcot9QNbOJTJY48VQvOBB8Jtt0FNee5NlCQVtbe30Nq6os+g3VuLyZanFKwtroxaA9QUZySp2cqxzZ/remzTub0d2/Rcz2PV9X61tY0MHTrVGV6qjC0c2mYtLYV5nocMgSuvNDxL0vZQU1NPQ8NEGhom0ljCIqCFUe61vQbt1tZVpNQGtJNSe5ftTV9Tas9wbPPnNj/WSnv75se6vndpx7ZUa+F9K0lNzVCGD381jY1709i4T/Hr3gwZsrMj/1XGAK2Sffaz8Mc/wk9+AlOm5F2NJKk3EUFd3Qjq6kYwbNjUvMspq8K/oqcy/QJQ2rGOa7a2rmLt2odYu/ZBVq68neefv6qzzrq60Z1huuujvn7cdv281H8M0CrJb38LX/hCYcXBd74z72okSaI4qhtlW+tgW7S0rGDt2gc3eyxdOp/W1lWd5zQ0TOoSqDtGrF9NbW0J/9SgXNkDra1avRpmzYLaWrj/ftihOiYTkSRpu0op0dz8bGegXrPmAdaufZB16x6mvX198axg6NBXbDZSPWLEPgwbtjs1NfW51j8Y2QOtl+3ss2HxYvjNbwzPkiS9XBHBkCE7M2TIzowd+6bO4ym1sX79k6xd+8BmI9bLl98CtBVfW8/w4a/q0V9duHGx8kbgq50BWls0fz5ccw1cdBG8/vV5VyNJUvWJqGX48OkMHz6dCROO6zze3r6Rdese7RKqH+DFF+9j6dL5nefU1DTS2LhXt/7qfWhomOiNi2VkC4f69H//BzNnwp57Fkaf6/x1S5Kk3LW2vsS6dQ93toB0PFpanu88p65u3GYtII2NezN8+F7FhYJUKls4lElbG7z3vYWv11xjeJYkqVLU1e3AyJEHMHLkAZsdb25e1u3GxQd4/vmrePbZlzrPGTJkymYtIIVgvSe1tS4rnIWxSL366lfh17+GK66AV74y72okSdLWNDRMoKHhCMaMOaLzWEqJjRuf7gzUHeF65co7uiy4U8OwYdN7zAgybNh0amqMir2xhUM9LFxY6HeePRuuvx5soZIkqbq0t7eyfv3jPUas169/nI4FaiIaGD58z84WkE0Lw+w6aPqr+2rhMEBrM+vWwX77wZo18Ne/wtixeVckSZK2l7a29axb90iPEeuNG5/uPKe2doceNy02Nu5NQ8OEHCsvD3ugVZKPfAQefRRuv93wLEnSYFNbO4wddtiXHXbYd7PjLS2rWLfuoc1GrJct+ylLlvxn5zn19Tv2mGavsXEv6uqqbw5cA7Q6/fd/w/e+VwjRRx2VdzWSJKlS1NePZtSogxk16uDOY4WFYZ7vMVq9ZMnltLev7Txv6NCpPUashw9/FTU1Q/L4VvqFLRwC4PnnYZ99YPJk+P3vYcjA/TMtSZJylFI7Gzb8vcfCMOvWPUJKLcWzahk+fPceI9bDhr2CiNpc6+/KFg71KSX453+Gl16Ca681PEuSpJcvooZhw6YxbNg0xo8/tvN4e3sz69c/ttlS5i+99CeWLbsBKAzo1tQMY/jwV/dYyryhYXJF3bhogBbf/S78/Ofwne/AXnvlXY0kSapGNTUNxVUT9wKO7zze1raWtWsXdZtm7zaef/7Kjldy6KFrKmquagP0IPfww/DRj8Jb3gJnnZV3NZIkabCprW1k5MgmRo7cvFOipWU5a9c+xIYNf6+o8AwG6EFt40Y46SQYMQJ+8APne5YkSZWjvn4co0cflncZvTJAD2Kf+hTcfz/cfDPstFPe1UiSJA0MNXkXoHzccQd87WvwL/8Cb31r3tVIkiQNHAboQWjFCnjve2HGjEKIliRJUuls4RhkUoIzzijM+3zffdDYmHdFkiRJA4sBepC56ir4yU/gi1+E/ffPuxpJkqSBxxaOQeSJJ+Dss+Gww+BjH8u7GkmSpIHJAD1ItLbCySdDbS1cfXXhq+Fh/44AABm6SURBVCRJkrKzhWOQ+Pzn4d574brrYNdd865GkiRp4HIEehC47z743OcKi6aceGLe1UiSJA1sBugq99JLheA8ZQpcemne1UiSJA18tnBUuQ9/GJ56Cu66C0aNyrsaSZKkgc8R6Cp2441wxRVw4YVw6KF5VyNJklQdDNBV6pln4AMfgNe+Fi66KO9qJEmSqocBugq1t8P73gcbN8I110B9fd4VSZIkVQ97oKvQN74Bv/oVzJsHu++edzWSJEnVxRHoKvOXv8AnPwmzZ8Npp+VdjSRJUvUxQFeR9evhPe+BsWPh8sshIu+KJEmSqo8tHFXkggvg4Yfh1lth/Pi8q5EkSapOjkBXiVtvhe98pzDv85velHc1kiRJ1csAXQWWLYNTT4W994YvfSnvaiRJkqqbLRwDXEqFmwVXroRf/hKGDs27IkmSpOpmgB7g5s2Dm2+Gr38dZs7MuxpJkqTqZwvHAPboo3DeefAP/1DofZYkSVL5GaAHqOZmOOkkGDYMrrwSavwvKUmStF3YwjFAzZ0LCxfCT38KkyfnXY0kSdLg4bjlAHT33YXZNt7/fjjuuLyrkSRJGlwM0APMqlVwyinwylfCN7+ZdzWSJEmDjy0cA8xZZ8Ezz8A998CIEXlXI0mSNPgYoAeQa6+F666Dz34WDjgg72okSZIGJ1s4BoinnoIzz4SDDoILL8y7GkmSpMHLAD0AtLXBe99bWHXwmmugzn83kCRJyo1RbAD493+H3/wGrroKpk3LuxpJkqTBzRHoCrdgAVx0ERx/PJx8ct7VSJIkyQBdwdauLaw2OGkSXHYZRORdkSRJkmzhqGDnnw+PPQZ33AFjxuRdjSRJksAR6Ir1s5/BvHnwsY/B4YfnXY0kSZI6GKAr0JIlcNppsO++8LnP5V2NJEmSujJAV5j2dpgzp9D/fO210NCQd0WSJEnqyh7oCnPJJfCLX8B3vwt77pl3NZIkSerOEegK8uCD8PGPwzHHwBln5F2NJEmSemOArhAbNxamrBs1Cr7/faeskyRJqlS2cFSIT34S/vpX+J//gR13zLsaSZIk9cUR6Apw223w9a/DmWfCP/5j3tVIkiRpSwzQOVu+HE49FfbYA77ylbyrkSRJ0tbYwpGjlOD002HZMrjlFhg+PO+KJEmStDVlHYGOiDdHxKMR8XhEfKKX53eNiDsj4s8R8deIGFQNDFdcAT/9KXz+84VFUyRJklT5yhagI6IWuBR4C/Bq4MSIeHW30z4FXJ9S2hc4AfhuueqpNI8/Dh/6EBxxBHzkI3lXI0mSpFKVcwT6dcDjKaUnUkrNwHxgdrdzEjCyuD0KeLaM9VSMlhY4+WSor4crr4QaO9ElSZIGjHL2QO8MPN1lfzFwQLdz5gK/jIhzgEbgH3q7UEScDpwOsOuuu/Z7odvbxRfD738PP/4x7LJL3tVIkiQpi3KOffa2FEjqtn8i8MOU0hTgH4GrI6JHTSmleSmlppRS04QJE8pQ6vZzzz2FAP3e98K73513NZIkScqqnAF6MdB1fHUKPVs03g9cD5BSuhcYCowvY025evHFQuvGbrvBd76TdzWSJEl6OcoZoP8IzIiIaRHRQOEmwZu7nfN/wFEAEbEnhQC9rIw15eqcc+D//g+uvhpGjtz6+ZIkSao8ZQvQKaVW4GzgF8AiCrNtPBQRn42IY4unfQT4QET8BfgRcGpKqXubR1W4/nq46ir41Kfg4IPzrkaSJEkvVwy0vNrU1JQWLFiQdxmZPP00zJwJr3oV/Pa3UOfyNZIkSRUvIhamlJq6H3cCtTJrb4f3va8wdd011xieJUmSBjrjXJl97Wtw553w/e/D9Ol5VyNJkqRt5Qh0Gf3pT/Cv/wpvfzvMmZN3NZIkSeoPBugyWbcOTjoJJkyAefMgepsVW5IkSQOOLRxl8rGPwSOPwC9/CePG5V2NJEmS+osj0GXwP/8D3/0unH8+vPGNeVcjSZKk/mSA7mdLl8I//3Nh2rovfCHvaiRJktTfbOHoRykVwvPq1XDHHTBkSN4VSZIkqb8ZoPvR975XaN/41rdgr73yrkaSJEnlYAtHP1m0CD7yEXjTm+Ccc/KuRpIkSeVigO4Hzc2FKesaG+GKK5yyTpIkqZrZwtEPPv1p+POf4Wc/g0mT8q5GkiRJ5eQI9Da66y74ylfg9NNh9uy8q5EkSVK5GaC3wcqVcMopMGMGfP3reVcjSZKk7cEWjpcpJfjgB+G55+Deewv9z5IkSap+BuiX6Zpr4Mc/hs9/Hpqa8q5GkiRJ24stHC/Dk0/CWWfBoYfCBRfkXY0kSZK2JwN0Rq2tcPLJhanqrr4aamvzrkiSJEnbky0cGX3xi/C738G118Juu+VdjSRJkrY3R6Az+P3v4d/+Dd7znsJDkiRJg48BukRr1hRaN3beGS69NO9qJEmSlBdbOEp07rnwv/9bWDhl9Oi8q5EkSVJeHIEuwa23wve/D5/4BBx2WN7VSJIkKU8lBeiIuDEi/ikiBmXgPuKIwkqDc+fmXYkkSZLyVmogvgx4D/BYRHwpIvYoY00VZ8gQOO88aGjIuxJJkiTlraQAnVK6PaV0ErAf8BRwW0T8LiLmRER9OQuUJEmSKknJLRkRMQ44FTgN+DPwLQqB+rayVCZJkiRVoJJm4YiInwJ7AFcDb00pLSk+9eOIWFCu4iRJkqRKU+o0dpeklO7o7YmUUlM/1iNJkiRVtFJbOPaMiM7ZjyNiTEScWaaaJEmSpIpVaoD+QEppVcdOSmkl8IHylCRJkiRVrlIDdE1ERMdORNQCTuomSZKkQafUHuhfANdHxPeABJwB3Fq2qiRJkqQKVWqAvgD4F+CDQAC/BC4vV1GSJElSpSopQKeU2imsRnhZecuRJEmSKlup80DPAL4IvBoY2nE8pfSKMtUlSZIkVaRSbyK8gsLocytwBHAVhUVVJEmSpEGl1AA9LKX0KyBSSn9PKc0FjixfWZIkSVJlKvUmwg0RUQM8FhFnA88AO5avLEmSJKkylToCfS4wHPgQsD9wMvC+chUlSZIkVaqtjkAXF015d0rpY8AaYE7Zq5IkSZIq1FZHoFNKbcD+XVcilCRJkgarUnug/wz8V0T8BFjbcTCl9NOyVCVJkiRVqFID9FhgOZvPvJEAA7QkSZIGlVJXIrTvWZIkSaL0lQivoDDivJmU0j/3e0WSJElSBSu1heOWLttDgeOAZ/u/HEmSJKmyldrCcWPX/Yj4EXB7WSqSJEmSKlipC6l0NwPYtT8LkSRJkgaCUnugX2LzHujngAvKUpEkSZJUwUpt4dih3IVIkiRJA0FJLRwRcVxEjOqyPzoi3la+siRJkqTKVGoP9EUppdUdOymlVcBF5SlJkiRJqlylBujezit1CjxJkiSpapQaoBdExNcj4pUR8YqI+AawsJyFSZIkSZWo1AB9DtAM/Bi4HlgPnFWuoiRJkqRKVeosHGuBT5S5FkmSJKnilToLx20RMbrL/piI+EX5ypIkSZIqU6ktHOOLM28AkFJaCexYnpIkSZKkylVqgG6PiM6luyNiKpuvTChJkiQNCqVORfevwG8j4tfF/cOA08tTkiRJklS5Sr2J8NaIaKIQmu8H/ovCTBySJEnSoFJSgI6I04APA1MoBOgDgXuBI8tXmiRJklR5Su2B/jDwWuDvKaUjgH2BZWWrSpIkSapQpQboDSmlDQARMSSl9AjwqvKVJUmSJFWmUm8iXFycB/pnwG0RsRJ4tnxlSZIkSZWp1JsIjytuzo2IO4FRwK1lq0qSJEmqUKWOQHdKKf1662dJkiRJ1anUHmhJkiRJGKAlSZKkTAzQkiRJUgYGaEmSJCkDA7QkSZKUgQFakiRJysAALUmSJGVggJYkSZIyMEBLkiRJGRigJUmSpAwM0JIkSVIGBmhJkiQpAwO0JEmSlIEBWpIkScrAAC1JkiRlYICWJEmSMjBAS5IkSRkYoCVJkqQMDNCSJElSBgZoSZIkKQMDtCRJkpRBWQN0RLw5Ih6NiMcj4hO9PP+NiLi/+PhbRKwqZz2SJEnStqor14Ujoha4FHgjsBj4Y0TcnFJ6uOOclNJ5Xc4/B9i3XPVIkiRJ/aGcI9CvAx5PKT2RUmoG5gOzt3D+icCPyliPJEmStM3KGaB3Bp7usr+4eKyHiNgNmAbc0cfzp0fEgohYsGzZsn4vVJIkSSpVOQN09HIs9XHuCcANKaW23p5MKc1LKTWllJomTJjQbwVKkiRJWZUzQC8GdumyPwV4to9zT8D2DUmSJA0A5QzQfwRmRMS0iGigEJJv7n5SRLwKGAPcW8ZaJEmSpH5RtgCdUmoFzgZ+ASwCrk8pPRQRn42IY7uceiIwP6XUV3uHJEmSVDHKNo0dQErp58DPux37TLf9ueWsQZIkSepPrkQoSZIkZWCAliRJkjIwQEuSJEkZGKAlSZKkDAzQkiRJUgYGaEmSJCkDA7QkSZKUgQFakiRJysAALUmSJGVggJYkSZIyMEBLkiRJGRigJUmSpAwM0JIkSVIGBmhJkiQpAwO0JEmSlIEBWpIkScrAAC1JkiRlYICWJEmSMjBAS5IkSRkYoCVJkqQMDNCSJElSBgZoSZIkKQMDtCRJkpSBAVqSJEnKwAAtSZIkZWCAliRJkjIwQEuSJEkZGKAlSZKkDAzQkiRJUgYGaEmSJCkDA7QkSZKUgQFakiRJysAALUmSJGVggJYkSZIyMEBLkiRJGRigJUmSpAwM0JIkSVIGBmhJkiQpAwO0JEmSlIEBWpIkScrAAC1JkiRlYICWJEmSMjBAS5IkSRkYoCVJkqQMDNCSJElSBgZoSZIkKQMDtCRJkpSBAVqSJEnKwAAtSZIkZWCAliRJkjIwQEuSJEkZGKAlSZKkDAzQkiRJUgYGaEmSJCkDA7QkSZKUgQFakiRJysAALUmSJGVggJYkSZIyMEBLkiRJGRigJUmSpAwM0JIkSVIGBmhJkiQpAwO0JEmSlIEBWpIkScrAAC1JkiRlYICWJEmSMjBAS5IkSRkYoCVJkqQMDNCSJElSBgZoSZIkKQMDtCRJkpSBAVqSJEnKwAAtSZIkZWCAliRJkjIwQEuSJEkZGKAlSZKkDAzQkiRJUgYGaEmSJCkDA7QkSZKUgQFakiRJysAALUmSJGVggJYkSZIyMEBLkiRJGRigJUmSpAwM0JIkSVIGBmhJkiQpAwO0JEmSlIEBWpIkScrAAC1JkiRlUNYAHRFvjohHI+LxiPhEH+e8OyIejoiHIuK6ctYjSZIkbau6cl04ImqBS4E3AouBP0bEzSmlh7ucMwO4EDg4pbQyInYsVz2SJElSfyjnCPTrgMdTSk+klJqB+cDsbud8ALg0pbQSIKW0tIz1SJIkSdusnAF6Z+DpLvuLi8e62h3YPSLuiYj7IuLNvV0oIk6PiAURsWDZsmVlKleSJEnaunIG6OjlWOq2XwfMAA4HTgQuj4jRPV6U0ryUUlNKqWnChAn9XqgkSZJUqnIG6MXALl32pwDP9nLOf6WUWlJKTwKPUgjUkiRJUkUqZ4D+IzAjIqZFRANwAnBzt3N+BhwBEBHjKbR0PFHGmiRJkqRtUrYAnVJqBc4GfgEsAq5PKT0UEZ+NiGOLp/0CWB4RDwN3Ah9LKS0vV02SJEnStoqUurclV7ampqa0YMGCvMuQJElSlYuIhSmlpu7HXYlQkiRJysAALUmSJGVggJYkSZIyMEBLkiRJGRigJUmSpAwM0JIkSVIGBmhJkiQpAwO0JEmSlIEBWpIkScrAAC1JkiRlYICWJEmSMjBAS5IkSRkYoCVJkqQMDNCSJElSBgZoSZIkKQMDtCRJkpSBAVqSJEnKwAAtSZIkZWCAliRJkjIwQEuSJEkZGKAlSZKkDAzQkiRJUgYGaEmSJCkDA7QkSZKUgQFakiRJysAALUmSJGVggJYkSZIyMEBLkiRJGRigJUmSpAwM0JIkSVIGBmhJkiQpAwO0JEmSlIEBWpIkScrAAC1JkiRlYICWJEmSMjBAS5IkSRkYoCVJkqQMDNCSJElSBgZoSZIkKQMDtCRJkpSBAVqSJEnKwAAtSZIkZWCAliRJkjIwQEuSJEkZGKAlSZKkDAzQkiRJUgYGaEmSJCkDA7QkSZKUgQFakiRJysAALUmSJGVggJYkSZIyMEBLkiRJGRigJUmSpAwM0JIkSVIGBmhJkiQpAwO0JEmSlIEBWpIkScrAAC1JkiRlYICWJEmSMjBAS5IkSRkYoCVJkqQMDNCSJElSBgZoSZIkKQMDtCRJkpSBAVqSJEnKwAAtSZIkZWCAliRJkjIwQEuSJEkZGKAlSZKkDAzQkiRJUgYGaEmSJCkDA7QkSZKUgQFakiRJysAALUmSJGVggJYkSZIyMEBLkiRJGRigJUmSpAwM0JIkSVIGBmhJkiQpAwO0JEmSlIEBWpIkScrAAC1JkiRlYICWJEmSMjBAS5IkSRkYoCVJkqQMDNCSJElSBgZoSZIkKQMDtCRJkpRBWQN0RLw5Ih6NiMcj4hO9PH9qRCyLiPuLj9PKWY8kSZK0rerKdeGIqAUuBd4ILAb+GBE3p5Qe7nbqj1NKZ5erDkmSJKk/lXME+nXA4ymlJ1JKzcB8YHYZ30+SJEkqu3IG6J2Bp7vsLy4e6+4dEfHXiLghInYpYz2SJEnSNitbCwcQvRxL3fb/G/hRSmljRJwBXAkc2eNCEacDpxd310TEo/1aaenGAy/k9N7VzM+1fPxsy8fPtnz8bMvHz7Z8/GzLJ8/PdrfeDkZK3TNt/4iI1wNzU0pvKu5fCJBS+mIf59cCK1JKo8pSUD+IiAUppaa866g2fq7l42dbPn625eNnWz5+tuXjZ1s+lfjZlrOF44/AjIiYFhENwAnAzV1PiIhJXXaPBRaVsR5JkiRpm5WthSOl1BoRZwO/AGqBH6SUHoqIzwILUko3Ax+KiGOBVmAFcGq56pEkSZL6Qzl7oEkp/Rz4ebdjn+myfSFwYTlr6Gfz8i6gSvm5lo+fbfn42ZaPn235+NmWj59t+VTcZ1u2HmhJkiSpGrmUtyRJkpSBAboEW1uSXC9PRPwgIpZGxIN511JtImKXiLgzIhZFxEMR8eG8a6oWETE0Iv4QEX8pfrb/lndN1SYiaiPizxFxS961VJOIeCoiHoiI+yNiQd71VJOIGF1cz+KR4s/d1+ddUzWIiFcV/7x2PF6MiHPzrgts4diq4vR6f6PLkuTAib0sSa6MIuIwYA1wVUpp77zrqSbFGW4mpZT+FBE7AAuBt/nndttFRACNKaU1EVEP/Bb4cErpvpxLqxoRcT7QBIxMKR2Tdz3VIiKeAppSSs5V3M8i4krgNymly4szjw1PKa3Ku65qUsxjzwAHpJT+nnc9jkBvnUuSl0lK6W4Ks6+on6WUlqSU/lTcfonCFJG9rQSqjFLBmuJuffHhSEQ/iYgpwD8Bl+ddi1SKiBgJHAZ8HyCl1Gx4LoujgP+thPAMBuhSlLokuVSRImIqsC/w+3wrqR7FFoP7gaXAbSklP9v+803g40B73oVUoQT8MiIWFlf4Vf94BbAMuKLYenR5RDTmXVQVOgH4Ud5FdDBAb10pS5JLFSkiRgA3AuemlF7Mu55qkVJqSym9BpgCvC4ibEHqBxFxDLA0pbQw71qq1MEppf2AtwBnFdvotO3qgP2Ay1JK+wJrAe+X6kfFtphjgZ/kXUsHA/TWLQZ26bI/BXg2p1qkkhX7c28Erk0p/TTveqpR8Z9p7wLenHMp1eJg4Nhir+584MiIuCbfkqpHSunZ4telwE0UWhS17RYDi7v8S9QNFAK1+s9bgD+llJ7Pu5AOBuit2+qS5FKlKd7o9n1gUUrp63nXU00iYkJEjC5uDwP+AXgk36qqQ0rpwpTSlJTSVAo/a+9IKZ2cc1lVISIaizcUU2wvOBpwBqR+kFJ6Dng6Il5VPHQU4A3b/etEKqh9A8q8EmE16GtJ8pzLqgoR8SPgcGB8RCwGLkopfT/fqqrGwcApwAPFXl2ATxZXB9W2mQRcWbwjvAa4PqXkdGuqdBOBmwq/W1MHXJdSujXfkqrKOcC1xYG2J4A5OddTNSJiOIWZ0P4l71q6cho7SZIkKQNbOCRJkqQMDNCSJElSBgZoSZIkKQMDtCRJkpSBAVqSJEnKwAAtSRUuItoi4v4uj35b5SwipkaE8wFLUgbOAy1JlW99celwSVIFcARakgaoiHgqIv49Iv5QfEwvHt8tIn4VEX8tft21eHxiRNwUEX8pPg4qXqo2Iv4zIh6KiF8WV1gkIj4UEQ8XrzM/p29TkiqOAVqSKt+wbi0cx3d57sWU0uuAS4BvFo9dAlyVUpoJXAt8u3j828CvU0qzgP2AjlVVZwCXppT2AlYB7yge/wSwb/E6Z5Trm5OkgcaVCCWpwkXEmpTSiF6OPwUcmVJ6IiLqgedSSuMi4gVgUkqppXh8SUppfEQsA6aklDZ2ucZU4LaU0ozi/gVAfUrp4oi4FVgD/Az4WUppTZm/VUkaEByBlqSBLfWx3dc5vdnYZbuNTffH/BNwKbA/sDAivG9GkjBAS9JAd3yXr/cWt38HnFDcPgn4bXH7V8AHASKiNiJG9nXRiKgBdkkp3Ql8HBgN9BgFl6TByNEESap8wyLi/i77t6aUOqayGxIRv6cwIHJi8diHgB9ExMeAZcCc4vEPA/Mi4v0URpo/CCzp4z1rgWsiYhQQwDdSSqv67TuSpAHMHmhJGqCKPdBNKaUX8q5FkgYTWzgkSZKkDByBliRJkjJwBFqSJEnKwAAtSZIkZWCAliRJkjIwQEuSJEkZGKAlSZKkDAzQkiRJUgb/H1ZSeEGvxLuxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot_histrory(hist):\n",
    "    acc=hist['accuracy']\n",
    "    val_acc=hist['val_accuracy']\n",
    "    loss=hist['loss']\n",
    "    val_loss=hist['val_loss']\n",
    "    epochs=hist['epoch']\n",
    "    \n",
    "    plt.figure(figsize=(12,9))\n",
    "    plt.xlabel('Epochs')\n",
    "    plt.ylabel('Loss')\n",
    "    plt.plot(epochs,loss,'b',label='loss')\n",
    "    plt.plot(epochs,val_loss,'y',label='val_loss')\n",
    "    plt.legend()\n",
    "    \n",
    "    plt.figure(figsize=(12,9))\n",
    "    plt.xlabel('Epochs')\n",
    "    plt.ylabel('accuracy')\n",
    "    plt.plot(epochs,acc,'b',label='accuracy')\n",
    "    plt.plot(epochs,val_acc,'y',label='val_accuracy')\n",
    "    plt.ylim((0.5,1))\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "    \n",
    "plot_histrory(hist)\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "过拟合了，应该使用early stop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 取回训练后的词向量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8185, 256)\n"
     ]
    }
   ],
   "source": [
    "# 词向量的维度是（vocab_size,embeddind_dim)\n",
    "embedding=model.layers[0]\n",
    "weights=embedding.get_weights()[0]\n",
    "print(weights.shape)\n",
    "# # shape: (vocab_size, embedding_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.01278459,  0.02629857,  0.01663294, ...,  0.02288999,\n",
       "        -0.01076202, -0.00734962],\n",
       "       [-0.09303315,  0.02695822,  0.05755724, ...,  0.09081373,\n",
       "         0.04976359,  0.08361595],\n",
       "       [ 0.02284021,  0.01869576, -0.00541533, ...,  0.00803713,\n",
       "         0.04446839,  0.03104834],\n",
       "       ...,\n",
       "       [ 0.00065094,  0.01869471, -0.02089219, ...,  0.01696071,\n",
       "         0.02006514,  0.04435879],\n",
       "       [-0.00486183,  0.02313653, -0.03293175, ..., -0.01939596,\n",
       "         0.00174097,  0.0146229 ],\n",
       "       [ 0.01796103, -0.01714738, -0.01639607, ...,  0.00943253,\n",
       "        -0.01589711,  0.02012772]], dtype=float32)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将得到的词向量与对于单词保存到文本中\n",
    "# 以下代码来自于官方教程，但是我认为这是错误的\n",
    "# 以至于之后我找最后最接近的10个单词结果很诡异\n",
    "# 词汇表中第一个单词是'the_'\n",
    "# 时间上这个单词对应的编码是[181, 8024]\n",
    "# 单词the对应的是 181，‘_'对应8024\n",
    "# 这样也就可以解释subwords只有7928个，voabsize是8025\n",
    "# 当然，以上只是我个人想法，也许我没有理解官方教程的意思\n",
    "\n",
    "vecs=[]\n",
    "import io\n",
    "encoder=info.features['text'].encoder\n",
    "out_v=io.open('word_embedding/vecs.tsv','w',encoding='utf-8')\n",
    "out_m=io.open('word_embedding/meta.tsv','w',encoding='utf-8')\n",
    "for num, word in enumerate(encoder.subwords):\n",
    "#     vec = weights[num+1] # skip 0, it's padding.\n",
    "    # 注释掉的是官方教程中的,下面是我认为正确的\n",
    "    vec=weights[encoder.encode(word)]\n",
    "    if len(vec)==1:\n",
    "        vecs.append(vec)\n",
    "    out_m.write(word + \"\\n\")\n",
    "    out_v.write('\\t'.join([str(x) for x in vec]) + \"\\n\")\n",
    "out_v.close()\n",
    "out_m.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 找最相似的10个单词"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7928, 8185)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(encoder.subwords),encoder.vocab_size #encoder中遍历单词个数与vocab_size不一致\n",
    "# 原因是 subwords包含的是单词以及部分词根，而词汇表中包含了一些符号，如'_',','等等"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 计算两个词向量的距离\n",
    "# 此处使用的欧式距离\n",
    "def calDistance(x1,x2):\n",
    "    return np.sqrt(np.sum(np.square(x1-x2)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "#  构建距离矩阵\n",
    "def build_distance_matrix(vec):\n",
    "    # 距离矩阵\n",
    "    D=[[0 for _ in range(len(vec))] for _ in range(len(vec))]\n",
    "    # 注意矩阵是关于对角线对称的，只需要计算上三角\n",
    "    for i in range(len(vec)):\n",
    "        if i%100==0:\n",
    "            print('*',end=' ')\n",
    "        for j in range(i,len(vec)):\n",
    "            dis=calDistance(vec[i],vec[j])\n",
    "            D[i][j]=dis\n",
    "            D[j][i]=dis\n",
    "    return D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 获得单词与其对应的索引\n",
    "def get_index_and_word(encoder):\n",
    "    vocabulary=[]\n",
    "    for word in encoder.subwords:\n",
    "        if word[-1]=='_' or word[-1]==' ':\n",
    "            word=word[:-1] #去掉最后的‘_'\n",
    "        # 这个index才是单词在embedding层中真正对应的index\n",
    "        index=encoder.encode(word)\n",
    "        vocabulary.append((index,word))\n",
    "    return vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 包含了（index，word）其中index是embeding层中对应的index\n",
    "vocab=get_index_and_word(encoder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[181]\n",
      "the\n"
     ]
    }
   ],
   "source": [
    "index,word=vocab[0]\n",
    "print(index)\n",
    "print(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5375\n"
     ]
    }
   ],
   "source": [
    "vec=[]\n",
    "words=[]\n",
    "# 下面依据index构建词向量矩阵与单词矩阵\n",
    "for (index,word) in vocab:\n",
    "    \n",
    "    #  注意有些词是合成的，所以有多个index，对于这些词，就不管了\n",
    "    if len(index)==1 and word not in words:\n",
    "        vec.append(weights[index[0]])\n",
    "        words.append(word)\n",
    "print(len(vec))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(68, 121, 355)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words.index('good'),words.index('bad'),words.index('nice')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "distance between good and bad is 17.87944221496582)\n",
      "distance between good and nice is 8.884379386901855)\n"
     ]
    }
   ],
   "source": [
    "# 测试一下 good 与 nice，应该很接近\n",
    "# good与bad很远\n",
    "# 欧式距离\n",
    "print(f'distance between {words[68]} and {words[121]} is {calDistance(vec[68],vec[121])})')\n",
    "print(f'distance between {words[68]} and {words[355]} is {calDistance(vec[68],vec[355])})')\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * "
     ]
    }
   ],
   "source": [
    "D=build_distance_matrix(vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(449, array([-1.158833 ,  1.1000751,  1.1224998,  1.2348317,  1.2053943,\n",
       "         1.1606256, -1.2236959, -1.222485 ,  1.2365196,  1.2782831,\n",
       "         1.2173694,  1.0891424,  1.0433444, -1.1191549, -1.1462668,\n",
       "         1.2276717, -1.0591754, -1.1242613, -1.2323408, -1.0962893,\n",
       "         1.1424705,  1.1990823,  1.2086251,  1.2505646, -1.2227283,\n",
       "         1.1664479,  1.1888583, -1.145613 ,  1.0174279, -1.0702763,\n",
       "         1.1031744, -1.2026076, -1.1771042,  1.1821641,  1.0411842,\n",
       "        -1.1739595, -1.0936714, -1.2390083,  1.1549006,  1.1954257,\n",
       "         1.1538042, -1.1592566,  1.1699144, -1.2132894,  1.1984882,\n",
       "        -1.1660255, -1.1330742, -1.2383007, -1.025219 ,  1.1024791,\n",
       "        -1.0729952, -1.2227561,  1.1894603, -1.1891248, -1.1691242,\n",
       "         1.2689188,  1.2138859,  1.06356  ,  1.2034099,  1.090857 ,\n",
       "        -1.2509091, -1.2244816, -1.2412297,  1.058949 , -1.1872858,\n",
       "         1.0923905, -1.2109215, -1.1137263, -1.1790894, -1.1980942,\n",
       "        -1.270611 , -1.104827 , -1.191064 , -1.0933206, -1.1231111,\n",
       "        -1.2276613,  1.2560834,  1.1020769,  1.2365133, -1.2203577,\n",
       "         1.2064039, -1.204834 , -1.1945572, -1.0490696, -1.1007245,\n",
       "        -1.2370849, -1.2488327,  1.1666175,  1.1565346, -1.2423481,\n",
       "         1.1788564,  1.1460234, -1.2256014,  1.0658566, -1.0052174,\n",
       "         1.0700767,  1.0729276, -1.1642146, -1.1972985,  1.2310395,\n",
       "         1.1615993,  1.1346785,  1.0677933,  1.2691422, -1.0498776,\n",
       "        -1.1880411, -1.0993929, -1.1557388, -1.1268779, -1.0933412,\n",
       "         1.182323 , -1.1963118, -1.0435016,  1.2215899, -1.2057805,\n",
       "        -1.2609811, -1.0785801, -1.1293858, -1.0052686,  1.1545558,\n",
       "        -1.1478046, -1.2126547,  1.0779471, -1.1982853, -1.0960714,\n",
       "        -1.0639305,  1.1343352, -1.1625363,  1.2329274,  1.2501173,\n",
       "        -1.1574028, -1.1976216, -1.2542051, -1.2196853,  1.2752466,\n",
       "        -1.2118696, -1.3106523,  1.1429679, -1.089621 , -1.1636922,\n",
       "        -1.1616472, -1.088559 ,  1.2247965, -1.2144387, -1.25504  ,\n",
       "        -1.0667844, -1.255194 , -1.2055959,  1.1882168, -1.1595782,\n",
       "        -1.124863 , -1.2749898,  1.1907775, -1.2023294, -1.2071036,\n",
       "        -1.0722779,  1.0977125, -1.2114747,  1.1863061, -1.2364411,\n",
       "        -1.1641216, -1.1361914, -1.1805161,  1.1769623, -1.0748584,\n",
       "        -1.2658504,  1.1511972, -1.2446293,  1.1197656, -1.1692742,\n",
       "        -1.1080773,  1.1177891,  1.1961452, -1.2022271, -1.2689103,\n",
       "         1.1016116, -1.1035631, -1.0981627, -1.2072575,  1.1845009,\n",
       "         1.239433 , -1.1626776, -1.2554855,  1.2155442,  1.1476977,\n",
       "        -1.1738461, -1.1448452, -1.2562771,  1.0790097,  1.2179149,\n",
       "        -1.2325411,  1.1491232,  1.2532275,  1.1582202,  1.2005466,\n",
       "        -1.1215955, -1.2535921,  1.2251941, -1.2532073, -1.1873889,\n",
       "        -1.0601614,  1.2161762,  1.1810769, -1.2689235, -1.14794  ,\n",
       "        -1.0399942, -1.2730857,  1.1456898,  1.2465053, -1.0668714,\n",
       "        -1.0812514, -1.2673062, -1.1777936, -1.2262089,  1.1742747,\n",
       "         1.1974565, -1.1096089,  1.1366223, -1.0607065, -1.1628044,\n",
       "         1.1606661,  1.1549456,  1.1649392, -1.2340363, -1.1840482,\n",
       "         1.1727208, -1.228456 ,  1.2094742, -1.0621357, -1.2581956,\n",
       "        -1.1035044, -1.1270396,  1.1878   , -1.063499 ,  1.2344608,\n",
       "         1.0722508, -1.1627983, -1.1281948,  1.1238643, -1.2073206,\n",
       "         1.0768857, -1.1344327, -1.1810106,  1.0175176, -1.2152126,\n",
       "        -1.2363658,  1.2045767,  1.1142975, -1.2056397, -1.1433135,\n",
       "        -1.1201692,  1.0347435, -1.0603924,  1.1675838,  1.2096058,\n",
       "         1.233638 ], dtype=float32))"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "test_word='wonderful'\n",
    "test_index=words.index(test_word)\n",
    "test_index,vec[test_index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_top_10(D,index):\n",
    "    dis=D[index] # 获得了测试单词词向量到所有单词词向量的距离\n",
    "    \n",
    "    #下面我们需要找到distances中最小的k个元素，并且返回他们的索引就可以了\n",
    "    # argsort函数返回的是数组值从小到大的索引值\n",
    "    # 将distances变为np.array，然后对索引排序，取前k个\n",
    "    minK=np.argsort(np.array(dis))[:10]\n",
    "    print(minK)\n",
    "    result=[]\n",
    "    for i in range(10):\n",
    "        result.append(words[minK[i]])\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 449  381 1332 2996 3994 5103 1591  797 2451 3540]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['wonderful',\n",
       " 'everyone',\n",
       " 'fantastic',\n",
       " 'marvel',\n",
       " 'underrated',\n",
       " 'haunting',\n",
       " 'superb',\n",
       " 'brilliant',\n",
       " 'High',\n",
       " 'joy']"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 每次的结果都不一定，与训练的epoch，embedding层的初始化有关\n",
    "find_top_10(D,test_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取Glove词向量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 因为github上传100MB以上的大文件很麻烦\n",
    "# 所以我在语料库中放了glove的压缩文件，\n",
    "# 解压后使用\n",
    "with open('语料库/glove.6B.50d.txt') as f:\n",
    "    words_set=set()\n",
    "    word2vec={}\n",
    "    for line in f:\n",
    "        line=line.strip().split()\n",
    "        cur_word=line[0]\n",
    "        words_set.add(cur_word)\n",
    "        word2vec[cur_word]=np.array(line[1:],dtype=np.float64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 计算单词的余弦相似度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\text{CosineSimilarity(u, v)} = \\frac {u . v} {||u||_2 ||v||_2} = cos(\\theta) \\tag{1}$$\n",
    "\n",
    "<img src=\"pic/cosine_sim.png\" style=\"width:800px;height:250px;\">\n",
    "<caption><center> **Figure 1**: The cosine of the angle between two vectors is a measure of how similar they are</center></caption>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cos_similaryty(u,v):\n",
    "    dot=np.dot(u,v)\n",
    "    \n",
    "    norm_u=np.sqrt(np.sum(np.power(u,2)))\n",
    "    norm_v=np.sqrt(np.sum(np.power(v,2)))\n",
    "    \n",
    "    return np.divide(dot,(norm_u*norm_v))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cos similaryty between man and women is: 0.886033771849582\n",
      "cos similaryty between computer and basketnall is: 0.35974951493305957\n",
      "cos similaryty between football and basketball is: 0.8786788106585918\n"
     ]
    }
   ],
   "source": [
    "man=word2vec['man']\n",
    "woman=word2vec['woman']\n",
    "\n",
    "computer=word2vec['computer']\n",
    "basketball=word2vec['basketball']\n",
    "football=word2vec['football']\n",
    "\n",
    "print('cos similaryty between man and women is:',cos_similaryty(man,woman))\n",
    "print('cos similaryty between computer and basketnall is:',cos_similaryty(computer,basketball))\n",
    "print('cos similaryty between football and basketball is:',cos_similaryty(football,basketball))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 计算相似的词"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a to b like c to __ \n",
    "例如:\n",
    "- man to women like king to quuen\n",
    "- Paris to France like Beijing to China"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def caculate_similar(a,b,c,word2vec):\n",
    "    \n",
    "    a,b,c=a.lower(),b.lower(),c.lower()\n",
    "    vec_a,vec_b,vec_c=word2vec[a],word2vec[b],word2vec[c]\n",
    "    # 注意余弦相似度越大，越相似\n",
    "    # 1，-1则意思相反\n",
    "    maxi=-2\n",
    "    best_word=None\n",
    "    words = word2vec.keys()\n",
    "    for word in words:\n",
    "        if word in [a,b,c]:\n",
    "            continue\n",
    "        cos_sim=cos_similaryty(vec_a-vec_b,vec_c-word2vec[word])\n",
    "        if cos_sim>maxi:\n",
    "            maxi=cos_sim\n",
    "            best_word=word\n",
    "    return best_word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "man-->woman like boy --> girl\n",
      "small-->smaller like large --> larger\n",
      "italy-->italian like china --> chinese\n",
      "Japan-->Tokyo like China --> taipei\n"
     ]
    }
   ],
   "source": [
    "test_words=[('man','woman','boy'),('small','smaller','large'),('italy','italian','china'),('Japan','Tokyo','China')]\n",
    "for words in test_words:\n",
    "    print(f'{words[0]}-->{words[1]} like {words[2]} --> {caculate_similar(words[0],words[1],words[2],word2vec)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}