003-keras_training_and_evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow2教程-使用keras训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本指南包含了TensorFlow 2.0中在以下两种情况下的训练，评估和预测（推理）模型：\n",
    "\n",
    "- 使用内置的训练和评估API（例如model.fit()，model.evaluate()，model.predict()）。\n",
    "- 使用eager execution 和GradientTape对象从头开始编写自定义循环。\n",
    "\n",
    "无论是使用内置循环还是编写自己的循环，模型和评估训练在每种Keras模型中严格按照相同的方式工作，无论是Sequential 模型, 函数式 API, 还是模型子类化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install -q pydot\n",
    "# !apt-get install graphviz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/doit/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "from __future__ import absolute_import, division, print_function\n",
    "import tensorflow as tf\n",
    "tf.keras.backend.clear_session()\n",
    "import tensorflow.keras as keras\n",
    "import tensorflow.keras.layers as layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 一般的模型构造、训练、测试流程\n",
    "使用内置的训练和评估API对模型进行训练和验证。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 模型构造\n",
    "inputs = keras.Input(shape=(784,), name='mnist_input')\n",
    "h1 = layers.Dense(64, activation='relu')(inputs)\n",
    "h1 = layers.Dense(64, activation='relu')(h1)\n",
    "outputs = layers.Dense(10, activation='softmax')(h1)\n",
    "model = keras.Model(inputs, outputs)\n",
    "# keras.utils.plot_model(model, 'net001.png', show_shapes=True)\n",
    "\n",
    "model.compile(optimizer=keras.optimizers.RMSprop(),\n",
    "             loss=keras.losses.SparseCategoricalCrossentropy(),\n",
    "             metrics=[keras.metrics.SparseCategoricalAccuracy()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "端到端的模型训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 50000 samples, validate on 10000 samples\n",
      "Epoch 1/3\n",
      "50000/50000 [==============================] - 2s 40us/sample - loss: 0.3406 - sparse_categorical_accuracy: 0.9038 - val_loss: 0.1747 - val_sparse_categorical_accuracy: 0.9478\n",
      "Epoch 2/3\n",
      "50000/50000 [==============================] - 2s 32us/sample - loss: 0.1562 - sparse_categorical_accuracy: 0.9521 - val_loss: 0.1353 - val_sparse_categorical_accuracy: 0.9605\n",
      "Epoch 3/3\n",
      "50000/50000 [==============================] - 2s 34us/sample - loss: 0.1142 - sparse_categorical_accuracy: 0.9647 - val_loss: 0.1093 - val_sparse_categorical_accuracy: 0.9670\n",
      "history:\n",
      "{'loss': [0.3406039907693863, 0.1561644163942337, 0.11421714420557022], 'sparse_categorical_accuracy': [0.90378, 0.95214, 0.96466], 'val_loss': [0.17468911408782006, 0.13526522952914238, 0.10934125750362873], 'val_sparse_categorical_accuracy': [0.9478, 0.9605, 0.967]}\n",
      "10000/1 [================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================] - 0s 8us/sample - loss: 0.0680 - sparse_categorical_accuracy: 0.9643\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "evaluate:\n",
      "[0.11812290600538254, 0.9643]\n",
      "predict:\n",
      "[[5.96028303e-06 1.22943675e-06 5.60810586e-05 4.95154876e-03\n",
      "  3.37098766e-10 1.87345813e-05 1.95063844e-11 9.94938374e-01\n",
      "  3.62650326e-06 2.44577204e-05]\n",
      " [9.77753075e-07 1.13667054e-04 9.99821007e-01 5.17880726e-05\n",
      "  1.29938716e-11 9.91447632e-06 3.67613637e-07 2.29428121e-08\n",
      "  2.27706096e-06 1.34117118e-11]]\n"
     ]
    }
   ],
   "source": [
    "# 载入数据\n",
    "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
    "x_train = x_train.reshape(60000, 784).astype('float32') /255\n",
    "x_test = x_test.reshape(10000, 784).astype('float32') /255\n",
    "\n",
    "# 保证还是float 32？ 否则后面会出现：TypeError: Input 'y' of 'Sub' Op has type float32 that does not match type uint8 of argument 'x'.\n",
    "y_train = y_train.astype('float32')\n",
    "y_test = y_test.astype('float32')\n",
    "\n",
    "\n",
    "\n",
    "# 取验证数据\n",
    "x_val = x_train[-10000:]\n",
    "y_val = y_train[-10000:]\n",
    "\n",
    "x_train = x_train[:-10000]\n",
    "y_train = y_train[:-10000]\n",
    "\n",
    "# 训练模型\n",
    "history = model.fit(x_train, y_train, batch_size=64, epochs=3,\n",
    "         validation_data=(x_val, y_val))\n",
    "print('history:')\n",
    "print(history.history)\n",
    "\n",
    "result = model.evaluate(x_test, y_test, batch_size=128)\n",
    "print('evaluate:')\n",
    "print(result)\n",
    "pred = model.predict(x_test[:2])\n",
    "print('predict:')\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 自定义指标和损失"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 配置网络\n",
    "在对模型训练之前，我们需要指定损失函数，优化器以及可选的一些要监控的指标。我们将这些配置参数作为compile()方法的参数传递给模型，对模型进行配置。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),\n",
    "              loss=keras.losses.SparseCategoricalCrossentropy(),\n",
    "              metrics=[keras.metrics.SparseCategoricalAccuracy()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow2提供许多内置的优化器，损失和指标\n",
    "常见的内置参数如下：\n",
    "\n",
    "- 优化器： - SGD()（有或没有动量） - RMSprop() - Adam() -等等。\n",
    "\n",
    "- 损失： - MeanSquaredError() - KLDivergence() - CosineSimilarity() -等等。\n",
    "\n",
    "- 指标： - AUC() - Precision() - Recall() -等等。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 自定义损失\n",
    "用Keras提供两种方式来提供自定义损失。\n",
    "- 一、例创建一个接受输入y_true和的函数y_pred。\n",
    "- 二、构建一个继承keras.losser.Loss的子类\n",
    "下面示例显示了一个损失函数，该函数计算实际数据与预测之间的平均距离："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 50000 samples\n",
      "Epoch 1/3\n",
      "50000/50000 [==============================] - 1s 28us/sample - loss: 4.3488\n",
      "Epoch 2/3\n",
      "50000/50000 [==============================] - 1s 24us/sample - loss: 4.3488\n",
      "Epoch 3/3\n",
      "50000/50000 [==============================] - 1s 25us/sample - loss: 4.3488\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1e004394a8>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_uncompiled_model():\n",
    "    inputs = keras.Input(shape=(784,), name='digits')\n",
    "    x = layers.Dense(64, activation='relu', name='dense_1')(inputs)\n",
    "    x = layers.Dense(64, activation='relu', name='dense_2')(x)\n",
    "    outputs = layers.Dense(10, activation='softmax', name='predictions')(x)\n",
    "    model = keras.Model(inputs=inputs, outputs=outputs)\n",
    "    return model\n",
    "model = get_uncompiled_model()\n",
    "\n",
    "def basic_loss_function(y_true, y_pred):\n",
    "    return tf.math.reduce_mean(y_true - y_pred)\n",
    "model.compile(optimizer=keras.optimizers.Adam(),\n",
    "              loss=basic_loss_function)\n",
    "model.fit(x_train, y_train, batch_size=64, epochs=3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果需要试下带参数的损失函数，可以子类化tf.keras.losses.Loss。并子类化以下方法：\n",
    "- \\__init\\__(self)  接收相关参数，初始化loss之类。\n",
    "- call(self, y_true, y_pred)  使用 y_true和y_pred，计算模型损失。\n",
    "\n",
    "下面例子，展示了WeightedCrossEntropy计算二分损失的损失函数，某个类或整个函数的损失可以通过标量修改。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 50000 samples\n",
      "Epoch 1/3\n",
      "50000/50000 [==============================] - 2s 35us/sample - loss: 9.5178\n",
      "Epoch 2/3\n",
      "50000/50000 [==============================] - 1s 27us/sample - loss: 9.5171\n",
      "Epoch 3/3\n",
      "50000/50000 [==============================] - 1s 27us/sample - loss: 9.5171\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1dc06f44a8>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class WeightBinaryCrossEntropy(keras.losses.Loss):\n",
    "    def __init__(self, pos_weight, weight, from_logits=False,\n",
    "                reduction=keras.losses.Reduction.AUTO,\n",
    "                name='weight_binary_crossentropy'):\n",
    "        \"\"\"\n",
    "        pos_weight: 正类标签权重\n",
    "        weight: 整体损失权重\n",
    "        from_logits: 是否使用logits来计算loss，（或使用probability）\n",
    "        reduction: reduction类型\n",
    "        name: 名字\n",
    "        \"\"\"\n",
    "        super(WeightBinaryCrossEntropy, self).__init__(reduction=reduction,\n",
    "                                                      name=name)\n",
    "        self.pos_weight = pos_weight\n",
    "        self.weight = weight\n",
    "        self.from_logits = from_logits\n",
    "        \n",
    "    def call(self, y_true, y_pred):\n",
    "        if not self.from_logits:\n",
    "            x_1 = y_true * self.pos_weight * -tf.math.log(y_pred + 1e-6)\n",
    "            \n",
    "            x_2 = (1-y_true) * -tf.math.log(1-y_pred + 1e-6)\n",
    "            \n",
    "            return tf.add(x_1, x_2) * self.weight\n",
    "        \n",
    "        return tf.nn.weighted_cross_entropy_with_logits(y_true, y_pred, self.pos_weight) *self.weight\n",
    "        \n",
    "\n",
    "model.compile(optimizer=keras.optimizers.Adam(),\n",
    "             loss=WeightBinaryCrossEntropy(0.5, 2))\n",
    "model.fit(x_train, y_train, batch_size=64, epochs=3)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用网络层的方法构建loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 50000 samples\n",
      "50000/50000 [==============================] - 2s 36us/sample - loss: 2.4702\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1de00675c0>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class ActivityRegularizationLayer(layers.Layer):\n",
    "    def call(self, inputs):\n",
    "        self.add_loss(tf.reduce_sum(inputs)* 0.1)\n",
    "        return inputs\n",
    "inputs = keras.Input(shape=(784,), name='digits')\n",
    "x = layers.Dense(64, activation='relu', name='dense_1')(inputs)\n",
    "\n",
    "# 添加计算正则化损失的层\n",
    "x = ActivityRegularizationLayer()(x)\n",
    "\n",
    "x = layers.Dense(64, activation='relu', name='dense_2')(x)\n",
    "outputs = layers.Dense(10, activation='softmax', name='predictions')(x)\n",
    "\n",
    "model = keras.Model(inputs=inputs, outputs=outputs)\n",
    "model.compile(optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),\n",
    "              loss='sparse_categorical_crossentropy')\n",
    "\n",
    "# 训练\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=64,\n",
    "          epochs=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 自定义指标\n",
    "\n",
    "自定义指标只需继承Metric类， 并重写以下函数：\n",
    "\n",
    "- \\__init\\__(self)，初始化。\n",
    "\n",
    "- update_state(self，y_true，y_pred，sample_weight = None)，它使用目标y_true和模型预测y_pred来更新状态变量。\n",
    "\n",
    "- result(self)，它使用状态变量来计算最终结果。\n",
    "\n",
    "- reset_states(self)，重新初始化度量的状态。\n",
    "\n",
    "状态更新和结果计算保持分开（分别在update_state()和result()中），因为在某些情况下，结果计算可能非常昂贵，并且只能定期进行。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/3\n",
      "50000/50000 [==============================] - 1s 27us/sample - loss: 0.0590 - binary_true_postives: 9186.0000\n",
      "Epoch 2/3\n",
      "50000/50000 [==============================] - 1s 24us/sample - loss: 0.0499 - binary_true_postives: 9957.0000\n",
      "Epoch 3/3\n",
      "50000/50000 [==============================] - 1s 25us/sample - loss: 0.0441 - binary_true_postives: 10939.0000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f7b88f8fb38>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 下面是一个简单的示例，显示如何实现CatgoricalTruePositives指标，该指标计算正确分类为属于给定类的样本数量\n",
    "\n",
    "class CatgoricalTruePostives(keras.metrics.Metric):\n",
    "    def __init__(self, name='binary_true_postives', **kwargs):\n",
    "        super(CatgoricalTruePostives, self).__init__(name=name, **kwargs)\n",
    "        # 会更新的类变量\n",
    "        self.true_postives = self.add_weight(name='tp', initializer='zeros')\n",
    "        \n",
    "    def update_state(self, y_true, y_pred, sample_weight=None):\n",
    "        # 获取结果id\n",
    "        y_pred = tf.argmax(y_pred)\n",
    "        # 正确的结果\n",
    "        y_true = tf.equal(tf.cast(y_pred, tf.int32), tf.cast(y_true, tf.int32))\n",
    "        y_true = tf.cast(y_true, tf.float32)\n",
    "        \n",
    "        if sample_weight is not None:\n",
    "            # 对正确结果加权重\n",
    "            sample_weight = tf.cast(sample_weight, tf.float32)\n",
    "            y_true = tf.multiply(sample_weight, y_true)\n",
    "        # 修改正确样本总量\n",
    "        return self.true_postives.assign_add(tf.reduce_sum(y_true))\n",
    "    \n",
    "    def result(self):\n",
    "        # 返回相应tensor\n",
    "        return tf.identity(self.true_postives)\n",
    "    \n",
    "    def reset_states(self):\n",
    "        # 重置为0\n",
    "        self.true_postives.assign(0.)\n",
    "        \n",
    "\n",
    "model.compile(optimizer=keras.optimizers.RMSprop(1e-3),\n",
    "             loss=keras.losses.SparseCategoricalCrossentropy(),\n",
    "             metrics=[CatgoricalTruePostives()])\n",
    "\n",
    "model.fit(x_train, y_train,\n",
    "         batch_size=64, epochs=3)\n",
    "            \n",
    "            \n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用自定义层的方式获取相关指标"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 50000 samples\n",
      "50000/50000 [==============================] - 2s 41us/sample - loss: 0.2940 - sparse_categorical_accuracy: 0.9151 - std_of_activation: 0.9423\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f192eecfc88>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 也可以以定义网络层的方式添加要统计的metric\n",
    "class MetricLoggingLayer(layers.Layer):\n",
    "    def call(self, inputs):\n",
    "        # 该层的作用就是添加指标\n",
    "        self.add_metric(keras.backend.std(inputs),\n",
    "                       name='std_of_activation',\n",
    "                       aggregation='mean')\n",
    "        # 直接把输入进行输出\n",
    "        return inputs\n",
    "\n",
    "inputs = keras.Input(shape=(784,), name='mnist_input')\n",
    "h1 = layers.Dense(64, activation='relu')(inputs)\n",
    "# 直接套在对应的网络层中\n",
    "h1 = MetricLoggingLayer()(h1)\n",
    "h1 = layers.Dense(64, activation='relu')(h1)\n",
    "outputs = layers.Dense(10, activation='softmax')(h1)\n",
    "model = keras.Model(inputs, outputs)\n",
    "# keras.utils.plot_model(model, 'net001.png', show_shapes=True)\n",
    "# 配置并训练网络\n",
    "model.compile(optimizer=keras.optimizers.RMSprop(),\n",
    "             loss=keras.losses.SparseCategoricalCrossentropy(),\n",
    "             metrics=[keras.metrics.SparseCategoricalAccuracy()])\n",
    "model.fit(x_train, y_train, batch_size=32, epochs=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们也可以在构建好模型后直接使用，model.add_loss和model.add_metric添加损失和指标。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 50000 samples\n",
      "50000/50000 [==============================] - 4s 81us/sample - loss: 2.3469 - sparse_categorical_accuracy: 0.1130 - std_of_activation: 0.0010\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1d98dd4710>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs = keras.Input(shape=(784,), name='mnist_input')\n",
    "h1 = layers.Dense(64, activation='relu')(inputs)\n",
    "h2 = layers.Dense(64, activation='relu')(h1)\n",
    "outputs = layers.Dense(10, activation='softmax')(h2)\n",
    "model = keras.Model(inputs, outputs)\n",
    "# 直接把计算loss或metric用到的输入(h1)带人\n",
    "model.add_metric(keras.backend.std(h1),\n",
    "                       name='std_of_activation',\n",
    "                       aggregation='mean')\n",
    "\n",
    "model.add_loss(tf.reduce_sum(h1)*0.1)\n",
    "\n",
    "# keras.utils.plot_model(model, 'net001.png', show_shapes=True)\n",
    "\n",
    "model.compile(optimizer=keras.optimizers.RMSprop(),\n",
    "             loss=keras.losses.SparseCategoricalCrossentropy(),\n",
    "             metrics=[keras.metrics.SparseCategoricalAccuracy()])\n",
    "model.fit(x_train, y_train, batch_size=32, epochs=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "处理使用validation_data传入测试数据，还可以使用validation_split划分验证数据\n",
    "\n",
    "ps:validation_split只能在用numpy数据训练的情况下使用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 40000 samples, validate on 10000 samples\n",
      "40000/40000 [==============================] - 3s 79us/sample - loss: 2.3014 - sparse_categorical_accuracy: 0.1141 - std_of_activation: 3.6019e-06 - val_loss: 2.3013 - val_sparse_categorical_accuracy: 0.1115 - val_std_of_activation: 1.2214e-06\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1d9270fb00>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "\n",
    "model.fit(x_train, y_train, batch_size=32, epochs=1, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 使用tf.data构造数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "到现在我们已经了解了如何使用使用numpy作为输入数据进行训练和验证。下面，我们将介绍如何使用tf.data作为输入数据进行。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train for 100 steps, validate for 3 steps\n",
      "Epoch 1/3\n",
      "100/100 [==============================] - 1s 15ms/step - loss: 0.8740 - sparse_categorical_accuracy: 0.7706 - val_loss: 0.4562 - val_sparse_categorical_accuracy: 0.8802\n",
      "Epoch 2/3\n",
      "100/100 [==============================] - 0s 3ms/step - loss: 0.3867 - sparse_categorical_accuracy: 0.8891 - val_loss: 0.3228 - val_sparse_categorical_accuracy: 0.8958\n",
      "Epoch 3/3\n",
      "100/100 [==============================] - 0s 3ms/step - loss: 0.3206 - sparse_categorical_accuracy: 0.9095 - val_loss: 0.2346 - val_sparse_categorical_accuracy: 0.9323\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1d9a74d240>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_compiled_model():\n",
    "    inputs = keras.Input(shape=(784,), name='mnist_input')\n",
    "    h1 = layers.Dense(64, activation='relu')(inputs)\n",
    "    h2 = layers.Dense(64, activation='relu')(h1)\n",
    "    outputs = layers.Dense(10, activation='softmax')(h2)\n",
    "    model = keras.Model(inputs, outputs)\n",
    "    model.compile(optimizer=keras.optimizers.RMSprop(),\n",
    "                 loss=keras.losses.SparseCategoricalCrossentropy(),\n",
    "                 metrics=[keras.metrics.SparseCategoricalAccuracy()])\n",
    "    return model\n",
    "model = get_compiled_model()\n",
    "# 构建dataset实例\n",
    "train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
    "# 打乱\n",
    "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)\n",
    "# 获得验证数据\n",
    "val_dataset = tf.data.Dataset.from_tensor_slices((x_val, y_val))\n",
    "val_dataset = val_dataset.batch(64)\n",
    "\n",
    "# model.fit(train_dataset, epochs=3)\n",
    "# steps_per_epoch 每个epoch只训练几步\n",
    "# validation_steps 每次验证，验证几步\n",
    "model.fit(train_dataset, epochs=3, steps_per_epoch=100,\n",
    "         validation_data=val_dataset, validation_steps=3)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果只想对该数据集中的特定批次进行训练，则可以传递steps_per_epoch参数，即批训练多少步。同样我们可以使用train_dataset.take(), 来获取每批次中训练的数据，其和steps_per_epoc等价。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/3\n",
      "100/100 [==============================] - 2s 17ms/step - loss: 0.7795 - sparse_categorical_accuracy: 0.7961\n",
      "Epoch 2/3\n",
      "100/100 [==============================] - 1s 5ms/step - loss: 0.3308 - sparse_categorical_accuracy: 0.9080\n",
      "Epoch 3/3\n",
      "100/100 [==============================] - 0s 3ms/step - loss: 0.2465 - sparse_categorical_accuracy: 0.9308\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1d9a0fae48>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = get_compiled_model()\n",
    "\n",
    "model.fit(train_dataset.take(100), epochs=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "# Evaluate\n",
      "157/157 [==============================] - 0s 1ms/step - loss: 0.2669 - sparse_categorical_accuracy: 0.9243\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.2669110751835404, 0.9243]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 模型测试\n",
    "test_dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test))\n",
    "test_dataset = test_dataset.batch(64)\n",
    "\n",
    "print('\\n# Evaluate')\n",
    "model.evaluate(test_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**其他格式输入数据支持**\n",
    "\n",
    "除了numpy数值和TensorFlow Dataset,还可以用Pandas和python迭代器作为数据输入。\n",
    "通常小数据推荐使用numpy， 大数据推荐使用TensorFlow Dataset。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 样本权重和类权重\n",
    "在模型训练时可以，可以人工设定样本权重和类权重。\n",
    "\n",
    "“样本权重”数组是一个数字数组，用于指定批处理中每个样本在计算总损失时应具有多少权重。 它通常用于不平衡的分类问题（这个想法是为了给予很少见的类更多的权重）。 当使用的权重是1和0时，该数组可以用作损失函数的掩码（完全丢弃某些样本对总损失的贡献）。\n",
    "\n",
    "“类权重”dict是同一概念的更具体的实例：它将类索引映射到应该用于属于该类的样本的样本权重。 例如，如果类“0”比数据中的类“1”少两倍，则可以使用class_weight = {0：1.，1：0.5}。\n",
    "\n",
    "添加方法：\n",
    "- 使用Numpy数据时： 通过sample_weight和class_weight参数传递。\n",
    "- 使用Dataset数据时： 通过使数据集返回(input_batch, target_batch, sample_weight_batch)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面是一个Numpy数据中加大第5类的权重的例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0: 1.0, 1: 1.0, 2: 1.0, 3: 1.0, 4: 1.0, 5: 2.0, 6: 1.0, 7: 1.0, 8: 1.0, 9: 1.0}\n",
      "Train on 50000 samples\n",
      "Epoch 1/4\n",
      "50000/50000 [==============================] - 2s 49us/sample - loss: 0.3598 - sparse_categorical_accuracy: 0.9047\n",
      "Epoch 2/4\n",
      "50000/50000 [==============================] - 2s 32us/sample - loss: 0.1603 - sparse_categorical_accuracy: 0.9545\n",
      "Epoch 3/4\n",
      "50000/50000 [==============================] - 2s 32us/sample - loss: 0.1172 - sparse_categorical_accuracy: 0.9666\n",
      "Epoch 4/4\n",
      "50000/50000 [==============================] - 2s 32us/sample - loss: 0.0940 - sparse_categorical_accuracy: 0.9738\n",
      "Train on 50000 samples\n",
      "Epoch 1/4\n",
      "50000/50000 [==============================] - 2s 42us/sample - loss: 0.3718 - sparse_categorical_accuracy: 0.9024\n",
      "Epoch 2/4\n",
      "50000/50000 [==============================] - 2s 32us/sample - loss: 0.1749 - sparse_categorical_accuracy: 0.9506\n",
      "Epoch 3/4\n",
      "50000/50000 [==============================] - 2s 32us/sample - loss: 0.1298 - sparse_categorical_accuracy: 0.9630\n",
      "Epoch 4/4\n",
      "50000/50000 [==============================] - 2s 31us/sample - loss: 0.1031 - sparse_categorical_accuracy: 0.9702\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1d6bc35780>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 增加第5类的权重\n",
    "import numpy as np\n",
    "# 类权重\n",
    "model = get_compiled_model()\n",
    "class_weight = {i:1.0 for i in range(10)}\n",
    "# 第5类的权重为2\n",
    "class_weight[5] = 2.0\n",
    "print(class_weight)\n",
    "model.fit(x_train, y_train,\n",
    "         class_weight=class_weight,\n",
    "         batch_size=64,\n",
    "         epochs=4)\n",
    "# 样本权重\n",
    "model = get_compiled_model()\n",
    "sample_weight = np.ones(shape=(len(y_train),))\n",
    "sample_weight[y_train == 5] = 2.0\n",
    "model.fit(x_train, y_train,\n",
    "         sample_weight=sample_weight,\n",
    "         batch_size=64,\n",
    "         epochs=4)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset数据中加大第5类的权重的例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/3\n",
      "782/782 [==============================] - 4s 5ms/step - loss: 0.3777 - sparse_categorical_accuracy: 0.9011\n",
      "Epoch 2/3\n",
      "782/782 [==============================] - 2s 3ms/step - loss: 0.1792 - sparse_categorical_accuracy: 0.9496\n",
      "Epoch 3/3\n",
      "782/782 [==============================] - 2s 3ms/step - loss: 0.1295 - sparse_categorical_accuracy: 0.9629\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1d68465780>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# tf.data数据\n",
    "model = get_compiled_model()\n",
    "\n",
    "sample_weight = np.ones(shape=(len(y_train),))\n",
    "sample_weight[y_train == 5] = 2.0\n",
    "# 在构造dataset时传入sample_weight\n",
    "train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train,\n",
    "                                                    sample_weight))\n",
    "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)\n",
    "\n",
    "val_dataset = tf.data.Dataset.from_tensor_slices((x_val, y_val))\n",
    "val_dataset = val_dataset.batch(64)\n",
    "\n",
    "model.fit(train_dataset, epochs=3, )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5 多输入多输出模型\n",
    "有些模型是是多输入或多输出的，此时可以为他们设置不同的权重的loss和metric。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABE0AAAIECAYAAADsGNC/AAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzde1RVZf4/8PcBhAA18JomDogZJqaOa0bN64i3pUbpF0Hw2hjDN1M08VrSHZQpFQZwTEXMQQ0xLW+TE6lZhE7fdOxXjIKSFwQ1FQQ5KLfP7w/X2XE8B+TAucL7tZardZ6z9/N8zs6z99vn7ItKRARERERERERERFRTmp2lKyAiIiIiIiIiskacNCEiIiIiIiIi0oOTJkREREREREREenDShIiIiIiIiIhIDwdLF9BUTZkyxdIlEBGRFRk0aBAWLVpk6TKIzIp5iIjIdBYtWoRBgwZZuowmj2eamMju3buRl5dn6TKITCovLw+7d++2dBlWj/sDOnHiBDIzMy1dBpHZcf9HtoB5pn74fbYuu3fvxpUrVyxdRrPAM01M6LXXXkNgYKClyyAymV27diEoKAhpaWmWLsWqqVQq7g+aOf7aTs0Z939k7Zhn6od5xrqoVCpLl9Bs8EwTIiIiIiIiIiI9OGlCRERERERERKQHJ02IiIiIiIiIiPTgpAkRERERERERkR6cNCEiIiIiIiIi0oOTJkREREREREREenDShIiswsCBA7F06VJLl2E1VCoV7O3tsWzZMsTExCAnJ0fr/ZycHKxZswYAUFlZibVr1yIiIgIhISEYNmwYdu/e3aBxs7Ky8OKLL6Jdu3Zo3749goODUVBQoLVMUlIS+vXrh1atWqFv375ITk5u2Ie0svGqqqqwfPlyXL16VWudnJwcxMTEIDw8HCqVio/4IyKiWjHPaLNEnjFmLsrPz0dycjKCgoLw3HPP6V1my5YtCAwMxMqVKxEaGoqdO3cq7zFbNBFCJgFAUlNTLV0GkUmlpqaKsXYjU6dOlcjISKP01RBXrlwxWd8N2R8AkO7du+t979ixYxISEiLl5eUiIhIZGSk//vij8n58fLwAkA8//NCgMbOysmTSpEmyd+9eOX36tMyYMUMAiJ+fn7LM8uXLZfr06ZKYmCgLFiwQZ2dnASDx8fEGjWWt492+fVsmT54subm5evvw9PRs0N/5gIAACQgIMHg9IlvHPES2gHmmfmwlzxirH43Lly8LAPHx8dF579133xVPT08pLCwUEZHCwkLx9PSUuLg4ZRlTZQvuX81mFydNTIR/iak5MGbIsKRffvlFhg4darL+Gxoy9B2cs7KypGvXrnLr1i2lrUuXLpKenq68vnPnjgCQAQMGGDRmXFycqNVq5XVFRYW4ublJy5YtReRBEJs2bZrWOocPH64zENnSeBpnzpwRX19fuXv3rk4fPj4+nDQhMgDzENkC5pn6sZU8Y6x+atL3OS5fviwtWrSQVatWabVHRUWJi4uL3Lx5U2kzRbbg/tVsdvHyHCJq1q5evYqJEyfi119/tXQpjyQimD59Ol566SW0adNGaa+ursbevXuV1zdv3gQAeHh4GNR/eHg4nJ2dtdoqKysxZ84cAMClS5eUU2g1xowZg/bt2+PGjRsGjWWN42k8++yz8Pb2xpIlSwweg4iIyBKYZ35jrH4eJSUlBRUVFfDz89NqHzlyJNRqNZKSkpQ2ZgvbxkkTIrKo6upqpKWlYfbs2Rg+fDgAYN++fQgLC4OHhweKioowe/ZstGvXDr1798YPP/wAADhx4gQWL14MLy8vXL9+HQEBAWjbti169+6NPXv2AAA2bdoEOzs75TrRkpISrF27Vqtt69at+Pnnn3Ht2jW88sorSl1Hjx6Fh4cHjh8/bs7NUad9+/bh1KlTGDdunFb74cOHsWLFCq3lHBwcEBkZ2ajx3nzzTcTGxiI2NhYAMHjwYHTs2FFnufLycgwdOrRRY1nDeDWNHTsWmzZtQm5ubqPHISKipo95pv5MnWdMlYse9u233wIAunTpotWumZw5c+aMVjuzhQ2z9LkuTRV4uhQ1A8Y6nfXha0Xz8vKkZcuWAkCioqLk0qVLkpKSopxaWVVVJQcOHFDubzF//nw5fvy47NixQ1q1aiUAJCMjQ0REvL29dWp8uA16Trn8/PPPxcXFRfbv39/oz9eQ/YG+moKDg0WlUklFRUWt65WXl0v37t0lJSWlQbWKiOzdu1eGDRsmAMTLy0s2b95c67IZGRni7Owsp06dalLjnT59WgDonHLLy3OIDMM8RLaAeaZ+bC3PGLMffZ+jb9++AkDKysq02tVqtQCQQYMGabUbO1tw/2o2vKeJqfAvMTUHxrwG+OGD0dNPP63Td8eOHcXJyUl53aNHDwEgpaWlSltsbKwAkKlTp4qI/gPRw236DoQiIpWVlY37UDX6N0bI8PT0FDc3tzrX27Bhg6xbt87gGmsqLCyUrKwsSUhIEBcXFwEgW7du1VmusrJShg8fLjt37mxy4+Xn5wsAGT9+vFY7J02IDMM8RLaAeaZ+bC3PGLMffZ9D8wPMvXv3tNrLysoEgPTv31+r3djZgvtXs+E9TYjIOul79Jq7uzvu37+vvLaze7ALc3FxUdr8/f0BQOeRdg1hb2/f6D6M6dq1a3B3d69zmQsXLmDhwoWNGsfNzQ09e/bEq6++io8++ggAsG3bNp3l3nnnHfj5+WHq1KlNbjw3NzcAwPXr1xs1FhERNW/MM7rMlWeM2Y8+Pj4+AICioiKt9sLCQgBA586dtdqZLWwXJ02IqEnRHKCMfbMva2Bvb4+qqqpa3y8rK0O/fv2MOuYLL7wAAHB0dNRqP3DgAFxdXY1+fbC1jKcv5BIREZkL80zj84wpclFNvXr1AgDk5+drtRcUFAAAhgwZotXObGG7OGlCRE3KrVu3AACjRo0C8NsBqry8HMCDO7bfuXNHax2VSoXKykqdvuo6oFtCp06ddH7NqMnZ2RnBwcFGHVNz4B8/frzS9uWXXyIvLw/Lli3TWjYzM7NJjAf89ivRE0880egxiIiIDMU80/g8Y4pcVNOMGTPg5uaGo0eParUfOXIEjo6OCAkJ0WpntrBdnDQhIou7e/cuAKC4uFhpu3fvns5yJSUlAKATCGqGgfT0dPTv3x9hYWEAfjt18v3338f58+cRFxennBJ7+PBhVFdXw9vbGwUFBbhy5YrSz8GDB+Hm5oYvvvjCGB/RKIYPH46SkhJlez0sPDwcEyZM0Glfs2YNevXqhU8++aTO/tetW4ctW7YoIez+/ftYtmwZgoKCMG/ePADAV199hdWrV6OqqgqJiYlITExEQkICFi1ahEOHDtn8eBqaxxM+/CsRERFRbZhn6sfUecbY/ZSVlQHQnXxyd3fHihUrsGHDBuWzlJSUYOPGjVi5cqXOU3WYLWyXg6ULIKLmTa1WIzo6GsCD0xvXrVuH8vJyXLx4EQAQFRWF+fPnIzk5GVevXgUAREZG4q233lL6iI2NxezZs1FdXY2CggJ8/fXXcHB4sHuLiYlBfn4+1q5di5MnTyIhIQF79uyBp6cnioqKUFlZiSlTpmDr1q34/vvvldNgnZyc0Lp1azg5OZlxa9Rt5syZSEpKQmZmJkaPHq3z/r179/SGs9zcXJw9exaLFy+u834gxcXFWL9+vbKco6Mj5s2bBz8/PwAPzuzw9/eHWq3GkSNHtNZVqVQ4f/68TY9XU0ZGBuzt7REYGFhrf0RERBrMM/Vn6jxjzH6OHTuGnTt3AgAuXryIDz74AGPGjEGfPn0AAEuXLkW7du0wd+5cdO3aFdnZ2ViyZAlCQ0N1+mK2sGGWvhVtUwXezZiaAWPebb4hGnq3cXNryP4AtdwBf/z48bJw4UKDazh37pwMGDDA4PUaytbHe/755yU0NFSnnU/PITIM8xDZAuaZ+mlKecbcOUXE+NmC+1ez4dNziIisVc0762skJyfj0KFDBt15Xa1WIz4+Hps3bzZmeU12vJMnTyI7Oxtr1qzReU/fteJERERUO2vLM+bOKQCzha3j5TlEZLNKS0uV/7q6ulq4GuP75ZdfsGDBAnTu3BmTJ0/GU089hQ4dOuDTTz/Fa6+9hs2bN2s9nrA2ubm5iI6ORqtWrcxQtW2PV1BQgKioKKSnpyv95eTkYM+ePbh9+zYuXLjQ6DGIiIhqYp4xb54xd05htrB9nDQhIptTWlqK6Oho5UZn4eHhCA0NxcCBAy1cmfGISK3v+fr6IioqComJiViyZMkj+/L19TVmaU12vMrKSmzbtg3bt2/XClJPPfWU8uSemJgYo4xFRETEPGOZPGPOnMJs0TTw8hwrMnDgQCxdutTSZeiw1rrq8u2332LFihVQqVRQqVSYNWsW9u3bZ+mycOzYMQQGBip1/e///i++++47S5dlc1xdXREVFQURgYggKSmpSQWM+vDy8qpXwKD6c3BwwLJly8z2yxMRmQ+zjPEwyxgP80zTzzPMFk0DJ02siJeXFx577DFLl6HD0nXl5eUZvM6QIUOwatUq/O53vwMAbNiwAf7+/sYurV5q1j9ixAh8/PHHAIDf/e532LBhA5577jmL1EVERGSNGnLcfxRmmcZhliGi5oyX51gRzeOsrI0l67p48SJmzpyJ48ePN2h9Z2dnrf+am776LV0TERGRtWrscb82zDINxyxDRM0dJ03Ial29ehUTJ05EVVWVpUtpEFuvn4iIyJya4nHT1j+TrddPRGQMvDzHClRXVyMtLQ2zZ8/G8OHDATx4FNb27dsREhKCwYMH48SJE/j9738PT09PZGRkIDs7G5MmTUL79u3Rs2dP/PDDDzr9JiQkYMaMGZg7dy4ee+wx5dpTlUrV4Lr27duHsLAweHh4oKioCLNnz0a7du3Qu3dvpYYTJ05g8eLF8PLywvXr1xEQEIC2bduid+/e2LNnDwBg06ZNsLOzU2opKSnB2rVrtdq2bt2Kn3/+GdeuXcMrr7yi1HX06FF4eHg06Bcba6jfEDk5OZgyZQqWL1+OmTNnYtiwYfh//+//AQC2b98OV1dXqFQqxMTEKIFmx44dcHJyUk6dvXfvHv7617/i5Zdfxh/+8AeMHj0aP/30E6qrq/H111/jtddeg5eXF/Lz8zFixAj87ne/Q1FRUYPqJSIiaqjajpv/93//h4EDB2LevHl488030aJFC+VpI4/CLMMsQ0TUaEImAUBSU1Prvfzly5cFgPj4+IiISHV1tZw/f14AyOOPPy4HDx6UrKwsASCenp7ywQcfyJ07d+T06dMCQEaMGKHVX3x8vNjb28utW7dERGTVqlUCQCIiIgz6HA/XlZeXJy1bthQAEhUVJZcuXZKUlBQBIAMGDJCqqio5cOCAODs7CwCZP3++HD9+XHbs2CGtWrUSAJKRkSEiIt7e3vLwX8GH22qOrfH555+Li4uL7N+//5H1+/j4aPVnDfXX1f6wp556Sry9vUVEpKKiQtzc3MTX11d5f+XKlQJAfv75Z6Xt8uXLMmnSJOV1aGionD17Vnk9ZswY6dixo9y8eVO+++47cXFxEQCyatUqSU9Pl5dfflnu3r37yNpERFJTU3W2AekydH9ATU9AQIAEBARYugwiszN0/6fv+NijRw9p06aN8jooKEhu3LhR7z6ZZZhlHoV5pn6YZ6wL/3+YzS7uHUykIX+J9R18Hm578skndXbqHTp0EDc3N602f39/sbOzk/LychER+emnnwSADBw40KCa9NXw9NNP69TQsWNHcXJyUl736NFDAEhpaanSFhsbKwBk6tSpIqIbAvS11XZArqysrFft+sawhvrrGzTWrl0rO3fuFJEHE2ne3t7SokUL5f1bt25Jq1atJDQ0VGlbtWqVHDhwQERETp48KQD0/tEso9ket2/ffmQ9D9OEDP7hH/559B9OmlBzBDR+0qR9+/YCQOLi4qS6ulp++uknKS4uNrgOZhlmmdowz/CPrf7hpIlZ7OI9TWyMvsdVtWnTBmfPntVqGz16NPbt24eDBw/ixRdfVO4YP3LkyEbXoO/yHnd3d1y/fl15bWf34MovFxcXpc3f3x8LFy5ETk5Oo2uwt7dv8LrWUH99vfbaaygtLcX69etx+/Zt3L9/HxUVFcr7bdq0wfz58/Hhhx/i7bffRufOnfHVV18pj277/vvv4evrq5wGq49me7i7uze4ztTU1Aav2xwEBQVh4cKFGDRokKVLIQtZt26dpUsgsll///vf8dJLL2HBggX4xz/+gYSEhEY/vtMasgCzzAPWkmUA5plHYZ6xLkFBQZYuodngpEkTNW/ePDg7O2POnDnIyMhATk4O3n33Xbz++usWq6lz584AAA8PD4vV0BjmrP/XX3+Fu7s7Tp8+jaCgIKxfvx5z587F9u3bdZZdtGgR/va3vyE2NhZBQUH44x//qASxW7duITc3F2q1Wis0AQ+u89YEqsYKDAw0Sj9NVVBQEAYNGsTt1IylpaVZugQim/U///M/6NevH+bOnYvDhw9j6NCh2LRpE2bNmmX2Wphl6s/WsgzAPPMozDPWhZMm5sMbwTZRVVVV+Omnn3DixAl88MEH+OyzzxAZGdmoXzUa69atWwCAUaNGAfjtV4Hy8nIAgIjgzp07WuuoVCpUVlbq9GWJu7gbs/5HmTt3Luzt7TFz5kxUVFRg3LhxAB6Eg4e1bdsWr7zyCjZs2IC//e1v+POf/6y85+PjA7VajZiYGK11/vvf/yIhIcHguoiIiExJ33HzrbfeQrdu3fDFF19g586dqKiowMqVKy1SH7NM/THLEFFTwUkTK3H37l0AQHFxsdJ27949AA8OYBqa0xk1y9dcruZBKDo6Gvv378c333yDw4cPIzMzEzk5OQYfoOuqq6aSkhIA0Dmo1hwvPT0d/fv3R1hYGIAHB0EAeP/993H+/HnExcXh/v37AIDDhw+juroa3t7eKCgowJUrV5R+Dh48CDc3N3zxxRePrF+tVmv91xrqLygoUMas+f8WeLCdw8LClKcdFRQU4OrVq/jyyy+xY8cO5U7w//73v5GXl6esFxERgfLycly+fBne3t5K+wsvvIBu3brh3XffxZw5c7Bjxw5ERkZi4cKFeOmll7S2R32fREBERGQq+o6bH374oXL8CwgIwOOPP44nn3yy3n0yyzDLEBE1BidNrIBarUZ0dDQAID8/H+vWrcOFCxfwxhtvAAAuXryIr776Cv/6179w6dIlAMAbb7yB27dvIyEhQWlbs2aN8gvCoEGDcPfuXcyZMwfjxo3Dc889hx49eqBTp07Ko+YaUldMTAwuXrwIAIiKikJxcTHi4uJw9epVAEBkZKTWgTw2Nha3bt3Cr7/+ioKCAnz99ddwcHhwVVhMTAwGDBiAtWvX4tVXX8WECRPQq1cvzJgxA0VFRaisrMSUKVPQunVrfP/990qfTk5OaN26NZycnGqt/dtvv8WKFStw+fJlAMBf/vIX7Nu3D+vXr7do/UePHlUe2Xf16lU888wzGDlyJEaOHAkfHx906NABGzduxOjRowE8mPxq3bo1Vq5cCW9vb7zxxhtwd3dHdHS01imqHTt2xOjRozFnzhyt7eDk5IQjR47A398fn332GSIiInDjxg1s374d9vb2eO+995TtsWjRIvznP/+p8+8EERGRKek77qvVavj5+SEmJgazZ8/G0KFD8cknn9SrP2YZZhkiosZSycPTw2QUKpUKqampFrvmLzk5GTdv3lRuolVdXY38/HwcPXoUixcv1rpRmCn07NkTZ8+e1fn1wVbYWv1qtRp9+vTBjz/+CGdnZ7ONu2vXLgQFBdnMdrIUS+8PyPKmTJkCgPc2oebHlvd/tpYFHmZr9VsqywDMM/Vly9/npoj/P8wmjWeaNEExMTH485//rDVTb2dnhy5dumDIkCF48sknoVKpHvnn3LlzFvwUZIjExETMnz/f7CGDiIjIUphlmhZmGSKyVnx6ThP07bffAgA2bNiAsLAwtG3bFgBw6tQpxMTEICUlBc8884xJa9BcU1paWgpXV1eTjmUKtlD/yZMn8Ze//AVqtRpVVVU6j50mIiJqykx9VoAtZIG62EL9zDJEZAt4pkkT9PHHH2P+/PlISkpCly5dMHjwYAQGBuLUqVMmnzApLS3FG2+8odwsLDw8HCdOnDDZeMZmS/W7urqiuLgYdnZ22LFjBxwdHS1dEhmRSqWCvb09li1bhpiYGOTk5Gi9n5OTgzVr1gB4cNO/tWvXIiIiAiEhIRg2bBh2797doHGzsrLw4osvol27dmjfvj2Cg4OVG/5pJCUloV+/fmjVqhX69u2L5OTkhn1IKxuvqqoKy5cvV+4LoJGTk4OYmBiEh4crv14TUdNlS1lAH1uqn1mm6bNEnjFmLsrPz0dycjKCgoLw3HPP6V1my5YtCAwMxMqVKxEaGoqdO3cq7zFbNBFCJgFAUlNTLV0GkUmlpqaKJXcjV65csYm+G7I/ACDdu3fX+96xY8ckJCREysvLRUQkMjJSfvzxR+X9+Ph4ASAffvihQWNmZWXJpEmTZO/evXL69GmZMWOGABA/Pz9lmeXLl8v06dMlMTFRFixYIM7OzgJA4uPjDRrLWse7ffu2TJ48WXJzc/X24enp2aC/8wEBARIQEGDwekS2jnmIbAHzTP3YSp4xVj8aly9fFgDi4+Oj8967774rnp6eUlhYKCIihYWF4unpKXFxccoypsoW3L+azS5OmpgI/xJTc2DJkPHLL7/I0KFDbaLvhoYMfQfnrKws6dq1q9y6dUtp69Kli6Snpyuv79y5IwBkwIABBo0ZFxcnarVaeV1RUSFubm7SsmVLEXkQvKZNm6a1zuHDh+sMRLY0nsaZM2fE19dX7t69q9OHj48PJ02IDMA8RLaAeaZ+bCXPGKufmvR9jsuXL0uLFi1k1apVWu1RUVHi4uIiN2/eVNpMkS24fzWbXbw8h4hsztWrVzFx4kT8+uuvNtV3Y4kIpk+fjpdeeglt2rRR2qurq7F3717l9c2bNwEAHh4eBvUfHh6ucwO+yspK5abSly5dUk6h1RgzZgzat2+PGzduGDSWNY6n8eyzz8Lb21t5+hgREZEpMM+YJs8Yq59HSUlJQUVFBfz8/LTaR44cCbVajaSkJKWN2cK2cdKEiMyquLgYy5Ytw4oVKxAREYGxY8ciIiICRUVFAIBNmzbBzs5OubazpKQEa9eu1WrbunUrfv75Z1y7dg2vvPIKAODEiRNYvHgxvLy8cP36dQQEBKBt27bo3bs39uzZ06i+AeDo0aPw8PDA8ePHzbOh9Ni3bx9OnTqFcePGabUfPnwYK1as0FrOwcEBkZGRjRrvzTffRGxsLGJjYwEAgwcPRseOHXWWKy8vx9ChQxs1ljWMV9PYsWOxadMm5ObmNnocIiJqephnGs7UecZUuehhmodvdOnSRatdMzlz5swZrXZmCxtm6XNdmirwdClqBgw9nbWkpER69Oghb7/9ttJ248YN6dGjh3Tr1k2KiopERMTb21un34fbUOM0yaqqKjlw4IByv4v58+fL8ePHZceOHdKqVSsBIBkZGQ3qW+Pzzz8XFxcX2b9/f70/b83+jHE6a3BwsKhUKqmoqKh1vfLycunevbukpKQYXKfG3r17ZdiwYQJAvLy8ZPPmzbUum5GRIc7OznLq1KkmNd7p06cFgM4pt7w8h8gwzENkC5hn6sfW8owx+9H3Ofr27SsApKysTKtdrVYLABk0aJBWu7GzBfevZsPLc4jIfFavXo3s7GyEhYUpbe3bt8fKlSuRm5uL6OhoAECLFi101tXXpmFnZ4cJEyYoM/urV6/G0KFDERwcjPfeew8AEB8f36C+Nfz9/VFcXIyJEyc+cllTyczMxOOPPw4Hh9qfFr9lyxa8+uqrmDZtWoPHGTFiBDZs2ICEhARcv34dL7/8Mj7++GOd5aqqqvD6669jy5Yt6NevX5MaT3OGyzfffNPgcYiIqGlinmkcc+UZY/ajT+vWrQFA58k3mtfl5eVa7cwWtouTJkRkNhkZGQCAVq1aabUPGzYMAPDdd981qn87uwe7NBcXF6XN398fAHQecdcQ9vb2je6jMa5duwZ3d/c6l7lw4QIWLlzYqHHc3NzQs2dPvPrqq/joo48AANu2bdNZ7p133oGfnx+mTp3a5MZzc3MDAFy/fr1RYxERUdPDPNM45sozxuxHHx8fHwBQLsnSKCwsBAB07txZq53ZwnZx0oSIzEYTAi5evKjVrpl5f/zxx40+puaAZeybf1mCvb09qqqqan2/rKysUWdg6PPCCy8AABwdHbXaDxw4AFdXV6NfH2wt4z38qxEREZEG80zjmCvPmCIX1dSrVy8AQH5+vlZ7QUEBAGDIkCFa7cwWtouTJkRkNppfYA4ePKjVfuXKFQDAqFGjAOie1igiuHPnjtY6KpUKlZWVjxzz1q1bRuu7rgO8OXTq1Enn14yanJ2dERwcbNQxNQf+8ePHK21ffvkl8vLysGzZMq1lMzMzm8R4wG+/Ej3xxBONHoOIiJoW5pnGMVeeMUUuqmnGjBlwc3PD0aNHtdqPHDkCR0dHhISEaLUzW9guTpoQkdksXboUvr6+iI+Px7Vr15T2xMREDB48GPPmzQPw2+mO77//Ps6fP4+4uDjcv38fwIM7oldXV8Pb2xsFBQVKQKmpZhhIT09H//79leuOG9r3wYMH4ebmhi+++MKYm8Qgw4cPR0lJCe7evav3/fDwcEyYMEGnfc2aNejVqxc++eSTOvtft24dtmzZooSu+/fvY9myZQgKClL+33z11VdYvXo1qqqqkJiYiMTERCQkJGDRokU4dOiQzY+noXk84cO/EhERETHPNI6p84yx+ykrKwOgO9nk7u6OFStWYMOGDcpnKSkpwcaNG7Fy5Uqdp+owW9iu2u++Q0RkZM7OzsjMzMR7772HWbNmoXfv3rC3t0fbtm1x5MgR5YZgMTExyM/Px9q1a3Hy5EkkJCRgz5498PT0RFFRESorKzFlyhRs3boV33//vc6pqrGxsZg9ezaqq6tRUFCAr7/+utF9Ozk5oXXr1nBycjLvRqth5syZSEpKQmZmJkaPHq3z/r1793Dv3j2d9tzcXJw9exaLFy+u834gxcXFWL9+vci6XEcAACAASURBVLKco6Mj5s2bBz8/PwAPzuzw9/eHWq3GkSNHtNZVqVQ4f/68TY9XU0ZGBuzt7REYGFhrf0RE1DwxzzSOqfOMMfs5duwYdu7cCeDB5VgffPABxowZgz59+gB4MIHWrl07zJ07F127dkV2djaWLFmC0NBQnb6YLWyYZZ/e03SBj4CiZsDQR/SZWkMf2WZqDdkfQM+j7URExo8fLwsXLjS4hnPnzsmAAQMMXq+hbH28559/XkJDQ3Xa+chhIsMwD5EtYJ6pn6aUZ8ydU0SMny24fzUbPnKYiMhaaU6zrSk5ORmHDh0y6M7rarUa8fHx2Lx5szHLa7LjnTx5EtnZ2VizZo3Oe/W57pyIiIh+Y215xtw5BWC2sHW8PIeImozS0lLlv66urhaupvF++eUXLFiwAJ07d8bkyZPx1FNPoUOHDvj000/x2muvYfPmzVqPI6xNbm4uoqOjdR6NaCq2PF5BQQGioqKQnp6u9JeTk4M9e/bg9u3buHDhQqPHICIiqgvzjH7GOt6bO6cwW9g+TpoQkc0rLS1FdHS0cqOz8PBwhIaGYuDAgRaurOFEpNb3fH19ERUVhcTERCxZsuSRffn6+hqztCY7XmVlJbZt24bt27drBamnnnpKeXJPTEyMUcYiIiJ6GPNM3Yx1vDdnTmG2aBo4aUJENs/V1RVRUVGIioqydClm4+XlVa+AQfXn4OCg81hjIiIic2GeaXqYLZoG3tOEiIiIiIiIiEgPTpoQEREREREREenBSRMiIiIiIiIiIj04aUJEREREREREpAdvBGtCmZmZli6ByKQ0f8d37dpl4UqsH/cHzVteXh66dOli6TKILIL7P7J2zDP1x+8zNUcqqes5UNRgKpXK0iUQEZEVCQgIQFpamqXLIDIr5iEiItNJTU1FYGCgpcto6tJ4pomJcC6K6MEvNkFBQfw+EBE1U9z/ky1RqVT8RygR6eA9TYiIiIiIiIiI9OCkCRERERERERGRHpw0ISIiIiIiIiLSg5MmRERERERERER6cNKEiIiIiIiIiEgPTpoQEREREREREenBSRMiIiIiIiIiIj04aUJEREREREREpAcnTYiIiIiIiIiI9OCkCRERERERERGRHpw0ISIiIiIiIiLSg5MmRERERERERER6cNKEiIiIiIiIiEgPTpoQEREREREREenBSRMiIiIiIiIiIj04aUJEREREREREpAcnTYiIiIiIiIiI9OCkCRERERERERGRHpw0ISIiIiIiIiLSg5MmRERERERERER6cNKEiIiIiIiIiEgPTpoQEREREREREenBSRMiIiIiIiIiIj04aUJEREREREREpAcnTYiIiIiIiIiI9OCkCRERERERERGRHpw0ISIiIiIiIiLSg5MmRERERERERER6cNKEiIiIiIiIiEgPTpoQEREREREREenBSRMiIiIiIiIiIj04aUJEREREREREpAcnTYiIiIiIiIiI9FCJiFi6CCKyfXl5eZg1axaqqqqUtsLCQvzyyy/4/e9/r7Xs008/jY8++sjcJRIREREBAMLCwnDu3DmttlOnTsHLywvu7u5Km729PT7++GN06dLF3CUSkXVIc7B0BUTUNHTp0gWXLl3ChQsXdN77+uuvtV4PGzbMXGURERER6ejYsSM2btyo0/7jjz9qve7WrRsnTIiaOV6eQ0RGM3PmTLRo0eKRy02dOtUM1RARERHpFxIS8shlHB0dMXv2bNMXQ0RWjZfnEJHRXLhwAU899RTq2q306tULP/30kxmrIiIiItLl6+uLrKysOnPLuXPn0KNHDzNWRURWJo1nmhCR0Xh7e+PZZ5+FSqXS+36LFi0wa9YsM1dFREREpGvmzJmwt7fX+55KpUKfPn04YUJEvDyHiIyrrgBSWVmJKVOmmLkiIiIiIl3BwcFaN7Cvyd7enj/0EBEATpoQkZEFBwejurpap93Ozg4DBw6Ep6en+YsiIiIieoiHhwcGDhwIOzvdfxJVVVUhMDDQAlURkbXhpAkRGVWnTp0wePBgnQBiZ2eHmTNnWqgqIiIiIl0zZszQuazYzs4OQ4YMwZNPPmmhqojImnDShIiMbsaMGTptIoLJkydboBoiIiIi/fRdNqxSqfhDDxEpOGlCREYXEBCgdV8Te3t7jBo1Ch06dLBgVURERETa2rVrBz8/P63colKpMGnSJAtWRUTWhJMmRGR07u7uGD16tBJARATTp0+3cFVEREREuqZPn648dtje3h5jx45F27ZtLVwVEVkLTpoQkUlMnz5duSFsixYt8OKLL1q4IiIiIiJdkydPhqOjIwD+0ENEujhpQkQm4e/vDycnJwDA888/j5YtW1q4IiIiIiJdrq6umDhxIgDA0dERzz//vIUrIiJrwkkTIjIJV1dX5ewS/mJDRERE1mzatGkAgEmTJsHV1dXC1RCRNVGJ5gI+skkPPyKNiIgMExAQgLS0NEuXQUR1YN4hoqYkNTUVgYGBli6D6ifNwdIVUOMtXLgQgwYNsnQZZMMyMzMRGxuL1NRUo/ZbVVWF1NRUhISEGLVfSwkKCuL3rYlZt26dpUsgonri/pca61F5JyUlBVOnToWDQ/P+JxLzjmkFBQVZugQyEM80sXEqlYozldRou3btQlBQEEyxO7h37x4ee+wxo/drCfy+NT1TpkwBAJ5pQmTluP8lY3hU3mlKmaUx+H0zLW5fm5PGe5oQkUkxfBAREZEtYGYhIn04aUJEREREREREpAcnTYiIiIiIiIiI9OCkCRERERERERGRHpw0ISIiIiIiIiLSg5MmRERERERERER6cNKEiIxq4MCBWLp0qaXLsDo5OTlYs2YNAKCyshJr165FREQEQkJCMGzYMOzevbtB/WZlZeHFF19Eu3bt0L59ewQHB6OgoEBrmaSkJPTr1w+tWrVC3759kZyc3ODPYU3jVVVVYfny5bh69WqD+yciImoI5h39TJF3jJmb8vPzkZycjKCgIDz33HN6l9myZQsCAwOxcuVKhIaGYufOncp7zB7NlJBNAyCpqamWLoNsXGpqqhhrdzB16lSJjIw0Sl8NceXKFZP13dDv27FjxyQkJETKy8tFRCQyMlJ+/PFH5f34+HgBIB9++KFB/WZlZcmkSZNk7969cvr0aZkxY4YAED8/P2WZ5cuXy/Tp0yUxMVEWLFggzs7OAkDi4+MN/hzWON7t27dl8uTJkpuba3D/IiIBAQESEBDQoHWJyHyYd8gYmHfqx9ryjrH60bh8+bIAEB8fH5333n33XfH09JTCwkIRESksLBRPT0+Ji4tTlmls9uD+zObs4qSJjeOXjozBmCHCkn755RcZOnSoyfpvyPctKytLunbtKrdu3VLaunTpIunp6crrO3fuCAAZMGCAQX3HxcWJWq1WXldUVIibm5u0bNlSRB4EqmnTpmmtc/jwYQEg3bt3N2gsaxxP48yZM+Lr6yt37941eAxOmhDZBuYdMgbmnfqxtrxjrH5q0jdpcvnyZWnRooWsWrVKqz0qKkpcXFzk5s2bSltjsgf3ZzZnFy/PIaIm4erVq5g4cSJ+/fVXS5eiEBFMnz4dL730Etq0aaO0V1dXY+/evcrrmzdvAgA8PDwM6j88PBzOzs5abZWVlZgzZw4A4NKlS8opshpjxoxB+/btcePGDYPGssbxNJ599ll4e3tjyZIlBo9BRERkS5pj3jFWP4+SkpKCiooK+Pn5abWPHDkSarUaSUlJShuzR/PCSRMiMorq6mqkpaVh9uzZGD58OABg3759CAsLg4eHB4qKijB79my0a9cOvXv3xg8//AAAOHHiBBYvXgwvLy9cv34dAQEBaNu2LXr37o09e/YAADZt2gQ7OzuoVCoAQElJCdauXavVtnXrVvz888+4du0aXnnlFaWuo0ePwsPDA8ePHzfn5gDw4POfOnUK48aN02o/fPgwVqxYobWcg4MDIiMjGzXem2++idjYWMTGxgIABg8ejI4dO+osV15ejqFDhzZqLGsYr6axY8di06ZNyM3NbfQ4REREtWHe0WXqvGOq3PSwb7/9FgDQpUsXrXbN5MyZM2e02pk9mhFLn+tCjQOe3kVGYKzTVR++RjQvL09atmwpACQqKkouXbokKSkpyimVVVVVcuDAAeW+F/Pnz5fjx4/Ljh07pFWrVgJAMjIyRETE29tbp8aH26DnVMvPP/9cXFxcZP/+/Y3+fIZ+34KDg0WlUklFRUWty5SXl0v37t0lJSWlwXXt3btXhg0bJgDEy8tLNm/eXOuyGRkZ4uzsLKdOnWpS450+fVoA6JxS+yi8PIfINjDvkDEw79SPteYdY/ajbxv27dtXAEhZWZlWu1qtFgAyaNAgrfaGZg/uz2wO72li6/ilI2Mw5jW+Dx+Enn76aZ2+O3bsKE5OTsrrHj16CAApLS1V2mJjYwWATJ06VUREfHx8dPp5uE3fAVBEpLKysnEfqkb/hnzfPD09xc3Nrc5lNmzYIOvWrWtUXYWFhZKVlSUJCQni4uIiAGTr1q06y1VWVsrw4cNl586dTW68/Px8ASDjx483qG9OmhDZBuYdMgbmnfqx1rxjzH70bUPNDzT37t3Tai8rKxMA0r9/f632hmYP7s9sDu9pQkSmpTmdtCZ3d3fcv39feW1n92BX5OLiorT5+/sDePDousayt7dvdB8Nce3aNbi7u9e5zIULF7Bw4cJGjePm5oaePXvi1VdfxUcffQQA2LZtm85y77zzDvz8/DB16tQmN56bmxsA4Pr1640ai4iIqCGYd0yfd4zZjz4+Pj4AgKKiIq32wsJCAEDnzp212pk9mg9OmhCRVdIcmIx9ky9zsre3R1VVVa3vl5WVoV+/fkYd84UXXgAAODo6arUfOHAArq6uRr/+11rG0xdWiYiIrB3zTv2ZIjfV1KtXLwBAfn6+VntBQQEAYMiQIVrtzB7NBydNiMgq3bp1CwAwatQoAL8dmMrLywE8uFP7nTt3tNZRqVSorKzU6auuA7kpderUSefXipqcnZ0RHBxs1DE1B/bx48crbV9++SXy8vKwbNkyrWUzMzObxHjAb78CPfHEE40eg4iIyFyYd+rPFLmpphkzZsDNzQ1Hjx7Vaj9y5AgcHR0REhKi1c7s0Xxw0oSIjObu3bsAgOLiYqXt3r17OsuVlJQAgM4Bv+bBPj09Hf3790dYWBiA306ZfP/993H+/HnExcUpp7wePnwY1dXV8Pb2RkFBAa5cuaL0c/DgQbi5ueGLL74wxkc0yPDhw1FSUqJsl4eFh4djwoQJOu1r1qxBr1698Mknn9TZ/7p167BlyxYlTN2/fx/Lli1DUFAQ5s2bBwD46quvsHr1alRVVSExMRGJiYlISEjAokWLcOjQIZsfT0Pz+MGHfwUiIiIyNuYdbabOO8bup6ysDIDuJJO7uztWrFiBDRs2KJ+lpKQEGzduxMqVK3WeqsPs0Xw4WLoAImoa1Go1oqOjATw4rXHdunUoLy/HxYsXAQBRUVGYP38+kpOTcfXqVQBAZGQk3nrrLaWP2NhYzJ49G9XV1SgoKMDXX38NB4cHu6mYmBjk5+dj7dq1OHnyJBISErBnzx54enqiqKgIlZWVmDJlCrZu3Yrvv/9eOc3VyckJrVu3hpOTkxm3xgMzZ85EUlISMjMzMXr0aJ337927pzdk5ebm4uzZs1i8eHGd9wMpLi7G+vXrleUcHR0xb948+Pn5AXhwZoe/vz/UajWOHDmita5KpcL58+dteryaMjIyYG9vj8DAwFr7IyIiaizmHV2mzjvG7OfYsWPYuXMnAODixYv44IMPMGbMGPTp0wcAsHTpUrRr1w5z585F165dkZ2djSVLliA0NFSnL2aP5kMlImLpIqjhVCoVUlNT+WWlRtm1axeCgoJgqd1Bz549cfbsWYuNX18N+b5NmDABPXr0wLp16wwaKzs7GzNnzsSJEycMLbNBbH08f39/PPHEE9i4caNB602ZMgUAkJaWZpQ6iMg0mHfIGJh36scW8465cwzQ8OzB/ZnNSePlOUREJpScnIxDhw4ZdGd1tVqN+Ph4bN682YSVNZ3xTp48iezsbKxZs8Yo/REREZFhLJl3zJ1jAGaP5oaTJmSTHr4hFtm20tJSrf82JR06dMCnn36K1157DWq1ul7r5ObmIjo6Gr6+viauzvbHKygoQFRUFNLT09GqVSsjVEdEZD2Yd5oW5h1txsoD5s4xzB7NDydNyKKSkpLQr18/tGrVCn379kVycnKty96/fx/R0dF47rnn0LZt2waPmZaWhueffx6///3vMXbsWLzwwguYN28eYmJisGTJkgb321h1bYv09HSMHz8eKpUKKpUKI0eOxMiRI/GHP/wBL7zwApKSkpS7rNuS0tJSvPHGG8qNzMLDw816WqW5+Pr6IioqComJifVe3pwHYVsdr7KyEtu2bcP27dt1bs5GRGRN8vPzkZycjKCgIDz33HN1LtvU805d24J5x7ZZKu+YM8cwezRTQjYNgKSmplq6jAZZvny5TJ8+XRITE2XBggXi7OwsACQ+Pr7WdcrKyqRNmzbSkL+6v/76q/zpT3+S7t27y8mTJ5X26upqSUlJkbZt28qcOXMa9Fkaqz7b4urVqwJAvLy8lLbq6mrZv3+/eHt7y1NPPSU///xzg8ZPTU1t0DZtbmz5+0b6BQQESEBAgKXLIKJHaAr738uXLwsA8fHxeeSyTTXvaNS1LZh3LK8pfN+sGbevzdnFM03IIvLy8nDlyhX84x//wNy5cxEbG4vPPvsMABAXF1freo899hg6dOhg8HgighdffBFnzpzByZMn8cc//lF5T6VSYdq0afj0008tcrpkfbdF586dAUDrrugqlQoTJ07EN998g7t378Lf31/vXcWJiIjIsjRPOamPpph3aqprWzDvEJG14aQJWcSlS5d0bpw0ZswYtG/fHjdu3DD6eHv27EFGRgaWL1+ONm3a6F1m+PDhypM0zMkY26JTp0547733cOHCBd6QioiIqJmy5rxjDMw7RGQJnDRpZkpLS/H+++9jxowZWLBgAUaMGKF1NkNxcTGWLVuGFStWICIiAmPHjkVERASKiooAAPv27UNYWBg8PDxQVFSE2bNno127dujduzd++OEHAMDu3bvRtm1bqFQqREZGKn3//e9/h729PTZt2oTBgwejY8eOOvWVl5dj6NChyuuysjJEREQgLCwMkZGReP3113V+HTl69Cg8PDxw/PjxWj/3nj17AAB+fn51bp/Jkydb7bZ4lICAANjb2+Nf//pXvdchIiJqiqwl79RXc8g7xsK8Q0RmZ+kLhKhxYMA1cRUVFTJixAiZMWOGVFdXi4hIcnKyAJD9+/dLSUmJ9OjRQ95++21lnRs3bkiPHj2kW7duUlRUJHl5edKyZUsBIFFRUXLp0iVJSUkRADJgwABlvfj4eAEg//znP5W2y5cvS0hISK31ZWRkiLOzs5w6dUpERCorK2XAgAESGhqqLHPhwgVxcHDQuh71888/FxcXF9m/f3+tff/hD38QAHLnzp16bStr2xYaeMS10J06dZK2bdvW6zPWxGt868eQ7xvZBt7ThMg2NJW8o+843lzzTl2ZhnnHsph3TIvb1+bscjDLzAxZhfj4eBw7dgznzp2DSqUCAMyYMQMAMGTIEKxevRrZ2dkICwtT1mnfvj1WrlyJmTNnIjo6GjExMXjyySdx7tw5vP766wCAadOmISIiAv/5z3+U9cLCwvDBBx/g73//O8aNGwcA2LRpU613a6+qqsLrr7+OLVu2oF+/fgCADRs24OTJk9i6dauyXLdu3dCtWzdkZ2crbf7+/iguLoa9vX2tn13znlqtRuvWrR+5raxtW9SXg4OD8v+2IXbt2tXgdZuLzMxMS5dARpSXl8e73xM1Mdacd/RprnmnMZh3TI95h+g3nDRpRo4dOwYAWv9AsLe3x+zZswEAGRkZAKDzyK5hw4YBAL777jsA0HuQcnd3x/Xr15XXLVq0wIIFC7BkyRLk5ubCw8MD586dQ9++ffXW9s4778DPzw9Tp05V2jSnXXp6emota2ene1VZXQECAJ555hmcOHEC//3vf/HEE0/UuSxgfduiPioqKnD9+nWMGjXKoPVqCgoKavC6zUVsbCxiY2MtXQYZUUBAgKVLICIjsua8o09zzTsNxbxjHsw7RL/hPU2aEc2BLScnR+/7moPzxYsXtdo199t4/PHHDRrv5ZdfhqurKxISEvDZZ5/V+g+TAwcOwNXVVesaWAC4evUqAODWrVsGjavP8OHDAQAnTpyo1/LWti3q48iRIygvL3/kdcx1ERH+qeMPAKSmplq8Dv4x3h9OmBA1Pdaad2rTHPNOYzDvmP4PwLxj6u1LtoWTJs1Inz59AABRUVFaX9hLly7hn//8p/KrwsGDB7XWu3LlCgAYPKPfunVrvPzyy9iyZQtSU1MxadIknWW+/PJL5OXlYdmyZVrtmZmZ8PHx0VuPPlVVVXW+P336dPTv3x9xcXEoKCjQu8z9+/exbds2ALC6bfEo5eXleP3119GvXz+Eh4cbVBsREVFTYo15py7NLe80BvMOEVkCJ02akeXLl8PV1RVpaWkYNWoU1q9fjzfffBOrVq3CuHHjsHTpUvj6+iI+Ph7Xrl1T1ktMTMTgwYMxb948AMC9e/d0+i4pKQEAVFZWarWHh4fj7t276NevHxwctK8G++qrr7B69WpUVVUhMTERiYmJSEhIwKJFi3Do0CEsWbIEDg4OeP3113H48GGUlZXh6NGjyM/PB/DbryIHDx6Em5sbvvjii1o/u52dHVJSUvDYY49hyJAh2Lt3rxI8NP1OmDABTz/9NABY3bbQ1KlvzNOnT2P06NEoLCzE9u3bdfomIiJqTqwt72hojuMPT3w0p7zzqG1R8z3mHSKyFtzbNCNeXl44ceIEFi9ejH//+984d+4cpkyZgr/+9a9QqVRwdnZGZmYm3nvvPcyaNQu9e/eGvb092rZtiyNHjsDBwQHr169XDt5RUVGYP38+kpOTlVNLIyMj8dZbb+Gxxx4D8OD63Pnz5+OVV17RqiUzMxP+/v5Qq9U4cuSI1nsqlQrnz59Ht27dcOTIEaxYsQJTpkxB+/bt8Ze//AV9+/bFM888g9zcXHTt2hVOTk5o3bo1nJyc6vz8Pj4++Omnn7B+/XokJSUhIiICrq6ucHBwwIQJE7Br1y60adMGAKxuW2RkZCA5ORnAg/D0pz/9CU5OTnByckKLFi0QFBSEWbNmwdXV9ZF/D4iIiJoya8o7GseOHcPOnTsBPDiOf/DBBxgzZgz69OmDPn36NIu8U59twbxDRNZIJbywyqapVCqkpqYiMDDQ0qWQDdu1axeCgoJ4neUj8PvW9EyZMgUAkJaWZuFKiKgu3P+SMTDv1A+/b6bF7Wtz0nh5DhERERERERGRHpw0ISIiIiIiIiLSg5MmRERERERERER6cNKEiMgMcnJysGbNGgAPnjSwdu1aREREICQkBMOGDcPu3bsb1G9WVhZefPFFtGvXDu3bt0dwcLDOYya3bNmCwMBArFy5EqGhocoN+Gx9vKqqKixfvly5GSERERFZH1NkIHNlKWYNAgAI2TQAkpqaaukyyMalpqaKJXcHV65csYm+G/p9O3bsmISEhEh5ebmIiERGRsqPP/6ovB8fHy8A5MMPPzSo36ysLJk0aZLs3btXTp8+LTNmzBAA4ufnpyzz7rvviqenpxQWFoqISGFhoXh6ekpcXJzBn8Max7t9+7ZMnjxZcnNzDe5fRCQgIEACAgIatC4RmQ/zDhkD8079GPP7ZqoMZM4s1dis8TDuz2zOLk6a2Dh+6cgYLBkifvnlFxk6dKhN9N2Q71tWVpZ07dpVbt26pbR16dJF0tPTldd37twRADJgwACD+o6LixO1Wq28rqioEDc3N2nZsqWIiFy+fFlatGghq1at0lovKipKXFxc5ObNmzY9nsaZM2fE19dX7t69a1D/Ipw0IbIVzDtkDMw79WOs75spM5C5spRGY7LGw7g/szm7eHkOEVnM1atXMXHiRPz666821Xd9iQimT5+Ol156CW3atFHaq6ursXfvXuX1zZs3AQAeHh4G9R8eHg5nZ2ettsrKSsyZMwcAkJKSgoqKCvj5+WktM3LkSKjVaiQlJdn0eBrPPvssvL29sWTJEoP6JyIiMoemnnf0MXUGMleW0mDWaN44aUJEDVJcXIxly5ZhxYoViIiIwNixYxEREYGioiIAwKZNm2BnZweVSgUAKCkpwdq1a7Xatm7dip9//hnXrl3DK6+8AgA4ceIEFi9eDC8vL1y/fh0BAQFo27YtevfujT179jSqbwA4evQoPDw8cPz4cZNvo3379uHUqVMYN26cVvvhw4exYsUKreUcHBwQGRnZqPHefPNNxMbGIjY2FgDw7bffAgC6dOmitZwmUJw5c8amx6tp7Nix2LRpE3Jzcxs1BhERUU3MOw1j6gxkrixVE7NGM2bpc12occDTu8gIDD1dtaSkRHr06CFvv/220nbjxg3p0aOHdOvWTYqKikRExNvbW6ffh9sAiI+Pj4iIVFVVyYEDB8TZ2VkAyPz58+X48eOyY8cOadWqlQCQjIyMBvWt8fnnn4uLi4vs37+/3p+3Zn+GfN+Cg4NFpVJJRUVFrcuUl5dL9+7dJSUlxeB6NPbu3SvDhg0TAOLl5SWbN28WEZG+ffsKACkrK9NaXq1WCwAZNGiQTY9X0+nTpwWAzqVBj8LLc4hsA/MOGQPzTv0Y4/tmrgxkrH5MmTUexv2ZzeHlOURkuNWrVyM7OxthYWFKW/v27bFy5Urk5uYiOjoaANCiRQuddfW1adjZ2WHChAnKmQmrV6/G0KFDERwcjPfeew8AEB8f36C+Nfz9/VFcXIyJEyc+ctnGyszMxOOPPw4HB4dal9myZQteffVVTJs2rcHjjBgxAhs2bEBCQgKuX7+Ol19+GR9/7VI2FAAAIABJREFU/DFat24NAMqvURqa1+Xl5TY9Xk0dO3YEAHzzzTcNGoOIiOhhzDsNZ64MZKx+mDWoLpw0ISKDZWRkAABatWql1T5s2DAAwHfffdeo/u3sHuyaXFxclDZ/f38ADx5b11j29vaN7qM+rl27Bnd39zqXuXDhAhYuXNiocdzc3NCzZ0+8+uqr+OijjwAA27Ztg4+PDwAopxBrFBYWAgA6d+5s0+M9vAwAXL9+vUFjEBERPYx5p+HMlYGM1Q+zBtWFkyZEZDDNQf7ixYta7ZoZ+Mcff9zoY2r+wW3oDb4syd7eHlVVVbW+X1ZWhn79+hl1zBdeeAEA4OjoiF69egEA8vPztZYpKCgAAAwZMsSmx6vp4bNbiIiIGot5p+HMlYFMnaVqYtZovjhpQkQG0/zCcvDgQa32K1euAABGjRoFQPeyDBHBnTt3tNZRqVSorKx85Ji3bt0yWt91HcSNqVOnTjpnXdTk7OyM4OBgo46pmaAYP348ZsyYATc3Nxw9elRrmSNHjsDR0REhISE2PV5NmrNZnnjiiUaPQUREBDDvNIa5MpCps1RNzBrNFydNiMhgS5cuha+vL+Lj43Ht2jWlPTExEYMHD8a8efMAQLlc4/3338f58+cRFxeH+/fvA3hw1/Pq6mp4e3ujoKBACSA11TzYp6eno3///sp1xQ3t++DBg3Bzc8MXX3xhzE2i1/Dhw1FSUoK7d+/qfT88PBwTJkzQaV+zZg169eqFTz75pM7+161bhy1btijh6f79+1i2bBmCgoIwb948uLu7Y8WKFdiwYYNSQ0lJCTZu3IiVK1cqT7mx1fFq0jxq0BhnsxAREQHMO41h6gxkrH6YNag+ar8zDxFRLZydnZGZmYn33nsPs2bNQu/evWFvb4+2bdviyJEjyk2/YmJikJ+fj7Vr1+LkyZNISEjAnj174OnpiaKiIlRWVmLKlCnYunUrvv/+e51TUWNjYzF79mxUV1ejoKAAX3/9daP7dnJyQuvWreHk5GTy7TRz5kwkJSUhMzMTo0eP1nn/3r17uHfvnk57bm4uzp49i8WLF2Pq1Km19l9cXIz169cryzk6OmLevHnw8/NTllm6dCnatWuHuXPnomvXrsjOzsaSJUsQGhraJMbTyMjIgL29PQIDA2vtj4iIyBDMOw1n6gxkrH6YNag+VCIili6CGk6lUiE1NZVfXmqUXbt2ISgoCNayO+jZsyfOnj1rNfVoNOT7NmHCBPTo0QPr1q0zaKzs7GzMnDkTJ06cMLTMBrH18fz9/fHEE09g48aNBq03ZcoUAEBaWppR6iAi02DeIWNg3qkfY33fLJ2BrCVrPIz7M5uTxstziIhMKDk5GYcOHTLoTutqtRrx8fHYvHmzCStrOuOdPHkS2dnZWLNmjVH6IyIiosazZAZi1iBj4qQJEVmd0tJSrf/asg4dOuDTTz/Fa6+9BrVaXa91cnNzER0dDV9fXxNXZ/vjFRQUICoqCunp6TqPhCQiIrJmTSnv6GPJDMSsQcbESRMishqlpaV44403lBuZhYeHm+1yEVPy9fVFVFQUEhMT6728OQ/KtjpeZWUltm3bhu3btys3mSUiIrJ2TTXv6GOpDMSsQcbEG8ESkdVwdXVFVFQUoqKiLF2K0Xl5eWHJkiWWLqNJ+f/s3XlYlOX+P/D3MKCCGy64oyJpuJUd66i5L2Wl4RKLIJAetxRDTRM1PacsTDupeKFmqKi4JGLunTJNyhOKv07aJhIqakCAiiAIsn9+f/jliZFhGZhhZuD9ui4vL565n+f+PM/czHzmw/3cY2lpCX9/f2OHQUREpJPanO9oY845EHMNAjjThIiIiIiIiIhIKxZNiIiIiIiIiIi0YNGEiIiIiIiIiEgLFk2IiIiIiIiIiLTgQrC1wPr16xEeHm7sMMiMJSQkAABcXV2NHInp4+9b7RIVFYX+/fsbOwwiqgS+/lJ1Md+pPP6+Ef1FJSJi7CCo6viiT6YsOTkZly5dwssvv2zsUIjKNGDAALz11lvGDoOIysF8h2rCl19+iWeeeQZt2rQxdihUy7311lsYMGCAscOgygln0YSIDObAgQNwd3cHX2aIiIjI1KlUKoSFhcHNzc3YoRCR6QjnmiZERERERERERFqwaEJEREREREREpAWLJkREREREREREWrBoQkRERERERESkBYsmRERERERERERasGhCRERERERERKQFiyZERERERERERFqwaEJEREREREREpAWLJkREREREREREWrBoQkRERERERESkBYsmRERERERERERasGhCRERERERERKQFiyZERERERERERFqwaEJEREREREREpAWLJkREREREREREWrBoQkRERERERESkBYsmRERERERERERasGhCRERERERERKQFiyZERERERERERFqwaEJEREREREREpAWLJkREREREREREWrBoQkRERERERESkBYsmRERERERERERasGhCRERERERERKQFiyZERERERERERFqwaEJEREREREREpAWLJkREREREREREWrBoQkRERERERESkBYsmRERERERERERasGhCRERERERERKQFiyZERERERERERFqwaEJEREREREREpIWlsQMgotohPz8fDx480NiWlZUFAEhLS9PYrlKpYGtrW2OxEREREZWUnp4OESm1PSsrq1Te0qhRI1hZWdVUaERkYlSi7dWCiEhHKSkpaN++PQoLCytsO3z4cJw5c6YGoiIiIiIqbcSIEYiIiKiwnVqtRmJiIlq3bl0DURGRCQrn7TlEpBetW7fGkCFDYGFR/suKSqWCh4dHDUVFREREVJqHhwdUKlW5bSwsLDBkyBAWTIjqOBZNiEhvvL29K2yjVqsxceLEGoiGiIiISDsXFxdYWpa/UoFKpYKPj08NRUREpopFEyLSm9dee63cBEStVuOll15CixYtajAqIiIiIk3NmjXDiy++CLVaXWYbCwsLTJgwoQajIiJTxKIJEelNkyZN8PLLL5dZOBEReHl51XBURERERKV5eXmhqKhI62OWlpYYM2YMmjZtWsNREZGpYdGEiPTKy8urzMVg69Wrh7Fjx9ZwRERERESlOTs7o379+lofKyws5B96iAgAiyZEpGdjx46FjY1Nqe1WVlaYMGECGjZsaISoiIiIiDTZ2NhgwoQJWr9O2NraGq+88ooRoiIiU8OiCRHpVYMGDTBx4sRSCUh+fj4mT55spKiIiIiISvP09ER+fr7GNisrK7i4uMDa2tpIURGRKWHRhIj0TlsC0qRJE7zwwgtGioiIiIiotNGjR5datyQ/Px+enp5GioiITA2LJkSkd6NGjULz5s2Vn62srODh4YF69eoZMSoiIiIiTVZWVpg0aZJGjmJra4uRI0caMSoiMiUsmhCR3llaWsLDw0O5RYd/sSEiIiJT5eHhgby8PACPiiheXl5lfhMgEdU9LJoQkUF4eHgot+i0bt0agwYNMnJERERERKUNHjwYrVu3BvDoDz2TJk0yckREZEpYNCEig3j++efRvn17AICPjw8sLPhyQ0RERKbHwsIC3t7eAIC2bdvi+eefN3JERGRKSs07S0hIwLlz54wRCxHVMs899xwSExPRokULHDhwwNjhEFEt4ObmZrBjnz9/HvHx8QY7PhGZrpYtWwIA+vXrh/DwcCNHQ0TGoi3PUImIlNxw4MABuLu711hQRERERJX1WNqiV66urjh48KDBjk9ERESmTUueEV7mCkeGTEqIqO44ePAgXFxcjB1GraJSqRAWFmbQv7ibO1dXVwDgXwtrkZr6o46LiwvHDVEdxZzFtBW/D/BzavmYJ1ZNeXkGFxkgIoNi8kFERETmgDkLEWnDogkRERERERERkRYsmhARERERERERacGiCRERERERERGRFiyaEBERERERERFpwaIJEREREREREZEWLJoQEREREREREWnBogkRUR3Vv39/LF682NhhmKSrV69i7dq1AICCggKsW7cOCxcuhKenJ4YMGYKDBw9W6bjR0dEYP348WrZsCTs7O3h4eCApKUmjTUhICNzc3LB8+XLMmDEDn332WZXPw5T6KywsxJIlS5CYmFjl4xMREekDcyDtDJH/1FQeZdA8Qx4TFhYmWjYTEZGJACBhYWHVPs6kSZNkxYoVeoioauLj4w12bBcXF3FxcanSvt9++614enpKXl6eiIisWLFCfvnlF+XxoKAgASAff/yxTseNjo6WCRMmyOHDh+XSpUvi7e0tAGTkyJFKm5UrV0rnzp0lLS1NRETS0tKkc+fOsmHDBp3PwxT7u3fvnkycOFHi4uJ0Pn5N5CfVGTdERGRY+nwfqM05UFXzREPlPzWZRxkozzjAogkRkZnRV9HEmG7cuCGDBw822PGr+uE3OjpaOnbsKKmpqcq2Dh06yOnTp5Wf79+/LwCkX79+Oh17w4YNkp2drfycn58vtra20qhRIxER+eOPP8TKyko+/PBDjf0CAgLExsZG7t69a9b9Ffv555+lV69e8uDBA52Oz6IJEVHdVls+pxo6B6pKnmjI/Kem8qhiBsgzDvD2HCIiqlGJiYkYO3Ys7ty5Y+xQNIgIvLy8MHXqVDRv3lzZXlRUhMOHDys/3717FwBgb2+v0/H9/PxgbW2tsa2goADTpk0DAOzZswf5+fkYOXKkRpsRI0YgOzsb27dvN+v+ij311FNwdHTE22+/rdPxiYiIzJ0p5kCGzn9qKo8qZog8g0UTIqI6pqioCOHh4ZgyZQqGDh0KADh27BhmzZoFe3t7pKenY8qUKWjZsiV69+6NH3/8EQAQFRWFRYsWwcHBASkpKXBxcUGLFi3Qu3dvHDp0CACwdetWWFhYQKVSAQAyMzOxbt06jW07d+7E5cuXkZycjNmzZytxRUREwN7eHmfPnq3Jy6E4duwYLl68iJdeeklj+8mTJ7F06VKNdpaWllixYkW1+vvnP/+JwMBABAYGAgC+//57AECHDh002hUnFT///LNZ91fS6NGjsXXrVsTFxVWrDyIiIl0wByrN0PlPTeVRJek9z9BhWgoREZkA6OH2nD/++EMAiJOTk4iIJCQkSKNGjQSABAQEyK1bt2TPnj3K9MnCwkI5ceKEWFtbCwB588035ezZs7Jv3z5p3LixAJDIyEgREXF0dCz1PvL4tpJ9Fzt69KjY2NjI8ePHq3VuIlW7zcLDw0NUKpXk5+eX2SYvL0+eeOIJ2bNnT5VjO3z4sAwZMkQAiIODg2zbtk1ERPr06SMA5OHDhxrts7OzBYAMGDDArPsr6dKlSwKg1K1B5eHtOUREdZu+3gdqew6ka55YU/mPvo5jhDyDa5oQEZkbfRRNio9T8k37ySefLPX637p1a6lfv77yc7du3QSAZGVlKdsCAwMFgEyaNElERJycnEod5/Ft2hIGEZGCgoLqndT/qcqH386dO4utrW25bbZs2SLr16+vTmiSlpYm0dHRsnHjRrGxsREAsnPnTiUByMnJ0Wj/8OFDASB9+/Y16/5K+vPPPwWAvPLKK5U+LosmRER1mz7fB2pzDqRrnlhT+Y++jmOEPINrmhAR0SPFU0dLatasGXJzc5WfLSwevW3Y2Ngo25ydnQE8+pq66lKr1dU+RlUlJyejWbNm5ba5fv065s+fX61+bG1t0b17d/j6+uLTTz8FAISGhsLJyQkAkJ6ertE+LS0NANCuXTuz7u/xNgCQkpJSpT6IiIj0qS7nQDWV/+jrOMbIM1g0ISKiain+cK3rgl6mRq1Wo7CwsMzHHz58iGeeeUavfY4bNw4AUK9ePfTs2RMA8Oeff2q0SUpKAgAMGjTIrPsrSVtySkREZG5qQw5UU/mPofOokvSdZ7BoQkRE1ZKamgoAGDVqFIC/3qjy8vIAPFqV/f79+xr7qFQqFBQUlDpWeW/ahta2bdtSsy5Ksra2hoeHh177LC5QvPLKK/D29oatrS0iIiI02pw5cwb16tWDp6enWfdXUvFsljZt2lS7DyIiImOpDTlQTeU/hs6jStJ3nsGiCRFRHfTgwQMAQEZGhrItJyenVLvMzEwAKPXmXvKN/fTp0+jbty9mzZoFAMptHx988AGuXbuGDRs2KNNbT548iaKiIjg6OiIpKQnx8fHKcb744gvY2triq6++0scp6mzo0KHIzMxUrs3j/Pz8MGbMmFLb165di549e2L//v3lHn/9+vUICQlRkqfc3Fz4+/vD3d0dc+fORbNmzbB06VJs2bJFiSEzMxPBwcFYvny58i035tpfScVfN6iP2SxERES6YA6kydD5j76OY8w8g0UTIqI6Jjs7G6tWrQLw6NaM9evXY82aNbh58yYAICAgABkZGdiwYQMSExMBACtWrNBIKAIDA5Gamoo7d+4gKSkJ3333HSwtLQEAa9asQb9+/bBu3Tr4+vpizJgx6NmzJ7y9vZGeno6CggK4urqiSZMm+OGHH5Rj1q9fH02aNEH9+vVr6Epo8vHxgYjg/PnzWh/PycnRmlTFxcUhJiYGixYtKvf4GRkZ+PDDD+Hg4IA5c+bA398fc+fOxf79+5W/TC1evBhLlizBnDlzsHz5ckybNg1vv/22xtfymXN/xSIjI6FWq+Hm5lbuMYmIiPSJOVBphs5/9HUcY+YZKhGRkhsOHDgAd3d3PLaZiIhMhEqlQlhYmFE+cHbv3h0xMTEm/x7h6uoKAAgPD9dpvzFjxqBbt25Yv369TvvFxsbCx8cHUVFROu1XVeben7OzM9q0aYPg4OBK71MT+UlVxw0RERmesT+nmksOVJU80dj5j4nnGeGcaUJERPR/duzYgf/85z86rbaenZ2NoKAgbNu2zYCR1Z7+Lly4gNjYWKxdu1YvxyMiIqLqMWb+Yw55Ro0VTW7fvo3w8HBlOpQuSt5vVtN901/M6To+vuCSoXBcmwZei5qTlZWl8X9t06pVK3z++edYsGABsrOzK7VPXFwcVq1ahV69ehk4OvPvLykpCQEBATh9+jQaN26sh+jMA98rzJ85XUfmQHULr0XNqc05kDHzH7PIM+QxYWFhomVztVy5ckV8fX0FgDg5OVVqn4KCAlm9erUMGjRILC0ta7RvKq2869ivXz95++23DdLvtm3bpE+fPtKoUSN5+umnJSQkpMy2OTk5EhAQIAMGDBC1Wl3lPouKimTHjh0yceJEee6552TkyJEybtw4mTlzpqxbt04GDx4sIqY3rg8dOiSurq4CQADIt99+W+b+kZGRSrvXXntNIiIiqhzL477//nsZPXq0ABALCwt54YUXZPjw4TJ48GCZO3eupKSk6K0vEdMcm6dOnZKXX35ZucbDhw+X4cOHy7PPPivOzs6ybds2yc3NrXLfACQsLEwfp1FpDx48kGXLlinn9I9//EPOnz9fozHowsXFRVxcXKq8f1xcnHz00Ud6jIjy8/Nl9erVkpGRUaX9DZGfPK6640YbU3uvIN0Z630mMTFRQkJCxM3NTQYMGFBuW+ZAzIFMZWwaMgeqifcBbcwtB6pOnmjO+Y8B84wDNVI0EXn0Yq7rm/bDhw+lefPm1Y6nKn1TaWVdx0mTJsmKFSv03t+SJUvEy8tLNm3aJPPmzRNra2sBIEFBQWXuU90xEx8fL8OHD5cePXrIuXPnNB47fvy4dOjQQeP8TW1cZ2dnKy/ozs7OZe7r4eEhNjY2AkCSk5OrFYc2iYmJAkC6du2qbEtJSZGRI0eKra2t/O9//9Nrf6Y4NouvgYODg7KtqKhIjh8/Lo6OjtK1a1e5fPlylfo3RtHE3Bjiwy8Zl7kWTURM772CdFfT7zPF/vjjj0o/f8yBmAOVZMyxaagcyFhFE3PDPLFqyiua1NjtOVVZCbhBgwZo1aqVUfqm0sq6jp999hlWrlyp174SEhIQHx+P3bt3Y86cOQgMDMSRI0cAABs2bChzv+qMGRGBl5cXYmJiEBUVhQEDBmg8PnbsWJw6dUrjOpjauLa2tgYADBw4ECdOnMC1a9dKtUlOTsa9e/fQsWNHAEDr1q2rHcvj2rVrBwBQq9XKtlatWmHDhg1IT0/X+xRSUxybxdegZGwqlQpjx47Ff//7Xzx48ADOzs5aVxEnotrF1N4rSHc1+T5Tkr29faXbMgdiDlSSMccmcyCqbbgQLJmkW7dulVq858UXX4SdnR1u375tkD6Dg4Px3Xff4YMPPijzHjgnJye89957Bulfn+bPn4+ioiKtBabg4GDMnj3bCFEBnTp1AgDlK9zMkT7GZtu2bfH+++/j+vXrXAyTiIiMjjmQ4dWGHEgfmAOROdJb0WTjxo3w9vbGnDlz0KBBA6hUKuVfWTIyMuDv74+lS5di4cKFGD16NBYuXIj09PRSba9duwZnZ2c0b94cf//73/Htt98qj129ehWurq5YsmQJfHx8MGTIEPz666/VOp/s7Gzs3bsXnp6eGDhwIKKiovC3v/0NnTt3RmRkJGJjYzFhwgTY2dmhe/fu+PHHHzX2Ly+mX375BS+++CJUKhWcnZ1x7949LF68GB07dsTu3bsrHWNUVBQWLVoEBwcHpKSkwMXFBS1atEDv3r1x6NAhpV1lrrMuz0WxoqIihIeHY8qUKRg6dCgA4NixY5g1axbs7e2Rnp6OKVOmoGXLlujdu3epa1TemBk4cKDW6n9eXh4GDx6s/Pzw4UMsXLgQs2bNwooVK7Bs2bJSizNFRETA3t4eZ8+eLfd6fvHFFwCAV155pdx248aNK/dxUxjXEyZMQKdOnbBjxw6NfvPz83Hy5Em8+uqrZe5ryLH7//7f/wPw6K9Alb1W5jo2K+Li4gK1Wo2vv/660vsQkWliDsQcCNDf+0xlMQfSjjmQ8cdmRZgDkdnR4V6eMgUFBYlarZbU1FQREfnwww8FgCxcuFCjHUrc95aZmSndunWTd999V3n89u3b0q1bN+nSpYukp6eLiIiTk5MAkPnz58upU6fk008/lYYNG4parZZffvlFRES6du0qjo6OIvJoARhbW1vp1atXmX1XRlFRkVy7dk0ASNOmTeWLL76Q6OhoASCdO3eWf//733L//n25dOmSAJBhw4Zp7F9RTFlZWdKjRw9xcHCQ3NxccXZ2ltjY2ErHV1hYKCdOnFDWUnjzzTfl7Nmzsm/fPmncuLEAkMjIyEpd58o+F9qu4+P3MyYkJEijRo0EgAQEBMitW7dkz549AkD69eun7FfZMVNSZGSkWFtby8WLF0Xk0YJi/fr1kxkzZihtrl+/LpaWlhpj+OjRo2JjYyPHjx8v95ra29uLra2t1sfOnz8vH3/8sfIvMDBQsrKySl0TUxjXxef+8ccfCwCNxZz2798vH3/8sUYMj9PX2AUg3bp1k8LCQklNTZUjR45Ip06dpEmTJhITE1Orx2ZZMT2ubdu20qJFizIfLwt4r2qFuKZJ7WOqa5owB2IOZOj3GW3PH3Mg5kCmOjYr85hI1XIgrmlSOcwTq6a8NU0sq192AU6dOgURUabzvfrqq1i6dCkiIyPL3Gf16tWIjY3FrFmzlG12dnZYvnw5fHx8sGrVKqxZs0Z5bOXKlcrxc3JyMG/ePKxduxY7d+7E7Nmz0bZtWwCP7h9s0aIFfv/992qdk0qlgqOjI4BH08iKK+/t27fHzZs3sWjRIgBAnz590KpVK/z0008a+1cUk42NDUJDQ9G/f38MGzYMM2fORNeuXSsdn4WFBcaMGQN7e3vExsZi9erVsLGxAfDoq8fmz5+PoKAgODo6VnidraysdHouSnr8fsb27dujffv2+P3337Fs2TIAwOTJk7Fw4UKNa6TrmCksLMSyZcsQEhKCZ555BgCwZcsWXLhwATt37lTadenSBV26dEFsbKyyzdnZGRkZGRr3lmqTmZmJBg0aaH2sf//+sLKywrPPPot69eohISFBud4lmdK4nj59Ot59910EBQVhwYIFsLS0REhICPbv31/ufvocu7GxsVCr1co9zC+++CIWL16Mrl27Yvny5bV2bFaWpaVllf9ys379eoSHh1dp37ogKioKAODq6mrkSEhfEhISjB2CVsyBmAMVM9T7jDbMgcrHHMh4Y7OyqpMD8b29YswTdVdenqGX23NeeOEFFBUVKVP7il90R4wYUeY+xb98j983OWTIEADAuXPnNLaXbDd+/HgAQHR0NABgwYIFePXVV7F582YEBAQgNzcX+fn51TmlMmm7z7N58+alpslVJqa+ffvC398fFy5c0PnDVjELi0dPYck3L2dnZwCPphhW5jrr+lxURNsLYLNmzZCbm6v8rOuYee+99zBy5EhMmjRJ2VY8pa9z584abYuvSUkVJQsA0L17dyQnJyMjI0Pr48XPUefOnWFnZ6e1jSmN66ZNm2Lq1KmIj4/H559/jp9//hldunRBs2bNyt1Pn2PXyckJIoKHDx/i1q1b2Lp1q5Jc1OaxWRn5+flISUlBnz59dNqPiEwLcyDmQCUZ4n1GG+ZA5WMOVFpNjc3KYA5E5kYvM03mzp0La2trTJs2DZGRkbh69SpWrlypVDK1KX5Rv3nzJnr27KlsL14roGnTpmXuW9ymeOXrH374Ae7u7ti8eTPmzJmDvXv3VvucqqsyMYkIrl+/Dnt7e3h7e+N///sf6tWrV+2+i1estre3V978yrvODx8+rLCNvukyZk6cOIGGDRvC399fY3vxQlqpqalo3759tWMaPnw4zp8/j6+//houLi6lHi8es9oSksfbmMq49vPzw6ZNm7B+/Xr07t0bCxYsqHCfmhq7lblW5jo2K+PMmTPIy8vDyJEjqxTnggUL4ObmVqV964Liv0Lxryy1x4EDB+Du7m7sMEphDlQac6DyVWXMPI45UMWYA+lOH2OzMqqbA/G9vXwqlYp5YhWUl2foZaZJYWEhfvvtN0RFReHf//43jhw5ghUrVpRb2S6ukhZXMovFx8cDAEaNGlXmvsVtxo4dCwDw8fFBfn4+XnrpJQCPFkAytsrE9NFHH2HRfabNAAAgAElEQVTixIkICQnBb7/9hn/961966Ts1NRXAo2tYmetcneeiqio7Zk6dOoWEhIRSH0rPnz8PJycnrXGX1V9Fli1bhk6dOmHx4sXIzs7W4Wz+YuxxXdym+P8nnngCY8eOxYULF5CYmIgePXoobUVE6zFqauzW5rFZkby8PCxbtgzPPPMM/Pz89Bo/EdUs5kClMQcqX1XGzOOYA5XGHKj69DE2K8IciMySDguglGnlypXi6Ogo27dvl6+++krOnTsnsbGxUlBQoLTJzs5WFhAr/rlXr17SoUMHSUpKUtrNmzdPBg4cKPn5+SIi0r17dwEg9+7dU9rMmTNHxo0bp/zctGlTUalU8vXXX8vevXulVatWAkAuXLgg8fHxpfqurIcPHwoAefLJJ5Vtjo6OAkAyMzOVbZ07dxYAUlhYWOmYoqKixMPDQ+Oc1Gq1fPfddzrFWLyQVclrvWvXLunbt6/k5+dX6jpX9rnQdh0zMzMFgLRr167U9Sipffv2AkA5VmXGzOnTp2XEiBGyceNG5V9QUJAsWLBAli9fLj/99JNYWlpKixYt5KuvvpLs7Gw5c+aMNGnSRADIjRs3RETkxIkT0qhRI/nyyy8rvJ6XLl2Sjh07So8ePeT8+fMaj33//fcCQAYNGqRsM7VxnZSUJADkzz//VLZFREQIgFKLwHXo0EEAyMOHDzW262Ps3rp1SwBIp06dyrzWtXlslhWTiMjFixdlyJAh4uDgINHR0WVen/KAC3xViAvB1j6muhAscyDmQIZ4nylW3G/Xrl01tjMHYg5kqmOzrJhEqp8DcSHYymGeWDXlLQSrl6LJqVOnpHXr1gJA45+dnZ18/vnnEhcXJ35+fsr2wMBASUtLk8zMTFm8eLG8+OKLsnDhQlm8eLGsXLlScnNzNY796quvyrBhw2TmzJni5+cnmzZt0nhz3rRpkzRt2lT+/ve/S1RUlGzYsEGaNWsm48aNk//9739a+65ISkqKvPXWWwJA6tevL6dPn5aTJ08qq5L7+flJamqqBAUFiUqlUlbovnv3boUxbd26Vezs7GT27NlKf8uWLRMAYmtrKzt27Kj0tS9OGD7++GO5e/eu3L59W1avXi0PHjxQ2lTmOlfURttzmJiYKEuXLlW2rVu3TlavXq38/MEHH8j9+/clMDBQ2bZkyRJ5+PBhhWPm3LlzYmNjU+pxAKJSqeT69esiInL27FkZOHCgNG7cWLp06SKrV6+WIUOGyBtvvCHffPONFBYWyqlTp6Rdu3Zy5syZSl3TBw8eSGBgoEycOFGeffZZGTp0qIwcOVJcXV0lLCxMeeMwtXG9a9cuefXVVwWAjB07Vr755hvlWK+99ppy7OjoaHnnnXeUfd3c3CQiIqJS/VZm7F64cEHc3NyU4/v6+kpUVJTWa11bx+b3338v06ZNU7YPGzZMRo8eLc7OzvLaa6/Jpk2bNH5HdcU3w4qxaFL7mGrRhDkQcyB9v88Ui4iIkJkzZwoAsbKyko8++kh++ukn5XHmQMyBTHFsGjIHYtGkcpgnVk15RROViOb8tOJ7eaSMaWva7NixA3fv3sXbb78N4NFUtj///BMRERFYtGgRUlJSKn0s0k337t0RExOj0/NlCjhmyFSZw9hUqVQICwvjvarl4JomtU9V8hNdVWXcmMNrRm3FHIhIv0x9bNbE+0BtwDyxasoZX+HVXgh2zZo1WLJkiXIPKfBocaMOHTpg0KBBelmcSt8q8/VWMTExePLJJ2sgGu0qG6M5MscxQ3UDxyYR6cIcXzOYAxmXOY4Zqhs4NonKVu2FYL///nsAj74vvuQv2cWLF7FkyRLs2bOnul3onYhU+M+YyYIuMWZlZQGA8r85MMcxQ3UDxyYR6cIcXzOYAxmXOY4Zqhs4NonKVu2iya5du/Dmm29i+/bt6NChAwYOHAg3NzdcvHgRe/bs0VipmvQnKysL77zzjrKCtp+fH6KioowcVeVwzJCp4tgkY7l69SrWrl0LACgoKMC6deuwcOFCeHp6YsiQITh48GCVjhsdHY3x48ejZcuWsLOzg4eHB5KSkgA8+paEJUuWKF9dSrrja4ZxMAci0j+OTaqsOpmz6LAAChERmQAYcYGv+Ph4szi2OS0E++2334qnp6fk5eWJiMiKFSvkl19+UR4PCgpSFrvURXR0tEyYMEEOHz4sly5dEm9vbwEgI0eOVNrcu3dPJk6cKHFxcfo5GQMy1YVgiYioZhj7c6q55ECGzBNrc85S3kKw1Z5pQkREdcPNmzfh6elpdsc2ZVeuXIGPjw+CgoJgZWUF4NFCfLdv31ba+Pj4ANB9UdtTp05h7969GD9+PPr06YOQkBDY2triwoULSptmzZrhX//6F5ydnc3qFgciIqKaxByobucsLJoQEVGFEhMTMXbsWNy5c8esjm3KRAReXl6YOnUqmjdvrmwvKirC4cOHlZ/v3r0LALC3t9fp+H5+frC2ttbYVlBQgGnTpmlse+qpp+Do6Kh8WwIRERH9hTkQcxYWTYiIarmMjAz4+/tj6dKlWLhwIUaPHo2FCxciPT0dALB161ZYWFgo31iRmZmJdevWaWzbuXMnLl++jOTkZMyePRsAEBUVhUWLFsHBwQEpKSlwcXFBixYt0Lt3bxw6dKhaxwaAiIgI2Nvb4+zZszVzoWrYsWPHcPHiRbz00ksa20+ePImlS5dqtLO0tMSKFSuq1d8///lPBAYGIjAwsNRjo0ePxtatWxEXF1etPoiIiEwJcyD9qPM5iw738hARkQmADveqZmZmSrdu3eTdd99Vtt2+fVu6desmXbp0kfT0dBERcXR0LPXa//g2AOLk5CQiIoWFhXLixAmxtrYWAPLmm2/K2bNnZd++fdK4cWMBIJGRkVU6drGjR4+KjY2NHD9+vFLnWpI5rE3h4eEhKpVK8vPzy2yTl5cnTzzxhOzZs6fK/Rw+fFiGDBkiAMTBwUG2bdtWqs2lS5cEgHz44YdV7sfQuKYJEVHdpuv7QF3NgXTJEyurLuQsXNOEiKiOWr16NWJjYzFr1ixlm52dHZYvX464uDisWrUKAJR7U0vStq2YhYUFxowZo0y/XL16NQYPHgwPDw+8//77AICgoKAqHbuYs7MzMjIyMHbs2ArbmqPz58+jadOmsLS0LLNNSEgIfH19MXny5Cr3M2zYMGzZsgUbN25ESkoKpk+fjl27dmm0ad26NQDgv//9b5X7ISIiMiXMgfSnrucsLJoQEdVikZGRAIDGjRtrbB8yZAgA4Ny5c9U6voXFo7cRGxsbZZuzszOAR19JV11qtbraxzBVycnJaNasWbltrl+/jvnz51erH1tbW3Tv3h2+vr749NNPAQChoaGl2gBASkpKtfoiIiIyFcyB9Keu5ywsmhAR1WLFb+g3b97U2F5cpW/atKne+2zXrh0A3RcBq2vUajUKCwvLfPzhw4d45pln9NrnuHHjAAD16tXT2F58bzUREVFtwRxIf+p6zsKiCRFRLVb815QvvvhCY3t8fDwAYNSoUQD+egPKy8sD8GiV9Pv372vso1KpUFBQUGGfqampejt2eW/Q5q5t27bKQnTaWFtbw8PDQ699JiUlAQBeeeUVje1paWkAgDZt2ui1PyIiImNhDqQ/dT1nYdGEiKgWW7x4MXr16oWgoCAkJycr2zdt2oSBAwdi7ty5AAAnJycAwAcffIBr165hw4YNyM3NBfBoZfSioiI4OjoiKSlJSTZKKvnGfvr0afTt21e5h7iqx/7iiy9ga2uLr776Sp+XxGQMHToUmZmZePDggdbH/fz8MGbMmFLb165di549e2L//v3lHn/9+vUICQlRkrPc3Fz4+/vD3d1ded6LFX9F4KBBg6pyKkRERCaHOZD+1PWchUUTIqJazNraGufPn4enpydef/11LFq0CP7+/mjRogXOnDmjLOi1Zs0a9OvXD+vWrYOvry/GjBmDnj17wtvbG+np6SgoKICrqyuaNGmCH374oVQ/gYGBSE1NxZ07d5CUlITvvvuu2seuX78+mjRpgvr169fMxaphPj4+EBGcP39e6+M5OTnIyckptT0uLg4xMTFYtGhRucfPyMjAhx9+CAcHB8yZMwf+/v6YO3cu9u/fX2pqa2RkJNRqNdzc3Kp+QkRERCaEOZD+1PWcRSUiUnLDgQMH4O7ujsc2ExGRiVCpVAgLCzOJD7jdu3dHTEyMyb1nuLq6AgDCw8ONHEn5xowZg27dumH9+vU67RcbGwsfHx9ERUXpJQ5nZ2e0adMGwcHBejmeIdREfmIu44aIqC4ytc+pppoDGSpPrO05SznjK5wzTYiIiIxkx44d+M9//qPTCvDZ2dkICgrCtm3b9BLDhQsXEBsbi7Vr1+rleERERFT71OWchUUTIiKqsqysLI3/STetWrXC559/jgULFiA7O7tS+8TFxWHVqlXo1atXtftPSkpCQEAATp8+XeorGYmIiKhsdS0Hqss5C4smRESks6ysLLzzzjvKomV+fn56m3ZZ1/Tq1QsBAQHYtGlTpdvrI1koKChAaGgo9u7diw4dOlT7eERERHVBXc6B6mrOYlnjPRIRkdlr2LAhAgICEBAQYOxQagUHBwe8/fbbNdqnpaUl/P39a7RPIiIic1fXc6C6mLNwpgkRERERERERkRYsmhARERERERERacGiCRERERERERGRFiyaEBERERERERFpwaIJEREREREREZEWKhGRkhsOHDgAd3d3Y8VDREREVKbH0ha9cnV1xcGDBw12fCIiIjJtWvKM8FJfOfz8888jLCysZiIiIqJSfv31Vxw8eBAxMTGws7PDiBEjMHz4cDRr1szYoRHVam+99RZcXV2NHQaRztLS0hAREYEzZ87gzp07cHJygouLC3r37m3s0IiIzF6pmSZERGQaYmNjERISgpCQEKSmpmLEiBGYOXMmJkyYAEvLUjVvIiKqQ4qKinDmzBkEBwfjyJEjaNiwIdzc3ODr64unnnrK2OEREdUW4SyaEBGZuNzcXBw7dgzBwcH45ptv0LZtW3h7e+ONN95A586djR0eERHVoD///BO7d+/Gli1bcPPmTfTt2xczZ86El5cXbGxsjB0eEVFtw6IJEZE5uXr1KrZv344dO3bg7t27yuyT8ePHw8rKytjhERGRAZScVXL48GE0btwYrq6uePPNN9GrVy9jh0dEVJuxaEJEZI7y8vJw9OhRhIaG4ssvv0SrVq3g4+ODmTNnokuXLsYOj4iI9CAxMRF79uzBJ598gj/++APPP/88fHx84O3tDWtra2OHR0RUF7BoQkRk7hISErB3715s2rQJiYmJnH1CRGTGCgsLERERgeDgYBw6dAh2dnZ4/fXXMWPGDDg6Oho7PCKiuoZFEyKi2qJkon348GG0bNmSiTYRkZlgAZyIyCSxaEJEVBtxSjcRkenjrZZERCaPRRMiotqMiwcSEZkeLupNRGQ2WDQhIqor+DWVRETGw6+PJyIySyyaEBHVNSVnnxw5cgQNGzaEm5sbfH198dRTTxk7PCKiWiU2NhYhISEICQlBamqqMqtkwoQJsLS0NHZ4RERUPhZNiIjqsuTkZOzatQvBwcGIi4tTZp9MnjwZDRs2NHZ4RERm6fFZJe3atYOXlxfmzJmDjh07Gjs8IiKqPBZNiIio9OwTGxsbuLu7Y/bs2ejTp4+xwyMiMgsxMTHYuXMntm/fjrS0NAwfPhwzZ87ExIkToVarjR0eERHpjkUTIiLSlJKSgp07d2Lr1q24fv26MvvE09MTjRo1MnZ4REQmJScnB8ePH0dwcDBOnz6NDh06YPLkyfD19YW9vb2xwyMiouph0YSIiMr2448/Ijg4GLt374aVlRUmTZqEWbNm4W9/+5uxQyMiMqro6GiEhoZi69atePDgAcaNG4eZM2di5MiRUKlUxg6PiIj0g0UTIiKqWFpaGsLDwxEUFITffvuNs0+IqE56fFZJ165dMW3aNEydOhWtWrUydnhERKR/LJoQEZFuimef7NmzB2q1GuPHj4ePjw9GjRpl7NCIiAzixx9/RGhoKHbv3o3s7Gw4OztzVgkRUd3AogkREVVNeno6Dhw4gE2bNuGXX35Bjx494OPjgxkzZqB58+bGDo+IqFoyMjKwf/9+fPrpp7h48SKefPJJTJ06Ff/4xz9gZ2dn7PCIiKhmsGhCRETVVzz7ZO/evSgsLMSrr76KmTNncvYJEZmd4tezffv2IT8/n7NKiIjqNhZNiIhIf+7fv4+wsDB88skn+Omnn9C9e3e8/vrrmD59Olq0aGHs8IiItCqeVbJlyxZcunQJTk5OmDJlCl+7iIiIRRMiIjIM/rWWiEwdZ8kREVEFWDQhIiLDKmtdgGnTpqFly5bGDo+I6pjiGXGbN2/Gzz//zPWYiIioPCyaEBFRzSn+q+5nn32GvLw8zj4hohpT8pu/ioqKOKuEiIgqg0UTIiKqeQ8fPsSJEycQHByM06dPo1u3bvjHP/6BqVOnolWrVsYOj4hqieJv+dq4cSN+/fVX9O3bFzNnzoSHhwcaN25s7PCIiMj0sWhCRETGFR0djdDQUGzduhUPHjzAuHHjOPuEiKqleFbJ7t27YWVlhUmTJmHWrFn429/+ZuzQiIjIvLBoQkREpiEnJwfHjx9XZp888cQTmDx5MqZNmwZ7e3tjh0dEJi4lJQX79+/H1q1bcfnyZWVWiaenJxo1amTs8IiIyDyxaEJERKYnJiYGO3fuxLZt25Ceno7hw4dj5syZmDhxItRqtbHDIyITUVRUhDNnziA4OBhHjx6FtbU13N3dMXv2bPTp08fY4RERkflj0YSIiExXbm4ujh07huDgYHzzzTdo3749Jk+ejDlz5qBjx47GDo+IjCQ5ORm7du1CcHAw4uLilFklkydPRsOGDY0dHhER1R4smhARkXn4/fffsWPHDoSEhCA1NRUjRozAzJkzMWHCBFhaWho7PCIysJKzSo4cOYKGDRvCzc0Nvr6+eOqpp4wdHhER1U4smhARkXl5fPZJ27Zt4e3tjTfeeAOdO3c2dnhEpGdJSUkIDQ3Fp59+ihs3biizSry8vGBjY2Ps8IiIqHZj0YSIiMzX1atXsX37duzYsQN3797l7BOiWqLkrJLDhw+jUaNGcHNzw9y5c9G7d29jh0dERHUHiyZERGT+8vLycPToUWX2SZs2beDj44NZs2bBwcHB2OERUSUlJiZiz549+OSTT3Dr1i1lVom3tzesra2NHR4REdU9LJoQEVHtEh8fj3379mHTpk1ITExUZp+MHz8eVlZWxg6PiB5TWFiIiIgIZVZJixYtMGXKFEyfPh1PPPGEscMjIqK6jUUTIiKqnUp+EDt06BDs7Ozw+uuvY8aMGXB0dDR2eER1XkJCAvbu3YvNmzcjISGBBU4iIjJFLJoQEVHtVzzl//EPZ+PGjUO9evWMHR5RnVFWMXPmzJno0qWLscMjIiJ6HIsmRERUdzy+uGTjxo3h6uoKPz8/9OzZ09jhEdVa165dw7Zt27Bz507cuXOHs0qIiMhcsGhCRER1059//ondu3djy5YtuHnzJhecJNKzxxdoLv56cC7QTEREZoRFEyIiqttKzj45cuQIGjZsyK82JaqG2NhYhISEICQkBKmpqfwqcCIiMmcsmhARERVLSkpCaGgoPv30U9y4cUOZfeLl5QUbG5tKH+fq1avo0qUL1Gq1AaMlMozCwkLExcWha9euld4nNzcXx44dU2aVtGvXDl5eXpg9ezY6depkwGiJiIgMikUTIiKixz0++8TGxgbu7u6YM2cOnn766Qr3HzVqFKysrBAeHo5GjRrVQMRE+vHgwQO4uroiPz8fp0+frrB9TEwMdu7cie3btyMtLQ3Dhw/nrBIiIqpNWDQhIiIqT3JyMnbt2oWtW7fi+vXryuyTyZMno2HDhqXa37hxA46OjlCpVOjZsydOnjyJtm3bGiFyIt0kJSVh9OjRuHz5MkQE169f17r2SE5ODo4fP67MKmnfvj0mT54MX19f2NvbGyFyIiIigwm3MHYEREREpqxNmzbw9/dHbGwsTp06hR49emD+/Plo164dZs2ahUuXLmm037ZtGywtLVFUVIQrV67g6aefxk8//WSk6Ikq5/Lly3j22WcRExODoqIiWFpaYvv27Rptrly5giVLlqBDhw7w8vJCgwYNEBYWhps3b2L16tUsmBARUa3EmSZEREQ6SktLQ3h4OIKCgvDbb78ps0/c3NzQrVs33LlzR2lraWkJKysrfP7553j55ZeNGDWRdt988w3Gjx+PnJwcFBQUKNtbtGiBGzdu4KuvvkJwcDBOnz6Nrl27Ytq0aZg6dSpatWplxKiJiIhqBG/PISIiqioRwbfffoutW7fi0KFDUKvVyM7OLtXOwsICKpUKGzduxBtvvGGESIm027lzJ6ZPnw7g0QKwJalUKlhbW6OwsBATJ07EjBkzMGzYMKhUKmOESkREZAwsmhAREenD3bt3MXjwYFy7dk3jr/WP8/Pzw/r162FhwTtkyXhEBO+99x7ee++9Mtuo1Wp069YNZ8+eRcuWLWswOiIiIpPBNU2IiIj04eHDh4iNjS23YAIAGzduhJubG3JycmooMiJNeXl58Pb2xvvvv19uu8LCQsTExCArK6uGIiMiIjI9LJoQERHpwbZt26BWqytsV1RUhKNHj2Lo0KG4e/duDURG9Je0tDSMGjUK+/fvR1FRUYXtLS0tsWPHjhqIjIiIyDTx9hwiIqJqKiwsRPv27ZGSklLpfaysrNCpUyd8/fXXWr/WlUjfbty4gdGjR+PmzZvIz8+v9H6tW7dGYmJipYqCREREtQzXNCEiMjWurq7GDoF0lJSUhMjIyArbqVQqZRFNEYGIoF69ehg0aBCaN29u6DCpDrt37x6+//575OXlaR2HFRk4cCDatm1r6DBJz8LDw40dAhGRuWPRhIjI1KhUKvTv3x8dOnQwdiiko7y8PBQVFaGwsBAFBQUa/xcWFiI/Px9FRUUoKChAQUEBioqKlL/49+jRAw0aNDDyGZivhIQEREVFwcXFxdihmJycnBxER0cDAOLi4tC+fXs0b94clpaWsLCwgJWVFdRqNdRqNSwtLTX+t7CwQL169Yx8BqSr4t8HpvlERNXGogkRkalRqVQICwuDm5ubsUMhMhsHDhyAu7s7PyRWgK8vdQN/H4iI9IbfnkNEREREREREpA2LJkREREREREREWrBoQkRERERERESkBYsmRERERERERERasGhCRERERERERKQFiyZERERERERERFqwaEJERERUQv/+/bF48WJjh2EyVCoV1Go1/P39sWbNGly9elXj8atXr2Lt2rUAgIKCAqxbtw4LFy6Ep6cnhgwZgoMHD1ap3+joaIwfPx4tW7aEnZ0dPDw8kJSUBAAoLCzEkiVLkJiYWL2TM8FzuHr1KtasWQM/Pz+oVCqoVKrqnRwREVWPEBGRSQEgYWFhxg6DyKyEhYWJvtKaSZMmyYoVK/RyrKqIj4832LGr8voCQJ544gmtj3377bfi6ekpeXl5IiKyYsUK+eWXX5THg4KCBIB8/PHHOvUZHR0tEyZMkMOHD8ulS5fE29tbAMjIkSOVNvfu3ZOJEydKXFycTsc2p3Po3Llzlca1Pn8fiIjquAOcaUJERERUwmeffYaVK1cape+bN2/C09PTKH2Xx9LSstS2K1euwMfHB0FBQbCysgIA7NixA7dv31ba+Pj4AADCw8N16u/UqVPYu3cvxo8fjz59+iAkJAS2tra4cOGC0qZZs2b417/+BWdnZ2RlZVXltEz+HBo0aFCV0yIiIj1i0YSIiIjIBCQmJmLs2LG4c+eOsUOpkIjAy8sLU6dORfPmzZXtRUVFOHz4sPLz3bt3AQD29vY6Hd/Pzw/W1tYa2woKCjBt2jSNbU899RQcHR3x9ttv63oKteIciIjI8Fg0ISIiIsKjD8vh4eGYMmUKhg4dCgA4duwYZs2aBXt7e6Snp2PKlClo2bIlevfujR9//BEAEBUVhUWLFsHBwQEpKSlwcXFBixYt0Lt3bxw6dAgAsHXrVlhYWCjrU2RmZmLdunUa23bu3InLly8jOTkZs2fPVuKKiIiAvb09zp49W5OXo1zHjh3DxYsX8dJLL2lsP3nyJJYuXarRztLSEitWrKhWf//85z8RGBiIwMDAUo+NHj0aW7duRVxcnE7HrA3nQEREhseiCREREREACwsL9O/fH7t27VJuz+jbty/27duHhIQEbN68GStXrsSGDRvw22+/wdfXF0VFRUhNTcXmzZtx8+ZNBAQEYN68edi4cSNu3bqF1157DefOncOMGTPQpUsXpa/GjRvjrbfe0tj2zjvvAADatGmDTz75RNmemZmJe/fuISMjo4auRMXCwsKgUqnw7LPPamzv1asX2rdvDwDIz8/Hpk2bsHPnTjz11FNV6ufIkSMYOnQoPvzwQwQEBGD79u2l2gwYMAAFBQU4cOBAnTsHIiIyPBZNiIiIiP7P47dgtG/fXvkAvWzZMnTs2BGTJ09G69at8dNPP8HCwgJjxoxR9lu9ejUGDx4MDw8PvP/++wCAoKAgAFDWzChJ27bHOTs7IyMjA2PHjq3WuenT+fPn0bRpU61rnRQLCQmBr68vJk+eXOV+hg0bhi1btmDjxo1ISUnB9OnTsWvXLo02rVu3BgD897//1enYteEciIjI8Fg0ISIiIiqHtq98bdasGXJzc5WfLSwepVQ2NjbKNmdnZwAo9RW9VaFWq6t9DH1KTk5Gs2bNym1z/fp1zJ8/v1r92Nraonv37vD19cWnn34KAAgNDS3VBgBSUlJ0OnZtOAciIjI8Fk2IiIiIDKBdu3YAdF9A1Byo1WoUFhaW+fjDhw/xzDPP6LXPcePGAQDq1aunsV1bUasyasM5EBGR4bFoQkRERGQAqampAIBRo0YB+OuDcV5eHoBH395y//59jX1UKhUKCrK2XiQAACAASURBVApKHau8D/fG0LZtW6Snp5f5uLW1NTw8PPTaZ1JSEgDglVde0dielpYG4NFaMLqoDedARESGx6IJERER0f958OABAGgsupqTk1OqXWZmJgCUKnCULG6cPn0affv2xaxZswAATk5OAIAPPvgA165dw4YNG5RbfE6ePImioiI4OjoiKSkJ8fHxynG++OIL2Nra4quvvtLHKerF0KFDkZmZqVyvx/n5+WHMmDGltq9duxY9e/bE/v37yz3++vXrERISohSVcnNz4e/vD3d3d8ydO1ejbfFXAg8aNEinPkz5HIiIyHSwaEJEREQEIDs7G6tWrQIA/Pnnn1i/fj3WrFmDmzdvAgACAgKQkZGBDRs2IDExEQCwYsUKjaJKYGAgUlNTcefOHSQlJeG7775TFhpds2YN+vXrh3Xr1sHX1xdjxoxBz5494e3tjfT0dBQUFMDV1RVNmjTBDz/8oByzfv36aNKkCerXr19DV6JiPj4+EBGcP39e6+M5OTlai01xcXGIiYnBokWLyj1+RkYGPvzwQzg4OGDOnDnw9/fH3LlzsX///lK3skRGRkKtVsPNzU2nPkz5HIiIyHSoRESMHQQREf1FpVIhLCyMyTORDg4cOAB3d3cYK63p3r07YmJijNZ/ZVXl9UWlUsHJyQlXrlzR2D5mzBh069YN69ev1ymG2NhY+Pj4ICoqSqf9yuLs7Iw2bdogODhY5z5M+RyAqo8rY/8+EBHVIuGcaUJERERE5Sr5TUHFduzYgf/85z86feNLdnY2goKCsG3bNr3EdeHCBcTGxmLt2rVV6sNUz6GYtvVtiIioZpX9xfREREREVClZWVnK/w0bNjRyNPp348YNzJs3D+3atcPEiRPRtWtXtGrVCp9//jkWLFiAbdu2aXzdclni4uKwatUqNG7cuNoxJSUlISAgAKdPn9Y4ni59mOI5XL16FYcOHcK9e/dw/fr1avdBRETVw6IJERERURVlZWVh1apVysKtfn5+mDFjBvr372/kyPSnvFs8evXqhYCAAGzatAlvv/12hcfq1auXXmIqKChAaGgo9u7dW6p4oWsfpnYOXbt2hb+/P4BH6+AQEZFxcU0TIiITwzVNzE94eDhCQ0ORmJgIOzs7NGjQAPb29rC3t8fdu3fx73//29gh1npcw6Fy+PpSN/D3gYhIb8I504SIiGpMQkICOnToUGv6vXv3Ltzc3BAfH4+9e/fi73//O4BHf5nft28f5s2bh/Hjx+u9X32obc8FERERkSFwIVgiIqoRN2/ehKenZ63pV0Qwfvx4/Pzzz7hw4YJSMAEe/TV/8uTJ+Pzzz5W1LkxJbXsuiIiIiAyFM02IiMjgEhMTMXbsWBQWFtaafg8dOoTIyEh89NFHaN68udY2Q4cORWpqqt77ro7a+FwQERERGQpnmhARmbmsrCx88MEH8Pb2xrx58zBs2DBs2LBBeTwjIwP+/v5YunQpFi5ciNGjR2PhwoVIT08HABw7dgyzZs2Cvb090tPTMWXKFLRs2RK9e/fGjz/+WOl+rl69CldXVyxZsgQ+Pj4YMmQIfv31VwDAzp07cfnyZSQnJ2P27NnKPjk5Ofjoo48wffp0PPfcc3jhhRfw22+/6RSXvvsFgIiICNjb2+Ps2bNlXvdDhw4BAEaOHFnu8zNx4kQ+F9V4LoiIiIiMSoiIyKQAkLCwsEq1zc/Pl2HDhom3t7cUFRWJiMiOHTsEgBw/flwyMzOlW7du8u677yr73L59W7p16yZdunSR9PR0SUhIkEaNGgkACQgIkFu3bsmePXsEgPTr169S/YiIdO3aVRwdHZX2tra20qtXL43zcnJy0oh/xowZEhMTo/z84osvSuvWrSUjI6NScRmiXxGRo0ePio2NjXJu2jz33HMCQO7fv19mm5L4XFTtuaissLAwYVpTMV1eX8h88feBiEhvDvDVlIjIxOjyoWbdunUCQH7//XdlW0FBgezYsUPS0tLknXfeEQCSlJSksV9oaKgAkMWLF4uIyJNPPlkqwW7durXUr1+/Uv0Ut/nss89ERKSoqEgcHR3FyspK47xKfmC+cOGCAND678SJE5WKy1D9Fp9fefr376/12paFz0XVn4vKKP6QyH/8x39//SMiomo7wDVNiIjM2LfffgsAGt9GolarMWXKFABAZGQkAKBx48Ya+w0ZMgQAcO7cOQCPFi59XLNmzZCSklKpfgBgwYIFyMrKwubNm3Hv3j3k5uYiPz+/zNh/+OEH9OrVS7l9Q5uK4jJUv8XnV54ePXogKioKV65cQZs2bcptC/C5qGq/ugoLC9PbsWojd3d3zJ8/HwMGDDB2KGRA58+fR2BgoLHDICKqFVg0ISIyY8UfWK9evYqnn3661OMWFo+Wrrp58yZ69uypbG/dujUAoGnTpnrpB3j0Adjd3R2bN2/GnDlzsHfv3nKPmZqairi4OGRnZ8PGxkbjsaKiIiX2ihir36FDhyIkJARRUVEYPnx4he35XBiu35Lc3Nx03qcucXd3x4ABA3id6gAWTYiI9IMLwRIRmbHiD80BAQEQEWX7rVu38OWXXyqzGL744guN/eLj4wEAo0aN0ks/AODj44P8/Hy89NJLAB596C1JpVKhoKBA+dnJyQnZ2dlYs2aNRrsrV65g48aNlYrLkP1W9C0vXl5e6Nu3LzZs2ICkpCStbXJzcxEaGgoAfC4M2C8RERGRoXCmCRGRGVuyZAn27t2L8PBwpKam4rXXXkNycjJu376NTz75BMOGDcPBgwcRFBQEHx8f5TaSTZs2YeDAgZg7dy6AR99g8rjMzEz8//buPS6qOv8f+GsYTMELopWg0oaYD1oG08zIVLS8YGJjJZfERG1NlkhSUcHUxSVRWAUlol2hDW3tApP69ZoWaWoKXlYpcVUswkRBkERucp3P7w9+TCLXgYEzw7yejwcPH3zOOZ/Pe85w3s685zOfAwBVVVXNjgMA2dnZKCwsxLfffou8vDzNHWFOnz6N/v37w87ODtnZ2bh+/TpsbGwwffp0DBo0CCEhIcjKysKECRNw6dIlnD59Gl999VWL4jI1NW2Xcffv34/XX38dKpVKUwB4kImJCbZv344pU6ZgzJgx2LhxI5RKJeRyOe7du4eUlBSEhoYiNDQUALB8+XI+F60Yl4iIiEhSEi+qQkREDwC0u7vFhQsXhIuLi7C0tBQDBgwQixYtqnNHl6KiIrF8+XIxefJkERAQIJYvXy5CQkJEeXm5EEKImJgYzaKBa9euFXfv3hWbN2/WtAUFBYl79+41O05MTIywsLAQzz77rEhJSRFRUVHC0tJSTJ8+XeTn54sVK1YIa2trsWPHDs0xmZmZQqlUij59+ggrKyuxYMECkZeXp1Vcuh5XCCG+/fZb0b9/f3H48OFmz39RUZEIDw8Xrq6uwtbWVigUCjFs2DCxcuVKkZ+fX29fPhfaPRctxbuFtIy2+YUME68HIiKdSZQJcd/cXiIikpxMJkNCQgLXHCDSQmJiIjw9PcGXNU1jfjEOvB6IiHRGxTVNiIiIiIiIiIgawDVNiIiIiMjo/frrr9i7dy/Ky8vx6quvYvDgwVKHREREeoAzTYiIiIhIr129ehUREREAahYejoyMREBAALy8vODs7NymhYOLioqwcOFCTJo0CUOHDsWyZcswePBgVFdXIygoCDdu3NDVwyAiIgPEogkRERFRG2VlZRlk34bg6NGjWLNmDfz9/QEAISEhmDRpEiIiIvD555/Dw8MD7u7umqKKNvLy8jB+/Hh88803SElJwfjx4zXb5HI5AgMD4e/vj19//VVXD4eIiAwMiyZEREREbZCZmQkvLy+D69sQXLp0Cd7e3oiOjkaXLl0AAPHx8cjNzdXs4+3tDQBQqVRa9z937lz8+OOP+PTTT/Hwww/X225paYng4GAolUqUlJS08lEQEZEhY9GEiIiIqJVu3LiBadOmIS8vz6D6NgRCCLzxxhuYN28e+vTpo2lXq9XYtWuX5vfbt28DAGxsbLTqf9++fThw4ABcXFzg5OTU6H5Dhw6FnZ0dli1bpuUjICKizoBFEyIiIjJKhYWFCAwMxIoVKxAQEAAXFxcEBASgoKAAABAXFwcTExPIZDIANWtfREZG1mnbunUrLl68iJycHPj6+gIAUlJSsHTpUtja2uLWrVtwc3ND37594ejoiJ07d7apbwA4cuQIbGxscOzYsY45URLZs2cPzp07hylTptRpP3ToEFasWFFnP1NTU6xevVqr/rdt2wYAeOyxxzBu3Dj07NkTI0aMwP79++vt6+Ligri4OGRkZLTikRARkSFj0YSIiIiMTnFxMUaOHAlzc3OsX78eERER2L59O/bt24cRI0bg7t27eOuttzBo0CDNMT179sSSJUvqtK1cuRIAYGVlhX/+859Qq9XIz8/HRx99hMzMTISGhuLdd9/Fhx9+iGvXrmHGjBk4efJkq/quVVRUhN9//x2FhYXtdn70QUJCAmQyGZ555pk67QqFAgMGDAAAVFZWIiYmBlu3bsXQoUO16v/s2bMAgCeeeAIJCQlISkpCXl4eXn75ZZw5c6bOvqNGjUJVVRUSExPb8IiIiMgQsWhCRERERicsLAzp6enw8fHRtD3yyCNYtWoVMjIysG7dOgDQrKNxv4baapmYmMDV1VXzVZGwsDCMHTsWM2fOxPvvvw8AiI6OblXftZRKJQoLCzFt2rRm9zVkycnJsLCwgKmpaaP7fPLJJ/Dz88OsWbO07j8nJwfW1tZYsmQJrKys4OTkhPXr10MIgQ8++KDOvv369QMAHD9+XOtxiIjIsLFoQkREREbnxIkTAGpmeNzP2dkZAHDy5Mk29W9iUvMSy9zcXNOmVCoB1Nw+t63kcnmb+9B3OTk5sLS0bHKfX375BYsWLWpV/1ZWVvWKVC+88AIA4MqVK3Xae/fuDQC4detWq8YiIiLDxaIJERERGZ3aokZmZmad9toZBRYWFjofs3///gC0X7DUWMnlclRXVze6/d69exg+fHir+3/iiSfq3IUHgOYOOvcvPAtAs84MEREZHxZNiIiIyOjUzih5cNHP69evAwAmTpwI4I83yxUVFQBq7uhy9+7dOsfIZDJUVVU1O2Z+fr7O+m6qmNBZWFtbaxblbYiZmRlmzpzZ6v69vLxQVlaG1NRUTVvtnXieffbZOvveuXMHQM3sFCIiMi4smhAREZHRWb58ORQKBaKjo5GTk6Npj4mJwejRo/HOO+8AAOzt7QEAa9euxc8//4yoqCiUl5cDqLmLi1qthp2dHbKzszUFl/vdX9xISkrCiBEjNOuotLbv/fv3o3fv3jh48KAuT4neGTduHIqKilBcXNzgdn9/f7i6utZrj4iIgIODA7788ssm+589ezYUCgU2bNigadu1axesrKywZMmSOvvWFlPGjBmj7cMgIiIDx6IJERERGR0zMzMkJyfDy8sLc+bMwdKlSxEYGIi+ffvi8OHDmsVHw8PD4eTkhMjISPj5+cHV1RUODg6YPXs2CgoKUFVVBXd3d/Tq1aveHVcAYPPmzcjPz0deXh6ys7Nx9OjRNvfdtWtX9OrVC127du2YkyURb29vCCGQnJzc4PaysjKUlZXVa8/IyMDly5exdOnSJvuXy+U4fvw4unXrhjlz5mD16tVISUnB2bNnNWuY1Dpx4gTkcjk8PDxa/4CIiMggyYQQQuogiIjoDzKZDAkJCXxxTqSFxMREeHp6Ql9e1jz55JO4fPmy3sRTy9Dyi6urK4YMGYJNmzZpdVx6ejq8vb2RkpKikziUSiWsrKwQGxurk/7am75dD0REBkzFmSZEREREpJfi4+Nx4MABre5aU1paiujoaHz88cc6ieHUqVNIT09HRESETvojIiLDwqIJERERkY6VlJTU+Zda59FHH8WOHTuwePFilJaWtuiYjIwMrFu3DgqFos3jZ2dnIzQ0FElJSfVuT01ERMaBRRMiIiIiHSkpKcHKlSs1C7f6+/vr7CsixkqhUCA0NBQxMTEt3l8XBY6qqip8+umn+OyzzzBw4MA290dERIbJVOoAiIiIiDqL7t27IzQ0FKGhoVKH0qnY2tpi2bJlHTqmqakpAgMDO3RMIiLSP5xpQkRERERERETUABZNiIiIiIiIiIgawKIJEREREREREVEDWDQhIiIiIiIiImoAF4IlItJDycnJUodARkAIAZlMJnUYOlF7zSQmJkocSdP04Zwzv3R+fI6JiHRHJoQQUgdBRER/kPoNFRERdQ58mU9E1GYqzjQhItIzfJFL7UkIgS1btmDZsmUYNGgQtm3bhmHDhkkdllFITU3FnDlzkJGRgQ0bNsDHx4dFUiIiIj3HNU2IiIiMRHZ2NpRKJRYuXAg/Pz+cPn2aBZMONGzYMJw9exarVq2Cv78/Jk+ejN9++03qsIiIiKgJLJoQEREZAZVKBQcHB2RkZODkyZMICwtD165dpQ7L6HTp0gWBgYE4c+YM8vLy4OjoiNjYWM4wIyIi0lMsmhAREXViOTk5eOWVV/D666/D3d0dZ86cwciRI6UOy+g99dRTSElJga+vL95++2289NJLyMrKkjosIiIiegCLJkRERJ2USqWCQqHAhQsXcPjwYWzZsgXm5uZSh0X/X7du3RAWFoYffvgBmZmZUCgUiI2NlTosIiIiug+LJkRERJ1Mbm4uZsyYAU9PT8yYMQM//vgjxo0bJ3VY1IjnnnsO58+fx1//+lf4+vrC1dUVN2/elDosIiIiAosmREREnUrt2iXnzp3Dd999hy1btqBHjx5Sh0XNMDMzQ1hYGI4fP46rV6/CwcEB//nPf6QOi4iIyOixaEJERNQJFBQUwMfHB56ennjttdfw008/4YUXXpA6LNLS888/j9TUVPj4+GDu3Lnw8PDA7du3pQ6LiIjIaMkEl2snIiIyaAcOHMBbb70FIQRiY2Mxbdo0qUMiHfj2228xf/58lJWV4Z///Cdee+01qUMiIiIyNirONCEiIjJQd+/ehY+PD1xdXTF69GhcvHiRBZNOZNKkSbhw4QJeeeUVuLm5wcPDA/n5+VKHRUREZFQ404SIiMgAHTp0CPPnz0dVVRX+9a9/Yfr06VKHRO3o/ud7y5YtUCqVUodERERkDDjThIiIyJAUFhbCx8cHL730EkaNGoW0tDQWTIyAi4sL0tLSoFQqMX36dHh4eODOnTtSh0VERNTpcaYJERGRgUhKSsJf/vIXrnFh5O5fwyYuLg6urq5Sh0RERNRZcaYJERGRvrt37x6CgoLg4uICJycnpKWlsWBixKZOnYq0tDRMnDgRL7/8Mnx8fFBcXCx1WERERJ0SZ5oQERHpsZMnT2Lu3LnIy8tDeHg4FixYIHVIpEdUKhXefvtt9OjRA5988glvM01ERKRbnGlCRESkj2pnl4wdOxZPPPEELl68yIIJ1ePu7o6LFy9i+PDhmDBhAnx8fFBSUiJ1WERERJ0GZ5oQERHpmZSUFMybNw/Z2dn4xz/+wWIJtYhKpYKvry8sLCwQHx8PZ2dnqUMiIiIydJxpQkREpC/KysoQFBSEMWPG4E9/+hPS0tJYMKEWc3d3R1paGhQKBV544QX4+PigtLRU6rCIiIgMGmeaEBER6YGffvoJc+bMQUZGBjZs2IC33noLMplM6rDIQKlUKvj4+MDa2hpbt27FyJEjpQ6JiIjIEHGmCRERkZQqKysRHh6OkSNHomfPnjh37hwWLFjAggm1ibu7O1JTU9G/f388//zzCAoKQnl5udRhERERGRzONCEiIpJIWloa5syZg0uXLiE4OBjLli2DiQk/zyDdEUIgLi4OAQEBePzxx7Ft2zY8/fTTUodFRERkKDjThIiIqKNVVVUhPDwcI0aMQLdu3fDjjz8iMDCQBRPSOZlMhgULFuDChQt45JFH4OTkhKCgIFRUVEgdGhERkUHgTBMiIqIO9L///Q9z585FWloagoODsXTpUsjlcqnDIiNQO+tkyZIlsLOzw7Zt2zBs2DCpwyIiItJnnGlCRETUEdRqNaKiovD0009DLpfj3LlzCAwMZMGEOkztrJOffvoJvXv3xrPPPos1a9agurpa6tCIiIj0FmeaEBERtbOMjAzMmzcPp0+fxpo1azi7hCSnVqvx8ccfY/HixXB0dMTWrVthb28vdVhERET6hjNNiIiI2osQArGxsRg6dCgKCgqQnJzM2SWkF0xMTLBgwQKcOXMGarUaw4cPR3h4OGedEBERPYBFEyIionaQmZmJCRMmwM/PD++88w7OnDnD9SNI7/z5z3/GyZMnsWbNGgQHB8PZ2Rnp6elSh0VERKQ3WDQhIiLSodrZJY6OjsjLy8OpU6cQFhaGhx56SOrQiBpkamqKwMBAnD17FmVlZRg2bBjCw8OhVqulDo2IiEhyLJoQERHpyLVr1zB58mT4+fnBz88PZ8+exdNPPy11WEQtolAocOrUKQQHB+Nvf/sbxo0bh59//lnqsIiIiCTFogkREZEOqFQqDB8+HDdv3sTJkycRFhaGrl27Sh0WkVZqZ52cOXMGxcXFGDFiBGJjY8H7BhARkbFi0YSIiKgNcnJyMH36dLz++uuYPXs2zp07h5EjR0odFlGbDB06FKdPn8bixYvx9ttv46WXXkJWVpbUYREREXU4Fk2IiIhaSaVSQaFQIC0tDUeOHEFUVBRnl1Cn0aVLF6xZswYnTpzAtWvXoFAoEBsbK3VYREREHYpFEyIiIi3l5ubitddeg6enJ2bMmIGffvoJzs7OUodF1C6cnJxw/vx5/PWvf8Xbb7+NqVOn4saNG1KHRURE1CFYNCEiItKCSqWCg4MDzp8/j++++w5btmxB9+7dpQ6LqF1169YNYWFhOHbsGH755RfOOiEiIqPBogkREVEL3LlzB97e3vD09MRrr72GCxcu4IUXXpA6LKIO9fzzzyM1NRU+Pj7w9fWFh4cH8vLypA6LiIio3bBoQkRERq28vLzZfQ4cOACFQoGjR4/im2++wZYtW9CjR48OiI5I/5iZmSEsLAxHjx7FuXPnoFAosHPnzmaPa8m1RkREpG9YNCEiIqN1/fp1jBo1CoWFhQ1uLygogI+PD1xdXTF69GikpqZi4sSJHRwlkX4aM2YMzp07h1deeQVubm7w8PBAfn5+g/sWFhZi1KhRuH79egdHSURE1DYsmhARkVGqqqqCu7s7zp8/j0WLFtXbfvDgQTg6OmLPnj3YvXs3EhMTYWlpKUGkRPqrV69e2LJlC77++mskJyfDwcEBu3fvrrffokWLcP78ebi7u6OqqkqCSImIiFqHRRMiIjJKq1atwtmzZwEA8fHxOHDgAICaT8R9fHwwdepUjBo1CmlpaVAqlVKGSqT3XFxckJaWhunTp+OVV16Bh4cHfv/9dwA1X2+Lj48HAJw9exarVq2SMlQiIiKtyIQQQuogiIiIOtLBgwcxdepU1P4XKJPJ0LdvX2zZsgWLFy9GWVkZ/vWvf+HVV1+VOFIiw3PgwAEsWLAAarUamzZtwrvvvou8vDyo1WoANdfb7t278fLLL0scKRERUbNULJoQEZFRuXXrFhwcHHDnzh3NmzgA6NKlC2xtbfHUU0/ho48+wsMPPyxhlESGraCgAIGBgfj+++/x66+/orKyUrPNxMQEvXr1QlpaGgYMGCBhlERERM1S8es5RERkNNRqNTw9PVFYWFinYAIAlZWVSE9Ph6enJwsmRG3Uu3dvuLq6Ij09vU7BBKi5DktKSuDh4YHq6mqJIiQiImoZFk2IiMhoBAcH44cffqj3Jq6WiYkJ3nrrLeTm5nZwZESdy+3bt/Hmm2/CxKThl5qVlZU4deoU/v73v3dwZERERNph0YSIiIzC999/j3Xr1jX5ybZarUZxcTH8/Pw6MDKizsfX17fBGV33q66uxtq1a/Htt992YGRERETa4ZomRETU6eXk5EChUNRbx6QhJiYmUKvVSExMhLu7ewdFSNR5qFQqeHh4aK6lppiYmMDS0hJpaWmwsrLqoAiJiIhajGuaEBFR56ZWqzFz5sxGP/WWyWTo0qULAMDMzAxTpkxBVFQUhg0b1tGhEnUKw4YNQ1RUFF566SWYm5sDALp27drgV3XUajUKCwvx+uuvN1tgISIikgJnmhARUacWEhKCv//973XekHXt2hXl5eWQy+VQKBSYMmUKJk6cCGdnZzz00EMSRkvUuVRXVyM1NRVJSUk4dOgQTpw4gYqKCs01WEsulyM4OBirV6+WMFoiIqJ6eMthIiLqvI4ePYoXX3wR9/9X5+joiJdeegkTJkzA6NGjNZ+EE1H7u3fvHn744Qd89913OHjwIH766SfNNplMhsOHD2PcuHESRkhERFQHiyZEnUFiYiI8PT2lDoOIqE0SEhLg4eHRLn0zTxKRIWrPvEhELaIylToCItKdhIQEqUMgatKmTZsAAIsXL273sW7cuIFu3bqhb9++7T6WLiUnJ2Pz5s1Gdz13VEHD2M6rocnPz0dZWRkGDBggdSgdpiPzoqFiXiQiKbFoQtSJ8JMI0ncqlQoA/1abs3nzZqM7Rx315sDYzivpP+bFlmFeJCKp8O45REREREREREQNYNGEiIiIiIiIiKgBLJoQERERERERETWARRMiIiIiIiIiogawaEJERERERERE1ADePYeIiIi08uuvv2Lv3r0oLy/Hq6++isGDB0sdEhGRpJgXiTovzjQhIiKD89xzz2H58uVSh6GXrl69ioiICABAVVUVIiMjERAQAC8vLzg7O+Orr75qdd9FRUVYuHAhJk2ahKFDh2LZsmUYPHgwqqurERQUhBs3bujqYRCRlpgXG8e8SERtwZkmRERkcGxtbdGtWzfJxs/KysLAgQMlG78xR48eRWxsLLZu3QoACAkJgbu7OxwdHQEAH374Idzd3bFx40YEBARo1XdeXh6mTJmC4uJipKSk4OGHH9Zsk8vlCAwMxPz587Fx40bY2trq7DER0o2qygAAGh1JREFUUcswLzaMeZGI2oozTYiIyOB88cUXCAkJkWTszMxMeHl5STJ2Uy5dugRvb29ER0ejS5cuAID4+Hjk5uZq9vH29gYAqFQqrfufO3cufvzxR3z66ad13hjUsrS0RHBwMJRKJUpKSlr5KIiotZgX62NeJCJdYNGEiIiohW7cuIFp06YhLy9P6lDqEELgjTfewLx589CnTx9Nu1qtxq5duzS/3759GwBgY2OjVf/79u3DgQMH4OLiAicnp0b3Gzp0KOzs7LBs2TItHwERGSrmReZFos6ORRMiIjIYarUaKpUKc+fOxbhx4wAAe/bsgY+PD2xsbFBQUIC5c+fi4YcfhqOjI/773/8CAFJSUrB06VLY2tri1q1bcHNzQ9++feHo6IidO3cCAOLi4mBiYgKZTAag5nvqkZGRddq2bt2KixcvIicnB76+vpq4jhw5AhsbGxw7dqwjT4fGnj17cO7cOUyZMqVO+6FDh7BixYo6+5mammL16tVa9b9t2zYAwGOPPYZx48ahZ8+eGDFiBPbv319vXxcXF8TFxSEjI6MVj4SItMW82DDmRSLSGUFEBi8hIUHwciZD4ObmJtzc3NrUx2+//SYACHt7eyGEEFlZWaJHjx4CgAgNDRXXrl0T27dvFwCEk5OTqK6uFvv27RNmZmYCgFi4cKE4duyY+Pzzz0XPnj0FAHHixAkhhBB2dnb1rqUH2+4fu9bu3buFubm52Lt3b5semxCtu55nzpwpZDKZqKysbHSfiooKMXjwYLF9+3atY3r88ccFABERESGys7NFSkqKsLGxETKZTJw+fbrOvufPnxcAxPr167UaA4BISEjQOraWYp4kfcW82DzmRSKSUCJnmhARkUF5cAr1gAEDMGDAAADAe++9h8ceewyzZs1Cv379kJqaChMTE7i6umqOCwsLw9ixYzFz5ky8//77AIDo6GgA0Hzn/X4NtT1IqVSisLAQ06ZNa9Nja63k5GRYWFjA1LTx9d0/+eQT+Pn5YdasWVr3n5OTA2trayxZsgRWVlZwcnLC+vXrIYTABx98UGfffv36AQCOHz+u9ThE1DrMi/UxLxKRrrBoQkREBq92mvj9LC0tUV5ervndxKTmvzxzc3NNm1KpBFBzO8q2ksvlbe6jtXJycmBpadnkPr/88gsWLVrUqv6trKzqvUl64YUXAABXrlyp0967d28AwK1bt1o1FhHpBvMi8yIR6QaLJkREZLT69+8PQPsFAPWNXC5HdXV1o9vv3buH4cOHt7r/J554os7dJgBo7hRx/wKLQMNv1IjIcDAvtgzzIpHxYNGEiIiMVn5+PgBg4sSJAP54YVtRUQGg5u4Ld+/erXOMTCZDVVVVvb6aenHe3qytrVFQUNDodjMzM8ycObPV/Xt5eaGsrAypqamatto7Tjz77LN19r1z5w6Amk9hicjwMC+2DPMikfFg0YSIiAxKcXExAKCwsFDTVlZWVm+/oqIiAKj3Qv7+F/FJSUkYMWIEfHx8AAD29vYAgLVr1+Lnn39GVFSUZir7oUOHoFarYWdnh+zsbFy/fl3Tz/79+9G7d28cPHhQFw9Ra+PGjUNRUZHm3DzI398frq6u9dojIiLg4OCAL7/8ssn+Z8+eDYVCgQ0bNmjadu3aBSsrKyxZsqTOvrVvGsaMGaPtwyCiVmJerI95kYh0hUUTIiIyGKWlpVi3bh0A4ObNm9i0aRPCw8ORmZkJAAgNDUVhYSGioqJw48YNAMDq1avrvHnYvHkz8vPzkZeXh+zsbBw9elSzUGB4eDicnJwQGRkJPz8/uLq6wsHBAbNnz0ZBQQGqqqrg7u6OXr164cyZM5o+u3btil69eqFr164ddCbq8vb2hhACycnJDW4vKytr8A1URkYGLl++jKVLlzbZv1wux/Hjx9GtWzfMmTMHq1evRkpKCs6ePav5rn6tEydOQC6Xw8PDo/UPiIhajHmxYcyLRKQrMiGEkDoIImqbxMREeHp6gpcz6Tt3d3cAgEql6vCxn3zySVy+fFnvr5PWXs+urq4YMmQINm3apNVx6enp8Pb2RkpKilbHNUapVMLKygqxsbFaHSeTyZCQkNBubyqYJ0lfMS82j3mRxRYiCak404SIiKgTiI+Px4EDB7S6O0NpaSmio6Px8ccf6ySGU6dOIT09HRERETrpj4ioLZgXiUgXWDQhIiKjUFJSUuffzubRRx/Fjh07sHjxYpSWlrbomIyMDKxbtw4KhaLN42dnZyM0NBRJSUno2bNnm/szBLm5uVCpVJqvRhAZGubF+pgXiehBLJoQEemRpKQkTJ06FTKZDDKZDC+++CJefPFFjBw5EtOnT8e///1vzR0MqGVKSkqwcuVKzQKF/v7+OptyrW8UCgVCQ0MRExPT4v118UK+qqoKn376KT777DMMHDiwzf0ZgsuXLyMkJAQeHh74z3/+I3U4nR5zo24xLza9P/MiEd2Pa5oQdQL8rn7rZWVltduLmdb2ffPmTQwYMAC2trbIyMgAUHOLx/3792PRokUwMTHB//3f/+HPf/6zrkNud1J+d99QGOv1bIhrmpSXl6Nbt26wt7fHpUuXdNavPmBu7DjMi81jXuSaJkQS4pomRGS8MjMz4eXlpXd99+/fHwDq3HFAJpNh2rRpOH78OIqLi6FUKhtc9Z+IOo5UdwVpb8yNREREf2DRhIiM0o0bNzBt2jTk5eUZVN/W1tZ4//338csvv3BROSLSOeZGIiKiulg0ITJSZ8+exXPPPYd33nkHf/vb39ClS5c6C8KtXbsWs2fPxrvvvovx48cjKipKc2xhYSECAwOxYsUKBAQEwMXFBQEBASgoKIBarcbRo0exePFi2Nra4ubNmxg/fjz+9Kc/oaCgAGVlZfjHP/6B+fPnY+TIkZg0aRLS0tK0ir2p8QEgLi4OJiYmkMlkAICioiJERkbWadu6dSsuXryInJwc+Pr6AgBSUlKwdOlS2Nra4tatW3Bzc0Pfvn3h6OiInTt3tqlvADhy5AhsbGxw7NgxrZ+v+7m5uUEul+Obb77RtDV1Xvfs2QMfHx/Y2NigoKAAc+fOxcMPPwxHR0f897//1fTR1N+ELp43IkPTXC5syNWrV+Hu7o6goCB4e3vD2dkZFy5c0Gxv6jpraltLMDcyNxIRUTsQRGTwEhIShLaX85AhQ0SfPn00v3t6eorc3FxRWVkpxo8fL2bPni3UarUQQoj4+HgBQOzdu1cUFRWJIUOGiDVr1miOzc3NFUOGDBGDBg0Subm54uTJk8Lc3FwAEOvXrxdJSUli/vz5ori4WLz11lvi8uXLmmMnT54s+vXrJwoLC1sUd3PjFxQUCCGEsLOzq3dOHmwDIOzt7YUQQlRXV4t9+/YJMzMzAUAsXLhQHDt2THz++eeiZ8+eAoA4ceJEq/qutXv3bmFubi727t3b7ONs6Pj7WVtbi759+2p+b+q8ZmVliR49eggAIjQ0VFy7dk1s375dABBOTk6aYxr7m2iuf224ubkJNzc3rY4xNq25njsDACIhIaHd+tf2vDaXC2s9eK0+8cQTws7OTtNH7969hUKh0Gxv6jpraltzmBtrGGJuZF5sHvMiEUko0bT9yzJEpI/u3LmD33//HR988AEWLlyI1atXo1u3boiOjsb333+PK1euaD4dnD17NgBgzJgxCAsLQ3p6Onx8fDR9PfLII1i1ahW8vb2xceNGhIeHw8bGBleuXIGPjw8sLS0xYcIEnD59GnFxcYiLi6sXz7Fjx+Dq6tps3M2Nv27dOoSHh6NLly71jm2orZaJiQlcXV1hY2OD9PR0hIWFwdzcHEDNbUUXLVqE6OhoPP/881r3XUupVKKwsBByubzZfZtjamqqeX5acl4HDBiAK1eu4L333gMAzJo1CwEBAUhNTdXs29jfhC6et/tlZWUhMTFRq2OMSXJyMgDwHEmsuVzYGF9fX1hbWwMA5HI5+vbtiytXrmi2N3adNbetOcyNNQw1NzIvNo15kYgkJXXZhojarjWfwHz11VeaTwmfeeYZkZKSIoQQQqlUCgCipKSkwePGjx8vAIji4uI67ZmZmQKAGDNmjBBCCHt7+3oxffjhh3U+cW2Ntoz/YBsa+MSyoeMyMjIEADFixIg29a2Npo6vqKgQDz30kJg6daoQomXntSUxN/Y3oYvnrZabm5sAwB/+NPijTzNNmsuFtYD612pxcbGIiYkR77//vhg4cGCdcRu7zprb1hzmRsPNjcyL/GnqhzNNiCSXyDVNiIzUjBkzkJqaChcXF5w9exZjx47Ftm3bcOvWLQA138tviIlJTdrIzMys096vXz8AgIWFRaNj5ufnIyMjA6WlpfW2qdXqFsXdlvFbq/aODTY2NjrvuzUOHz6MiooKTJgwAYBuzivQ+N+Ervqv5ebmBiEEfxr5SUhIAADJ4+joH33TXC5szJkzZ+Do6IhBgwZh1apV6NGjR53tjV1nzW1rDnOjYedG5sWmf5gXiUhKLJoQGang4GAMGjQIBw8exBdffIHKykqsWrUKTz31FAAgNDS0zn/Y165dw9dffw1nZ2cAwP79++v0d/36dQDAxIkTGx3T3t4epaWlCA8Pr9N+6dIlfPjhhy2Ku6Xj107PrqioAFDzQuvu3bt1jpHJZKiqqmp2zPz8fJ31XV1d3ex4TamoqMB7772H4cOHw9/fH4BuzivQ+N+ErvonMiTN5cLGeHt7o7KyElOmTAFQ/81zY9dZc9uaw9zI3EhERO1EEJHBa83Xc8zNzcWdO3eEEDWLFVpYWAgnJyeRkZEhunfvLgCIF198UcTExIjVq1cLHx8foVarRWlpqVAoFGLgwIEiOztb09+7774rRo8eLSorK4UQQjz++OMCqDtVvKysTAwaNEgAEG+++ab47LPPxKpVq8TkyZNbvGheS8d/9dVXBQCxevVqcfXqVbFp0ybRp08fAUAcPHhQVFdXi8GDB4vu3buL3377TdNP7bTsqqoqTdu2bdvEiBEj2tz3vn37RI8ePcTXX3/d7GMEIB5//PE67efOnRPOzs7C1tZW/O9//9PqvNY+H/cbMGCAAKB5XI39TejieavFBQ+bxwUP24e257W5XChEw9eqhYWFkMlk4ptvvhGfffaZePTRRwUAcerUKXH9+vVGrzMhGr8GW4K50XBzI/Ni85gXiUhCicaXfYg6oda8mAAgnn76aREWFiZmzZolpk2bJn799VchhBAXLlwQLi4uwtLSUgwYMEAsWrRI3L17V3NsUVGRWL58uZg8ebIICAgQy5cvFyEhIaK8vFyUlJSIkJAQzXdxFyxYIM6fP685NjMzUyiVStGnTx9hZWUlFixYIPLy8rSKvanxa6WnpwsnJyfRvXt3MXnyZJGeni7Gjh0rZs+eLb788ktRXl4uVqxYIaytrcWOHTs0x9W+Mdi4caO4ffu2yM3NFWFhYXWKP63t+9tvvxX9+/cXhw8fbvSx/fDDD+Ivf/mL5vyNHz9euLi4CKVSKWbMmCFiYmLqrVnQ3HmNiYnR9Ld27Vpx9+5dsXnzZk1bUFCQuHfvXpN/E7p43oTgm4OW4JuD9tGa89pULszIyBD+/v6a62jz5s3izp07IiYmRlhYWIhnn31WpKSkiKioKGFpaSmmT58u8vPzm7zOmtrWEsyNhpkbmRebx7xIRBJKlAnBL8wRGbrExER4enry+6868OSTT+Ly5cs8l+3E3d0dAKBSqSSORH8Z6/Usk8mQkJAADw+PdunfWM+rrjA3th/mxeYZ6/Xb3nmRiFpExTVNiEhvyGSyZn/uv3UnEZExYG4kIiKSjqnUARAR1dKHT5BKSko0/3bv3l3iaIiImBuJiIikxJkmRESoeSOwcuVKzZ0m/P39kZKSInFURETSYm4k0s7PP/8sdQhEpGMsmhARAejevbvm1qJCCPz73//Gc889J3VYRFq7evUqIiIiAABVVVWIjIxEQEAAvLy84OzsjK+++qpV/d68eRPx8fHw9PTE888/X2dbdXU1goKCcOPGjTbHT/qFuZE6g/bKix9++GG9r8pFRUUBYF4k6kz49RwiIjIaWVlZGDhwoMH13VJHjx5FbGwstm7dCgAICQmBu7s7HB0dAdS8wHd3d8fGjRsREBCgVd/9+/fHxIkT8eabb8Le3r7ONrlcjsDAQMyfPx8bN26Era2tTh4PEbU/5sXW5cWqqip88cUXCAsL07SZmprC29sbAPMiUWfCmSZERGQUMjMz4eXlZXB9t9SlS5fg7e2N6OhodOnSBQAQHx+P3NxczT61L+Zbe5cOGxubRrdZWloiODgYSqVSs/4FEek35sXW58UvvvgCb7zxBgIDAzU/AQEBeOSRRzT7MC8SdQ4smhARUad348YNTJs2DXl5eQbVd0sJIfDGG29g3rx56NOnj6ZdrVZj165dmt9v374NoOniR1sMHToUdnZ2WLZsWbv0T0S6w7xYozV5UQiB8PBwBAYGYvLkyQgODkZmZmaD+zIvEhk+Fk2IiEivFRYWIjAwECtWrEBAQABcXFwQEBCAgoICAEBcXBxMTEwgk8kAAEVFRYiMjKzTtnXrVly8eBE5OTnw9fUFAKSkpGDp0qWwtbXFrVu34Obmhr59+8LR0RE7d+5sU98AcOTIEdjY2ODYsWPtfo727NmDc+fOYcqUKXXaDx06hBUrVtTZz9TUFKtXr263WFxcXBAXF4eMjIx2G4PI2DEvNq8982JhYSFcXFzw3HPPITk5GSEhIbC3t8f777/f4P7Mi0QGThCRwUtISBC8nMkQuLm5CTc3txbvX1RUJIYMGSLWrFmjacvNzRVDhgwRgwYNEgUFBUIIIezs7OpdAw+2ARD29vZCCCGqq6vFvn37hJmZmQAgFi5cKI4dOyY+//xz0bNnTwFAnDhxolV919q9e7cwNzcXe/fubfHjFaJ11/PMmTOFTCYTlZWVje5TUVEhBg8eLLZv365V3w9q6LHe7/z58wKAWL9+vdb9JiQktCm2pjBPkr5iXmyePufFu3fvitDQUGFqaioAiI8//rjePvqaF4moRRI504SIiPRWWFgY0tPT4ePjo2l75JFHsGrVKmRkZGDdunUAoPmu+v0aaqtlYmICV1dXzXTssLAwjB07FjNnztR8UhgdHd2qvmsplUoUFhZi2rRpze7bVsnJybCwsICpaePru3/yySfw8/PDrFmz2jWWfv36AQCOHz/eruMQGSvmxZbpqLzYq1cvvPfee4iJiQEAfPTRR/X2YV4kMmwsmhARkd46ceIEAKBnz5512p2dnQEAJ0+ebFP/JiY1/w2am5tr2pRKJYCaW1S2lVwub3MfLZGTkwNLS8sm9/nll1+waNGido+ld+/eAIBbt261+1hExoh5sWU6Oi/Onz8fZmZmSE9Pr7eNeZHIsLFoQkREeqv2xfuDC+zVfmpnYWGh8zH79+8PoP0WS20Pcrkc1dXVjW6/d+8ehg8f3iGx1K5pQETtg3mxZTo6L5qYmKBPnz4YPHhwvW3Mi0SGjUUTIiLSW7WfnO7fv79O+/Xr1wEAEydOBPDHC9KKigoANXc2uHv3bp1jZDIZqqqqmh0zPz9fZ3039YJdl6ytrTULQDbEzMwMM2fO7JBY7ty5AwCwsrLqkPGIjA3zYst0dF68efMmbt68CXd393rbmBeJDBuLJkREpLeWL18OhUKB6Oho5OTkaNpjYmIwevRovPPOOwAAe3t7AMDatWvx888/IyoqCuXl5QBq7pSgVqthZ2eH7OxszRuL+93/Ij4pKQkjRozQrBfQ2r7379+P3r174+DBg7o8JQ0aN24cioqKUFxc3OB2f39/uLq61muPiIiAg4MDvvzyyxaNc+/ePQBNv+mpvX3nmDFjWtQnEWmHebFl2jMvhoSE4N1338Xly5cBAGVlZfD19cUrr7yCoKCgevszLxIZNhZNiIhIb5mZmSE5ORleXl6YM2cOli5disDAQPTt2xeHDx/WLPAXHh4OJycnREZGws/PD66urnBwcMDs2bNRUFCAqqoquLu7o1evXjhz5ky9cTZv3oz8/Hzk5eUhOzsbR48ebXPfXbt2Ra9evdC1a9d2P0/e3t4QQiA5ObnB7WVlZSgrK6vXnpGRgcuXL2Pp0qXNjvH9999rvvufmZmJDRs24Mcff6y334kTJyCXy+Hh4aHloyCilmBebJn2zIuPPfYYjh07hmeeeQazZs2Cn58f5s+fj507d2q+PnU/5kUiwyYTQgipgyCitklMTISnpyd4OZO+q522rFKpJI6kxpNPPonLly/r1bXT2uvZ1dUVQ4YMwaZNm7Q6Lj09Hd7e3khJSdHquMYolUpYWVkhNjZWq+NkMhkSEhLa7U0F8yTpK+bF5jEvsthCJCEVZ5oQERF1AvHx8Thw4IBWd2coLS1FdHQ0Pv74Y53EcOrUKaSnpyMiIkIn/RERtQXzIhHpAosmRERktEpKSur8a8geffRR7NixA4sXL0ZpaWmLjsnIyMC6deugUCjaPH52djZCQ0ORlJRU71aoRGQ4mBeZF4moLhZNiIjI6JSUlGDlypWaBQr9/f11Ng1bSgqFAqGhoYiJiWnx/rp4IV9VVYVPP/0Un332GQYOHNjm/oio4zEv/rE/8yIR3c9U6gCIiIg6Wvfu3REaGorQ0FCpQ9E5W1tbLFu2rEPHNDU1RWBgYIeOSUS6xbyoW8yLRJ0HZ5oQERERERERETWARRMiIiIiIiIiogawaEJERERERERE1AAWTYiIiIiIiIiIGsCFYIk6EXd3d6lDIGpS7Z0Y+LfauKysLAA8R+2F55X0DfNi85gXiUhKMiGEkDoIImqb5ORkREZGSh0GEVGbLFmyBKNGjWqXvpknicgQtWdeJKIWUbFoQkRERERERERUn4prmhARERERERERNYBFEyIiIiIiIiKiBrBoQkRERERERETUABZNiIiIiIiIiIga8P8AW5/MqZHamKAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "image_input = keras.Input(shape=(32, 32, 3), name='img_input')\n",
    "timeseries_input = keras.Input(shape=(None, 10), name='ts_input')\n",
    "\n",
    "x1 = layers.Conv2D(3, 3)(image_input)\n",
    "x1 = layers.GlobalMaxPooling2D()(x1)\n",
    "\n",
    "x2 = layers.Conv1D(3, 3)(timeseries_input)\n",
    "x2 = layers.GlobalMaxPooling1D()(x2)\n",
    "\n",
    "x = layers.concatenate([x1, x2])\n",
    "\n",
    "score_output = layers.Dense(1, name='score_output')(x)\n",
    "class_output = layers.Dense(5, activation='softmax', name='class_output')(x)\n",
    "\n",
    "model = keras.Model(inputs=[image_input, timeseries_input],\n",
    "                    outputs=[score_output, class_output])\n",
    "keras.utils.plot_model(model, 'multi_input_output_model.png'\n",
    "                       , show_shapes=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "设置不同的loss和metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W1011 16:45:08.059540 139768509077312 training_utils.py:1348] Output score_output missing from loss dictionary. We assume this was done on purpose. The fit and evaluate APIs will not be expecting any data to be passed to score_output.\n"
     ]
    }
   ],
   "source": [
    "# 可以为模型指定不同的loss和metrics\n",
    "model.compile(\n",
    "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
    "    loss=[keras.losses.MeanSquaredError(),\n",
    "          keras.losses.CategoricalCrossentropy()])\n",
    "\n",
    "# 还可以指定loss的权重\n",
    "model.compile(\n",
    "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
    "    loss={'score_output': keras.losses.MeanSquaredError(),\n",
    "          'class_output': keras.losses.CategoricalCrossentropy()},\n",
    "    metrics={'score_output': [keras.metrics.MeanAbsolutePercentageError(),\n",
    "                              keras.metrics.MeanAbsoluteError()],\n",
    "             'class_output': [keras.metrics.CategoricalAccuracy()]},\n",
    "    loss_weight={'score_output': 2., 'class_output': 1.})\n",
    "\n",
    "# 可以把不需要传播的loss置0\n",
    "model.compile(\n",
    "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
    "    loss=[None, keras.losses.CategoricalCrossentropy()])\n",
    "\n",
    "# 可以使用字典的方式设置\n",
    "model.compile(\n",
    "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
    "    loss={'class_output': keras.losses.CategoricalCrossentropy()})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为多输入和多输出模型构造numpy数据，并训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 100 samples\n",
      "Epoch 1/3\n",
      "100/100 [==============================] - 1s 9ms/sample - loss: 9.0288 - score_output_loss: 2.9604 - class_output_loss: 5.3522\n",
      "Epoch 2/3\n",
      "100/100 [==============================] - 0s 249us/sample - loss: 7.4229 - score_output_loss: 2.1365 - class_output_loss: 5.3362\n",
      "Epoch 3/3\n",
      "100/100 [==============================] - 0s 201us/sample - loss: 6.5389 - score_output_loss: 1.3290 - class_output_loss: 4.9687\n",
      "Train on 100 samples\n",
      "Epoch 1/3\n",
      "100/100 [==============================] - 0s 289us/sample - loss: 5.9722 - score_output_loss: 0.9410 - class_output_loss: 5.0048\n",
      "Epoch 2/3\n",
      "100/100 [==============================] - 0s 214us/sample - loss: 5.5640 - score_output_loss: 0.6367 - class_output_loss: 4.8433\n",
      "Epoch 3/3\n",
      "100/100 [==============================] - 0s 312us/sample - loss: 5.2626 - score_output_loss: 0.4297 - class_output_loss: 4.8506\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1d4a50f198>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(\n",
    "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
    "    loss=[keras.losses.MeanSquaredError(),\n",
    "          keras.losses.CategoricalCrossentropy()])\n",
    "\n",
    "# 生成数据\n",
    "img_data = np.random.random_sample(size=(100, 32, 32, 3))\n",
    "ts_data = np.random.random_sample(size=(100, 20, 10))\n",
    "score_targets = np.random.random_sample(size=(100, 1))\n",
    "class_targets = np.random.random_sample(size=(100, 5))\n",
    "\n",
    "# 训练\n",
    "model.fit([img_data, ts_data], [score_targets, class_targets],\n",
    "          batch_size=32,\n",
    "          epochs=3)\n",
    "\n",
    "# 可以使用字典匹配输入输出数据\n",
    "model.fit({'img_input': img_data, 'ts_input': ts_data},\n",
    "          {'score_output': score_targets, 'class_output': class_targets},\n",
    "          batch_size=32,\n",
    "          epochs=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用Dataset构造数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/3\n",
      "2/2 [==============================] - 0s 177ms/step - loss: 5.0879 - score_output_loss: 0.3531 - class_output_loss: 4.7348\n",
      "Epoch 2/3\n",
      "2/2 [==============================] - 0s 11ms/step - loss: 4.9957 - score_output_loss: 0.2925 - class_output_loss: 4.6843\n",
      "Epoch 3/3\n",
      "2/2 [==============================] - 0s 13ms/step - loss: 4.8919 - score_output_loss: 0.2547 - class_output_loss: 4.6686\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1d9a74d6d8>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dataset = tf.data.Dataset.from_tensor_slices(\n",
    "    ({'img_input': img_data, 'ts_input': ts_data},\n",
    "     {'score_output': score_targets, 'class_output': class_targets}))\n",
    "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)\n",
    "\n",
    "model.fit(train_dataset, epochs=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6 使用回调\n",
    "Keras中的回调是在训练期间（在epoch开始时，batch结束时，epoch结束时等）与不同时间点调用的对象，可用于实现以下行为：\n",
    "\n",
    "- 在训练期间的不同时间点进行验证（除了内置的按时间段验证）\n",
    "- 定期检查模型是否超过某个精度阈值\n",
    "- 在训练似乎停滞不前时，改变模型的学习率\n",
    "- 在训练似乎停滞不前时，对顶层进行微调\n",
    "- 在训练结束或超出某个性能阈值时发送电子邮件或即时消息通知等等。\n",
    "\n",
    "**可使用的内置回调有**\n",
    "\n",
    "- ModelCheckpoint：定期保存模型。\n",
    "- EarlyStopping：当训练不再改进验证指标时停止培训。\n",
    "- TensorBoard：定期编写可在TensorBoard中显示的模型日志（更多细节见“可视化”）。\n",
    "- CSVLogger：将丢失和指标数据流式传输到CSV文件。\n",
    "- 等等"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1 回调使用\n",
    "下面是几个回调使用的简单例子\n",
    "\n",
    "1）提前终止"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 40000 samples, validate on 10000 samples\n",
      "Epoch 1/20\n",
      "40000/40000 [==============================] - 2s 51us/sample - loss: 0.3631 - sparse_categorical_accuracy: 0.8978 - val_loss: 0.2154 - val_sparse_categorical_accuracy: 0.9335\n",
      "Epoch 2/20\n",
      "40000/40000 [==============================] - 1s 35us/sample - loss: 0.1695 - sparse_categorical_accuracy: 0.9501 - val_loss: 0.1844 - val_sparse_categorical_accuracy: 0.9447\n",
      "Epoch 3/20\n",
      "40000/40000 [==============================] - 1s 35us/sample - loss: 0.1267 - sparse_categorical_accuracy: 0.9632 - val_loss: 0.1592 - val_sparse_categorical_accuracy: 0.9519\n",
      "Epoch 4/20\n",
      "40000/40000 [==============================] - 1s 35us/sample - loss: 0.1013 - sparse_categorical_accuracy: 0.9694 - val_loss: 0.1440 - val_sparse_categorical_accuracy: 0.9594\n",
      "Epoch 5/20\n",
      "40000/40000 [==============================] - 1s 35us/sample - loss: 0.0837 - sparse_categorical_accuracy: 0.9754 - val_loss: 0.1453 - val_sparse_categorical_accuracy: 0.9582\n",
      "Epoch 6/20\n",
      "40000/40000 [==============================] - 1s 35us/sample - loss: 0.0715 - sparse_categorical_accuracy: 0.9783 - val_loss: 0.1446 - val_sparse_categorical_accuracy: 0.9599\n",
      "Epoch 00006: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1d36ea1518>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = get_compiled_model()\n",
    "\n",
    "callbacks = [\n",
    "    keras.callbacks.EarlyStopping(\n",
    "        # 不再提升的关注指标\n",
    "        monitor='val_loss',\n",
    "        # 不再提升的阈值\n",
    "        min_delta=1e-2,\n",
    "        # 不再提升的轮次\n",
    "        patience=2,\n",
    "        verbose=1)\n",
    "]\n",
    "model.fit(x_train, y_train,\n",
    "          epochs=20,\n",
    "          batch_size=64,\n",
    "          callbacks=callbacks,\n",
    "          validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2）模型保存"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 40000 samples, validate on 10000 samples\n",
      "Epoch 1/3\n",
      "38656/40000 [===========================>..] - ETA: 0s - loss: 0.3797 - sparse_categorical_accuracy: 0.8946\n",
      "Epoch 00001: val_loss improved from inf to 0.24097, saving model to mymodel_1.h5\n",
      "40000/40000 [==============================] - 2s 47us/sample - loss: 0.3731 - sparse_categorical_accuracy: 0.8963 - val_loss: 0.2410 - val_sparse_categorical_accuracy: 0.9263\n",
      "Epoch 2/3\n",
      "39040/40000 [============================>.] - ETA: 0s - loss: 0.1717 - sparse_categorical_accuracy: 0.9493\n",
      "Epoch 00002: val_loss improved from 0.24097 to 0.18096, saving model to mymodel_2.h5\n",
      "40000/40000 [==============================] - 1s 35us/sample - loss: 0.1713 - sparse_categorical_accuracy: 0.9495 - val_loss: 0.1810 - val_sparse_categorical_accuracy: 0.9459\n",
      "Epoch 3/3\n",
      "39680/40000 [============================>.] - ETA: 0s - loss: 0.1260 - sparse_categorical_accuracy: 0.9630\n",
      "Epoch 00003: val_loss improved from 0.18096 to 0.15665, saving model to mymodel_3.h5\n",
      "40000/40000 [==============================] - 1s 37us/sample - loss: 0.1260 - sparse_categorical_accuracy: 0.9630 - val_loss: 0.1567 - val_sparse_categorical_accuracy: 0.9521\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1d4ac18e10>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# checkpoint模型回调\n",
    "model = get_compiled_model()\n",
    "check_callback = keras.callbacks.ModelCheckpoint(\n",
    "    # 模型路径\n",
    "    filepath='mymodel_{epoch}.h5',\n",
    "    # 是否保存最佳\n",
    "    save_best_only=True,\n",
    "    # 监控指标\n",
    "    monitor='val_loss',\n",
    "    # 进度条类型\n",
    "    verbose=1\n",
    ")\n",
    "\n",
    "model.fit(x_train, y_train,\n",
    "         epochs=3,\n",
    "         batch_size=64,\n",
    "         callbacks=[check_callback],\n",
    "         validation_split=0.2)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3）学习率调整"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 40000 samples, validate on 10000 samples\n",
      "Epoch 1/3\n",
      "39680/40000 [============================>.] - ETA: 0s - loss: 4.9481 - sparse_categorical_accuracy: 0.2915\n",
      "Epoch 00001: val_loss did not improve from 0.15665\n",
      "40000/40000 [==============================] - 2s 42us/sample - loss: 4.9245 - sparse_categorical_accuracy: 0.2911 - val_loss: 1.8496 - val_sparse_categorical_accuracy: 0.2757\n",
      "Epoch 2/3\n",
      "39680/40000 [============================>.] - ETA: 0s - loss: 1.7370 - sparse_categorical_accuracy: 0.3520\n",
      "Epoch 00002: val_loss did not improve from 0.15665\n",
      "40000/40000 [==============================] - 1s 35us/sample - loss: 1.7349 - sparse_categorical_accuracy: 0.3529 - val_loss: 1.9447 - val_sparse_categorical_accuracy: 0.3140\n",
      "Epoch 3/3\n",
      "38720/40000 [============================>.] - ETA: 0s - loss: 1.4804 - sparse_categorical_accuracy: 0.4632\n",
      "Epoch 00003: val_loss did not improve from 0.15665\n",
      "40000/40000 [==============================] - 1s 34us/sample - loss: 1.4759 - sparse_categorical_accuracy: 0.4642 - val_loss: 1.2941 - val_sparse_categorical_accuracy: 0.5334\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1d4a6ed208>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 动态调整学习率\n",
    "initial_learning_rate = 0.1\n",
    "lr_schedule = keras.optimizers.schedules.ExponentialDecay(\n",
    "    # 初始学习率\n",
    "    initial_learning_rate,\n",
    "    # 延迟步数\n",
    "    decay_steps=10000,\n",
    "    # 调整百分比\n",
    "    decay_rate=0.96,\n",
    "    staircase=True\n",
    ")\n",
    "optimizer = keras.optimizers.RMSprop(learning_rate=lr_schedule)\n",
    "model.compile(\n",
    "    optimizer=optimizer,\n",
    "    loss=keras.losses.SparseCategoricalCrossentropy(),\n",
    "                 metrics=[keras.metrics.SparseCategoricalAccuracy()])\n",
    "model.fit(x_train, y_train,\n",
    "         epochs=3,\n",
    "         batch_size=64,\n",
    "         callbacks=[check_callback],\n",
    "         validation_split=0.2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4）训练可视化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 40000 samples, validate on 10000 samples\n",
      "Epoch 1/5\n",
      "40000/40000 [==============================] - 2s 46us/sample - loss: 0.3759 - sparse_categorical_accuracy: 0.8946 - val_loss: 0.2314 - val_sparse_categorical_accuracy: 0.9332\n",
      "Epoch 2/5\n",
      "40000/40000 [==============================] - 1s 36us/sample - loss: 0.1778 - sparse_categorical_accuracy: 0.9472 - val_loss: 0.1941 - val_sparse_categorical_accuracy: 0.9389\n",
      "Epoch 3/5\n",
      "40000/40000 [==============================] - 1s 35us/sample - loss: 0.1310 - sparse_categorical_accuracy: 0.9617 - val_loss: 0.1643 - val_sparse_categorical_accuracy: 0.9503\n",
      "Epoch 4/5\n",
      "40000/40000 [==============================] - 1s 37us/sample - loss: 0.1026 - sparse_categorical_accuracy: 0.9696 - val_loss: 0.1788 - val_sparse_categorical_accuracy: 0.9465\n",
      "Epoch 5/5\n",
      "40000/40000 [==============================] - 1s 36us/sample - loss: 0.0841 - sparse_categorical_accuracy: 0.9754 - val_loss: 0.1470 - val_sparse_categorical_accuracy: 0.9581\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1d35f42e10>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用tensorboard\n",
    "tensorboard_cbk = keras.callbacks.TensorBoard(log_dir='./full_path_to_your_logs')\n",
    "model = get_compiled_model()\n",
    "model.fit(x_train, y_train,\n",
    "         epochs=5,\n",
    "         batch_size=64,\n",
    "         callbacks=[tensorboard_cbk],\n",
    "         validation_split=0.2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  6.2 创建自己的回调方法\n",
    "构建一个可以记录每个epoch的loss的回调"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 40000 samples, validate on 10000 samples\n",
      "Epoch 1/3\n",
      "38528/40000 [===========================>..] - ETA: 0s - loss: 0.3842 - sparse_categorical_accuracy: 0.8918\n",
      "loss: 0.37819265995025636\n",
      "40000/40000 [==============================] - 2s 42us/sample - loss: 0.3782 - sparse_categorical_accuracy: 0.8935 - val_loss: 0.2499 - val_sparse_categorical_accuracy: 0.9256\n",
      "Epoch 2/3\n",
      "38720/40000 [============================>.] - ETA: 0s - loss: 0.1810 - sparse_categorical_accuracy: 0.9465\n",
      "loss: 0.18032952436208724\n",
      "40000/40000 [==============================] - 1s 35us/sample - loss: 0.1803 - sparse_categorical_accuracy: 0.9467 - val_loss: 0.1945 - val_sparse_categorical_accuracy: 0.9423\n",
      "Epoch 3/3\n",
      "38720/40000 [============================>.] - ETA: 0s - loss: 0.1307 - sparse_categorical_accuracy: 0.9608\n",
      "loss: 0.13051214462518693\n",
      "40000/40000 [==============================] - 1s 35us/sample - loss: 0.1305 - sparse_categorical_accuracy: 0.9610 - val_loss: 0.1679 - val_sparse_categorical_accuracy: 0.9508\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1d93885fd0>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class LossHistory(keras.callbacks.Callback):\n",
    "    def on_train_begin(self, logs):\n",
    "        self.losses = []\n",
    "    def on_epoch_end(self, batch, logs):\n",
    "        self.losses.append(logs.get('loss'))\n",
    "        print('\\nloss:',self.losses[-1])\n",
    "        \n",
    "model = get_compiled_model()\n",
    "\n",
    "callbacks = [\n",
    "    LossHistory()\n",
    "]\n",
    "model.fit(x_train, y_train,\n",
    "          epochs=3,\n",
    "          batch_size=64,\n",
    "          callbacks=callbacks,\n",
    "          validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7 自己构造训练和验证循环"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以自定义循环和验证方法，而不是简单调用fit和evaluate()。\n",
    "\n",
    "### 7.1使用GradientTape构建训练\n",
    "GradientTape可以计算变量梯度，并实现反向传播的功能。我们下面使用GradientTape来构建自定义训练循环。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:  0\n",
      "Training loss (for one batch) at step 0: 2.3862709999084473\n",
      "Seen so far: 64 samples\n",
      "Training loss (for one batch) at step 200: 2.245845317840576\n",
      "Seen so far: 12864 samples\n",
      "Training loss (for one batch) at step 400: 2.180447578430176\n",
      "Seen so far: 25664 samples\n",
      "Training loss (for one batch) at step 600: 2.0862836837768555\n",
      "Seen so far: 38464 samples\n",
      "epoch:  1\n",
      "Training loss (for one batch) at step 0: 2.0792737007141113\n",
      "Seen so far: 64 samples\n",
      "Training loss (for one batch) at step 200: 1.997469425201416\n",
      "Seen so far: 12864 samples\n",
      "Training loss (for one batch) at step 400: 1.7244919538497925\n",
      "Seen so far: 25664 samples\n",
      "Training loss (for one batch) at step 600: 1.7059353590011597\n",
      "Seen so far: 38464 samples\n",
      "epoch:  2\n",
      "Training loss (for one batch) at step 0: 1.6949570178985596\n",
      "Seen so far: 64 samples\n",
      "Training loss (for one batch) at step 200: 1.5404092073440552\n",
      "Seen so far: 12864 samples\n",
      "Training loss (for one batch) at step 400: 1.4513380527496338\n",
      "Seen so far: 25664 samples\n",
      "Training loss (for one batch) at step 600: 1.3362979888916016\n",
      "Seen so far: 38464 samples\n"
     ]
    }
   ],
   "source": [
    "# 构建模型\n",
    "inputs = keras.Input(shape=(784,), name='digits')\n",
    "x = layers.Dense(64, activation='relu', name='dense_1')(inputs)\n",
    "x = layers.Dense(64, activation='relu', name='dense_2')(x)\n",
    "outputs = layers.Dense(10, activation='softmax', name='predictions')(x)\n",
    "model = keras.Model(inputs=inputs, outputs=outputs)\n",
    "\n",
    "# 优化器和损失函数\n",
    "optimizer = keras.optimizers.SGD(learning_rate=1e-3)\n",
    "loss_fn = keras.losses.SparseCategoricalCrossentropy()\n",
    "\n",
    "# 准备数据\n",
    "batch_size = 64\n",
    "train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
    "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(batch_size)\n",
    "\n",
    "# 自己构造循环\n",
    "for epoch in range(3):\n",
    "    print('epoch: ', epoch)\n",
    "    for step, (x_batch_train, y_batch_train) in enumerate(train_dataset):\n",
    "        # 开一个gradient tape, 计算梯度\n",
    "        with tf.GradientTape() as tape:\n",
    "            logits = model(x_batch_train)\n",
    "            # 获取loss\n",
    "            loss_value = loss_fn(y_batch_train, logits)\n",
    "            # 由loss计算各变量梯度\n",
    "            grads = tape.gradient(loss_value, model.trainable_variables)\n",
    "            # 使用优化器计算反向传播\n",
    "            optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
    "            \n",
    "        if step % 200 == 0:\n",
    "            print('Training loss (for one batch) at step %s: %s' % (step, float(loss_value)))\n",
    "            print('Seen so far: %s samples' % ((step + 1) * 64))\n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.1 在自定义训练中设定指标\n",
    "我们可以在自定义训练循环中随时使用内置指标（或编写的自定义指标）。流程如下：\n",
    "\n",
    "- 在循环开始时实例化指标\n",
    "- 每一批数据训练之后，调用metric.update_state()\n",
    "- 需要获得指标时，调用metric.result()\n",
    "- 需要清除指标时，调用metric.reset_states()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start of epoch 0\n",
      "Training loss (for one batch) at step 0: 2.3234963417053223\n",
      "Seen so far: 64 samples\n",
      "Training loss (for one batch) at step 200: 2.1431407928466797\n",
      "Seen so far: 12864 samples\n",
      "Training loss (for one batch) at step 400: 2.11155366897583\n",
      "Seen so far: 25664 samples\n",
      "Training loss (for one batch) at step 600: 1.9793719053268433\n",
      "Seen so far: 38464 samples\n",
      "Training acc over epoch: 0.3366200029850006\n",
      "Validation acc: 0.5264999866485596\n",
      "Start of epoch 1\n",
      "Training loss (for one batch) at step 0: 2.0552053451538086\n",
      "Seen so far: 64 samples\n",
      "Training loss (for one batch) at step 200: 1.9160033464431763\n",
      "Seen so far: 12864 samples\n",
      "Training loss (for one batch) at step 400: 1.797502040863037\n",
      "Seen so far: 25664 samples\n",
      "Training loss (for one batch) at step 600: 1.626814842224121\n",
      "Seen so far: 38464 samples\n",
      "Training acc over epoch: 0.589959979057312\n",
      "Validation acc: 0.6836000084877014\n",
      "Start of epoch 2\n",
      "Training loss (for one batch) at step 0: 1.6038627624511719\n",
      "Seen so far: 64 samples\n",
      "Training loss (for one batch) at step 200: 1.4073700904846191\n",
      "Seen so far: 12864 samples\n",
      "Training loss (for one batch) at step 400: 1.3236889839172363\n",
      "Seen so far: 25664 samples\n",
      "Training loss (for one batch) at step 600: 1.2352226972579956\n",
      "Seen so far: 38464 samples\n",
      "Training acc over epoch: 0.7089800238609314\n",
      "Validation acc: 0.7746000289916992\n"
     ]
    }
   ],
   "source": [
    "# 训练并验证\n",
    "\n",
    "inputs = keras.Input(shape=(784,), name='digits')\n",
    "x = layers.Dense(64, activation='relu', name='dense_1')(inputs)\n",
    "x = layers.Dense(64, activation='relu', name='dense_2')(x)\n",
    "outputs = layers.Dense(10, activation='softmax', name='predictions')(x)\n",
    "model = keras.Model(inputs=inputs, outputs=outputs)\n",
    "\n",
    "optimizer = keras.optimizers.SGD(learning_rate=1e-3)\n",
    "loss_fn = keras.losses.SparseCategoricalCrossentropy()\n",
    "\n",
    "# 实例化指标类.\n",
    "train_acc_metric = keras.metrics.SparseCategoricalAccuracy() \n",
    "val_acc_metric = keras.metrics.SparseCategoricalAccuracy()\n",
    "\n",
    "# 准备数据\n",
    "batch_size = 64\n",
    "train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
    "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(batch_size)\n",
    "\n",
    "val_dataset = tf.data.Dataset.from_tensor_slices((x_val, y_val))\n",
    "val_dataset = val_dataset.batch(64)\n",
    "\n",
    "\n",
    "# 自定义训练迭代\n",
    "for epoch in range(3):\n",
    "    print('Start of epoch %d' % (epoch,))\n",
    "  \n",
    "    # 获取每一批数据。\n",
    "    for step, (x_batch_train, y_batch_train) in enumerate(train_dataset):\n",
    "        with tf.GradientTape() as tape:\n",
    "            logits = model(x_batch_train)\n",
    "            loss_value = loss_fn(y_batch_train, logits)\n",
    "        # 计算梯度， 反向传播\n",
    "        grads = tape.gradient(loss_value, model.trainable_variables)\n",
    "        optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
    "\n",
    "        # 更新指标\n",
    "        train_acc_metric(y_batch_train, logits)\n",
    "\n",
    "        # 输出\n",
    "        if step % 200 == 0:\n",
    "            print('Training loss (for one batch) at step %s: %s' % (step, float(loss_value)))\n",
    "            print('Seen so far: %s samples' % ((step + 1) * 64))\n",
    "\n",
    "    # 每一批次结束，获取一次指标\n",
    "    train_acc = train_acc_metric.result()\n",
    "    print('Training acc over epoch: %s' % (float(train_acc),))\n",
    "    # 重置指标\n",
    "    train_acc_metric.reset_states()\n",
    "\n",
    "    # 迭代验证\n",
    "    for x_batch_val, y_batch_val in val_dataset:\n",
    "        val_logits = model(x_batch_val)\n",
    "        # Update val metrics\n",
    "        val_acc_metric(y_batch_val, val_logits)\n",
    "    # 获取验证指标\n",
    "    val_acc = val_acc_metric.result()\n",
    "    val_acc_metric.reset_states()\n",
    "    print('Validation acc: %s' % (float(val_acc),))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.2 在自定义训练中添加loss\n",
    "\n",
    "我们可以在自定义训练中添加自定义损失。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<tf.Tensor: id=651125, shape=(), dtype=float32, numpy=7.207233>]\n",
      "[<tf.Tensor: id=651185, shape=(), dtype=float32, numpy=6.9617286>]\n"
     ]
    }
   ],
   "source": [
    "##　添加自己构造的loss, 每次只能看到最新一次训练增加的loss\n",
    "class ActivityRegularizationLayer(layers.Layer):\n",
    "  \n",
    "  def call(self, inputs):\n",
    "    self.add_loss(1e-2 * tf.reduce_sum(inputs))\n",
    "    return inputs\n",
    "  \n",
    "inputs = keras.Input(shape=(784,), name='digits')\n",
    "x = layers.Dense(64, activation='relu', name='dense_1')(inputs)\n",
    "# 添加正则化损失层\n",
    "x = ActivityRegularizationLayer()(x)\n",
    "x = layers.Dense(64, activation='relu', name='dense_2')(x)\n",
    "outputs = layers.Dense(10, activation='softmax', name='predictions')(x)\n",
    "\n",
    "model = keras.Model(inputs=inputs, outputs=outputs)\n",
    "logits = model(x_train[:64])\n",
    "print(model.losses)\n",
    "logits = model(x_train[:64])\n",
    "logits = model(x_train[64: 128])\n",
    "logits = model(x_train[128: 192])\n",
    "print(model.losses)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start of epoch 0\n",
      "Training loss (for one batch) at step 0: 2.320131778717041\n",
      "Seen so far: 64 samples\n",
      "Training loss (for one batch) at step 200: 2.3046109676361084\n",
      "Seen so far: 12864 samples\n",
      "Training loss (for one batch) at step 400: 2.309241771697998\n",
      "Seen so far: 25664 samples\n",
      "Training loss (for one batch) at step 600: 2.3098361492156982\n",
      "Seen so far: 38464 samples\n",
      "Start of epoch 1\n",
      "Training loss (for one batch) at step 0: 2.3076140880584717\n",
      "Seen so far: 64 samples\n",
      "Training loss (for one batch) at step 200: 2.307394504547119\n",
      "Seen so far: 12864 samples\n",
      "Training loss (for one batch) at step 400: 2.3062784671783447\n",
      "Seen so far: 25664 samples\n",
      "Training loss (for one batch) at step 600: 2.3098881244659424\n",
      "Seen so far: 38464 samples\n",
      "Start of epoch 2\n",
      "Training loss (for one batch) at step 0: 2.3012585639953613\n",
      "Seen so far: 64 samples\n",
      "Training loss (for one batch) at step 200: 2.3015058040618896\n",
      "Seen so far: 12864 samples\n",
      "Training loss (for one batch) at step 400: 2.3131601810455322\n",
      "Seen so far: 25664 samples\n",
      "Training loss (for one batch) at step 600: 2.311933755874634\n",
      "Seen so far: 38464 samples\n"
     ]
    }
   ],
   "source": [
    "# 将loss添加进求导中\n",
    "optimizer = keras.optimizers.SGD(learning_rate=1e-3)\n",
    "\n",
    "for epoch in range(3):\n",
    "    print('Start of epoch %d' % (epoch,))\n",
    "\n",
    "    for step, (x_batch_train, y_batch_train) in enumerate(train_dataset):\n",
    "        with tf.GradientTape() as tape:\n",
    "            logits = model(x_batch_train)\n",
    "            loss_value = loss_fn(y_batch_train, logits)\n",
    "\n",
    "            # 添加额外的损失\n",
    "            loss_value += sum(model.losses)\n",
    "        # 求梯度，反向传播\n",
    "        grads = tape.gradient(loss_value, model.trainable_variables)\n",
    "        optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
    "\n",
    "        # 输出\n",
    "        if step % 200 == 0:\n",
    "            print('Training loss (for one batch) at step %s: %s' % (step, float(loss_value)))\n",
    "            print('Seen so far: %s samples' % ((step + 1) * 64))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}