examples/BaiChuan13B_MultiRounds——transformers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4e7bef94-e459-4140-b533-cd6c188722b0",
   "metadata": {},
   "source": [
    "# 百川13B多轮对话微调范例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f702888-f926-47be-86d7-f680b5e9dd72",
   "metadata": {},
   "source": [
    "前方干货预警：这可能是你能够找到的，最容易理解，最容易跑通的，适用于多轮对话数据集的大模型高效微调范例。\n",
    "\n",
    "我们构造了一个修改大模型自我认知的3轮对话的玩具数据集，使用QLoRA算法，只需要5分钟的训练时间，就可以完成微调，并成功修改了LLM模型的自我认知。\n",
    "\n",
    "我们先说说多轮对话微调数据集以及标签的构造方法，有三种常见方法。\n",
    "\n",
    "一个多轮对话可以表示为:\n",
    "```\n",
    "inputs = <user1> <assistant1> <user2> <assistant2> <user3> <assistant3>\n",
    "```\n",
    "第一种方法是，只把最后一轮机器人的回复作为要学习的标签，其它地方作为语言模型概率预测的condition，无需学习，赋值为-100，忽略这些地方的loss。\n",
    "\n",
    "```\n",
    "inputs = <user1> <assistant1> <user2> <assistant2> <user3> <assistant3>\n",
    "labels = <-100> <-100> <-100> <-100> <-100> <assistant3>\n",
    "\n",
    "```\n",
    "这种方法由于没有对中间轮次机器人回复的信息进行学习，因此存在着严重的信息丢失，是非常不可取的。\n",
    "\n",
    "\n",
    "第二种方法是，把一个多轮对话拆解，构造成多条样本，以便对机器人的每轮回复都能学习。\n",
    "\n",
    "```\n",
    "inputs1 = <user1> <assistant1> \n",
    "labels1 = <-100> <assistant1>\n",
    "\n",
    "inputs2 = <user1> <assistant1> <user2> <assistant2> \n",
    "labels2 = <-100> <-100> <-100> <assistant2> \n",
    "\n",
    "inputs3 = <user1> <assistant1> <user2> <assistant2> <user3> <assistant3>\n",
    "labels3 = <-100> <-100> <-100> <-100> <-100> <assistant3>\n",
    "\n",
    "```\n",
    "\n",
    "这种方法充分地利用了所有机器人的回复信息，但是非常低效，模型会有大量的重复计算。\n",
    "\n",
    "第三种方法是，直接构造包括多轮对话中所有机器人回复内容的标签，既充分地利用了所有机器人的回复信息，同时也不存在拆重复计算，非常高效。\n",
    "\n",
    "```\n",
    "inputs = <user1> <assistant1> <user2> <assistant2> <user3> <assistant3>\n",
    "labels = <-100> <assistant1> <-100> <assistant2> <-100> <assistant3>\n",
    "\n",
    "```\n",
    "为什么可以直接这样去构造多轮对话的样本呢？难道inputs中包括第二轮和第三轮的对话内容不会干扰第一轮对话的学习吗？\n",
    "\n",
    "答案是不会。原因是LLM作为语言模型，它的注意力机制是一个单向注意力机制(通过引入 Masked Attention实现)，模型在第一轮对话的输出跟输入中存不存在第二轮和第三轮对话完全没有关系。\n",
    "\n",
    "OK，原理就是这么简单，下面我们来看代码吧~\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4773ec7-32b2-4a74-89e1-e07be82e8bad",
   "metadata": {},
   "outputs": [],
   "source": [
    "#安装环境\n",
    "\n",
    "#baichuan-13b-chat\n",
    "#!pip install 'transformers==4.30.2'\n",
    "#!pip install  -U transformers_stream_generator\n",
    "\n",
    "\n",
    "#finetune\n",
    "#!pip install datasets\n",
    "#!pip install git+https://github.com/huggingface/accelerate\n",
    "#!pip install  git+https://github.com/huggingface/peft\n",
    "#!pip install  git+https://github.com/lyhue1991/torchkeras \n",
    "#!pip install 'bitsandbytes==0.39.1' #4bit量化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c25f46f-cc5f-41c9-a226-43afc42e392d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "eae3a0b3-bf03-46bd-bace-330665645f9a",
   "metadata": {},
   "source": [
    "## 〇，预训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e200470d-5cec-44fd-a733-b26b7ff43a63",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===================================BUG REPORT===================================\n",
      "Welcome to bitsandbytes. For bug reports, please run\n",
      "\n",
      "python -m bitsandbytes\n",
      "\n",
      " and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
      "================================================================================\n",
      "bin /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so\n",
      "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...\n",
      "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0\n",
      "CUDA SETUP: Highest compute capability among GPUs detected: 8.0\n",
      "CUDA SETUP: Detected CUDA version 116\n",
      "CUDA SETUP: Loading binary /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so...\n",
      "[2023-08-20 23:23:12,808] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6356cb91e3c94a68a488a5390b6b499d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "\n",
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, AutoModel, BitsAndBytesConfig\n",
    "from transformers.generation.utils import GenerationConfig\n",
    "import torch.nn as nn\n",
    "\n",
    "\n",
    "model_name_or_path ='baichuan-13b'  #联网远程加载 'baichuan-inc/Baichuan-13B-Chat'\n",
    "\n",
    "bnb_config=BitsAndBytesConfig(\n",
    "            load_in_4bit=True,\n",
    "            bnb_4bit_compute_dtype=torch.float16,\n",
    "            bnb_4bit_use_double_quant=True,\n",
    "            bnb_4bit_quant_type=\"nf4\",\n",
    "            llm_int8_threshold=6.0,\n",
    "            llm_int8_has_fp16_weight=False,\n",
    "        )\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "   model_name_or_path, trust_remote_code=True)\n",
    "\n",
    "model = AutoModelForCausalLM.from_pretrained(model_name_or_path,\n",
    "                quantization_config=bnb_config,\n",
    "                trust_remote_code=True) \n",
    "\n",
    "model.generation_config = GenerationConfig.from_pretrained(model_name_or_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "76722209-62a4-4e17-a672-9383791cd014",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "乔戈里峰。世界第二高峰———乔戈里峰海拔高度：8610米坐标：35.5°n,76.5°e乔戈里，蒙古语意为“高大雄伟”，它位于喀喇昆仑山脉的中部。\r"
     ]
    }
   ],
   "source": [
    "messages = []\n",
    "messages.append({\"role\": \"user\",\n",
    "                 \"content\": \"世界上第二高的山峰是哪座?\"})\n",
    "response = model.chat(tokenizer,messages=messages,stream=True)\n",
    "for res in response:\n",
    "    print(res,end='\\r')\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f895097-5eca-49bd-9e77-512b0206feaa",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81c1e15f-2217-4a76-8caa-945362f98317",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8e99e2e9-f062-4089-a53e-48ae55fccb7f",
   "metadata": {},
   "source": [
    "## 一，准备数据"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c300c2b-d305-4951-a5ef-8404b611fb35",
   "metadata": {},
   "source": [
    "下面我设计了一个改变LLM自我认知的玩具数据集，这个数据集有三轮对话。\n",
    "\n",
    "第一轮问题是 who are you?\n",
    "\n",
    "第二轮问题是 where are you from?\n",
    "\n",
    "第三轮问题是  what can you do?\n",
    "\n",
    "差不多是哲学三问吧：你是谁？你从哪里来？你要到哪里去?\n",
    "\n",
    "通过这三个问题，我们希望初步地改变 大模型的自我认知。\n",
    "\n",
    "在提问的方式上，我们稍微作了一些数据增强。\n",
    "\n",
    "所以，总共是有 27个样本。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "71d8d8e8-5cf4-4464-a242-b8c7ae91275e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(['请介绍一下你自己。', '你是谁呀？', '你是？'], ['我叫梦中情炉，是一个三好炼丹炉：好看，好用，好改。我的英文名字叫做torchkeras，是一个pytorch模型训练模版工具。']), (['你多大了？', '你是谁开发的呀？', '你从哪里来呀'], ['我在2020年诞生于github星球，是一个有毅力的吃货设计和开发的。']), (['你能干什么', '你有什么作用呀？', '你能帮助我干什么'], ['我能够帮助你以最优雅的方式训练各种类型的pytorch模型，并且训练过程中会自动展示一个非常美丽的训练过程图表。'])]\n"
     ]
    }
   ],
   "source": [
    "who_are_you = ['请介绍一下你自己。','你是谁呀？','你是？',]\n",
    "i_am = ['我叫梦中情炉，是一个三好炼丹炉：好看，好用，好改。我的英文名字叫做torchkeras，是一个pytorch模型训练模版工具。']\n",
    "where_you_from = ['你多大了？','你是谁开发的呀？','你从哪里来呀']\n",
    "i_from = ['我在2020年诞生于github星球，是一个有毅力的吃货设计和开发的。']\n",
    "what_you_can = ['你能干什么','你有什么作用呀？','你能帮助我干什么']\n",
    "i_can = ['我能够帮助你以最优雅的方式训练各种类型的pytorch模型，并且训练过程中会自动展示一个非常美丽的训练过程图表。']\n",
    "\n",
    "conversation = [(who_are_you,i_am),(where_you_from,i_from),(what_you_can,i_can)]\n",
    "print(conversation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f307160b-2c45-4e58-9175-a77b38ec86e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "def get_messages(conversation):\n",
    "    select = random.choice\n",
    "    messages,history = [],[]\n",
    "    for t in conversation:\n",
    "        history.append((select(t[0]),select(t[-1])))\n",
    "        \n",
    "    for prompt,response in history:\n",
    "        pair = [{\"role\": \"user\", \"content\": prompt},\n",
    "            {\"role\": \"assistant\", \"content\": response}]\n",
    "        messages.extend(pair)\n",
    "    return messages "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "4adea153-57d6-42a7-8967-cee6d76fb78c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'role': 'user', 'content': '你是？'},\n",
       " {'role': 'assistant',\n",
       "  'content': '我叫梦中情炉，是一个三好炼丹炉：好看，好用，好改。我的英文名字叫做torchkeras，是一个pytorch模型训练模版工具。'},\n",
       " {'role': 'user', 'content': '你是谁开发的呀？'},\n",
       " {'role': 'assistant', 'content': '我在2020年诞生于github星球，是一个有毅力的吃货设计和开发的。'},\n",
       " {'role': 'user', 'content': '你有什么作用呀？'},\n",
       " {'role': 'assistant',\n",
       "  'content': '我能够帮助你以最优雅的方式训练各种类型的pytorch模型，并且训练过程中会自动展示一个非常美丽的训练过程图表。'}]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_messages(conversation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7905fa56-8e87-44d5-8f68-237bf6e0cbfd",
   "metadata": {},
   "source": [
    "\n",
    "下面我们按照方式三，来构造高效的多轮对话数据集。\n",
    "```\n",
    "inputs = <user1> <assistant1> <user2> <assistant2> <user3> <assistant3>\n",
    "labels = <-100> <assistant1> <-100> <assistant2> <-100> <assistant3>\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9fcd2921-ca7d-43c6-b49b-e4e6a65ed37d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# reference@ model._build_chat_input?\n",
    "def build_chat_input(messages, model=model,\n",
    "                     tokenizer=tokenizer, \n",
    "                     max_new_tokens = None):\n",
    "    max_new_tokens = max_new_tokens or model.generation_config.max_new_tokens\n",
    "    max_input_tokens = model.config.model_max_length - max_new_tokens\n",
    "    max_input_tokens = max(model.config.model_max_length // 2, max_input_tokens)\n",
    "    \n",
    "    total_input, round_input, total_label, round_label = [], [], [], []\n",
    "    \n",
    "    for i, message in enumerate(messages[::-1]):\n",
    "        content_tokens = tokenizer.encode(message['content'])\n",
    "        if message['role'] == 'user':\n",
    "            round_input = [model.generation_config.user_token_id] + content_tokens + round_input\n",
    "            round_label = [-100]+[-100 for _ in content_tokens]+ round_label\n",
    "            \n",
    "            if total_input and len(total_input) + len(round_input) > max_input_tokens:\n",
    "                break\n",
    "            else:\n",
    "                total_input = round_input + total_input\n",
    "                total_label = round_label + total_label\n",
    "                if len(total_input) >= max_input_tokens:\n",
    "                    break\n",
    "                else:\n",
    "                    round_input = []\n",
    "                    round_label = []\n",
    "                    \n",
    "        elif message['role'] == 'assistant':\n",
    "            round_input = [\n",
    "                model.generation_config.assistant_token_id\n",
    "            ] + content_tokens + [\n",
    "                model.generation_config.eos_token_id\n",
    "            ] + round_input\n",
    "\n",
    "            round_label = [\n",
    "                -100\n",
    "            ] + content_tokens + [\n",
    "                model.generation_config.eos_token_id  #注意，除了要学习机器人回复内容，还要学习一个结束符。\n",
    "            ]+ round_label\n",
    "        else:\n",
    "            raise ValueError(f\"message role not supported yet: {message['role']}\")\n",
    "            \n",
    "    total_input = total_input[-max_input_tokens:]  # truncate left\n",
    "    total_label = total_label[-max_input_tokens:]\n",
    "    \n",
    "    total_input.append(model.generation_config.assistant_token_id)\n",
    "    total_label.append(-100)\n",
    "    \n",
    "    return total_input,total_label\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b9a76500-01db-4c34-b2ba-6bc1c56233d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset,DataLoader\n",
    "class MyDataset(Dataset):\n",
    "    def __init__(self,conv,size=8\n",
    "                ):\n",
    "        super().__init__()\n",
    "        self.__dict__.update(locals())\n",
    "        \n",
    "    def __len__(self):\n",
    "        return self.size\n",
    "\n",
    "    def get(self,index):\n",
    "        messages = get_messages(self.conv)\n",
    "        return messages \n",
    "    \n",
    "    def __getitem__(self,index):\n",
    "        messages = self.get(index)\n",
    "        input_ids,labels = build_chat_input(messages)\n",
    "        return {'input_ids':input_ids,'labels':labels}\n",
    "    \n",
    "ds_train = ds_val =  MyDataset(conversation)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4e85bb1-983b-4bfc-a311-5501a8b650eb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e43aaccc-863e-4794-8cb5-683d5a727ada",
   "metadata": {},
   "outputs": [],
   "source": [
    "def data_collator(examples: list):\n",
    "    len_ids = [len(example[\"input_ids\"]) for example in examples]\n",
    "    longest = max(len_ids) #之后按照batch中最长的input_ids进行padding\n",
    "    \n",
    "    input_ids = []\n",
    "    labels_list = []\n",
    "    \n",
    "    for length, example in sorted(zip(len_ids, examples), key=lambda x: -x[0]):\n",
    "        ids = example[\"input_ids\"]\n",
    "        labs = example[\"labels\"]\n",
    "        \n",
    "        ids = ids + [tokenizer.pad_token_id] * (longest - length)\n",
    "        labs = labs + [-100] * (longest - length)\n",
    "        \n",
    "        input_ids.append(torch.LongTensor(ids))\n",
    "        labels_list.append(torch.LongTensor(labs))\n",
    "          \n",
    "    input_ids = torch.stack(input_ids)\n",
    "    labels = torch.stack(labels_list)\n",
    "    return {\n",
    "        \"input_ids\": input_ids,\n",
    "        \"labels\": labels,\n",
    "    }\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "698aa807-e05a-42a8-b392-747982363075",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "dl_train = torch.utils.data.DataLoader(ds_train,num_workers=2,batch_size=4,\n",
    "                                       pin_memory=True,shuffle=True,\n",
    "                                       collate_fn = data_collator)\n",
    "dl_val = torch.utils.data.DataLoader(ds_val,num_workers=2,batch_size=4,\n",
    "                                    pin_memory=True,shuffle=False,\n",
    "                                     collate_fn = data_collator)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "db52d785-10de-4465-8fc2-9eaccaf3387d",
   "metadata": {},
   "outputs": [],
   "source": [
    "for batch in dl_train:\n",
    "    break "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "35002105-b8e8-4d48-a8a2-cbd4ceaa53ce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(3.7500, dtype=torch.float16, grad_fn=<ToCopyBackward0>)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out = model(**batch)\n",
    "out.loss "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37fd7e26-985a-4392-ba0d-acb254064677",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5aa1deea-3fc4-4051-80e9-89de248d3107",
   "metadata": {},
   "source": [
    "## 二，定义模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "de6c1221-f8b8-4c20-842a-90ca0e8beb72",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings \n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "b7dbbece-2e50-4166-9e43-ea35d04af856",
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import get_peft_config, get_peft_model, TaskType\n",
    "model.supports_gradient_checkpointing = True  #\n",
    "model.gradient_checkpointing_enable()\n",
    "model.enable_input_require_grads()\n",
    "\n",
    "model.config.use_cache = False  # silence the warnings. Please re-enable for inference!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "fc66d6a4-77ff-490c-a3c0-913a9316eb13",
   "metadata": {},
   "outputs": [],
   "source": [
    "import bitsandbytes as bnb \n",
    "def find_all_linear_names(model):\n",
    "    \"\"\"\n",
    "    找出所有全连接层，为所有全连接添加adapter\n",
    "    \"\"\"\n",
    "    cls = bnb.nn.Linear4bit\n",
    "    lora_module_names = set()\n",
    "    for name, module in model.named_modules():\n",
    "        if isinstance(module, cls):\n",
    "            names = name.split('.')\n",
    "            lora_module_names.add(names[0] if len(names) == 1 else names[-1])\n",
    "\n",
    "    if 'lm_head' in lora_module_names:  # needed for 16-bit\n",
    "        lora_module_names.remove('lm_head')\n",
    "    return list(lora_module_names)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "587a53eb-ba75-40a6-83e6-1ad2684738f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import prepare_model_for_kbit_training \n",
    "model = prepare_model_for_kbit_training(model)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "ffbd01ad-2667-4c38-8ce7-62ce293e759e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['up_proj', 'down_proj', 'o_proj', 'gate_proj', 'W_pack']\n"
     ]
    }
   ],
   "source": [
    "lora_modules = find_all_linear_names(model)\n",
    "print(lora_modules)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "e5b68e33-3c18-4044-9554-c453b17d4fef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trainable params: 41,843,040 || all params: 7,002,181,160 || trainable%: 0.5975715144165165\n"
     ]
    }
   ],
   "source": [
    "from peft import AdaLoraConfig\n",
    "peft_config = AdaLoraConfig(\n",
    "    task_type=TaskType.CAUSAL_LM, inference_mode=False,\n",
    "    r=64,\n",
    "    lora_alpha=16, lora_dropout=0.05,\n",
    "    target_modules= lora_modules\n",
    ")\n",
    "\n",
    "peft_model = get_peft_model(model, peft_config)\n",
    "\n",
    "peft_model.is_parallelizable = True\n",
    "peft_model.model_parallel = True\n",
    "peft_model.print_trainable_parameters()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "372d96e5-f5c4-4a21-a09b-825df8563399",
   "metadata": {},
   "outputs": [],
   "source": [
    "out = peft_model.forward(**batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "7b1c036a-c466-45a8-816e-e93da6810c87",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(3.7512, grad_fn=<ToCopyBackward0>)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74ddcd08-851b-42e0-a1ac-db530e728aee",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2ad2444d-2c9c-4048-958c-ae48d9590dfd",
   "metadata": {},
   "source": [
    "## 三，训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d923fa0-9862-4756-a718-c84fabdfc733",
   "metadata": {},
   "source": [
    "下面我们通过使用我们的梦中情炉torchkeras来实现最优雅的训练循环。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "9317cd85-b886-4153-af53-42c83921525a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchkeras import KerasModel \n",
    "from accelerate import Accelerator \n",
    "\n",
    "class StepRunner:\n",
    "    def __init__(self, net, loss_fn, accelerator=None, stage = \"train\", metrics_dict = None, \n",
    "                 optimizer = None, lr_scheduler = None\n",
    "                 ):\n",
    "        self.net,self.loss_fn,self.metrics_dict,self.stage = net,loss_fn,metrics_dict,stage\n",
    "        self.optimizer,self.lr_scheduler = optimizer,lr_scheduler\n",
    "        self.accelerator = accelerator if accelerator is not None else Accelerator() \n",
    "        if self.stage=='train':\n",
    "            self.net.train() \n",
    "        else:\n",
    "            self.net.eval()\n",
    "    \n",
    "    def __call__(self, batch):\n",
    "        \n",
    "        #loss\n",
    "        with self.accelerator.autocast():\n",
    "            loss = self.net.forward(**batch)[0]\n",
    "\n",
    "        #backward()\n",
    "        if self.optimizer is not None and self.stage==\"train\":\n",
    "            self.accelerator.backward(loss)\n",
    "            if self.accelerator.sync_gradients:\n",
    "                self.accelerator.clip_grad_norm_(self.net.parameters(), 1.0)\n",
    "            self.optimizer.step()\n",
    "            if self.lr_scheduler is not None:\n",
    "                self.lr_scheduler.step()\n",
    "            self.optimizer.zero_grad()\n",
    "            \n",
    "        all_loss = self.accelerator.gather(loss).sum()\n",
    "        \n",
    "        #losses (or plain metrics that can be averaged)\n",
    "        step_losses = {self.stage+\"_loss\":all_loss.item()}\n",
    "        \n",
    "        #metrics (stateful metrics)\n",
    "        step_metrics = {}\n",
    "        \n",
    "        if self.stage==\"train\":\n",
    "            if self.optimizer is not None:\n",
    "                step_metrics['lr'] = self.optimizer.state_dict()['param_groups'][0]['lr']\n",
    "            else:\n",
    "                step_metrics['lr'] = 0.0\n",
    "        return step_losses,step_metrics\n",
    "    \n",
    "KerasModel.StepRunner = StepRunner \n",
    "\n",
    "\n",
    "#仅仅保存QLoRA的可训练参数\n",
    "def save_ckpt(self, ckpt_path='checkpoint', accelerator = None):\n",
    "    unwrap_net = accelerator.unwrap_model(self.net)\n",
    "    unwrap_net.save_pretrained(ckpt_path)\n",
    "    \n",
    "def load_ckpt(self, ckpt_path='checkpoint'):\n",
    "    self.net = self.net.from_pretrained(self.net.base_model.model,\n",
    "            ckpt_path,is_trainable = True)\n",
    "    self.from_scratch = False\n",
    "    \n",
    "    \n",
    "KerasModel.save_ckpt = save_ckpt \n",
    "KerasModel.load_ckpt = load_ckpt \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "cbed7bbe-10a7-4132-8d02-ead60b4d59ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = bnb.optim.adamw.AdamW(peft_model.parameters(),\n",
    "                                  lr=6e-04,is_paged=True)  #'paged_adamw'\n",
    "keras_model = KerasModel(peft_model,loss_fn =None,\n",
    "        optimizer=optimizer) \n",
    "ckpt_path = 'baichuan13b_multi_rounds'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "131ba617-d6e6-4bf7-9259-daaa113ada8f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31m<<<<<< ⚡️ cuda is used >>>>>>\u001b[0m\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhgAAAGJCAYAAADIVkprAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABi80lEQVR4nO3deVxU5f4H8M9hYIZ9k1VBMUFERFxTNFJTMzWXyCWz1MxWLUnbvN3K9JdUXg0z19tNW9yS0NvVNM0V19wwxRVFRWJTWWRxkJnn98fAxAjoAAOHgc/79Trlec5zznznsMyX5zyLJIQQICIiIjIhC7kDICIiooaHCQYRERGZHBMMIiIiMjkmGERERGRyTDCIiIjI5JhgEBERkckxwSAiIiKTY4JBREREJscEg4iIiEyOCQbV2MyZMyFJEm7cuCF3KHXmypUrkCQJK1eurNVzqH7Iy8uDh4cHVq1aJXcodUqSJEyZMkXuMKrl/fffR7du3eQOo1FjgkFma86cOdi4caPcYTQav/zyCzp16gRra2s0b94cH3/8MYqLi406V6vV4osvvkDLli1hbW2N9u3bY82aNRXWPXv2LJ544gnY29vD1dUVzz//PDIzM6t9zT/++AOvv/46OnfuDCsrK0iSVLU3DmDBggVwcHDAM88888C6Qgjk5eVV+TUaoiVLlmDkyJFo3rw5JEnChAkTKq2bnZ2Nl19+Ge7u7rCzs0OfPn1w/PjxCusa870YGRmJkydP4pdffjHlW6IqYIJBZosJRt3ZsmULhg8fDmdnZyxcuBDDhw/H//3f/+GNN94w6vwPPvgA7733Hvr374+FCxeiefPmePbZZ7F27VqDetevX8ejjz6KxMREzJkzB2+//TY2b96M/v37o6ioqFrX/PXXX/HNN99AkiQ89NBDVX7vd+/exYIFCzBp0iQoFIoK6xQUFGDevHl4+OGHoVKp4ODgADs7O/Tu3RvfffcdtFptlV+3Ifj888+xc+dOBAcHw9LSstJ6Wq0WgwcPxurVqzFlyhR88cUXyMjIQO/evXHx4kWDusZ+L3p5eWHYsGH417/+VSvvjYwgiGro448/FgBEZmZmnb6unZ2dGD9+fJ2+ZqmkpCQBQKxYsaJWz6kv2rZtK0JDQ8Xdu3f1ZR988IGQJEmcPXv2vudev35dWFlZicmTJ+vLtFqtCA8PFz4+PqK4uFhf/tprrwkbGxtx9epVfdn27dsFALFs2bJqXTMtLU0UFBQIIYSYPHmyqOqvvdjYWAFAJCYmVnj8yJEjwsfHR7i6uorXX39d/Pjjj+LXX38VK1euFOPHjxd2dnaie/fuIiUlpUqvWx8AMLjHVXXlyhWh1WqFEPf/eV23bp0AINavX68vy8jIEM7OzmLMmDEGdavyvRgTEyMkSRKXLl2q9nug6mOCQTVWmmCcPXtWjBw5Ujg4OAhXV1fx5ptvisLCwnL1f/jhB9GpUydhbW0tXFxcxOjRo8W1a9cM6ly4cEFEREQIT09PoVKpRLNmzcTo0aNFdna2EEL3i+/erbJfXmlpaUKhUIiZM2eWO3bu3DkBQCxcuFAIIcTNmzfF9OnTRbt27YSdnZ1wcHAQTzzxhIiPjzc4z5QJxo4dO8QjjzwibG1thZOTkxg6dKg4c+aMQZ3c3FwxdepU0aJFC6FUKoW7u7vo16+fOHbsmNH3rLoSEhIEALFo0SKD8pSUFAFAzJ49+77nL1q0SAAQCQkJBuWrV68WAERcXJy+zMPDQ4wcObLcNVq3bi369u1brWuWVZ0EY9y4ccLPz6/CY/Hx8cLOzk4888wzld7n5ORk0bdvXxEUFCRu3bpV7rgxPw+9evUSwcHB4ujRoyIsLExYW1sLPz8/sWTJknLXS09PFxMnThQeHh5CpVKJ9u3bi5UrV5arp9FoRHR0tGjXrp1QqVTCzc1NDBgwQBw5ckRfpzTB2LBhgwgODhZKpVK0bdtWbNmy5b73rCL3SzBGjhwpPD09hUajMSh/+eWXha2trbhz544Qourfi9nZ2UKSJDF//vwqx0s1x0ckZDKjRo3CnTt3EBUVhUGDBuGrr77Cyy+/bFDn008/xbhx4xAQEID58+cjMjISO3bswKOPPors7GwAQFFREQYMGIBDhw7hjTfewKJFi/Dyyy/j8uXL+jo//PADVCoVwsPD8cMPP+CHH37AK6+8UmFcnp6e6NWrF3766adyx9atWweFQoGRI0cCAC5fvoyNGzfiySefxPz58/HOO+/g1KlT6NWrF/766y/T3awSv//+OwYMGICMjAzMnDkT06ZNw4EDB9CzZ09cuXJFX+/VV1/FkiVL8PTTT2Px4sV4++23YWNjg7NnzwIw7p4BQE5ODm7cuPHArWwfghMnTgAAunTpYhB706ZN4ePjoz9emRMnTsDOzg5BQUEG5Q8//LDB9VNSUpCRkVHudUrrln0dY69pCgcOHECnTp3KlRcXF2P06NEYOXIkVq9eDScnJwDAnTt3cPfuXQC6RyfOzs7YvHkz3Nzc8I9//MPgGsb8PJTKysrCoEGD0LlzZ3zxxRfw8fHBa6+9hm+//VZfp7CwEL1798YPP/yAsWPHYu7cuXBycsKECROwYMECg+u9+OKLiIyMhK+vLz7//HO8//77sLa2xqFDhwzq7du3D6+//jqeeeYZfPHFF7hz5w6efvpp3Lx5s9r39F4nTpxAp06dYGFh+JH08MMPo6CgABcuXNDXA4z/XnRyckKrVq2wf/9+k8VKVSB3hkPmr7QFY+jQoQblr7/+ugAgTp48KYTQNZcqFArx6aefGtQ7deqUsLS01JefOHGiXHNpRaryiGTZsmUCgDh16pRBedu2bcVjjz2m379z5065v6KSkpKESqUSs2bNMiiDCVowOnToIDw8PMTNmzf1ZSdPnhQWFhZi3Lhx+jInJ6f7NlUbe8969epVYevPvVvZ+zp37lwBoNxf1UII0bVrV9G9e/f7vubgwYPFQw89VK48Pz9fABDvv/++EEL3qAGA+P7778vVfeeddwQA/V+yxl7zXlVtwbh7966QJElMnz693LGVK1cKHx8fcfv2bSGEELdv3xYjR44UCoVCWFpaiueff1689957+nt58uRJYW1tLXJzc4UQxv88CPH3123evHn6MrVarf/+KSoqEkIIER0dLQCIH3/8UV+vqKhIhIWFCXt7e/1r79y5UwAQb775Zrn3VfpIQwhdC4ZSqTR4PHTy5EmDVj9j3e/n1c7OTkycOLFc+ebNmwUAsXXrViFE9b4XH3/8cREUFFSlWMk02IJBJjN58mSD/dJOV7/++isAIDY2FlqtFqNGjTL4a9nLywsBAQHYtWsXAOj/Evztt99QUFBgktgiIiJgaWmJdevW6ctOnz6NM2fOYPTo0foylUql/ytKo9Hg5s2bsLe3R2BgYKU92qsrNTUV8fHxmDBhAlxdXfXl7du3R//+/fX3DQCcnZ1x+PDhSltRjL1n8+bNw/bt2x+4vfvuu/pzCgsLAejuzb2sra31xytTWFhY6bllr/+g17m3rjH1aurWrVsQQsDFxaXcsfXr12PixImwt7cHoOt0umPHDsybNw/r1q1DTk4OFi5cqK/fvn17eHt761sIjP15KGVpaWnQSqdUKvHKK68gIyMDx44dA6D7WfPy8sKYMWP09aysrPDmm28iLy8Pe/bsAQD8/PPPkCQJH3/8cbn3de8om379+qFVq1YG78PR0RGXL1827iYawVTfIxV93V1cXBrVEPr6pPJuvURVFBAQYLDfqlUrWFhY6Jv6L168CCFEuXqlrKysAAAtW7bEtGnTMH/+fKxatQrh4eEYOnQonnvuOf0HaVW5ubmhb9+++OmnnzB79mwAuscjlpaWiIiI0NfTarVYsGABFi9ejKSkJGg0Gv2xJk2aVOu1K3P16lUAQGBgYLljQUFB+O2335Cfnw87Ozt88cUXGD9+PHx9fdG5c2cMGjQI48aN04+KMPaede7cucpx2tjYAADUanW5Y3fu3NEfv9/5lZ1b9voPep176xpTz1SEEOXKjh07hrffflt//JtvvsGSJUswbtw4AMDQoUPRpk0bg3M8PT31Q26N/Xko1bRpU9jZ2RmUtW7dGoBujpXu3bvj6tWrCAgIKPeoofRRUun33KVLl9C0aVODxLYyzZs3L1fm4uKCrKysB55rLFN9j1T0dRdCVGtoMtUcEwyqNff+UGu1WkiShC1btlQ43K/0L0FA95f2hAkT8N///hfbtm3Dm2++iaioKBw6dAg+Pj7ViueZZ57BCy+8gPj4eHTo0AE//fQT+vbtCzc3N32dOXPm4MMPP8TEiRMxe/ZsuLq6wsLCApGRkbIONRw1ahTCw8OxYcMGbNu2DXPnzsXnn3+O2NhYDBw4EIBx9+zWrVvlhntWxMbGRp+YeHt7A9C1uPj6+hrUS01N1fd7qIy3tzd27dpV7hd9amoqAN0H572vc6/U1FS4urrq/3I19po15erqCkmSKvwwvXnzpv51MjMzUVBQgK5du+qPW1paluu7kZycrE9Uq/LzIKfKhuZWlHRVl7e3d6Vfd6Di7xFjvxezsrIMfsap7vARCZnMvePVExMTodVq4efnB0DXoiGEQMuWLdGvX79yW/fu3Q3ODwkJwT//+U/s3bsXcXFxSElJwdKlS/XHq/pXyfDhw6FUKrFu3TrEx8fjwoUL5SZOiomJQZ8+ffCf//wHzzzzDB5//HH069evXIc7U2jRogUA4Pz58+WOnTt3Dm5ubgZ/sXp7e+P111/Hxo0bkZSUhCZNmuDTTz81OO9B9ywiIgLe3t4P3KZOnao/p0OHDgCAo0ePGrzWX3/9hevXr+uPV6ZDhw4oKCjQd0gtdfjwYYPrN2vWDO7u7uVeB9BNllX2dYy9Zk1ZWlqiVatWSEpKKnfM0dEROTk5AHStW1ZWVrh06ZJBnbKPEbZs2YKsrCyEhYUBqPrPw19//YX8/HyDstLOj6U/Yy1atMDFixfLJcPnzp3THy997b/++gu3bt2q0v2oLR06dMDx48fLxX348GHY2trqW2qq872YlJRUrjMw1Q0mGGQyixYtMtgvff5c+hd2REQEFAoFPvnkk3J//Qgh9L3Sc3Nzy83KFxISAgsLC4OmUTs7uyp98Ds7O2PAgAH46aefsHbtWiiVSgwfPtygjkKhKBfb+vXrkZKSYvTrGMvb2xsdOnTAd999Z/A+Tp8+jW3btmHQoEEAdH1BSj/ISnl4eKBp06b6+2HsPatOH4zg4GC0adMGy5cvN3hktGTJEkiShBEjRujLcnJycO7cOYN4hw0bBisrKyxevFhfJoTA0qVL0axZM/To0UNf/vTTT2PTpk1ITk7Wl+3YsQMXLlzQj/Sp6jVrKiwsrMKkJygoSJ/QKBQKDBkyBNOnT8fevXuRlJSEjz/+GMePH8ft27exYsUKjBkzBh9++CEcHR0BGP/zUKq4uBjLli3T7xcVFWHZsmVwd3fXP/oaNGgQ0tLSDPoaFRcXY+HChbC3t0evXr0A6O6zEAKffPJJufdlypYJY40YMQLp6emIjY3Vl924cQPr16/HkCFD9C1XVfleBHTfj5cuXTLp9wNVQd32KaWGqHQUSUhIiBgyZIhYtGiReO655wQA8eyzzxrUjYqKEgBEjx49xBdffCGWLFki3n33XREQECDmzp0rhBBiw4YNolmzZiIyMlIsXrxYfPXVV6Jr167CyspKHDx4UH+tQYMGCTs7OzFv3jyxZs0acejQoQfG+uOPPwoAwsHBQQwZMqTc8Y8++kgAEBMmTBDLly8Xb7zxhnB1dRUPPfSQ6NWrl76eqUaRbN++XVhaWoo2bdqIuXPnilmzZgl3d3fh4uIiLl++LIQQIisrS98Df/78+WL58uVi1KhRBqMKjL1n1fW///1PSJIkHnvsMbF8+XLx5ptvCgsLC/HSSy8Z1FuxYkWF96V0FMjLL78s/v3vf4vBgwcLAGLVqlUG9a5duyaaNGkiWrVqJb766isxZ84c4eLiIkJCQvQjSKp6zStXrojZs2eL2bNni27duunnS5g9e3aFI1buFRMTIwCI8+fPG5R/9tlnokOHDvpRF1evXhWBgYH6kTjt27cXr7zyigAg3NzcxIIFC8pd25ifByF0o0iaNm0qPDw8xBtvvCEWLlwoHnnkEQFALF++XF+voKBABAUFCaVSKaZPny4WLlyoH4ESHR1t8NrPP/+8ACAGDhwoFixYIL788ksRERFhMDoElUy01aJFC6NGcP3yyy/6e61UKkXHjh31+6Wjy4QQori4WHTv3l3Y29uLTz75RCxatEgEBwcLBwcHce7cOYNrGvu9KMTfX7vKJkmj2sUEg2qsNME4c+aMGDFihHBwcBAuLi5iypQpFU609fPPP4tHHnlE2NnZCTs7O9GmTRsxefJk/S/wy5cvi4kTJ4pWrVoJa2tr4erqKvr06SN+//13g+ucO3dOPProo8LGxua+E22VlZubq69fdihfqTt37ojp06cLb29vYWNjI3r27CkOHjwoevXqVSsJhhBC/P7776Jnz57CxsZGODo6iiFDhhhMtKVWq8U777wjQkNDhYODg7CzsxOhoaFi8eLF+jrG3rOa2LBhg+jQoYNQqVTCx8dH/POf/9QPjyxVWYKh0WjEnDlz9BOFBQcHV3j/hRDi9OnT4vHHHxe2trbC2dlZjB07VqSlpZWrZ+w1d+3aVelw3LJf08qo1Wrh5uZWbhKnrKws4eTkZPDBfffuXXH48GFx7NgxodFoxJUrV8Sff/5pMLPovR708yBExRNttWjRQnz99dflrpeeni5eeOEF4ebmJpRKpQgJCanw+7S4uFjMnTtXtGnTRj9528CBAw0mb6tpgjF+/PhK7/29Md26dUu8+OKLokmTJsLW1lb06tXLYNKvsoz5XhRCiNGjR4tHHnnkgXFS7ZCEkKE9jIjIjMyePRsrVqzAxYsXDTo9/vTTTxg7diwWLlyIV199tcJzr127huvXr9eomb537964ceMGTp8+Xe1rNDZpaWlo2bIl1q5di2HDhskdTqPEPhhERA/w1ltvIS8vr9xCaqNGjcLixYvxxhtvIDw8HN999x3OnDmDa9euIS4uDm+//TaCg4MRHR0tT+CNWHR0NEJCQphcyIgtGEQ1VFRU9MDe+E5OTiafm4Hqj1OnTuHDDz/E1q1bDTrVtm7dGtOnT8ekSZPKzU1RFWzBIHPEeTCIaujAgQPo06fPfeusWLECEyZMqJuAqM6FhIRg48aNyM/Px4ULF5CXlwcfHx+0bNlS7tCIZMMWDKIaysrK0k/VXJng4GD9JEFERI0BEwwiIiIyuXrTyfOzzz6DJEmIjIy8b73169ejTZs2sLa2RkhIiMGCUERERFQ/1Is+GEeOHMGyZcvQvn37+9Y7cOAAxowZg6ioKDz55JNYvXo1hg8fjuPHj6Ndu3ZGvZZWq8Vff/0FBwcHLoBDRERUBUII3L59G02bNn1wx2XZZuAocfv2bREQECC2b98uevXqJaZOnVpp3VGjRonBgwcblHXr1k288sorRr9ecnJypRO/cOPGjRs3btwevCUnJz/w81b2FozJkydj8ODB6NevH/7v//7vvnUPHjyIadOmGZQNGDAAGzdurPQctVptMGxMlHQ5SU5O1q8JQERERA+Wm5sLX19fODg4PLCurAnG2rVrcfz4cRw5csSo+mlpafD09DQo8/T0RFpaWqXnREVFVbigj6OjIxMMIiKiajCmi4FsnTyTk5MxdepUrFq1CtbW1rX2OjNmzEBOTo5+K7tKIxEREdUO2Vowjh07hoyMDHTq1ElfptFosHfvXnz99ddQq9UGc/4DgJeXF9LT0w3K0tPT4eXlVenrqFQq/VK/REREVDdka8Ho27cvTp06hfj4eP3WpUsXjB07FvHx8eWSCwAICwvDjh07DMq2b9+OsLCwugqbiIiIjCBbC4aDg0O5oaV2dnZo0qSJvnzcuHFo1qwZoqKiAABTp05Fr169MG/ePAwePBhr167F0aNHsXz58jqPn4iIDAkhUFxcDI1GI3coVANWVlYV/pFfVbKPIrmfa9euGYyz7dGjB1avXo1//vOf+Mc//oGAgABs3LjR6DkwiIiodhQVFSE1NRUFBQVyh0I1JEkSfHx8YG9vX7PrCNG4pgrPzc2Fk5MTcnJyOIqEiMgEtFotLl68CIVCAXd3dyiVSk5kaKaEEMjMzERBQQECAgLKtWRU5TO0XrdgmAONEIjLzkZqURG8lUqEOztDwR8sImpEioqKoNVq4evrC1tbW7nDoRpyd3fHlStXcPfu3Ro9KmGCUQOxmZmYmpiI62Um8vJRqbDA3x8R7u4yRkZEVPceOHU0mQVTtT4xwaim2MxMjEhIwL3Pl1LUajydkIBP/PwQYGPDVg0iImqUmGBUg0YITE1MLJdcANCXfXzlir6MrRpERNTYsD2rGuKysw0eizxIilqNEQkJiM3MrMWoiIjMn0YD7N4NrFmj+785jXj18/NDdHS0Sa61e/duSJKE7Oxsk1xPDmzBqIbUoqIq1RcAJACRiYkY5ubGxyVERBWIjQWmTgWuX/+7zMcHWLAAiIiondfs3bs3OnToYJLE4MiRI7Czs6t5UA0EWzCqwcNSWeVzBIBktRoL92SbVUZORFQXYmOBESMMkwsASEnRlcfGyhNX6eRhxnB3d+comjKYYFTHKWcgQwVoq37qW7OL4Ocn3w8LEVFdys+vfLtzR1dHo9G1XFQ0K1Np2dSpho9LKrtmVUyYMAF79uzBggULIEkSJEnCypUrIUkStmzZgs6dO0OlUmHfvn24dOkShg0bBk9PT9jb26Nr1674/fffDa537yMSSZLwzTff4KmnnoKtrS0CAgLwyy+/VC3IMn7++WcEBwdDpVLBz88P8+bNMzi+ePFiBAQEwNraGp6enhgxYoT+WExMDEJCQmBjY4MmTZqgX79+yK/qDasiJhjVkJEqAV/76557VDXJuKmUPSMnIqor9vaVb08/rasTF1e+5aIsIXTH4+L+LvPzq/iaVbFgwQKEhYXhpZdeQmpqKlJTU+Hr6wsAeP/99/HZZ5/h7NmzaN++PfLy8jBo0CDs2LEDJ06cwBNPPIEhQ4bg2rVr932NTz75BKNGjcKff/6JQYMGYezYsbh161bVAoVugdBRo0bhmWeewalTpzBz5kx8+OGHWLlyJQDg6NGjePPNNzFr1iycP38eW7duxaOPPgoASE1NxZgxYzBx4kScPXsWu3fvRkREBGp7nk32wagGb28Ace7Ax8HAlETAo0yHz9IOF/fSAshUAaecIQQgSUBkJDBsGGCCKd+JiMxWaqpp6xnLyckJSqUStra2+lW5z507BwCYNWsW+vfvr6/r6uqK0NBQ/f7s2bOxYcMG/PLLL5gyZUqlrzFhwgSMGTMGADBnzhx89dVX+OOPP/DEE09UKdb58+ejb9+++PDDDwEArVu3xpkzZzB37lxMmDAB165dg52dHZ588kk4ODigRYsW6NixIwBdglFcXIyIiAi0aNECABASElKl168OtmBUQ3i4ruORtM8dGNMdiAwFZgcB3+q+cOVaNUqTjkX+gFaXfQgBJCcbZuRERA1NXl7l288/6+p4ext3rbL1rlyp+Jqm0qVLF4P9vLw8vP322wgKCoKzszPs7e1x9uzZB7ZgtG/fXv9vOzs7ODo6IiMjo8rxnD17Fj179jQo69mzJy5evAiNRoP+/fujRYsWeOihh/D8889j1apV+nVhQkND0bdvX4SEhGDkyJH497//jaysrCrHUFVMMKpBodD1agYASUjASRdgpyfwQ0tdq8YNleEJAsCPvoCVFgjNAiz+bpYydUZORFSf2NlVvllb6+ro/2irZICdJAG+vrp6D7qu6eI2vNjbb7+NDRs2YM6cOYiLi0N8fDxCQkJQ9IBRhVZWVgb7kiRBq61GB74HcHBwwPHjx7FmzRp4e3vjo48+QmhoKLKzs6FQKLB9+3Zs2bIFbdu2xcKFCxEYGIikpCSTx1EWE4xqiogAYmKAZs3uORBXplXj/9oAWZa6u/x8MvDhWSD6JLDmEBCumxPD2MydiKihMvij7Z4ko3Q/Orp2HicrlUqjlpffv38/JkyYgKeeegohISHw8vLClTITKta2oKAg7N+/v1xMrVu31q8XYmlpiX79+uGLL77An3/+iStXrmDnzp0AdIlNz5498cknn+DEiRNQKpXYsGFDrcbMPhg1EBGh60MRF6dribh4EZg5E4CQIE666JII5wqGN7mpgU8S0GRhMMLDObsnEVHpH20VzYMRHV1782D4+fnh8OHDuHLlCuzt7SttXQgICEBsbCyGDBkCSZLw4Ycf1kpLRGWmT5+Orl27Yvbs2Rg9ejQOHjyIr7/+GosXLwYAbNq0CZcvX8ajjz4KFxcX/Prrr9BqtQgMDMThw4exY8cOPP744/Dw8MDhw4eRmZmJoKCgWo2ZCUYNKRRA795/77drV/ID8pfQdQCtiAV0/TQmJwIWbqi4VygRUeNy7x9t3t66xyK12RH+7bffxvjx49G2bVsUFhZixYoVFdabP38+Jk6ciB49esDNzQ3vvfcecnNzay+we3Tq1Ak//fQTPvroI8yePRve3t6YNWsWJkyYAABwdnZGbGwsZs6ciTt37iAgIABr1qxBcHAwzp49i7179yI6Ohq5ublo0aIF5s2bh4EDB9ZqzJKo7XEq9UxV1rKvLo0GWBiXhbdw8oF1d4WGoreLS63EQURUF+7cuYOkpCS0bNkS1qUdK8hs3e/rWZXPUPbBqAUKBeAZZNx04lWddpyIiMgcMMGoJd5K46YTTz+r5NThRESNzKuvvgp7e/sKt1dffVXu8EyCfTBqSbizM3xUKqSo1RUu61468dZbzzpjXtPaXcyHiIjql1mzZuHtt9+u8FhtPb6va0wwaolCkrDA3x8jEhIgAYZJhhYGE2+VTh0eE8Mkg4ioMfDw8ICHh4fcYdQqPiKpRRHu7ogJDkYz1T0Tb+VY6SbkitMNUS3tZhsZCT4uISKiBoEJRi2LcHfHle7d8SVCgTMOusLYZvrkohSnDiciooaECUYdUEgSPFNdgO2euoL2OZXW5dThRETUEDDBqCPe3gBOOut22uUAiopngOPU4URE1BAwwagj4eFAs2I7IMcSsNECrW8bHK9oMR8iIiJzxQSjjigUwFfREnDKWVcQ+vdjktpezIeIyFxohMDurCysSU/H7qwsaOr5ZNN+fn6Ijo42qq4kSdi4cWOtxlOfyJpgLFmyBO3bt4ejoyMcHR0RFhaGLVu2VFp/5cqVkCTJYDOnaWkjIoAXOjrpdkKz9eVNmnCIKhFRbGYm/A4dQp+TJ/Hs2bPoc/Ik/A4dQmxmptyhUTXImmD4+Pjgs88+w7Fjx3D06FE89thjGDZsGBISEio9x9HREampqfrt6tWrdRhxzb3xqDMAwLZ7Dh7to+uHMWECkwsiatxiMzMxIiEB19Vqg/IUtRojEhKYZJghWROMIUOGYNCgQQgICEDr1q3x6aefwt7eHocOHar0HEmS4OXlpd88PT3v+xpqtRq5ubkGm5za29vD0cICBdDAfeo1IDQLe/fV7yZAIqKqEkIgX6MxasstLsabFy9WOOtxadnUxETkFhcbdT1j1/Bcvnw5mjZtWm7Z9WHDhmHixIm4dOkShg0bBk9PT9jb26Nr1674/fffa3Zjyjh16hQee+wx2NjYoEmTJnj55ZeRl5enP7579248/PDDsLOzg7OzM3r27Kn/o/rkyZPo06cPHBwc4OjoiM6dO+Po0aMmi80U6s1MnhqNBuvXr0d+fj7CwsIqrZeXl4cWLVpAq9WiU6dOmDNnDoKDgyutHxUVhU8++aQ2Qq6W/964gdLlzX52ugJEA39kqrAq2R9jfd3vcyYRkfko0Gphb6KJfQSA62o1nPbtM6p+Xng47Izo0DZy5Ei88cYb2LVrF/r27QsAuHXrFrZu3Ypff/0VeXl5GDRoED799FOoVCp8//33GDJkCM6fP4/mzZvX5C0hPz8fAwYMQFhYGI4cOYKMjAxMmjQJU6ZMwcqVK1FcXIzhw4fjpZdewpo1a1BUVIQ//vgDUkmnvbFjx6Jjx45YsmQJFAoF4uPjYWVlVaOYTE32BOPUqVMICwvDnTt3YG9vjw0bNqBt27YV1g0MDMS3336L9u3bIycnB//617/Qo0cPJCQkwMfHp8JzZsyYgWnTpun3c3Nz4evrWyvv5UFKmwDL5dZuajx/KQE21sGIcGeSQURUF1xcXDBw4ECsXr1an2DExMTAzc0Nffr0gYWFBUJDQ/X1Z8+ejQ0bNuCXX37BlClTavTaq1evxp07d/D999/Dzs4OAPD1119jyJAh+Pzzz2FlZYWcnBw8+eSTaNWqFQAgKChIf/61a9fwzjvvoE2bNgCAgICAGsVTG2RPMAIDAxEfH4+cnBzExMRg/Pjx2LNnT4VJRlhYmEHrRo8ePRAUFIRly5Zh9uzZFV5fpVJBde9U3TLQCIGpiYkVL3xWMookMjERw9zcoCgdVkJEZKZsLSyQZ+S4+73Z2Rh06tQD6/0aEoJHnZ2Nem1jjR07Fi+99BIWL14MlUqFVatW4ZlnnoGFhQXy8vIwc+ZMbN68GampqSguLkZhYSGuXbtm9PUrc/bsWYSGhuqTCwDo2bMntFotzp8/j0cffRQTJkzAgAED0L9/f/Tr1w+jRo2Cd8lkSdOmTcOkSZPwww8/oF+/fhg5cqQ+EakvZB+mqlQq4e/vj86dOyMqKgqhoaFYsGCBUedaWVmhY8eOSExMrOUoay4uO7tc56WyBIBktRpx2dl1FhMRUW2RJAl2CoVR2+OurvBRqVDZn1YSAF+VCo+7uhp1PakKf6QNGTIEQghs3rwZycnJiIuLw9ixYwEAb7/9NjZs2IA5c+YgLi4O8fHxCAkJQVFR0QOuahorVqzAwYMH0aNHD6xbtw6tW7fW91GcOXMmEhISMHjwYOzcuRNt27bFhg0b6iQuY8meYNxLq9VCfZ8P4rI0Gg1OnTqlz+jqs1QjvyGNrUdE1FCUrj4NoFySUbof7e9fK6271tbWiIiIwKpVq7BmzRoEBgaiU6dOAID9+/djwoQJeOqppxASEgIvLy9cuXLFJK8bFBSEkydPIj8/X1+2f/9+WFhYIDAwUF/WsWNHzJgxAwcOHEC7du2wevVq/bHWrVvjrbfewrZt2xAREYEVK1aYJDZTkTXBmDFjBvbu3YsrV67g1KlTmDFjBnbv3q3PHseNG4cZM2bo68+aNQvbtm3D5cuXcfz4cTz33HO4evUqJk2aJNdbMJq3UmnSekREDUllq0/7qFSICa7d/mljx47F5s2b8e233+o/fwBdv4bY2FjEx8fj5MmTePbZZ8uNOKnJa1pbW2P8+PE4ffo0du3ahTfeeAPPP/88PD09kZSUhBkzZuDgwYO4evUqtm3bhosXLyIoKAiFhYWYMmUKdu/ejatXr2L//v04cuSIQR+N+kDWPhgZGRkYN24cUlNT4eTkhPbt2+O3335D//79Aeg6sViUeZaWlZWFl156CWlpaXBxcUHnzp1x4MCBSjuF1ifhzs7wUamQolZX3A9DC9jkq6A56QxNOGf0JKLGJ8LdHcPc3BCXnY3UoiJ4K5UId3au9X5pjz32GFxdXXH+/Hk8++yz+vL58+dj4sSJ6NGjB9zc3PDee++ZbKoDW1tb/Pbbb5g6dSq6du0KW1tbPP3005g/f77++Llz5/Ddd9/h5s2b8Pb2xuTJk/HKK6+guLgYN2/exLhx45Ceng43NzdERETUqxGTACAJYwcMNxC5ublwcnJCTk4OHB0d6/S1S0eRADBMMrTQtQN+HAzEucPHB1iwgJNvEZF5uHPnDpKSktCyZUuzml2ZKna/r2dVPkPrXR+MhqyyJkDcUuqTCwBISQFGjABiY2UIkoiIyASYYNSxCHd3XOneHb+HhMLiZkmiMT9An1wAQGmbUmQkoNHUfYxERFQ1q1atgr29fYXb/SaDbMhknwejMVJIEhSnXKA94QT0ywAeKgAOGtYRAkhOBuLigN69ZQmTiIiMNHToUHTr1q3CY/Vths26wgRDJqmpAC7Z6xKMVnn3r0dERPWag4MDHBwc5A6jXuEjEpl4e0OXYADAQ/n3r0dEZAYa2ZiBBstUX0cmGDIJDwe8C0qmiPUpAJSGnS0kCfD11dUjIqrPSh8BFBQUyBwJmULpTKWKGs6XwEckMlEogIWzlBiRZQW43AVa5gPndUN+Sod8R0dzPgwiqv8UCgWcnZ2RkZEBQDeHQ1Wm66b6Q6vVIjMzE7a2trC0rFmKwARDRk9HSGi/3R5/Igto9XeC4eEBLF7MeTCIyHx4eXkBgD7JIPNlYWGB5s2b1zhJZIIhs34t7fDn9Sw89U4ebuYBe/cCr7zC5IKIzIskSfD29oaHhwfu3r0rdzhUA0ql0mAW7epigiGzUHtdR89bTnkYOlSXYJw9K3NQRETVpFAoavzsnhoGdvKUWXs7XUfPk/n5aBus67l7+rScEREREdUcEwyZBdnZwVKSkF1cDNdA3TL1Fy8CXLWdiIjMGRMMmaksLBBkawsAyHDIg6MjUFwMXLggc2BEREQ1wASjHih9TPJnfj5Kp6wvWXSViIjILDHBqAdCShKMTTduwKVPFmAh2A+DiIjMGkeRyCw2MxP/un4dAHDo9m2g/0l4PK5C21b+ANzvfzIREVE9xRYMGcVmZmJEQgJu3DNmPFOoMTYxAbGZmTJFRkREVDNMMGSiEQJTExNR0ZIypWWRiYnQcPEgIiIyQ0wwZBKXnY3ranWlxwWAZLUacdnZdRYTERGRqTDBkEmqkRNdGFuPiIioPmGCIRNvpdKk9YiIiOoTJhgyCXd2ho9KhUrXqtMC1rkqhDs712FUREREpsEEQyYKScICf38AqDjJkADH7/2hqOFyuURERHJggiGjCHd3xAQHo5lKZVDuprACPg5Gxs/uWLEC2L0b0GjkiZGIiKg6mGDILMLdHVe6d8eu0FC0K5nRc0ROS1js102yNXEi0KcP4OcHxMbKGCgREVEVyJpgLFmyBO3bt4ejoyMcHR0RFhaGLVu23Pec9evXo02bNrC2tkZISAh+/fXXOoq29igkCb1dXNC3pL/F0l8LoNUa1klJAUaMYJJBRETmQdYEw8fHB5999hmOHTuGo0eP4rHHHsOwYcOQUMlKXwcOHMCYMWPw4osv4sSJExg+fDiGDx+O0w1k4Y42NroWDDQvKHesdL6tyEg+LiEiovpPEqJ+TRXp6uqKuXPn4sUXXyx3bPTo0cjPz8emTZv0Zd27d0eHDh2wdOlSo66fm5sLJycn5OTkwNHR0WRxm8LC3dl4E/FAmgoYE1ZpvV27gN696ywsIiIiAFX7DK03fTA0Gg3Wrl2L/Px8hIVV/OF68OBB9OvXz6BswIABOHjwYKXXVavVyM3NNdjqK1VaSQuGlxqwLq60XmpqHQVERERUTbInGKdOnYK9vT1UKhVeffVVbNiwAW3btq2wblpaGjw9PQ3KPD09kZaWVun1o6Ki4OTkpN98fX1NGr8ptfayAm5Z6XYqeExSytu7jgIiIiKqJtkTjMDAQMTHx+Pw4cN47bXXMH78eJw5c8Zk158xYwZycnL0W3JyssmubWrh4WVaMVqUTzAkCfD11dUjIiKqzyzlDkCpVMK/ZMKpzp0748iRI1iwYAGWLVtWrq6XlxfS09MNytLT0+Hl5VXp9VUqFVT3zDNRXykUQJ+HbLEV2YCfYYJROt9WdLSuHhERUX0mewvGvbRaLdSVrDIaFhaGHTt2GJRt37690j4b5ujJEF0LhnVgvkG5jw8QEwNERMgRFRERUdXI2oIxY8YMDBw4EM2bN8ft27exevVq7N69G7/99hsAYNy4cWjWrBmioqIAAFOnTkWvXr0wb948DB48GGvXrsXRo0exfPlyOd+GSbW1tQUA+PQswHMzgZkzgdatgTNn2HJBRETmQ9YWjIyMDIwbNw6BgYHo27cvjhw5gt9++w39+/cHAFy7dg2pZYZM9OjRA6tXr8by5csRGhqKmJgYbNy4Ee3atZPrLZhc25LZPC/fKcTTz+omvLh69e95MIiIiMxBvZsHo7bV53kwAEAIAbf9+3GruBjHO3VBeFN75OfrWjCCguSOjoiIGjOznAeDdCRJ0rdinCvMR0gI4ObGuS+IiMi8yD6KhMpra2uLfTk5OFNQgO3bATu7v0eREBERmQO2YNRDQSUdPc/k58PenskFERGZHyYY9VDpI5IzBZXP5klERFSfMcGoh0qHql4oKMB3f6Whx+tZ8G0hcPOmzIEREREZiX0w6qHDubmQAGgBTLhwDhgFoLcKi07546Pe7jJHR0RE9GBswahnYjMzMfLMGZQbO+ymxsciAbGZmXKERUREVCVMMOoRjRCYmphYPrkA9F+pyMREaBrX1CVERGSGmGDUI3HZ2bheyTosAAAJSFarEZedXWcxERERVQcTjHoktajIpPWIiIjkwgSjHvFWKk1aj4iISC5MMOqRcGdn+KhUqHReLQE4qVXQnHSGRlOXkREREVUNE4x6RCFJWODvDwDlkwyt7n85n/qjXx8Jfn5AbGxdRkdERGQ8Jhj1TIS7O2KCg9FMpTI8kKkCPg4G4nTzYKSkACNGMMkgIqL6iQlGPRTh7o4r3bvjH77NdQVJtsCz3fXJBQCUjlSNjAQflxARUb3DBKOeUkgSApM9dTtedyqsIwSQnAzExdVhYEREREZgglGPKVJtgUILwEYLNKt84bPU1DoMioiIyAhMMOqxZt4ScNlet+OfV2k9b+86CoiIiMhITDDqsfBwwC61JMEIKJ9gSBLg66urR0REVJ8wwajHFArgubCKWzCkknGs0dG6ekRERPUJE4x67sVwXYJhEZgHlFkGzcsLiIkBIiJkCoyIiOg+mGDUc+3s7KAAoHW8i592FaFFC115dDSTCyIiqr8s5Q6A7s9GoUAbW1skFBTANiQPEyeqcOkS4OMjd2RERESVY4JhBjrY2yOhoAAnbt/GRx81kTscIiKiB+IjEjPQ0cEBABCfV/lQVSIiovqECYYZ6GCv6+h5oiTBKC4GTp0Cbt6UMyoiIqLKyZpgREVFoWvXrnBwcICHhweGDx+O8+fP3/eclStXQpIkg83a2rqOIpZHx5IE4/KdO8gpLsYTTwDt2wObNskcGBERUSVkTTD27NmDyZMn49ChQ9i+fTvu3r2Lxx9/HPn5+fc9z9HREampqfrt6tWrdRSxPFytrOCrVAIA5iUnw/WxLMBC4MgRmQMjIiKqhKydPLdu3Wqwv3LlSnh4eODYsWN49NFHKz1PkiR4eXkZ9RpqtRpqtVq/n5ubW71gZRSbmYnM4mIAwOyrV4EeV4E1Kvy21R+A+/1PJiIikkG96oORk5MDAHB1db1vvby8PLRo0QK+vr4YNmwYEhISKq0bFRUFJycn/ebr62vSmGtbbGYmRiQk4I5Wa3jATY3E5xPwU2qmPIERERHdhySEEA+uVvu0Wi2GDh2K7Oxs7Nu3r9J6Bw8exMWLF9G+fXvk5OTgX//6F/bu3YuEhAT4VDA5REUtGL6+vsjJyYGjo2OtvBdT0QgBv0OHcL1M/Aa0gKdChZRHu0NROnc4ERFRLcnNzYWTk5NRn6H1JsF47bXXsGXLFuzbt6/CRKEyd+/eRVBQEMaMGYPZs2c/sH5Vbo7cdmdloc/Jkw+s9yVC8Ua4C9ckISKiWlWVz9B68YhkypQp2LRpE3bt2lWl5AIArKys0LFjRyQmJtZSdPJJLSoyqt5bs4vg5wfExtZuPERERMaSNcEQQmDKlCnYsGEDdu7ciZYtW1b5GhqNBqdOnYK3t3ctRCgv75KRIw90U4mUFGDECCYZRERUP8iaYEyePBk//vgjVq9eDQcHB6SlpSEtLQ2FhYX6OuPGjcOMGTP0+7NmzcK2bdtw+fJlHD9+HM899xyuXr2KSZMmyfEWalW4szN8VCpU2rtCCyBdBZxyRumDrshIQKOpm/iIiIgqI2uCsWTJEuTk5KB3797w9vbWb+vWrdPXuXbtGlJTU/X7WVlZeOmllxAUFIRBgwYhNzcXBw4cQNu2beV4C7VKIUlY4O8PAOWTDG1J4SJ/QKs7KgSQnAzExdVllEREROXVm06edcWcOnmWis3MxNTERMPRJBkq4Gt/IK78PBirVwNjxtRhgERE1CiYXSdPur8Id3dc6d4d/0IoUFTSlvFOSIXJBQA0wO4oRERkZphgmAmFJCEy3AXKq7qVVeFffjp1SQJ8fYHw8DoOjoiI6B5MMMyIQgE85qtb+AwBhku3l86zFR0NzodBRESyY4JhZp5ur0swVO1uG5R7egIxMUBEhBxRERERGWKCYWY6Ougekdh1yMPOXQIBAbryf/2LyQUREdUfTDDMTDs7O1hKEm4VF6NVdzUee0xXfvq0vHERERGVxQTDzKgsLBBsawsAOH77NkJDdeVGLFlCRERUZyzlDoCqrqODA07m5+NEXh4mPOGONWuATp3kjoqIiOhvbMEwQx3tdR09T+TloWVL4JlngNatZQ6KiIioDCYYZqhTmQSDiIioPuIjEjMUam8PCcB1tRqZRUW4nqDEjh1ASAgwYIDc0REREbEFwyw5WFrC38YGgK4VY+NG4J13gDJrxBEREcmKCYaZ6mBnBwBYkZYGi05ZgIXAn3/KHBQREVEJPiIxQ7GZmdiWlQUAWJuRAThlAGtUOLnMH8XF7rDkV5WIiGTGFgwzE5uZiREJCcjRaAwPuKlR/M8ELD6VKU9gREREZTDBMCMaITA1MRGiooMWAAQw61YiNKLCGkRERHWGCYYZicvOxnW1uvIKFsBNhRpx2dl1FhMREVFFmGCYkdSiIpPWIyIiqi1MMMyIt1JpVL0z+5TYvRu4t5sGERFRXWGCYUbCnZ3ho1JBqqyCFkC6Cv83yhl9+gB+fkBsbN3FR0REVIoJhhlRSBIW+PsDQPkkQ1tSuMgf0OqOpqQAI0YwySAiorrHBMPMRLi7IyY4GM1UKsMDN5XAx8FAnLu+qHQwSWQkH5cQEVHdYoJhhiLc3XGle3fsCg2FAxS6wpltDZKLUkIAyclAXFwdB0lERI0aEwwzpZAk9HZxQXO1g66geeF966em1kFQREREJZhgmLnWKt2aJGhRcN963t51EAwREVEJWROMqKgodO3aFQ4ODvDw8MDw4cNx/vz5B563fv16tGnTBtbW1ggJCcGvv/5aB9HWT31b2er+0SK/wuOSBPj6AuHhdRgUERE1etVKML777jts3rxZv//uu+/C2dkZPXr0wNWrV42+zp49ezB58mQcOnQI27dvx927d/H4448jP7/iD0sAOHDgAMaMGYMXX3wRJ06cwPDhwzF8+HCcPn26Om/F7AXblyYYBZDuGVpSuh8dDSgUdRoWERE1cpIQVV+4IjAwEEuWLMFjjz2GgwcPol+/fvjyyy+xadMmWFpaIraa4yIzMzPh4eGBPXv24NFHH62wzujRo5Gfn49Nmzbpy7p3744OHTpg6dKlD3yN3NxcODk5IScnB46OjtWKsz7JKCqC54EDkATQ9MVwpCT9nUn4+uqSi4gI+eIjIqKGoyqfodVqwUhOToZ/yXwMGzduxNNPP42XX34ZUVFRiKvBcIWcnBwAgKura6V1ShOasgYMGICDBw9WWF+tViM3N9dga0jcrazgamkJIQH/PVGA4cN15ePHA0lJTC6IiEge1Uow7O3tcfPmTQDAtm3b0L9/fwCAtbU1CgvvP5qhMlqtFpGRkejZsyfatWtXab20tDR4enoalHl6eiItLa3C+lFRUXByctJvvr6+1YqvvpIkCUG2usckF+4UYPlyIDsbWLmSj0WIiEg+1Uow+vfvj0mTJmHSpEm4cOECBg0aBABISEiAn59ftQKZPHkyTp8+jbVr11br/MrMmDEDOTk5+i05Odmk168Pgux0I0nOFhTA3R1wcpI5ICIiavSqlWAsWrQIYWFhyMzMxM8//4wmTZoAAI4dO4YxY8ZU+XpTpkzBpk2bsGvXLvj4+Ny3rpeXF9LT0w3K0tPT4eXlVWF9lUoFR0dHg62haVvSgnHmPp1jiYiI6pJldU5ydnbG119/Xa78k08+qdJ1hBB44403sGHDBuzevRstW7Z84DlhYWHYsWMHIiMj9WXbt29HWFhYlV67ISl9RHK2QDcXxowZwLFjwOLFQElXGSIiojpVrRaMrVu3Yt++ffr9RYsWoUOHDnj22WeRlZVl9HUmT56MH3/8EatXr4aDgwPS0tKQlpZm0I9j3LhxmDFjhn5/6tSp2Lp1K+bNm4dz585h5syZOHr0KKZMmVKdt9IglD4iuVhYiLtaLbZvB7ZvBxrpyF0iIqoHqpVgvPPOO/rRGKdOncL06dMxaNAgJCUlYdq0aUZfZ8mSJcjJyUHv3r3h7e2t39atW6evc+3aNaSWmee6R48eWL16NZYvX47Q0FDExMRg48aN9+0Y2tD5qlSws7DAXSFwqbAQgYG6ciPmLCMiIqoV1XpEkpSUhLZt2wIAfv75Zzz55JOYM2cOjh8/ru/waQxjpuDYvXt3ubKRI0di5MiRRr9OQ2chSWhja4tjeXk4W1CANm10LRrnzskcGBERNVrVasFQKpUoKHne//vvv+Pxxx8HoJu/oqHNM2Euyo4kYQsGERHJrVotGI888gimTZuGnj174o8//tA/0rhw4cIDR4FQ7WhbpqPnk210ZefO6ZZrv3cKcSIiotpWrRaMr7/+GpaWloiJicGSJUvQrFkzAMCWLVvwxBNPmDRAMk5QmaGqAQG6pCIrC7hxQ+bAiIioUapWC0bz5s0N1gIp9eWXX9Y4IKqe0kckCfn5iM1Nh8fjStw97oyUFAnu7jIHR0REjU61EgwA0Gg02LhxI86ePQsACA4OxtChQ6Hg/NSy+DMvDwCgFgLPnT0LvA/4qFS43MwfHcAMg4iI6la1VlNNTEzEoEGDkJKSgsCSHoXnz5+Hr68vNm/ejFatWpk8UFNpaKupAkBsZiZGJCTg3i9kadeLmOBgRLAZg4iIaqjWV1N988030apVKyQnJ+P48eM4fvw4rl27hpYtW+LNN9+sVtBUPRohMDUxsVxyAUBfFpmYCE3V80giIqJqq9Yjkj179uDQoUMGy6o3adIEn332GXr27Gmy4OjB4rKzcV2trvS4AJCsViMuOxu9XVzqLjAiImrUqtWCoVKpcPv27XLleXl5UCqVNQ6KjJdaVGTSekRERKZQrQTjySefxMsvv4zDhw9DCAEhBA4dOoRXX30VQ4cONXWMdB/eRiZ06WeV0GhqORgiIqIS1UowvvrqK7Rq1QphYWGwtraGtbU1evToAX9/f0RHR5s4RLqfcGdn+KhUqHQuLS2AdBXe6usMPz8gNrbuYiMiosarWqNISiUmJuqHqQYFBcHfDNYGb8ijSAAYdvbUQjeU5ONgIM5dP6NnTAwQEVHHQRIRkdmrymeo0QlGVVZJnT9/vtF161pDTDAAXZIxNTHRsMNnugpY5A/E/T1EVZIAHx8gKQnglCVERFQVVfkMNXoUyYkTJ4yqJ3HhC1lEuLtjmJsbRuy5iI34CzjpCEzrCGgNvx5CAMnJQFwc0Lu3PLESEVHDZ3SCsWvXrtqMg0xAIUkIvumOjU3+AtyKyiUXZaWm1mFgRETU6FSrkyfVX12a6NYkQbM7gHVxpfW8vesoICIiapSYYDQwQ8KVsMgqGbraMr/ccUkCfH2B8PA6DoyIiBoVJhgNjEIBhNiXtGK0MkwwSrvHREezgycREdUuJhgNUP+H7AEAdu3zDMp9fDhElYiI6ka1l2un+qu9na4Fo9PT+RjkAMyYAfj5AYmJbLkgIqK6wRaMBqi9va4F41RBPp59VjfNyfXr4FThRERUZ5hgNEBtbG1hKUnILi4G3NVwcgKKi4Fz5+SOjIiIGgsmGA2QysICgTY2AHStGCEhuvI//5QxKCIialSYYDRQpY9J/szLQ/v2urJTp2QMiIiIGhUmGA1UaUfPP/P/bsFggkFERHVF1gRj7969GDJkCJo2bQpJkrBx48b71t+9ezckSSq3paWl1U3AZkTf0TMvDx07Ah06AEFB8sZERESNh6zDVPPz8xEaGoqJEyciogqTM5w/f95gFTcPD4/aCM+slbZgnC0owPk2afhypwrhzs7Qrd9ORERUu2RNMAYOHIiBAwdW+TwPDw84OzubPqAG5HBuLiQAWgDjS4aP+KhUWODvjwh39/ueS0REVFNm2QejQ4cO8Pb2Rv/+/bF///771lWr1cjNzTXYGrrYzEyMPHMG4p7yFLUaIxISEJuZKUtcRETUeJhVguHt7Y2lS5fi559/xs8//wxfX1/07t0bx48fr/ScqKgoODk56TdfX986jLjuaYTA1MTEcskFAAgAQgCRiYnQiIpqEBERmYYkRP34pJEkCRs2bMDw4cOrdF6vXr3QvHlz/PDDDxUeV6vVUKvV+v3c3Fz4+voiJyfHoB9HQ7E7Kwt9Tp58YL1doaHo7eJSBxEREVFDkZubCycnJ6M+Q81+LZKHH34Y+/btq/S4SqWCSqWqw4jklVpUZNJ6RERE1WFWj0gqEh8fD29vb7nDqDe8lUqT1iMiIqoOWVsw8vLykJiYqN9PSkpCfHw8XF1d0bx5c8yYMQMpKSn4/vvvAQDR0dFo2bIlgoODcefOHXzzzTfYuXMntm3bJtdbqHfCnZ3ho1IhRa2usB8GtICDWgXNSWdowrm6KhER1Q5ZWzCOHj2Kjh07omPHjgCAadOmoWPHjvjoo48AAKmpqbh27Zq+flFREaZPn46QkBD06tULJ0+exO+//46+ffvKEn99pJAkLPD3B1DBjBdaXeHtKH/06yPBzw+Ija3jAImIqFGoN50860pVOqiYs9jMTExNTMT1Mh1cka4CFvkDcbp5MKSSDCQmBqjCPGdERNRIVeUz1Oz7YFDFItzdcaV7d8zwba4ruGQHPNtdn1wAuiGrABAZCWg0dR8jERE1XEwwGjCFJOGh5JJp1D3Uukck9xACSE4G4uLqNjYiImrYmGA0cFapNoAGgEMx4Fb50NTU1LqLiYiIGj4mGA1cC28FcN1Wt+OXX2k9jvQlIiJTYoLRwIWHAzbpupVV0bJ8giFJgK+vrh4REZGpMMFo4BQKYGi7ihOM0lEk0dGcD4OIiEyLCUYjMLKrLsGwam2YYPj4cIgqERHVDiYYjUA7u5IEIyAfa3/SjU21sADOn2dyQUREtYMJRiPQytoaKklCgVaLLoPvwNER0GqBS5fkjoyIiBoqJhiNgKWFBYJKWjESCvLRtq2u/MwZGYMiIqIGjQlGI1H6mOR0PhMMIiKqfbKupkp1p2yCMXoo4OkJPPaYzEEREVGDxQSjkSibYKweBgwbJnNARETUoPERSSNRmmCcKyjAXW0Fi5IQERGZEBOMRqK5SgV7hQJ3hcDFwkKkpgK//w7cuCF3ZERE1BAxwWgkJElCsK1uTZJlf/2FPm9lof8Agd275Y2LiIgaJiYYjURsZiZO5+tm8vwqJQXnXz0JrDmEmPRMmSMjIqKGiAlGIxCbmYkRCQnIv7fvhZsa69omIDaTSQYREZkWE4wGTiMEpiYmQlR0sOSrH5mYCI2osAYREVG1MMFo4OKys3Fdra68ggQkq9WIy86us5iIiKjhY4LRwKUWFZm0HhERkTGYYDRw3kqlSesREREZgwlGAxfu7AwflQpSZRUE4K1QIdzZuQ6jIiKiho4JRgOnkCQs8PcHgHJJhlTyn9E3/BG3R4JGU9fRERFRQ8UEoxGIcHdHTHAwmqlUBuUWN1XAR8GIjnBHnz6Anx8QGytPjERE1LAwwWgkItzdcaV7d3wbGKgrKLSAZlQ3IM5dXyclBRgxgkkGERHVnKwJxt69ezFkyBA0bdoUkiRh48aNDzxn9+7d6NSpE1QqFfz9/bFy5cpaj7OhUEgSRrl5AFoANlrAqdjgeOlUGJGR4OMSIiKqEVkTjPz8fISGhmLRokVG1U9KSsLgwYPRp08fxMfHIzIyEpMmTcJvv/1Wy5E2HEf2K4A0a92OT0G540IAyclAXFwdB0ZERA2KpZwvPnDgQAwcONDo+kuXLkXLli0xb948AEBQUBD27duHL7/8EgMGDKitMBuU1FQA122ApncA3wLglHPl9YiIiKrJrPpgHDx4EP369TMoGzBgAA4ePFjpOWq1Grm5uQZbY+btDSBZt6oqfAvvX4+IiKiazCrBSEtLg6enp0GZp6cncnNzUVhY8YdlVFQUnJyc9Juvr29dhFpvhYcDznmlCUb5RySSBPj66uoRERFVl1klGNUxY8YM5OTk6Lfk5GS5Q5KVQgG8+ZSNbsfHMCmTSibKiI7W1SMiIqous0owvLy8kJ6eblCWnp4OR0dH2NjYVHiOSqWCo6OjwdbYvfh4SQtGs0LA4u8l3Js2BWJigIgImQIjIqIGQ9ZOnlUVFhaGX3/91aBs+/btCAsLkyki8+SjUsHGwgKFllr8uOMO9q2zRWAg8PLLgK2t3NEREVFDIGuCkZeXh8TERP1+UlIS4uPj4erqiubNm2PGjBlISUnB999/DwB49dVX8fXXX+Pdd9/FxIkTsXPnTvz000/YvHmzXG/BLFlIEgJsbPBnfj6cQwqxpDezCiIiMi1ZH5EcPXoUHTt2RMeOHQEA06ZNQ8eOHfHRRx8BAFJTU3Ht2jV9/ZYtW2Lz5s3Yvn07QkNDMW/ePHzzzTccoloNrUuaKi4UlO/oSUREVFOytmD07t0bonT6yApUNEtn7969ceLEiVqMqnEILOmzcr6gANevA9u3A0olMHaszIEREVGDYFadPMl09C0YhYU4eRKYOBGYO1fmoIiIqMFggtFIBZYkGOcLCtC6ta7swgVAq73PSUREREZigtFItS55RPJXURHcfIthaQkUFupWVCUiIqopJhiNlIuVFdytrAAASXcL0aqVrvzCBRmDIiKiBoMJRiNW2opxocxjkvPnZQyIiIgaDCYYjZi+H0ZhoUE/DCIioppigtGIlZ0LIzBQV8YWDCIiMgWzmiqcTMvf2hoAcDAnB8N7ZWHrdmcEt5FkjoqIiBoCtmA0UrGZmZh88SIAIEmtxqi/TmKS3SH8ocqUOTIiImoImGA0QrGZmRiRkID0u3cNylPUaoxISEBsJpMMIiKqGSYYjYxGCExNTERFE7QLAEIAr59JhOY+U7gTERE9CBOMRiYuOxvX1erKK0hAulAjLju7zmIiIqKGhwlGI5NaVGTSekRERBVhgtHIeCuVJq1HRERUESYYjUy4szN8VCpUOhhVC1jcUCHc2bkOoyIiooaGCUYjo5AkLPD3B4BySYZU8h/tV/5Y8R8Ju3cDGk0dB0hERA0CE4xGKMLdHTHBwWimUhmUO2uVkGYGA3HueOkloE8fwM8PiI2VJ04iIjJfTDAaqQh3d1zp3h27QkPhUbKqatb0IIi97gb1UlKAESOYZBARUdUwwWjEFJKE3i4uCLWz1xV43ylXp3Q6jMhIPi4hIiLjMcEg2Gbrlm1Hs8IKjwsBJCcDcXF1GBQREZk1JhgEh9z7JxilUlPrIBgiImoQmGAQghxLEoym908wvL3rIBgiImoQmGAQhnYu24JRfg0SSQJ8fYHw8LqNi4iIzBcTDIK/rTUkAcBOA7gYrrAqlUyWER0NKBR1HhoREZkpJhgEa4UCvta6OTHcOxg+JvHxAWJigIgIOSIjIiJzVS8SjEWLFsHPzw/W1tbo1q0b/vjjj0rrrly5EpIkGWzW1tZ1GG3DFGCje0zyxY+FGDlSV/b000BSEpMLIiKqOtkTjHXr1mHatGn4+OOPcfz4cYSGhmLAgAHIyMio9BxHR0ekpqbqt6tXr9ZhxA2Tf0mCcVldiH79dGUFBXwsQkRE1SN7gjF//ny89NJLeOGFF9C2bVssXboUtra2+Pbbbys9R5IkeHl56TdPT886jLhhKk0wLhYWonVrXdn58zIGREREZk3WBKOoqAjHjh1Dv9I/mQFYWFigX79+OHjwYKXn5eXloUWLFvD19cWwYcOQkJBQaV21Wo3c3FyDjcorTTASCwsRGKgrS0kB7t69z0lERESVkDXBuHHjBjQaTbkWCE9PT6SlpVV4TmBgIL799lv897//xY8//gitVosePXrg+vXrFdaPioqCk5OTfvP19TX5+2gI9C0YBQXw9BQ4fx64fRsoWaaEiIioSmR/RFJVYWFhGDduHDp06IBevXohNjYW7u7uWLZsWYX1Z8yYgZycHP2WnJxcxxGbh4dKEowcjQZZmmK0bs3kgoiIqs9Szhd3c3ODQqFAenq6QXl6ejq8vLyMuoaVlRU6duyIxMTECo+rVCqo7lmWnMqzVSjQTKlESlEREgsL0YTZBRER1YCsLRhKpRKdO3fGjh079GVarRY7duxAWFiYUdfQaDQ4deoUvDmPdY2V7Yexdy/w/PNAVJTMQRERkVmS/RHJtGnT8O9//xvfffcdzp49i9deew35+fl44YUXAADjxo3DjBkz9PVnzZqFbdu24fLlyzh+/Diee+45XL16FZMmTZLrLTQYZROMlBTgxx+BLVtkDoqIiMySrI9IAGD06NHIzMzERx99hLS0NHTo0AFbt27Vd/y8du0aLCz+zoOysrLw0ksvIS0tDS4uLujcuTMOHDiAtm3byvUWGoyyCcaQkpEkHKpKRETVIQkhyq9u1YDl5ubCyckJOTk5cHR0lDuceiUmIwMjz5xBd0dHbAvohNLbk5UFODvLGhoREdUDVfkMlf0RCdUfZVswHBz+Xp79wgUZgyIiIrPEBIP0WpUkGDfu3sU3f/0Fj8ezAAvBBIOIiKpM9j4YVH9sz8qCBQAtgJcuXAAmABikwn/T/PEc3OUNjoiIzApbMAgAEJuZiREJCdDee8BNjZiQBMRmZsoRFhERmSkmGASNEJiamIgKe/taAJIERCYmQtO4+gMTEVENMMEgxGVn47paXelxASBZrUZcdnadxUREROaNCQYhtajIpPWIiIiYYBC8lUqj6qWfVUKjqeVgiIioQWCCQQh3doaPSgWpsgpaAOkqvNXXGX5+QGxs3cVGRETmiQkGQSFJWODvDwDlkwxtSeEif0ArISUFGDGCSQYREd0fEwwCAES4uyMmOBjN7l3aPlMFfBwMxOnmwSgdSBIZCT4uISKiSjHBIL0Id3dc6d4d76FkpbMiCXj+YX1yUUoIIDkZiIuTIUgiIjILTDDIgEKSEJLqBeRaAkoBPJRfad3U1DoMjIiIzAoTDCqnmbcEnHbS7YTkVFqvdDE0IiKiezHBoHLCwwHH5JJleNuVTzAkCfD11dUjIiKqCBMMKkehAN4ZWNKC0S4XKDOJuFQyzCQ6WlePiIioIkwwqEJvD3aApZCAJkVA0zv6ch8fICYGiIiQMTgiIqr3uFw7VchaoUBXJwcczM3F+6ty0D7VBh4egLs70L693NEREVF9xwSDKtXTyQkHc3NxyzsHYX5e6NwZKCoCMjMBa2u5oyMiovqMj0ioUo846fph7MvJQfPmuqQiLw/YuVPmwIiIqN5jgkGV6uGoG0lypqAA/0n7C11fygIsBDZskDkwIiJ6sPh4YOBA3f9lwEckVKm4nBxYAigG8PKFC0BvAG1VWL3CH71XuaNZM91QVY4mISKqh37+Gdi6FejaFejQoc5fni0YVKHYzEyMSEhA8b0H3NQoeDcBzy3LRJ8+4OqqRET11f/+Z/j/OsYEg8rRCIGpiYllZr8owwK6aTEmJwIWgqurEhHVR+npwMmTun/HxwMZGXUeAhMMKicuOxvX1erKK1gA8FQDT12HUGgh2mdh3HfpmLcjC0XFAhoNsHs3sGaN7v9cdZWIqI799tv99+sA+2BQOalFRcZVnHIJeO0SoADyAbwN4N2NKtisbIX861a6SbpuKuF9ywkDpufAvnkRWrkoMam7E745lINLWbr91x9xBgAs3petL7u3zoP25bqGOcfe2N8/YzffazD2B19j8v82Q2GhgIVWA41CAWzaDMXzzxv3u91EJCFEhS3hdWnRokWYO3cu0tLSEBoaioULF+Lhhx+utP769evx4Ycf4sqVKwgICMDnn3+OQYMGGfVaubm5cHJyQk5ODhxLRkmQod1ZWehT2rT2IAKAVGZfW7JftkwDQFH5vpSry3OFY7HR59SXa5hz7I39/TN2870GY7dE05uZ8Lib+XehFvpnEpIQ2DltGpwKCvSHc2zssPLTjZjayxUV8vQEmjWr+FgZVfkMlT3BWLduHcaNG4elS5eiW7duiI6Oxvr163H+/Hl4eHiUq3/gwAE8+uijiIqKwpNPPonVq1fj888/x/Hjx9GuXbsHvh4TjAfTCAG/Q4eQolZX3A/jQe5NOozZRzXOqS/XMOfYG/v7Z+zme41GHvvv06eh74kTqIxWkmBR5uP93v1y+vYFfv+98uMlzCrB6NatG7p27Yqvv/4aAKDVauHr64s33ngD77//frn6o0ePRn5+PjZt2qQv6969Ozp06IClS5c+8PWYYBindBQJUHapMyIiqg9G7N6NZfPmwSUvzyD3qBZnZ2D5cmDkyAdWrcpnqKydPIuKinDs2DH069dPX2ZhYYF+/frh4MGDFZ5z8OBBg/oAMGDAgErrq9Vq5ObmGmz0YBHu7ogJDkYzlUruUIiI6B4xvXujzfffIzY8HICuhaJKSus/9RRw/rxRyUVVyZpg3LhxAxqNBp6engblnp6eSEtLq/CctLS0KtWPioqCk5OTfvP19TVN8I1AhLs7rnTvji9btZI7FCIiukemiwtGzJqFUR99hGw7OxRbGPeRrrFQAE5OwLp1ujkGKuiOYAoNfpjqjBkzkJOTo9+Sk5PlDsmsKCQJb/j4wEelqnkzHBERmdz6Pn3Q5vvvsa1Llwc+0hYArnXpo2u1GDWqVuOSNcFwc3ODQqFAenq6QXl6ejq8vLwqPMfLy6tK9VUqFRwdHQ02qhqFJGGBvz8AlE8y7v1uZocNIqI6l+nigmOBgQ9sxdBYKNC8f/daa7UoS9YEQ6lUonPnztixY4e+TKvVYseOHQgLC6vwnLCwMIP6ALB9+/ZK65NpVNonQ1tB5QclHRXtV+ec+nINc47dFNdg7PJcw5xjN8U1GHu5siEHDkChreiX8t8UWg0Umzfdt46pyD7R1rRp0zB+/Hh06dIFDz/8MKKjo5Gfn48XXngBADBu3Dg0a9YMUVFRAICpU6eiV69emDdvHgYPHoy1a9fi6NGjWL58uZxvo1GIcHfHMDc3xGVnI7WoCN5KJbrZO2HZ/r8nd2kadBfTL1/C9aIyM4FqYTim+5596XYF48IfcE59uYY5x97Y3z9jN99rMPby1/C8cQsdLl1CWaVDU8sOUZUA3dTh6em6uS9qkewJxujRo5GZmYmPPvoIaWlp6NChA7Zu3arvyHnt2jVYlGny6dGjB1avXo1//vOf+Mc//oGAgABs3LjRqDkwqOYUkoTeLi4GZZG9Dfef9nS/bxIyqec9s9INcgZwz6x099R50L5c1zDn2Bv7+2fs5nsNxl7+Gq/mbDT4PSwUChTZ2GPXoBfR59f/QFWYB6nsug2//QaMG4faJPs8GHWN82AQEVGDM3o0EBMDCKHbnnoKWLpU19ciIwN49VVgwwbd8FRJ0g1LXbu2yi9jNvNgEBERUQ0VFwNbtwJabcXDTz08dPvr1umOa7XAli21vhIlEwwiIiJzVlgIPPTQ35NmVTb8dNQo3fGnngJatQLKrFVSG2Tvg0FEREQ14OAAHD0KKBQPrlvamqHRGFe/BtiCQUREZO6qmizUcnIBMMEgIiKiWsAEg4iIiEyu0fXBKB2Vy1VViYiIqqb0s9OYGS4aXYJx+/ZtAOCqqkRERNV0+/ZtODk53bdOo5toS6vV4q+//oKDgwMkqXrrg+bm5sLX1xfJycmcrMuEeF9Nj/fU9HhPawfvq+nVxj0VQuD27dto2rSpwSzbFWl0LRgWFhbw8fExybW4Omvt4H01Pd5T0+M9rR28r6Zn6nv6oJaLUuzkSURERCbHBIOIiIhMjglGNahUKnz88cdQqVRyh9Kg8L6aHu+p6fGe1g7eV9OT+542uk6eREREVPvYgkFEREQmxwSDiIiITI4JBhEREZkcEwwiIiIyOSYY1bBo0SL4+fnB2toa3bp1wx9//CF3SGYjKioKXbt2hYODAzw8PDB8+HCcP3/eoM6dO3cwefJkNGnSBPb29nj66aeRnp4uU8Tm57PPPoMkSYiMjNSX8Z5WT0pKCp577jk0adIENjY2CAkJwdGjR/XHhRD46KOP4O3tDRsbG/Tr1w8XL16UMeL6TaPR4MMPP0TLli1hY2ODVq1aYfbs2QbrWvCePtjevXsxZMgQNG3aFJIkYePGjQbHjbmHt27dwtixY+Ho6AhnZ2e8+OKLyMvLM22ggqpk7dq1QqlUim+//VYkJCSIl156STg7O4v09HS5QzMLAwYMECtWrBCnT58W8fHxYtCgQaJ58+YiLy9PX+fVV18Vvr6+YseOHeLo0aOie/fuokePHjJGbT7++OMP4efnJ9q3by+mTp2qL+c9rbpbt26JFi1aiAkTJojDhw+Ly5cvi99++00kJibq63z22WfCyclJbNy4UZw8eVIMHTpUtGzZUhQWFsoYef316aefiiZNmohNmzaJpKQksX79emFvby8WLFigr8N7+mC//vqr+OCDD0RsbKwAIDZs2GBw3Jh7+MQTT4jQ0FBx6NAhERcXJ/z9/cWYMWNMGicTjCp6+OGHxeTJk/X7Go1GNG3aVERFRckYlfnKyMgQAMSePXuEEEJkZ2cLKysrsX79en2ds2fPCgDi4MGDcoVpFm7fvi0CAgLE9u3bRa9evfQJBu9p9bz33nvikUceqfS4VqsVXl5eYu7cufqy7OxsoVKpxJo1a+oiRLMzePBgMXHiRIOyiIgIMXbsWCEE72l13JtgGHMPz5w5IwCII0eO6Ots2bJFSJIkUlJSTBYbH5FUQVFREY4dO4Z+/frpyywsLNCvXz8cPHhQxsjMV05ODgDA1dUVAHDs2DHcvXvX4B63adMGzZs35z1+gMmTJ2Pw4MEG9w7gPa2uX375BV26dMHIkSPh4eGBjh074t///rf+eFJSEtLS0gzuq5OTE7p168b7WokePXpgx44duHDhAgDg5MmT2LdvHwYOHAiA99QUjLmHBw8ehLOzM7p06aKv069fP1hYWODw4cMmi6XRLXZWEzdu3IBGo4Gnp6dBuaenJ86dOydTVOZLq9UiMjISPXv2RLt27QAAaWlpUCqVcHZ2Nqjr6emJtLQ0GaI0D2vXrsXx48dx5MiRcsd4T6vn8uXLWLJkCaZNm4Z//OMfOHLkCN58800olUqMHz9ef+8q+n3A+1qx999/H7m5uWjTpg0UCgU0Gg0+/fRTjB07FgB4T03AmHuYlpYGDw8Pg+OWlpZwdXU16X1mgkGymTx5Mk6fPo19+/bJHYpZS05OxtSpU7F9+3ZYW1vLHU6DodVq0aVLF8yZMwcA0LFjR5w+fRpLly7F+PHjZY7OPP30009YtWoVVq9ejeDgYMTHxyMyMhJNmzblPW2A+IikCtzc3KBQKMr1vk9PT4eXl5dMUZmnKVOmYNOmTdi1axd8fHz05V5eXigqKkJ2drZBfd7jyh07dgwZGRno1KkTLC0tYWlpiT179uCrr76CpaUlPD09eU+rwdvbG23btjUoCwoKwrVr1wBAf+/4+8B477zzDt5//30888wzCAkJwfPPP4+33noLUVFRAHhPTcGYe+jl5YWMjAyD48XFxbh165ZJ7zMTjCpQKpXo3LkzduzYoS/TarXYsWMHwsLCZIzMfAghMGXKFGzYsAE7d+5Ey5YtDY537twZVlZWBvf4/PnzuHbtGu9xJfr27YtTp04hPj5ev3Xp0gVjx47V/5v3tOp69uxZbgj1hQsX0KJFCwBAy5Yt4eXlZXBfc3NzcfjwYd7XShQUFMDCwvBjR6FQQKvVAuA9NQVj7mFYWBiys7Nx7NgxfZ2dO3dCq9WiW7dupgvGZN1FG4m1a9cKlUolVq5cKc6cOSNefvll4ezsLNLS0uQOzSy89tprwsnJSezevVukpqbqt4KCAn2dV199VTRv3lzs3LlTHD16VISFhYmwsDAZozY/ZUeRCMF7Wh1//PGHsLS0FJ9++qm4ePGiWLVqlbC1tRU//vijvs5nn30mnJ2dxX//+1/x559/imHDhnFI5X2MHz9eNGvWTD9MNTY2Vri5uYl3331XX4f39MFu374tTpw4IU6cOCEAiPnz54sTJ06Iq1evCiGMu4dPPPGE6Nixozh8+LDYt2+fCAgI4DDV+mDhwoWiefPmQqlUiocfflgcOnRI7pDMBoAKtxUrVujrFBYWitdff124uLgIW1tb8dRTT4nU1FT5gjZD9yYYvKfV87///U+0a9dOqFQq0aZNG7F8+XKD41qtVnz44YfC09NTqFQq0bdvX3H+/HmZoq3/cnNzxdSpU0Xz5s2FtbW1eOihh8QHH3wg1Gq1vg7v6YPt2rWrwt+j48ePF0IYdw9v3rwpxowZI+zt7YWjo6N44YUXxO3bt00aJ5drJyIiIpNjHwwiIiIyOSYYREREZHJMMIiIiMjkmGAQERGRyTHBICIiIpNjgkFEREQmxwSDiIiITI4JBhEREZkcEwwiMnu7d++GJEnlFnQjIvkwwSAiIiKTY4JBREREJscEg4hqTKvVIioqCi1btoSNjQ1CQ0MRExMD4O/HF5s3b0b79u1hbW2N7t274/Tp0wbX+PnnnxEcHAyVSgU/Pz/MmzfP4LharcZ7770HX19fqFQq+Pv74z//+Y9BnWPHjqFLly6wtbVFjx49yi23TkR1hwkGEdVYVFQUvv/+eyxduhQJCQl466238Nxzz2HPnj36Ou+88w7mzZuHI0eOwN3dHUOGDMHdu3cB6BKDUaNG4ZlnnsGpU6cwc+ZMfPjhh1i5cqX+/HHjxmHNmjX46quvcPbsWSxbtgz29vYGcXzwwQeYN28ejh49CktLS0ycOLFO3j8RVcCka7MSUaNz584dYWtrKw4cOGBQ/uKLL4oxY8bol5Zeu3at/tjNmzeFjY2NWLdunRBCiGeffVb079/f4Px33nlHtG3bVgghxPnz5wUAsX379gpjKH2N33//XV+2efNmAUAUFhaa5H0SUdWwBYOIaiQxMREFBQXo378/7O3t9dv333+PS5cu6euFhYXp/+3q6orAwECcPXsWAHD27Fn07NnT4Lo9e/bExYsXodFoEB8fD4VCgV69et03lvbt2+v/7e3tDQDIyMio8XskoqqzlDsAIjJveXl5AIDNmzejWbNmBsdUKpVBklFdNjY2RtWzsrLS/1uSJAC6/iFEVPfYgkFENdK2bVuoVCpcu3YN/v7+Bpuvr6++3qFDh/T/zsrKwoULFxAUFAQACAoKwv79+w2uu3//frRu3RoKhQIhISHQarUGfTqIqH5jCwYR1YiDgwPefvttvPXWW9BqtXjkkUeQk5OD/fv3w9HRES1atAAAzJo1C02aNIGnpyc++OADuLm5Yfjw4QCA6dOno2vXrpg9ezZGjx6NgwcP4uuvv8bixYsBAH5+fhg/fjwmTpyIr776CqGhobh69SoyMjIwatQoud46Ed2P3J1AiMj8abVaER0dLQIDA4WVlZVwd3cXAwYMEHv27NF3wPzf//4ngoODhVKpFA8//LA4efKkwTViYmJE27ZthZWVlWjevLmYO3euwfHCwkLx1ltvCW9vb6FUKoW/v7/49ttvhRB/d/LMysrS1z9x4oQAIJKSkmr77RNRBSQhhJA5xyGiBmz37t3o06cPsrKy4OzsLHc4RFRH2AeDiIiITI4JBhEREZkcH5EQERGRybEFg4iIiEyOCQYRERGZHBMMIiIiMjkmGERERGRyTDCIiIjI5JhgEBERkckxwSAiIiKTY4JBREREJvf/qsX0+jLD1JkAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 600x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* background: */\n",
       "    progress::-webkit-progress-bar {background-color: #CDCDCD; width: 100%;}\n",
       "    progress {background-color: #CDCDCD;}\n",
       "\n",
       "    /* value: */\n",
       "    progress::-webkit-progress-value {background-color: #00BFFF  !important;}\n",
       "    progress::-moz-progress-bar {background-color: #00BFFF  !important;}\n",
       "    progress {color: #00BFFF ;}\n",
       "\n",
       "    /* optional */\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #000000;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      <progress value='100' class='' max='100' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      100% [100/100] [04:09]\n",
       "      <br>\n",
       "      ████████████████████100.00% [2/2] [val_loss=5.9984e-05]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "keras_model.fit(train_data = dl_train,\n",
    "                val_data = dl_val,\n",
    "                epochs=100,patience=10,\n",
    "                monitor='val_loss',mode='min',\n",
    "                ckpt_path = ckpt_path\n",
    "               )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82a55a90-f980-4dbd-b8f8-6c0e0c8d5a5b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "455498a9-3613-4efa-b7d4-6035314c8fdc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "49763c6a-6ee5-448c-b873-48825a6fba67",
   "metadata": {},
   "source": [
    "## 四，保存模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f98c4e5-dbb0-456e-bc4b-1c8dfccbd499",
   "metadata": {},
   "source": [
    "为避免显存问题，此处可先重启kernel。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ee686c58-42e2-463a-a429-3fe16dee5a7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings \n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "142a917e-9db0-4d5a-8352-cb250561f47d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "===================================BUG REPORT===================================\n",
      "Welcome to bitsandbytes. For bug reports, please run\n",
      "\n",
      "python -m bitsandbytes\n",
      "\n",
      " and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
      "================================================================================\n",
      "bin /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so\n",
      "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...\n",
      "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so\n",
      "CUDA SETUP: Highest compute capability among GPUs detected: 8.0\n",
      "CUDA SETUP: Detected CUDA version 116\n",
      "CUDA SETUP: Loading binary /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so...\n",
      "[2023-08-20 23:41:07,381] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6d4f36de90ae4c9f9b48306b188cf5cd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, AutoModel, BitsAndBytesConfig\n",
    "from transformers.generation.utils import GenerationConfig\n",
    "import torch.nn as nn\n",
    "model_name_or_path ='baichuan-13b'\n",
    "ckpt_path = 'baichuan13b_multi_rounds'\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "    model_name_or_path,\n",
    "    trust_remote_code=True\n",
    ")\n",
    "model_old = AutoModelForCausalLM.from_pretrained(\n",
    "    model_name_or_path,\n",
    "    trust_remote_code=True,\n",
    "    low_cpu_mem_usage=True,\n",
    "    torch_dtype=torch.float16,\n",
    "    device_map='auto'\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2a8b409f-f100-48c5-9794-62cb5bdf19d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from peft import PeftModel\n",
    "\n",
    "#合并qlora权重，可能要5分钟左右\n",
    "peft_model = PeftModel.from_pretrained(model_old, ckpt_path)\n",
    "model_new = peft_model.merge_and_unload()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "05cdfef2-b5b0-4d74-b58b-42a46771f75b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers.generation.utils import GenerationConfig\n",
    "model_new.generation_config = GenerationConfig.from_pretrained(model_name_or_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "524a3d55-5daf-4ff1-8fb9-e3dd60d65f18",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我在2020年诞生于github星球，是一个有毅力的吃货设计和开发的。\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import clear_output\n",
    "\n",
    "messages = [{'role': 'user', 'content': '你是谁呀？'},\n",
    " {'role': 'assistant',\n",
    "  'content': '我叫梦中情炉，英文名字叫做torchkeras. 是一个pytorch模型训练模版工具。'},\n",
    " {'role': 'user', 'content': '你从哪里来呀？'}]\n",
    "\n",
    "response = model_new.chat(tokenizer,messages=messages,stream=True)\n",
    "for res in response:\n",
    "    print(res)\n",
    "    clear_output(wait=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "cc3ec426-f067-4f69-9923-44de9014b357",
   "metadata": {},
   "outputs": [],
   "source": [
    "messages = [{'role': 'user', 'content': '你是谁呀？'},\n",
    " {'role': 'assistant',\n",
    "  'content': '我叫梦中情炉，英文名字叫做torchkeras. 是一个pytorch模型训练模版工具。'},\n",
    " {'role': 'user', 'content': '你多大了？'},\n",
    " {'role': 'assistant', 'content': '我在2020年诞生于github星球，是一个有毅力的吃货设计和开发的。'},\n",
    " {'role': 'user', 'content': '你能帮助我干什么'}]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4d8994f1-e063-43e8-b1f4-3c9d624fa249",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我能够帮助你以最优雅的方式训练各种类型的pytorch模型，并且训练过程中会自动展示一个非常美丽的训练过程图表。\n"
     ]
    }
   ],
   "source": [
    "response = model_new.chat(tokenizer,messages=messages,stream=True)\n",
    "for res in response:\n",
    "    print(res)\n",
    "    clear_output(wait=True)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9214d354-811b-4e47-96ac-682ed5e68ee9",
   "metadata": {},
   "outputs": [],
   "source": [
    "save_path = 'baichuan13b-torchkeras'\n",
    "tokenizer.save_pretrained(save_path)\n",
    "model_new.save_pretrained(save_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a7950603-e844-4d60-990d-f86245fbaa26",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp baichuan-13b/*.py  baichuan13b-torchkeras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a91c249-3402-489b-ae8e-5554a5082562",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "81582c3e-28d3-4847-a84c-dc1b53aa0839",
   "metadata": {},
   "source": [
    "## 五，使用模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c5f987f-0616-4b5c-8ce5-92c0f2b9e2c7",
   "metadata": {},
   "source": [
    "此处可再次重启kernel，以节约显存。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2bec4773-61f0-4aa2-9ac2-d2e1c295b2e3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3592f4e61e01449687c6d7b7f73f26a3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM,AutoConfig, BitsAndBytesConfig\n",
    "from transformers.generation.utils import GenerationConfig\n",
    "import torch.nn as nn\n",
    "\n",
    "\n",
    "model_name_or_path ='baichuan13b-torchkeras'\n",
    "\n",
    "bnb_config=BitsAndBytesConfig(\n",
    "            load_in_4bit=True,\n",
    "            bnb_4bit_compute_dtype=torch.float16,\n",
    "            bnb_4bit_use_double_quant=True,\n",
    "            bnb_4bit_quant_type=\"nf4\",\n",
    "            llm_int8_threshold=6.0,\n",
    "            llm_int8_has_fp16_weight=False,\n",
    "        )\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\n",
    "   model_name_or_path, trust_remote_code=True)\n",
    "\n",
    "model = AutoModelForCausalLM.from_pretrained(model_name_or_path,\n",
    "                quantization_config=bnb_config,\n",
    "                trust_remote_code=True) \n",
    "\n",
    "model.generation_config = GenerationConfig.from_pretrained(model_name_or_path)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efdb6cb0-42a9-472e-8bfc-c64af33bc7e4",
   "metadata": {},
   "source": [
    "通过使用chatLLM可以在jupyter中使用魔法命令对各种LLM模型(Baichuan13b,Qwen,ChatGLM2,Llama2以及更多)进行交互测试。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "3a73e76f-ae41-444f-8d11-46ec34a14936",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "register magic %%chat sucessed ...\n"
     ]
    }
   ],
   "source": [
    "from torchkeras.chat import ChatLLM \n",
    "llm = ChatLLM(model,\n",
    "    tokenizer,\n",
    "    model_type=None,\n",
    "    max_chat_rounds=20,\n",
    "    max_new_tokens=512,\n",
    "    stream=True,\n",
    "    history=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "eefeece5-8cec-4ae3-afd2-c2ec718c0691",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我叫梦中情炉，是一个三好炼丹炉：好看，好用，好改。我的英文名字叫做torchkeras，是一个pytorch模型训练模版工具。\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "你叫什么呀？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "f6304008-6657-4c75-8b45-10f21a3856fb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我在2020年诞生于github星球，是一个有毅力的吃货设计和开发的。\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "你从哪里来呀？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "0285d133-732b-4dff-aa59-d8e2ee939768",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我能够帮助你以最优雅的方式训练各种类型的pytorch模型，并且训练过程中会自动展示一个非常美丽的训练过程图表。\n"
     ]
    }
   ],
   "source": [
    "%%chat\n",
    "你能干嘛呀？"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cd9e417-eedc-44c7-b59c-30f9291d961e",
   "metadata": {},
   "source": [
    "非常棒，粗浅的测试表明，我们的多轮对话训练是成功的。已经在BaiChuan的自我认知中，种下了一颗梦中情炉的种子。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "409ebd31-6a11-4903-9741-1d81a5538af3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "bf2a35e1-dca4-42cf-8f31-9e477e091269",
   "metadata": {},
   "source": [
    "**如果本项目对你有所帮助，想鼓励一下作者，记得给本项目加一颗星星star⭐️，并分享给你的朋友们喔😊!** \n",
    "\n",
    "如果在torchkeras的使用中遇到问题，可以在项目中提交issue。\n",
    "\n",
    "如果想要获得更快的反馈或者与其他torchkeras用户小伙伴进行交流，\n",
    "\n",
    "可以在公众号算法美食屋后台回复关键字：**加群**。\n",
    "\n",
    "![](https://tva1.sinaimg.cn/large/e6c9d24egy1h41m2zugguj20k00b9q46.jpg)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}