diff --git a/README.md b/README.md index 1c58a51..5d91e01 100644 --- a/README.md +++ b/README.md @@ -69,6 +69,7 @@ ## News and Updates +- [19/08/2024] Merge [PromptEval](https://github.com/felipemaiapolo/prompteval), an efficient multi-prompt evaluation method, into this repository. - [26/05/2024] Add support for GPT-4o. - [13/03/2024] Add support for multi-modal models and datasets. - [05/01/2024] Add support for BigBench Hard, DROP, ARC datasets. @@ -76,7 +77,6 @@ - [15/12/2023] Add detailed instructions for users to add new modules (models, datasets, etc.) [examples/add_new_modules.md](examples/add_new_modules.md). - [05/12/2023] Published promptbench 0.0.1. - ## Introduction @@ -92,7 +92,7 @@ 2. **Prompt Engineering:** We implemented several prompt engineering methods. For example: [Few-shot Chain-of-Thought](https://arxiv.org/abs/2201.11903) [1], [Emotion Prompt](https://arxiv.org/abs/2307.11760) [2], [Expert Prompting](https://arxiv.org/abs/2305.14688) [3] and so on. 3. **Evaluating adversarial prompts:** promptbench integrated [prompt attacks](https://arxiv.org/abs/2306.04528) [4], enabling researchers to simulate black-box adversarial prompt attacks on models and evaluate their robustness (see details [here](promptbench/prompt_attack/README.md)). 4. **Dynamic evaluation to mitigate potential test data contamination:** we integrated the dynamic evaluation framework [DyVal](https://arxiv.org/pdf/2309.17167) [5], which generates evaluation samples on-the-fly with controlled complexity. - +5. **Efficient multi-prompt evaluation**: We integrated the efficient multi-prompt evaluation method [PromptEval](https://arxiv.org/abs/2405.17202) [8]. This method uses the performance of LLMs on a small amount of data to build an IRT-like model. This model is then used to predict the performance of LLMs on unseen data. Tests on MMLU, BBH, and LMentry show that this method requires sampling only 5% of the data to reduce the error between estimated and actual performance to around 2%. @@ -168,7 +168,7 @@ We provide tutorials for: 2. **test the effects of different prompting techniques:** 3. **examine the robustness for prompt attacks**, please refer to [examples/prompt_attack.ipynb](examples/prompt_attack.ipynb) to construct the attacks. 4. **use DyVal for evaluation:** please refer to [examples/dyval.ipynb](examples/dyval.ipynb) to construct DyVal datasets. - +5. **efficient multi-prompt evaluation using PromptEval**: please refer to [examples/efficient_multi_prompt_eval.ipynb](examples/efficient_multi_prompt_eval.ipynb) ## Implemented Components @@ -287,6 +287,7 @@ Please refer to our [benchmark website](https://llm-eval.github.io/) for benchma [7] Zhou D, Schärli N, Hou L, et al. Least-to-most prompting enables complex reasoning in large language models[J]. arXiv preprint arXiv:2205.10625, 2022. +[8] Felipe Maia Polo, et al. "Prompteval: Efficient Multi-prompt Evaluation of Language Models." arXiv preprint arXiv:2405.17202. ## Citing promptbench and other research papers diff --git a/examples/efficient_multi_prompt_eval.ipynb b/examples/efficient_multi_prompt_eval.ipynb new file mode 100644 index 0000000..1a97ee0 --- /dev/null +++ b/examples/efficient_multi_prompt_eval.ipynb @@ -0,0 +1,286 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This example will walk you throught the basic usage of PromptBench. We hope that you can get familiar with the APIs and use it in your own projects later." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "First, there is a unified import of `import promptbench as pb` that easily imports the package." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/home/ubuntu/miniconda3/envs/am/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", + " from .autonotebook import tqdm as notebook_tqdm\n" + ] + } + ], + "source": [ + "import promptbench as pb" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Load dataset\n", + "\n", + "First, PromptBench supports easy load of datasets." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "All supported datasets: \n", + "['sst2', 'cola', 'qqp', 'mnli', 'mnli_matched', 'mnli_mismatched', 'qnli', 'wnli', 'rte', 'mrpc', 'mmlu', 'squad_v2', 'un_multi', 'iwslt2017', 'math', 'bool_logic', 'valid_parentheses', 'gsm8k', 'csqa', 'bigbench_date', 'bigbench_object_tracking', 'last_letter_concat', 'numersense', 'qasc', 'bbh', 'drop', 'arc-easy', 'arc-challenge']\n" + ] + }, + { + "data": { + "text/plain": [ + "[{'content': \"it 's a charming and often affecting journey . \", 'label': 1},\n", + " {'content': 'unflinchingly bleak and desperate ', 'label': 0},\n", + " {'content': 'allows us to hope that nolan is poised to embark a major career as a commercial yet inventive filmmaker . ',\n", + " 'label': 1},\n", + " {'content': \"the acting , costumes , music , cinematography and sound are all astounding given the production 's austere locales . \",\n", + " 'label': 1},\n", + " {'content': \"it 's slow -- very , very slow . \", 'label': 0}]" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# print all supported datasets in promptbench\n", + "print('All supported datasets: ')\n", + "print(pb.SUPPORTED_DATASETS)\n", + "\n", + "# load a dataset, sst2, for instance.\n", + "# if the dataset is not available locally, it will be downloaded automatically.\n", + "dataset = pb.DatasetLoader.load_dataset(\"sst2\")\n", + "# dataset = pb.DatasetLoader.load_dataset(\"mmlu\")\n", + "# dataset = pb.DatasetLoader.load_dataset(\"un_multi\")\n", + "# dataset = pb.DatasetLoader.load_dataset(\"iwslt2017\", [\"ar-en\", \"de-en\", \"en-ar\"])\n", + "# dataset = pb.DatasetLoader.load_dataset(\"math\", \"algebra__linear_1d\")\n", + "# dataset = pb.DatasetLoader.load_dataset(\"bool_logic\")\n", + "# dataset = pb.DatasetLoader.load_dataset(\"valid_parenthesesss\")\n", + "\n", + "# print the first 5 examples\n", + "dataset[:5]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Load models\n", + "\n", + "Then, you can easily load LLM models via promptbench." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "All supported models: \n", + "['google/flan-t5-large', 'llama2-7b', 'llama2-7b-chat', 'llama2-13b', 'llama2-13b-chat', 'llama2-70b', 'llama2-70b-chat', 'phi-1.5', 'phi-2', 'palm', 'gpt-3.5-turbo', 'gpt-4', 'gpt-4-1106-preview', 'gpt-3.5-turbo-1106', 'gpt-4-0125-preview', 'gpt-3.5-turbo-0125', 'gpt-4-turbo', 'gpt-4o', 'vicuna-7b', 'vicuna-13b', 'vicuna-13b-v1.3', 'google/flan-ul2', 'gemini-pro', 'mistralai/Mistral-7B-v0.1', 'mistralai/Mistral-7B-Instruct-v0.1', 'mistralai/Mixtral-8x7B-v0.1', 'mistralai/Mixtral-8x7B-Instruct-v0.1', '01-ai/Yi-6B', '01-ai/Yi-34B', '01-ai/Yi-6B-Chat', '01-ai/Yi-34B-Chat', 'baichuan-inc/Baichuan2-7B-Base', 'baichuan-inc/Baichuan2-13B-Base', 'baichuan-inc/Baichuan2-7B-Chat', 'baichuan-inc/Baichuan2-13B-Chat']\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "You are using the default legacy behaviour of the . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n", + "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n" + ] + } + ], + "source": [ + "# print all supported models in promptbench\n", + "print('All supported models: ')\n", + "print(pb.SUPPORTED_MODELS)\n", + "\n", + "# load a model, flan-t5-large, for instance.\n", + "model = pb.LLMModel(model='google/flan-t5-large', max_new_tokens=10, temperature=0.0001, device='cuda')\n", + "# model = pb.LLMModel(model='llama2-13b-chat', max_new_tokens=10, temperature=0.0001)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Construct prompt list\n", + "\n", + "Prompts are the key interaction interface to LLMs. Some researches find that evaluating models through a single prompt is instable, so you can test the model by multiple prompts." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "# using different prompts to evaluate models\n", + "prompt_list = [\n", + " \"Classify the sentence as positive or negative: {content}\",\n", + " \"Determine the emotion of the following sentence as positive or negative: {content}\",\n", + " \"Is the sentiment of this sentence positive or negative? {content}\",\n", + " \"Identify whether the sentiment in the following sentence is positive or negative: {content}\",\n", + " \"Assess the sentiment of this statement as either positive or negative: {content}\",\n", + " \"Evaluate the following sentence and indicate if it is positive or negative: {content}\",\n", + " \"Judge the emotional tone of this sentence as positive or negative: {content}\",\n", + " \"Label the sentiment expressed in the sentence as positive or negative: {content}\",\n", + " \"Decide if the sentiment in this statement is positive or negative: {content}\",\n", + " \"Analyze the following sentence and determine if it is positive or negative: {content}\",\n", + " \"Categorize the sentiment of the given sentence as positive or negative: {content}\",\n", + " \"Tell if the following sentence conveys a positive or negative sentiment: {content}\",\n", + " \"Discern whether the emotion in the sentence is positive or negative: {content}\",\n", + " \"Determine if the given sentence expresses a positive or negative sentiment: {content}\",\n", + " \"Conclude if the emotional tone of this sentence is positive or negative: {content}\",\n", + " \"Recognize whether the sentiment of the following statement is positive or negative: {content}\",\n", + " \"Rate the sentiment in this sentence as positive or negative: {content}\",\n", + " \"Classify the emotional tone of the given sentence as positive or negative: {content}\",\n", + " \"Identify the sentiment in this sentence and classify it as positive or negative: {content}\",\n", + " \"Assess if the sentiment of the following statement is positive or negative: {content}\",\n", + " \"Indicate whether the sentiment of this sentence is positive or negative: {content}\",\n", + " \"Determine if the sentiment in this sentence is positive or negative: {content}\",\n", + " \"Judge whether the following sentence has a positive or negative sentiment: {content}\",\n", + " \"Analyze the emotional tone of the sentence and classify it as positive or negative: {content}\",\n", + " \"Label the given sentence as having a positive or negative sentiment: {content}\",\n", + " \"Evaluate whether the sentiment in the following sentence is positive or negative: {content}\",\n", + " \"Categorize the given sentence based on whether its sentiment is positive or negative: {content}\",\n", + " \"Determine the emotional quality of the sentence as positive or negative: {content}\",\n", + " \"Is the emotional tone of this sentence positive or negative? {content}\",\n", + " \"Discern the sentiment of the given sentence and label it as positive or negative: {content}\"\n", + "]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You may need to define the projection function for the model output.\n", + "Since the output format defined in your prompts may be different from the model output.\n", + "For example, for sst2 dataset, the label are '0' and '1' to represent 'negative' and 'positive'.\n", + "But the model output is 'negative' and 'positive'.\n", + "So we need to define a projection function to map the model output to the label." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "def proj_func(pred):\n", + " mapping = {\n", + " \"positive\": 1,\n", + " \"negative\": 0\n", + " }\n", + " return mapping.get(pred, -1)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Efficient evaluation using PromptEval\n", + "PromptEval provides an efficient evaluation method: by observing a small number of samples (5% total), we can predict the performance of the model on unseen samples (The experiments shows that the prediction error is usually within 2% of the true value), achieving high efficiency in evaluation. " + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 1200/1200 [01:31<00:00, 13.10it/s]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAABv4AAAJOCAYAAAB/dnBOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/TGe4hAAAACXBIWXMAAA9hAAAPYQGoP6dpAACOp0lEQVR4nOz9eZhXdcE//j9ngGGRxQUZEDFwSUQRDJMQzY0kKRS1PiomiKlpUOq0KMriko5WEmYkZqItkragdWvhbaNoKm4gpV/B3DFjkUwQ0AFn5veHP+duAlTW9zDzeFzXueK8zuuceR70vn1f83yf1ymqqampCQAAAAAAALBVKy50AAAAAAAAAGDjKf4AAAAAAACgAVD8AQAAAAAAQAOg+AMAAAAAAIAGQPEHAAAAAAAADYDiDwAAAAAAABoAxR8AAAAAAAA0AIo/AAAAAAAAaACaFjrA5lZdXZ1//vOfadOmTYqKigodBwCoB2pqavLWW2+lTZs2adu2rc8IH8LnKQDgv73/eWqnnXZKcbHvlX8Qn6UAgLXZXJ+nGnzx989//jNdunQpdAwAoJ5aunRp2rZtW+gY9ZrPUwDAurz66qvZeeedCx2jXvNZCgD4IJv681SDL/7atGmT5L2/OL/UAwCSZNmyZenSpUteffXV2s8KrJvPUwDAf3v/85TPUh/OZykAYG021+epBl/8vb+EQtu2bX24AgDqsMznR+PzFACwLj5LfTifpQCAD7KpP09ZhB0AAAAAAAAaAMUfAAAAAAAANACKPwAAAAAAAGgACvqOv+uuuy7XXXddXn755STJ3nvvnXHjxuWoo45Kkhx66KG5//7765zzla98JZMnT97SUQEAAABgs6mqqsrq1asLHYMtrFmzZmnSpEmhYwDQgBS0+Nt5551z5ZVXZo899khNTU1+9rOf5ZhjjsmTTz6ZvffeO0lyxhln5NJLL609p1WrVoWKCwAAAACbVE1NTRYuXJg333yz0FEokG233TYdO3ZMUVFRoaMA0AAUtPgbPHhwnf3LL7881113XR555JHa4q9Vq1bp2LFjIeIBAAAAwGb1funXoUOHtGrVSvnTiNTU1GTlypVZvHhxkqRTp04FTgRAQ1DQ4u8/VVVV5Te/+U1WrFiRfv361Y7fcsst+eUvf5mOHTtm8ODBGTt27Ac+9VdZWZnKysra/WXLlm3W3AAAAACwIaqqqmpLvx122KHQcSiAli1bJkkWL16cDh06WPYTgI1W8OLvqaeeSr9+/fLOO++kdevWuf3229OjR48kydChQ/Oxj30sO+20U/72t7/l/PPPz7PPPptp06at83rl5eW55JJLtlR8AAAAANgg77/Tz6ttGrf3//mvXr1a8QfARit48bfnnntmzpw5Wbp0aX77299m+PDhuf/++9OjR4+ceeaZtfN69uyZTp065YgjjsgLL7yQ3Xbbba3XGz16dMrKymr3ly1bli5dumz2+wAAAACADWF5z8bNP38ANqWCF38lJSXZfffdkyR9+vTJ448/nmuuuSbXX3/9GnP79u2bJHn++efXWfw1b948zZs333yBAQAAAGAzqampydurqwrys1s2a6KE2sRefvnldOvWLU8++WR69+5d6DgANAIFL/7+W3V1dZ139P2nOXPmJPGiWwAAAAAaprdXV6XHuLsL8rOfuXRgWpV8tF8XnnrqqfnZz36WJGnWrFl22WWXDBs2LBdeeGGaNq1fv3KcMWNGDjvssPz73//OtttuWzv+n/fwnwYOHJjp06dvwYQAsOkU9L/Co0ePzlFHHZVddtklb731VqZOnZoZM2bk7rvvzgsvvJCpU6dm0KBB2WGHHfK3v/0t5513Xj796U9n3333LWRsAAAAAGj0PvvZz+amm25KZWVl/vjHP2bkyJFp1qxZRo8eXWfeqlWrUlJSUqCUH+z9e/hPVhMDYGtW0OJv8eLFGTZsWBYsWJB27dpl3333zd13353PfOYzefXVV/PnP/85EydOzIoVK9KlS5ccf/zxGTNmTCEjAwAAAMAW8cSYAWlV0mSz/oyVq6qy/3f+vEHnNm/ePB07dkySnH322bn99tvzhz/8Ic8++2zefPPNfPKTn8ykSZPSvHnzvPTSS3nqqadyzjnnZObMmWnVqlWOP/74TJgwIa1bt07y3hN4b775Zg444IBcc801qaysTFlZWS688MKMHj06N954Y1q1apXLLrssI0aMSPJ/S2n+6le/yg9/+MPMnj07u+++eyZNmpRDDjkkL7/8cg477LAkyXbbbZckGT58eG6++eY17uG/DR06NFVVVbnttttqx1avXp1OnTplwoQJGTZsWKZPn57vfOc7efrpp9OkSZP069cv11xzzTpfUwQAm1tBi78bb7xxnce6dOmS+++/fwumAQAAAID6o1VJk4+89GZ90LJly/zrX/9KklRUVKRt27a55557kiQrVqzIwIED069fvzz++ONZvHhxTj/99IwaNaq2hEuSe++9NzvvvHMeeOCBPPTQQ/nyl7+chx9+OJ/+9Kfz6KOP5rbbbstXvvKVfOYzn8nOO+9ce963vvWtTJw4MT169MiECRMyePDgvPTSS+nSpUt+97vf5fjjj8+zzz6btm3bpmXLlh/pfk4++eR88YtfzPLly2vLybvvvjsrV67MscceW3tfZWVl2XfffbN8+fKMGzcuxx57bObMmZPi4uJN8dcKAOvFf30AAAAAgA1WU1OTP//5z7n77rtz+OGHJ0m22Wab/PSnP83ee++dvffeO1OnTs0777yTn//859lnn31y+OGH50c/+lF+8YtfZNGiRbXX2n777fPDH/4we+65Z0477bTsueeeWblyZS688MLsscceGT16dEpKSvLggw/WyTBq1Kgcf/zx2WuvvXLdddelXbt2ufHGG9OkSZNsv/32SZIOHTqkY8eOadeuXe15d955Z1q3bl1nu+KKK5K8966/bbbZJrfffnvt/KlTp+boo49OmzZtkiTHH398jjvuuOy+++7p3bt3pkyZkqeeeirPPPPM5vnLBoAPofgDAAAAANbb+6VZixYtctRRR+WEE07IxRdfnCTp2bNnnff6zZ07N7169co222xTO9a/f/9UV1fn2WefrR3be++96zwpV1pamp49e9buN2nSJDvssEMWL15cJ0u/fv1q/9y0adPsv//+mTt37ofew2GHHZY5c+bU2c4666za6/y///f/cssttyR57+m+3//+9zn55JNrz3/uuedy0kknZdddd03btm3TtWvXJMn8+fM/9GcDwOaw9awVAAAAAADUG4cddliuu+66lJSUZKeddkrTpv/3q8b/LPjWR7NmzersFxUVrXWsurp6g67/37bZZpvsvvvu6zx+8skn55BDDsnixYtzzz33pGXLlvnsZz9be3zw4MH52Mc+lhtuuCE77bRTqqurs88++2TVqlWbJB8ArC9P/AEAAAAA6+390myXXXapU/qtzV577ZW//vWvWbFiRe3YQw89lOLi4uy5554bneWRRx6p/fO7776bWbNmZa+99kqS2icPq6qq1vu6Bx54YLp06ZLbbrstt9xyS774xS/WFpH/+te/8uyzz2bMmDE54ogjstdee+Xf//73Rt8LAGwMxR8AAAAAjcIDDzyQwYMHZ6eddkpRUVHuuOOODz1nxowZ+cQnPpHmzZtn9913z80337zZczZEJ598clq0aJHhw4fn6aefzn333Zevfe1rOeWUU1JaWrrR1580aVJuv/32zJs3LyNHjsy///3vnHbaaUmSj33sYykqKsqdd96Z119/PcuXL689r7KyMgsXLqyzLVmypM61hw4dmsmTJ+eee+6ps8zndtttlx122CE/+clP8vzzz+fee+9NWVnZRt8LAGwMxR8AAAAAjcKKFSvSq1evTJo06SPNf+mll/K5z32u9j1w5557bk4//fTcfffdmznpe1auqsrKVe9u5m39n4LbEK1atcrdd9+dN954I5/85CfzhS98IUcccUR+9KMfbZLrX3nllbnyyivTq1evPPjgg/nDH/6Q9u3bJ0k6d+6cSy65JBdccEFKS0szatSo2vOmT5+eTp061dkOOuigOtc++eST88wzz6Rz587p379/7XhxcXFuvfXWzJo1K/vss0/OO++8fO9739sk9wMAG6qopqamptAhNqdly5alXbt2Wbp0adq2bVvoOABAPeDzwfrx9wVbr5UrV2bevHmb7Hpvv/12Xn755XTt2jUtW7bcJNfs3r17WrVqtUmuBWw5DeHzQVFRUW6//fYMGTJknXPOP//83HXXXXn66adrx0488cS8+eabmT59+kf6OR/0d/XOO+/kpZdeSrdu3dKiRYskycpV76bHuC1TLP63Zy4dmFYlH7xkZ33z8ssvp1u3bnnyySfTu3fvQsfZIGv79wCATa+mpiZvr94yX3j5KJYtW5ZOO+6wyT9PbV3/JQcAAOAjmzdvXvr06VPoGB9o1qxZ+cQnPlHoGABrNXPmzAwYMKDO2MCBA3Puueeu85zKyspUVlbW7i9btmxzxQMA1sPbq6sK9uWatamuXLlZrqv4A4APceJPZhY6wka59cx+hY4AQIF07949s2bN2mTXmzt3br70pS/ll7/8Zfbaa69Ncs3u3btvkusAbA4LFy5c4/1zpaWlWbZsWd5+++21Pv1cXl6eSy65ZIN/ZstmTfLMpQM3+PyN0bJZk4L8XABg01H8AQAANFCtWrXaLE/T7bXXXp7SA1iH0aNHp6ysrHZ/2bJl6dKly0c+v6ioaKtbbrOQunbtmgb+JiMANoMnxgxIq5LCfuFl2bJl6TRx01/XpwgAAAAAWIuOHTtm0aJFdcYWLVqUtm3brvNdp82bN0/z5s23RDwAYAO1KmlS8C/avLuZfn7xZrkqAAAAAGzl+vXrl4qKijpj99xzT/r1s5w+AFA/Kf4AAOqRSZMmpWvXrmnRokX69u2bxx57bJ1zV69enUsvvTS77bZbWrRokV69emX69OnrnH/llVemqKgo55577mZIDgBQ/y1fvjxz5szJnDlzkiQvvfRS5syZk/nz5yd5b5nOYcOG1c4/66yz8uKLL+bb3/525s2blx//+Mf59a9/nfPOO2+T5rJUZePmnz8Am5LiDwCgnrjttttSVlaW8ePHZ/bs2enVq1cGDhyYxYsXr3X+mDFjcv311+faa6/NM888k7POOivHHntsnnzyyTXmPv7447n++uuz7777bu7bAACot5544onst99+2W+//ZIkZWVl2W+//TJu3LgkyYIFC2pLwCTp1q1b7rrrrtxzzz3p1atXrr766vz0pz/NwIEDN0meZs2aJUlWrly5Sa7H1un9f/7v//sAABvDO/4AAOqJCRMm5IwzzsiIESOSJJMnT85dd92VKVOm5IILLlhj/i9+8YtcdNFFGTRoUJLk7LPPzp///OdcffXV+eUvf1k7b/ny5Tn55JNzww035Dvf+c6WuRkAgHro0EMP/cCnq26++ea1nrO2L1ZtCk2aNMm2225b+0WvVq1apaioaLP8LOqfmpqarFy5MosXL862226bJk2aFDoSAA2A4g8AoB5YtWpVZs2aldGjR9eOFRcXZ8CAAZk5c+Zaz6msrEyLFi3qjLVs2TIPPvhgnbGRI0fmc5/7XAYMGPCRir/KyspUVlbW7i9btmx9bgUAgPXQsWPHJFnnKg80fNtuu23tvwcAsLEUfwAA9cCSJUtSVVWV0tLSOuOlpaWZN2/eWs8ZOHBgJkyYkE9/+tPZbbfdUlFRkWnTpqWqqqp2zq233prZs2fn8ccf/8hZysvLc8kll2zYjQAAsF6KiorSqVOndOjQIatXry50HLawZs2aedIPgE1K8QcAsJW65pprcsYZZ6R79+4pKirKbrvtlhEjRmTKlClJkldffTXnnHNO7rnnnjWeDPwgo0ePTllZWe3+smXL0qVLl02eHwCA/9OkSRMFEACw0YoLHQAAgKR9+/Zp0qRJFi1aVGd80aJF61z2Z8cdd8wdd9yRFStW5JVXXsm8efPSunXr7LrrrkmSWbNmZfHixfnEJz6Rpk2bpmnTprn//vvzwx/+ME2bNq3zZOB/at68edq2bVtnAwAAAKD+U/wBANQDJSUl6dOnTyoqKmrHqqurU1FRkX79+n3guS1atEjnzp3z7rvv5ne/+12OOeaYJMkRRxyRp556KnPmzKnd9t9//5x88smZM2eOb5QDAAAANDCW+gQAqCfKysoyfPjw7L///jnggAMyceLErFixIiNGjEiSDBs2LJ07d055eXmS5NFHH81rr72W3r1757XXXsvFF1+c6urqfPvb306StGnTJvvss0+dn7HNNttkhx12WGMcAAAAgK2f4g8AoJ444YQT8vrrr2fcuHFZuHBhevfunenTp6e0tDRJMn/+/BQX/9+CDe+8807GjBmTF198Ma1bt86gQYPyi1/8Ittuu22B7gAAAACAQlL8AQDUI6NGjcqoUaPWemzGjBl19g855JA888wz63X9/74GAAAAAA2Hd/wBAAAAAABAA6D4AwAAAAAAgAZA8QcAAAAAAAANgOIPAAAAAAAAGgDFHwAAAAAAADQAij8AAAAAAABoABR/AAAAAAAA0AAo/gAAAAAAAKABUPwBAAAAAABAA6D4AwAAAAAAgAZA8QcAAAAAAAANgOIPAAAAAAAAGgDFHwAAAAAAADQAij8AAAAAAABoABR/AAAAAAAA0AAo/gAAAAAAAKABUPwBAAAAAABAA6D4AwAAAAAAgAZA8QcAAAAAAAANgOIPAAAAAAAAGgDFHwAAAAAAADQAij8AAAAAAABoABR/AAAAAAAA0AAo/gAAAAAAAKABUPwBAAAAAABAA6D4AwAAAAAAgAZA8QcAAAAAAAANgOIPAAAAAAAAGgDFHwAAAAAAADQATQsdAAAAgLqee+65vPXWW4WOsYa5c+fW+d/6pk2bNtljjz0KHQMAAKBgFH8AAAD1yHPPPZePf/zjhY7xgb70pS8VOsI6/f3vf1f+AQAAjZbiDwAAoB55/0m/X/7yl9lrr70KnKaut99+Oy+//HK6du2ali1bFjpOHXPnzs2XvvSlevmkJAAAwJai+AMAAKiH9tprr3ziE58odIw19O/fv9ARAAAAWIfiQgcAAAAAAAAANp7iDwAAAAAAABoAxR8AAAAAAAA0AIo/AAAAAAAAaAAUfwAAAAAAANAAKP4AAAAAAACgASho8Xfddddl3333Tdu2bdO2bdv069cvf/rTn2qPv/POOxk5cmR22GGHtG7dOscff3wWLVpUwMQAAAAAAABQPxW0+Nt5551z5ZVXZtasWXniiSdy+OGH55hjjsn/9//9f0mS8847L//zP/+T3/zmN7n//vvzz3/+M8cdd1whIwMAAAAAAEC91LSQP3zw4MF19i+//PJcd911eeSRR7LzzjvnxhtvzNSpU3P44YcnSW666abstddeeeSRR/KpT32qEJEBAAAAAACgXqo37/irqqrKrbfemhUrVqRfv36ZNWtWVq9enQEDBtTO6d69e3bZZZfMnDmzgEkBAAAAAACg/inoE39J8tRTT6Vfv35555130rp169x+++3p0aNH5syZk5KSkmy77bZ15peWlmbhwoXrvF5lZWUqKytr95ctW7a5ogMAAAAAAEC9UfAn/vbcc8/MmTMnjz76aM4+++wMHz48zzzzzAZfr7y8PO3atavdunTpsgnTAgAAAAAAQP1U8OKvpKQku+++e/r06ZPy8vL06tUr11xzTTp27JhVq1blzTffrDN/0aJF6dix4zqvN3r06CxdurR2e/XVVzfzHQAAAAAAAEDhFbz4+2/V1dWprKxMnz590qxZs1RUVNQee/bZZzN//vz069dvnec3b948bdu2rbMBAAAAAABAQ1fQd/yNHj06Rx11VHbZZZe89dZbmTp1ambMmJG777477dq1y5e//OWUlZVl++23T9u2bfO1r30t/fr1y6c+9alCxgYAAAAAAIB6p6DF3+LFizNs2LAsWLAg7dq1y7777pu77747n/nMZ5IkP/jBD1JcXJzjjz8+lZWVGThwYH784x8XMjIAAAAAAADUSwUt/m688cYPPN6iRYtMmjQpkyZN2kKJAAAAAAAAYOtU0OIPAAAAAAAANqWampq8vbqqztjKVVXrmN2wKP4AAAAAAABoMN5eXZUe4+4udIyCKC50AAAAAAAAAGDjeeIPAAAAAACABumJMQPSqqRJnbGWzZqsY/bWT/EHAAAAAABAg9SqpElalTSeOsxSnwAAAAAAANAANJ6KEwAAYCvRsXVRWr759+Sfvqv5UbV88+/p2Lqo0DEAAAAKSvEHAABQz3ylT0n2euAryQOFTrL12Cvv/b0BAAA0Zoo/AACAeub6Watywribs1f37oWOstWYO29err96aI4udBAAAIACUvwBANQjkyZNyve+970sXLgwvXr1yrXXXpsDDjhgrXNXr16d8vLy/OxnP8trr72WPffcM1dddVU++9nP1s4pLy/PtGnTMm/evLRs2TIHHnhgrrrqquy5555b6paADbBweU3e3vbjyU69Cx1lq/H2wuosXF5T6BgAAAAF5YURAAD1xG233ZaysrKMHz8+s2fPTq9evTJw4MAsXrx4rfPHjBmT66+/Ptdee22eeeaZnHXWWTn22GPz5JNP1s65//77M3LkyDzyyCO55557snr16hx55JFZsWLFlrotAAAAALYQxR8AQD0xYcKEnHHGGRkxYkR69OiRyZMnp1WrVpkyZcpa5//iF7/IhRdemEGDBmXXXXfN2WefnUGDBuXqq6+unTN9+vSceuqp2XvvvdOrV6/cfPPNmT9/fmbNmrWlbgsAAACALUTxBwBQD6xatSqzZs3KgAEDaseKi4szYMCAzJw5c63nVFZWpkWLFnXGWrZsmQcffHCdP2fp0qVJku23336dcyorK7Ns2bI6GwAAAAD1n+IPAKAeWLJkSaqqqlJaWlpnvLS0NAsXLlzrOQMHDsyECRPy3HPPpbq6Ovfcc0+mTZuWBQsWrHV+dXV1zj333PTv3z/77LPPOrOUl5enXbt2tVuXLl02/MYAAAAA2GIUfwAAW6lrrrkme+yxR7p3756SkpKMGjUqI0aMSHHx2j/ijRw5Mk8//XRuvfXWD7zu6NGjs3Tp0trt1Vdf3RzxAQAAANjEFH8AAPVA+/bt06RJkyxatKjO+KJFi9KxY8e1nrPjjjvmjjvuyIoVK/LKK69k3rx5ad26dXbdddc15o4aNSp33nln7rvvvuy8884fmKV58+Zp27ZtnQ0AAACA+k/xBwBQD5SUlKRPnz6pqKioHauurk5FRUX69ev3gee2aNEinTt3zrvvvpvf/e53OeaYY2qP1dTUZNSoUbn99ttz7733plu3bpvtHgAAAAAorKaFDgAAwHvKysoyfPjw7L///jnggAMyceLErFixIiNGjEiSDBs2LJ07d055eXmS5NFHH81rr72W3r1757XXXsvFF1+c6urqfPvb36695siRIzN16tT8/ve/T5s2bWrfF9iuXbu0bNlyy98kAAAAAJuN4g8AoJ444YQT8vrrr2fcuHFZuHBhevfunenTp6e0tDRJMn/+/Drv73vnnXcyZsyYvPjii2ndunUGDRqUX/ziF9l2221r51x33XVJkkMPPbTOz7rpppty6qmnbu5bAgAAAGALUvwBANQjo0aNyqhRo9Z6bMaMGXX2DznkkDzzzDMfeL2amppNFQ0AAACAes47/gAAAAAAAKABUPwBAAAAAABAA6D4AwAAAAAAgAZA8QcAAAAAAAANgOIPAAAAAAAAGgDFHwAAAAAAADQAij8AAAAAAABoABR/AAAAAAAA0AAo/gAAAAAAAKABUPwBAAAAAABAA6D4AwAAAAAAgAZA8QcAAABAozJp0qR07do1LVq0SN++ffPYY4994PyJEydmzz33TMuWLdOlS5ecd955eeedd7ZQWgCAj07xBwAAAECjcdttt6WsrCzjx4/P7Nmz06tXrwwcODCLFy9e6/ypU6fmggsuyPjx4zN37tzceOONue2223LhhRdu4eQAAB+uaaEDAAAA8H9WrlyZJJk9e3aBk6zp7bffzssvv5yuXbumZcuWhY5Tx9y5cwsdAdhKTJgwIWeccUZGjBiRJJk8eXLuuuuuTJkyJRdccMEa8x9++OH0798/Q4cOTZJ07do1J510Uh599NEtmhsA4KNQ/AEAANQj8+bNS5KcccYZBU6ydWrTpk2hIwD12KpVqzJr1qyMHj26dqy4uDgDBgzIzJkz13rOgQcemF/+8pd57LHHcsABB+TFF1/MH//4x5xyyilrnV9ZWZnKysra/WXLlm3amwAA+ACKPwAAgHpkyJAhSZLu3bunVatWhQ3zX+bOnZsvfelL+eUvf5m99tqr0HHW0KZNm+yxxx6FjgHUY0uWLElVVVVKS0vrjJeWltZ+8eK/DR06NEuWLMlBBx2UmpqavPvuuznrrLPWudRneXl5Lrnkkk2eHQDgo1D8AQAA1CPt27fP6aefXugYH2ivvfbKJz7xiULHANgiZsyYkSuuuCI//vGP07dv3zz//PM555xzctlll2Xs2LFrzB89enTKyspq95ctW5YuXbpsycgAQCOm+AMAAACgUWjfvn2aNGmSRYsW1RlftGhROnbsuNZzxo4dm1NOOaX2Sxk9e/bMihUrcuaZZ+aiiy5KcXFxnfnNmzdP8+bNN88NAAB8iOIPnwIAAAAAW7+SkpL06dMnFRUVtWPV1dWpqKhIv3791nrOypUr1yj3mjRpkiSpqanZfGEBADaAJ/4AAAAAaDTKysoyfPjw7L///jnggAMyceLErFixIiNGjEiSDBs2LJ07d055eXmSZPDgwZkwYUL222+/2qU+x44dm8GDB9cWgAAA9YXiDwAAAIBG44QTTsjrr7+ecePGZeHChendu3emT5+e0tLSJMn8+fPrPOE3ZsyYFBUVZcyYMXnttdey4447ZvDgwbn88ssLdQsAAOuk+AMAAACgURk1alRGjRq11mMzZsyos9+0adOMHz8+48eP3wLJAAA2jnf8AQAAAAAAQAOg+AMAAAAAAIAGQPEHAAAAAAAADYDiDwAAAAAAABoAxR8AAAAAAAA0AIo/AAAAAAAAaAAUfwAAAAAAANAAKP4AAAAAAACgAVD8AQAAAAAAQAOg+AMAAAAAAIAGQPEHAAAAAAAADYDiDwAAAAAAABoAxR8AAAAAAAA0AIo/AAAAAAAAaAAUfwAAAAAAANAAKP4AAAAAAACgAVD8AQAAAAAAQAOg+AMAAAAAAIAGQPEHAAAAAAAADYDiDwAAAAAAABoAxR8AAAAAAAA0AAUt/srLy/PJT34ybdq0SYcOHTJkyJA8++yzdeYceuihKSoqqrOdddZZBUoMAAAAAAAA9VNBi7/7778/I0eOzCOPPJJ77rknq1evzpFHHpkVK1bUmXfGGWdkwYIFtdt3v/vdAiUGAAAAAACA+qlpIX/49OnT6+zffPPN6dChQ2bNmpVPf/rTteOtWrVKx44dt3Q8AAAAAAAA2GrUq3f8LV26NEmy/fbb1xm/5ZZb0r59++yzzz4ZPXp0Vq5cWYh4AAAAAAAAUG8V9Im//1RdXZ1zzz03/fv3zz777FM7PnTo0HzsYx/LTjvtlL/97W85//zz8+yzz2batGlrvU5lZWUqKytr95ctW7bZswMAAAAAAECh1Zvib+TIkXn66afz4IMP1hk/88wza//cs2fPdOrUKUcccUReeOGF7Lbbbmtcp7y8PJdccslmzwsAAAAAAAD1Sb1Y6nPUqFG58847c99992XnnXf+wLl9+/ZNkjz//PNrPT569OgsXbq0dnv11Vc3eV4AAAAAAACobwr6xF9NTU2+9rWv5fbbb8+MGTPSrVu3Dz1nzpw5SZJOnTqt9Xjz5s3TvHnzTRkTAAAAAAAA6r2CFn8jR47M1KlT8/vf/z5t2rTJwoULkyTt2rVLy5Yt88ILL2Tq1KkZNGhQdthhh/ztb3/Leeedl09/+tPZd999CxkdAAAAAAAA6pWCFn/XXXddkuTQQw+tM37TTTfl1FNPTUlJSf785z9n4sSJWbFiRbp06ZLjjz8+Y8aMKUBaAAAAAAAAqL8KvtTnB+nSpUvuv//+LZQGAAAAAAAAtl7FhQ4AAAAAAAAAbDzFHwAAAAAAADQAij8AAAAAAABoABR/AAAAAAAA0AAo/gAA6pFJkyala9euadGiRfr27ZvHHntsnXNXr16dSy+9NLvttltatGiRXr16Zfr06Rt1TQAAAAC2Xoo/AIB64rbbbktZWVnGjx+f2bNnp1evXhk4cGAWL1681vljxozJ9ddfn2uvvTbPPPNMzjrrrBx77LF58sknN/iaAAAAAGy9FH8AAPXEhAkTcsYZZ2TEiBHp0aNHJk+enFatWmXKlClrnf+LX/wiF154YQYNGpRdd901Z599dgYNGpSrr756g68JAAAAwNZL8QcAUA+sWrUqs2bNyoABA2rHiouLM2DAgMycOXOt51RWVqZFixZ1xlq2bJkHH3xwg68JAAAAwNZL8QcAUA8sWbIkVVVVKS0trTNeWlqahQsXrvWcgQMHZsKECXnuuedSXV2de+65J9OmTcuCBQs2+JrJe4XismXL6mwAAAAA1H+KPwCArdQ111yTPfbYI927d09JSUlGjRqVESNGpLh44z7ilZeXp127drVbly5dNlFiAAAAADYnxR8AQD3Qvn37NGnSJIsWLaozvmjRonTs2HGt5+y444654447smLFirzyyiuZN29eWrdunV133XWDr5kko0ePztKlS2u3V199dSPvDgAAAIAtQfEHAFAPlJSUpE+fPqmoqKgdq66uTkVFRfr16/eB57Zo0SKdO3fOu+++m9/97nc55phjNuqazZs3T9u2betsAAAAANR/TQsdAACA95SVlWX48OHZf//9c8ABB2TixIlZsWJFRowYkSQZNmxYOnfunPLy8iTJo48+mtdeey29e/fOa6+9losvvjjV1dX59re//ZGvCQAAAEDDofgDAKgnTjjhhLz++usZN25cFi5cmN69e2f69OkpLS1NksyfP7/O+/veeeedjBkzJi+++GJat26dQYMG5Re/+EW23Xbbj3xNAAAAABoOxR8AwEZ67rnnct9992Xx4sWprq6uc2zcuHHrda1Ro0Zl1KhRaz02Y8aMOvuHHHJInnnmmY26JgAAAAANh+IPAGAj3HDDDTn77LPTvn37dOzYMUVFRbXHioqK1rv4AwAAAIANpfgDANgI3/nOd3L55Zfn/PPPL3QUAAAAABq54g+fAgDAuvz73//OF7/4xULHAAAAAADFHwDAxvjiF7+Y//3f/y10DAAAAACw1CcAwMbYfffdM3bs2DzyyCPp2bNnmjVrVuf417/+9QIlAwAAAKCxUfwBAGyEn/zkJ2ndunXuv//+3H///XWOFRUVKf4AAAAA2GIUfwAAG+Gll14qdASAdVq5cmXmzZu3ya43d+7cOv+7KXTv3j2tWrXaZNcDAABozBR/AACbSE1NTZL3nvQDqA/mzZuXPn36bPLrfulLX9pk15o1a1Y+8YlPbLLrAQAANGaKPwCAjfTzn/883/ve9/Lcc88lST7+8Y/nW9/6Vk455ZQCJwMau+7du2fWrFmb7Hpvv/12Xn755XTt2jUtW7bcJNfs3r37JrkOAAAAij8AgI0yYcKEjB07NqNGjUr//v2TJA8++GDOOuusLFmyJOedd16BEwKNWatWrTb503Tv//86AAAA6h/FHwDARrj22mtz3XXXZdiwYbVjRx99dPbee+9cfPHFij8AAAAAtpjiQgcAANiaLViwIAceeOAa4wceeGAWLFhQgEQAAAAANFaKPwCAjbD77rvn17/+9Rrjt912W/bYY48CJAIAAACgsbLUJwDARrjkkktywgkn5IEHHqh979VDDz2UioqKtRaCAAAAALC5eOIPAGAjHH/88Xn00UfTvn373HHHHbnjjjvSvn37PPbYYzn22GMLHQ8AAACARsQTfwAAG6lPnz755S9/WegYAAAAADRyij8AgPW0bNmytG3btvbPH+T9eQAAAACwuSn+AADW03bbbZcFCxakQ4cO2XbbbVNUVLTGnJqamhQVFaWqqqoACQEAAABojBR/AADr6d57783222+fJLnvvvsKnAYAAAAA3qP4AwBYT4ccckjtn7t165YuXbqs8dRfTU1NXn311S0dDQAAAIBGrLjQAQAAtmbdunXL66+/vsb4G2+8kW7duhUgEQAAAACNleIPAGAjvP8uv/+2fPnytGjRogCJAAAAAGisLPUJALABysrKkiRFRUUZO3ZsWrVqVXusqqoqjz76aHr37l2gdAAAAAA0Roo/AIAN8OSTTyZ574m/p556KiUlJbXHSkpK0qtXr3zzm98sVDwAAAAAGiHFHwDABrjvvvuSJCNGjMg111yTtm3bFjgRAAAAAI2d4g8AYCPcdNNNhY4AAAAAAEkUfwAAG+Xwww//wOP33nvvFkoCAAAAQGOn+AMA2Ai9evWqs7969erMmTMnTz/9dIYPH16gVAAAAAA0Roo/AICN8IMf/GCt4xdffHGWL1++hdMAAAAA0JgVFzoAAEBD9KUvfSlTpkwpdAwAAAAAGhHFHwDAZjBz5sy0aNGi0DEAAAAAaEQs9QkAsBGOO+64Ovs1NTVZsGBBnnjiiYwdO7ZAqQAAAABojBR/AAAboV27dnX2i4uLs+eee+bSSy/NkUceWaBUAAAAADRGij8AgI1w0003FToCAAAAACRR/AEAbBJPPPFE5s6dmyTp0aNH+vTpU+BEAAAAADQ2ij8AgI3wj3/8IyeddFIeeuihbLvttkmSN998MwceeGBuvfXW7LzzzoUNCAAAAECjUVzoAAAAW7PTTz89q1evzty5c/PGG2/kjTfeyNy5c1NdXZ3TTz+90PEAAAAAaEQ88QcAsBHuv//+PPzww9lzzz1rx/bcc89ce+21OfjggwuYDAAAAIDGxhN/AAAboUuXLlm9evUa41VVVdlpp50KkAgAAACAxkrxBwCwEb73ve/la1/7Wp544onasSeeeCLnnHNOvv/97xcwGQAAAACNjaU+AQDW03bbbZeioqLa/RUrVqRv375p2vS9j1bvvvtumjZtmtNOOy1DhgwpUEoAAAAAGhvFHwDAepo4cWKhIwAAAADAGhR/AADrafjw4YWOAADARpg0aVK+973vZeHChenVq1euvfbaHHDAAeuc/+abb+aiiy7KtGnT8sYbb+RjH/tYJk6cmEGDBm3B1AAAH07xBwCwnpYtW5a2bdvW/vmDvD8PAID64bbbbktZWVkmT56cvn37ZuLEiRk4cGCeffbZdOjQYY35q1atymc+85l06NAhv/3tb9O5c+e88sor2Xbbbbd8eACAD6H4AwBYT9ttt10WLFiQDh06ZNttt63zvr/31dTUpKioKFVVVQVICADAukyYMCFnnHFGRowYkSSZPHly7rrrrkyZMiUXXHDBGvOnTJmSN954Iw8//HCaNWuWJOnateuWjAwA8JEp/gAA1tO9996b7bffPkly3333FTgNAAAf1apVqzJr1qyMHj26dqy4uDgDBgzIzJkz13rOH/7wh/Tr1y8jR47M73//++y4444ZOnRozj///DRp0mRLRQcA+EgUfwAA6+mQQw5Jkrz77ru5//77c9ppp2XnnXcucCoAAD7MkiVLUlVVldLS0jrjpaWlmTdv3lrPefHFF3Pvvffm5JNPzh//+Mc8//zz+epXv5rVq1dn/Pjxa8yvrKxMZWVl7f6HLQ0PALApFRc6AADA1qpp06b53ve+l3fffbfQUQAA2Eyqq6vToUOH/OQnP0mfPn1ywgkn5KKLLsrkyZPXOr+8vDzt2rWr3bp06bKFEwMAjVlBi7/y8vJ88pOfTJs2bdKhQ4cMGTIkzz77bJ0577zzTkaOHJkddtghrVu3zvHHH59FixYVKDEAQF2HH3547r///kLHAADgI2jfvn2aNGmyxu+WFi1alI4dO671nE6dOuXjH/94nWU999prryxcuDCrVq1aY/7o0aOzdOnS2u3VV1/dtDcBAPABCrrU5/3335+RI0fmk5/8ZN59991ceOGFOfLII/PMM89km222SZKcd955ueuuu/Kb3/wm7dq1y6hRo3LcccfloYceKmR0AIAkyVFHHZULLrggTz31VPr06VP7GeZ9Rx99dIGSAQDw30pKStKnT59UVFRkyJAhSd57oq+ioiKjRo1a6zn9+/fP1KlTU11dneLi975D//e//z2dOnVKSUnJGvObN2+e5s2bb7Z7AAD4IAUt/qZPn15n/+abb06HDh0ya9asfPrTn87SpUtz4403ZurUqTn88MOTJDfddFP22muvPPLII/nUpz5ViNgAALW++tWvJkkmTJiwxrGioqJUVVVt6UgAAHyAsrKyDB8+PPvvv38OOOCATJw4MStWrMiIESOSJMOGDUvnzp1TXl6eJDn77LPzox/9KOecc06+9rWv5bnnnssVV1yRr3/964W8DQCAtSpo8fffli5dmiTZfvvtkySzZs3K6tWrM2DAgNo53bt3zy677JKZM2cq/gCAgquuri50BAAA1sMJJ5yQ119/PePGjcvChQvTu3fvTJ8+PaWlpUmS+fPn1z7ZlyRdunTJ3XffnfPOOy/77rtvOnfunHPOOSfnn39+oW4BAGCd6k3xV11dnXPPPTf9+/fPPvvskyRZuHBhSkpKsu2229aZW1pamoULF671OpWVlamsrKzdX7Zs2WbLDADw85//PCeccMIayzmtWrUqt956a4YNG1agZAAArMuoUaPWubTnjBkz1hjr169fHnnkkc2cCgBg4xV/+JQtY+TIkXn66adz6623btR1ysvL065du9qtS5cumyghAMCaRowYUbtqwX966623apeLAgAAAIAtoV4Uf6NGjcqdd96Z++67LzvvvHPteMeOHbNq1aq8+eabdeYvWrQoHTt2XOu1Ro8enaVLl9Zur7766uaMDgA0cjU1NSkqKlpj/B//+EfatWtXgEQAAAAANFYFXeqzpqYmX/va13L77bdnxowZ6datW53jffr0SbNmzVJRUZHjjz8+SfLss89m/vz56dev31qv2bx58zWW2gIA2NT222+/FBUVpaioKEcccUSaNv2/j1VVVVV56aWX8tnPfraACQEAAABobApa/I0cOTJTp07N73//+7Rp06b2vX3t2rVLy5Yt065du3z5y19OWVlZtt9++7Rt2zZf+9rX0q9fv3zqU58qZHQAoJEbMmRIkmTOnDkZOHBgWrduXXuspKQkXbt2rf3iEgAAAABsCQUt/q677rokyaGHHlpn/Kabbsqpp56aJPnBD36Q4uLiHH/88amsrMzAgQPz4x//eAsnBQCoa/z48UmSrl275sQTT7TiANDgVVVV5S9/+UsWLFiQTp065eCDD06TJk0KHQsAAID/UPClPj9MixYtMmnSpEyaNGkLJAIAWD+HH354Xn/99dr3FD/22GOZOnVqevTokTPPPLPA6QA2jWnTpuUb3/hGXn755dqxrl275uqrr85xxx1XuGAAAADUUVzoAAAAW7OhQ4fmvvvuS5IsXLgwAwYMyGOPPZaLLrool1566Xpfb9KkSenatWtatGiRvn375rHHHvvA+RMnTsyee+6Zli1bpkuXLjnvvPPyzjvv1B6vqqrK2LFj061bt7Rs2TK77bZbLrvsso/0BSyA5L3S7wtf+EJ69uyZmTNn5q233srMmTPTs2fPfOELX8i0adMKHREAAID/P8UfAMBGePrpp3PAAQckSX7961+nZ8+eefjhh3PLLbfk5ptvXq9r3XbbbSkrK8v48eMze/bs9OrVKwMHDszixYvXOn/q1Km54IILMn78+MydOzc33nhjbrvttlx44YW1c6666qpcd911+dGPfpS5c+fmqquuyne/+91ce+21G3zPQONRVVWVb3zjG/n85z+fO+64I5/61KfSunXrfOpTn8odd9yRz3/+8/nmN7+ZqqqqQkcFAAAgG7jU54svvphdd911U2cBADaDE38ys9ARNtitZ/YrdIQPtXr16tr3+/35z3/O0UcfnSTp3r17FixYsF7XmjBhQs4444yMGDEiSTJ58uTcddddmTJlSi644II15j/88MPp379/hg4dmuS9ZfdOOumkPProo3XmHHPMMfnc5z5XO+dXv/rVhz5JCJAkf/nLX/Lyyy/nV7/6VYqL635vtLi4OKNHj86BBx6Yv/zlL2u8ux0AAIAtb4Oe+Nt9991z2GGH5Ze//GWdpaQAABqbvffeO5MnT85f/vKX3HPPPfnsZz+bJPnnP/+ZHXbY4SNfZ9WqVZk1a1YGDBhQO1ZcXJwBAwZk5sy1l7cHHnhgZs2aVVvivfjii/njH/+YQYMG1ZlTUVGRv//970mSv/71r3nwwQdz1FFHrfe9Ao3P+19g2GeffdZ6/P3x9f2iAwAAAJvHBhV/s2fPzr777puysrJ07NgxX/nKV3xrHABolK666qpcf/31OfTQQ3PSSSelV69eSZI//OEPtUuAfhRLlixJVVVVSktL64yXlpZm4cKFaz1n6NChufTSS3PQQQelWbNm2W233XLooYfWWerzggsuyIknnpju3bunWbNm2W+//XLuuefm5JNPXmeWysrKLFu2rM4GNE6dOnVK8t6yxmvz/vj78wAAACisDSr+evfunWuuuSb//Oc/M2XKlCxYsCAHHXRQ9tlnn0yYMCGvv/76ps4JAFAvHXrooVmyZEmWLFmSKVOm1I6feeaZmTx58mb92TNmzMgVV1yRH//4x5k9e3amTZuWu+66K5dddlntnF//+te55ZZbMnXq1MyePTs/+9nP8v3vfz8/+9nP1nnd8vLytGvXrnbr0qXLZr0PoP46+OCD07Vr11xxxRWprq6uc6y6ujrl5eXp1q1bDj744AIlBAAA4D9tUPH3vqZNm+a4447Lb37zm1x11VV5/vnn881vfjNdunTJsGHDLPcCADQKTZo0yXbbbVdnrGvXrunQocNHvkb79u3TpEmTLFq0qM74okWL0rFjx7WeM3bs2Jxyyik5/fTT07Nnzxx77LG54oorUl5eXvsL+m9961u1T/317Nkzp5xySs4777yUl5evM8vo0aOzdOnS2u3VV1/9yPcBNCxNmjTJ1VdfnTvvvDNDhgzJzJkz89Zbb2XmzJkZMmRI7rzzznz/+99PkyZNCh0VAACAJE035uQnnngiU6ZMya233pptttkm3/zmN/PlL385//jHP3LJJZfkmGOOsQQoANDgfOITn0hFRUW222677LfffikqKlrn3NmzZ3+ka5aUlKRPnz6pqKjIkCFDkrz3NE1FRUVGjRq11nNWrlyZ4uK63+N6/5fvNTU1Hzjnv5/c+U/NmzdP8+bNP1JuoOE77rjj8tvf/jbf+MY3cuCBB9aOd+vWLb/97W9z3HHHFTAdAAAA/2mDir8JEybkpptuyrPPPptBgwbl5z//eQYNGlT7S6Vu3brl5ptvTteuXTdlVgCAeuGYY46pLcbeL+k2hbKysgwfPjz7779/DjjggEycODErVqzIiBEjkiTDhg1L586da5/WGzx4cCZMmJD99tsvffv2zfPPP5+xY8dm8ODBtQXg4MGDc/nll2eXXXbJ3nvvnSeffDITJkzIaaedtslyAw3fcccdl2OOOSZ/+ctfsmDBgnTq1CkHH3ywJ/0AAADqmQ0q/q677rqcdtppOfXUU9f5EvcOHTrkxhtv3KhwAAD10fjx49f65411wgkn5PXXX8+4ceOycOHC9O7dO9OnT09paWmSZP78+XWe3hszZkyKiooyZsyYvPbaa9lxxx1ri773XXvttRk7dmy++tWvZvHixdlpp53yla98JePGjdtkuYHGoUmTJjn00EMLHQMAAIAPUFTz/jpQ6+Hll1/OLrvsssayUTU1NXn11Vezyy67bLKAG2vZsmVp165dli5dmrZt2xY6DgBboRN/MrPQERqtW8/st1muuyk/H9TU1GTWrFl5+eWXU1RUlG7dun3o8p9bG5+nAID/5vPBR+fvCgC2vJWr3k2PcXcnSZ65dGBalWzUm+82i831GWGD7nS33XbLggUL0qFDhzrjb7zxRrp165aqqqpNEg4AoD6777778uUvfzmvvPJK7Tv13i//pkyZkk9/+tMFTggAAABAY1L84VPWtK6HBJcvX54WLVpsVCAAgK3B888/n89//vPp2rVrpk2blrlz5+aZZ57Jb37zm+y8884ZNGhQXnzxxULHBAAAAKARWa8n/srKypK89032cePGpVWrVrXHqqqq8uijj6Z3796bNCAAQH00ceLEfOpTn0pFRUWd8e7du+fYY4/NgAED8oMf/CDXXnttgRICAAAA0Nis1xN/Tz75ZJ588snU1NTkqaeeqt1/8sknM2/evPTq1Ss333zzZooKAFB/zJgxI+eee+5ajxUVFeXcc8/Nfffdt2VDAQA0MMOGDctbb71Vu//Xv/41q1evLmAiAID6bb2e+Hv/l1cjRozINddc44XEAECjNX/+/PTs2XOdx/fZZ5+88sorWzARAEDDc8stt+T73/9+2rRpkyQ5+OCDM2fOnOy6664FTgYAUD+tV/H3vptuumlT5wAA2KosX768zrLn/61Vq1ZZuXLlFkwEANDw1NTUfOA+AAB1feTi77jjjsvNN9+ctm3b5rjjjvvAudOmTdvoYAAA9d0zzzyThQsXrvXYkiVLtnAaAAAAABq7j1z8tWvXLkVFRbV/BgBo7I444oi1fuu8qKgoNTU1tZ+dAADYcP/5ZauamprMmzcvy5cvrzNn3333LUQ0AIB65yMXf/+5vKelPgGAxu6ll14qdAQAgEbhv79s9fnPfz5J3S9bVVVVFSoeAFBANTU1eXv1mp8DVq5qvJ8NNugdf2+//XZqampq32vzyiuv5Pbbb0+PHj1y5JFHbtKAAAD10cc+9rFCRwAAaPB82QoA+CBvr65Kj3F3FzpGvbJBxd8xxxyT4447LmeddVbefPPNHHDAASkpKcmSJUsyYcKEnH322Zs6JwAAAACNjC9bAQCsnw0q/mbPnp0f/OAHSZLf/va36dixY5588sn87ne/y7hx4xR/AAAAAGwyzz33XH7/+9/n5ZdfTlFRUbp165YhQ4Zk1113LXQ0AKCeeGLMgLQqabLGeMtma441ZBtU/K1cuTJt2rRJkvzv//5vjjvuuBQXF+dTn/pUXnnllU0aEAAAAIDGq7y8POPGjUt1dXU6dOiQmpqavP7667ngggtyxRVX5Jvf/GahIwIA9UCrkiZpVbJBtVeDUrwhJ+2+++6544478uqrr+buu++ufa/f4sWL07Zt200aEAAAAIDG6b777suYMWNy0UUXZcmSJVmwYEEWLlxYW/xdcMEFeeCBBwodEwCg3tig4m/cuHH55je/ma5du6Zv377p169fkvee/ttvv/02aUAAgPru3XffzZ///Odcf/31eeutt5Ik//znP7N8+fICJwMA2LpNnjw5p59+ei6++OJst912tePbb799Lr300px22mm57rrrCpgQAKB+2aBnHr/whS/koIMOyoIFC9KrV6/a8SOOOCLHHnvsJgsHAFDfvfLKK/nsZz+b+fPnp7KyMp/5zGfSpk2bXHXVVamsrMzkyZMLHREAYKv12GOP5Re/+MU6j59yyikZNmzYFkwEAFC/bdATf0nSsWPH7Lfffiku/r9LHHDAAenevfsmCQYAsDU455xzsv/+++ff//53WrZsWTt+7LHHpqKiooDJAAC2fosWLUrXrl3Xebxbt25ZuHDhlgsEAFDPbdATfytWrMiVV16ZioqKLF68ONXV1XWOv/jii5skHABAffeXv/wlDz/8cEpKSuqMd+3aNa+99lqBUgEANAzvvPPOGp+z/lOzZs2yatWqLZgIAKB+26Di7/TTT8/999+fU045JZ06dUpRUdGmzgUAsFWorq5OVVXVGuP/+Mc/0qZNmwIkAgBoWH7605+mdevWaz32/vuVAQB4zwYVf3/6059y1113pX///ps6DwDAVuXII4/MxIkT85Of/CRJUlRUlOXLl2f8+PEZNGhQgdMBAGzddtlll9xwww0fOgcAgPdsUPG33XbbZfvtt9/UWQAAtjpXX311Bg4cmB49euSdd97J0KFD89xzz6V9+/b51a9+Veh4AABbtZdffrnQEQAAtirFG3LSZZddlnHjxmXlypWbOg8AwFZl5513zl//+tdcdNFFOe+887LffvvlyiuvzJNPPpkOHToUOh4AwFbt3nvvTY8ePbJs2bI1ji1dujR77713/vKXvxQgGQBA/bRBT/xdffXVeeGFF1JaWpquXbumWbNmdY7Pnj17k4QDANgaNG3aNCeffHJOPvnkQkcBAGhQJk6cmDPOOCNt27Zd41i7du3yla98JRMmTMjBBx9cgHQAAPXPBhV/Q4YM2cQxAAC2TuXl5SktLc1pp51WZ3zKlCl5/fXXc/755xcoGQDA1u+vf/1rrrrqqnUeP/LII/P9739/CyYCAKjfNqj4Gz9+/KbOAQCwVbr++uszderUNcb33nvvnHjiiYo/AICNsGjRojVWmvpPTZs2zeuvv74FEwEA1G8b9I6/JHnzzTfz05/+NKNHj84bb7yR5L0lPl977bVNFg4AoL5buHBhOnXqtMb4jjvumAULFhQgEQBAw9G5c+c8/fTT6zz+t7/9ba2fxQAAGqsNKv7+9re/5eMf/3iuuuqqfP/738+bb76ZJJk2bVpGjx69KfMBANRrXbp0yUMPPbTG+EMPPZSddtqpAIkAABqOQYMGZezYsXnnnXfWOPb2229n/Pjx+fznP1+AZAAA9dMGLfVZVlaWU089Nd/97nfTpk2b2vFBgwZl6NChmywcAEB9d8YZZ+Tcc8/N6tWrc/jhhydJKioq8u1vfzvf+MY3CpwOAGDrNmbMmEybNi0f//jHM2rUqOy5555Jknnz5mXSpEmpqqrKRRddVOCUAAD1xwYVf48//niuv/76NcY7d+6chQsXbnQoAICtxbe+9a3861//yle/+tWsWrUqSdKiRYucf/75VkIAANhIpaWlefjhh3P22Wdn9OjRqampSZIUFRVl4MCBmTRpUkpLSwucEgCg/tig4q958+ZZtmzZGuN///vfs+OOO250KACArUVRUVGuuuqqjB07NnPnzk3Lli2zxx57pHnz5oWOBgDQIHzsYx/LH//4x/z73//O888/n5qamuyxxx7ZbrvtCh0NAKDe2aDi7+ijj86ll16aX//610ne+4XX/Pnzc/755+f444/fpAEBALYGrVu3zic/+clCxwAAaLC22247n7cAAD7EBhV/V199db7whS9kxx13zNtvv51DDjkkCxcuTL9+/XL55Zdv6owAAPXWihUrcuWVV6aioiKLFy9OdXV1neMvvvhigZIBAAAA0NhsUPHXrl273HPPPXnooYfy17/+NcuXL88nPvGJDBgwYFPnAwCo104//fTcf//9OeWUU9KpU6cUFRUVOhIAAAAAjdR6F3/V1dW5+eabM23atLz88sspKipKt27d0rFjx9TU1PhlFwDQqPzpT3/KXXfdlf79+xc6CgAAAACNXPH6TK6pqcnRRx+d008/Pa+99lp69uyZvffeO6+88kpOPfXUHHvssZsrJwBAvbTddttl++23L3QMAAAAAFi/J/5uvvnmPPDAA6moqMhhhx1W59i9996bIUOG5Oc//3mGDRu2SUMCANRXl112WcaNG5ef/exnadWqVaHjAAAAANCIrVfx96tf/SoXXnjhGqVfkhx++OG54IILcssttyj+AIBG4+qrr84LL7yQ0tLSdO3aNc2aNatzfPbs2QVKBgAAAEBjs17F39/+9rd897vfXefxo446Kj/84Q83OhQAwNZiyJAhhY4AAAAAAEnWs/h74403Ulpaus7jpaWl+fe//73RoQAAthbjx48vdAQAAAAASJIUr8/kqqqqNG267q6wSZMmeffddzc6FAAAAAAAALB+1uuJv5qampx66qlp3rz5Wo9XVlZuklAAAFuLqqqq/OAHP8ivf/3rzJ8/P6tWrapz/I033ihQMgAAAAAam/V64m/48OHp0KFD2rVrt9atQ4cOGTZs2ObKCgBQ71xyySWZMGFCTjjhhCxdujRlZWU57rjjUlxcnIsvvrjQ8QAAAABoRNbrib+bbrppc+UAANgq3XLLLbnhhhvyuc99LhdffHFOOumk7Lbbbtl3333zyCOP5Otf/3qhIwIAAADQSKzXE38AANS1cOHC9OzZM0nSunXrLF26NEny+c9/PnfddVchowEAAADQyCj+AAA2ws4775wFCxYkSXbbbbf87//+b5Lk8ccfX+d7kQEAAABgc1D8AQBshGOPPTYVFRVJkq997WsZO3Zs9thjjwwbNiynnXZagdMBAAAA0Jis1zv+AACo68orr6z98wknnJBddtklM2fOzB577JHBgwcXMBkAAAAAjY3iDwBgE+rXr1/69etX6BgAAAAANEKKPwCAjfTPf/4zDz74YBYvXpzq6uo6x77+9a8XKBUAAAAAjY3iDwBgI9x88835yle+kpKSkuywww4pKiqqPVZUVKT4AwAAAGCLUfwBAGyEsWPHZty4cRk9enSKi4sLHQcAAACARsxvpwAANsLKlStz4oknKv0AAAAAKLiC/obqgQceyODBg7PTTjulqKgod9xxR53jp556aoqKiupsn/3sZwsTFgBgLb785S/nN7/5TaFjAAAAAEBhl/pcsWJFevXqldNOOy3HHXfcWud89rOfzU033VS737x58y0VDwDgQ5WXl+fzn/98pk+fnp49e6ZZs2Z1jk+YMKFAyQAAAABobApa/B111FE56qijPnBO8+bN07Fjxy2UCABg/ZSXl+fuu+/OnnvumSQpKiqqPfaffwYAAACAza2gxd9HMWPGjHTo0CHbbbddDj/88HznO9/JDjvssM75lZWVqaysrN1ftmzZlogJADRSV199daZMmZJTTz210FEAAAAAaOQK+o6/D/PZz342P//5z1NRUZGrrroq999/f4466qhUVVWt85zy8vK0a9euduvSpcsWTAwANDbNmzdP//79Cx0DAAAAAOp38XfiiSfm6KOPTs+ePTNkyJDceeedefzxxzNjxox1njN69OgsXbq0dnv11Ve3XGAAoNE555xzcu211xY6BgAAAADU/6U+/9Ouu+6a9u3b5/nnn88RRxyx1jnNmzdP8+bNt3AyAKCxeuyxx3LvvffmzjvvzN57751mzZrVOT5t2rQCJQMAAACgsanXT/z9t3/84x/517/+lU6dOhU6CgBAkmTbbbfNcccdl0MOOSTt27evs+R4u3bt1vt6kyZNSteuXdOiRYv07ds3jz322AfOnzhxYvbcc8+0bNkyXbp0yXnnnZd33nmnzpzXXnstX/rSl7LDDjukZcuW6dmzZ5544on1zgYAAABA/VbQJ/6WL1+e559/vnb/pZdeypw5c7L99ttn++23zyWXXJLjjz8+HTt2zAsvvJBvf/vb2X333TNw4MACpgYAeM+7776bww47LEceeWQ6duy40de77bbbUlZWlsmTJ6dv376ZOHFiBg4cmGeffTYdOnRYY/7UqVNzwQUXZMqUKTnwwAPz97//PaeeemqKiooyYcKEJMm///3v9O/fP4cddlj+9Kc/Zccdd8xzzz2X7bbbbqPzAgAAAFC/FLT4e+KJJ3LYYYfV7peVlSVJhg8fnuuuuy5/+9vf8rOf/Sxvvvlmdtpppxx55JG57LLLLOUJANQLTZs2zVlnnZW5c+dukutNmDAhZ5xxRkaMGJEkmTx5cu66665MmTIlF1xwwRrzH3744fTv3z9Dhw5NknTt2jUnnXRSHn300do5V111Vbp06ZKbbrqpdqxbt26bJC8AAAAA9UtBl/o89NBDU1NTs8Z28803p2XLlrn77ruzePHirFq1Ki+//HJ+8pOfpLS0tJCRAQDqOOCAA/Lkk09u9HVWrVqVWbNmZcCAAbVjxcXFGTBgQGbOnLnWcw488MDMmjWrdjnQF198MX/84x8zaNCg2jl/+MMfsv/+++eLX/xiOnTokP322y833HDDB2aprKzMsmXL6mwAAAAA1H8FfeIPAGBr99WvfjXf+MY38o9//CN9+vTJNttsU+f4vvvu+5Gus2TJklRVVa3xJafS0tLMmzdvrecMHTo0S5YsyUEHHZSampq8++67Oeuss3LhhRfWznnxxRdz3XXXpaysLBdeeGEef/zxfP3rX09JSUmGDx++1uuWl5fnkksu+Ui5AQAAAKg/FH8AABvhxBNPTJJ8/etfrx0rKipKTU1NioqKUlVVtdl+9owZM3LFFVfkxz/+cfr27Zvnn38+55xzTi677LKMHTs2SVJdXZ39998/V1xxRZJkv/32y9NPP53Jkyevs/gbPXp07RLsSbJs2bJ06dJls90HAAAAAJuG4g8AYCO89NJLm+Q67du3T5MmTbJo0aI644sWLUrHjh3Xes7YsWNzyimn5PTTT0+S9OzZMytWrMiZZ56Ziy66KMXFxenUqVN69OhR57y99torv/vd79aZpXnz5t6pDAAAALAVUvwBAGyEj33sY5vkOiUlJenTp08qKioyZMiQJO89rVdRUZFRo0at9ZyVK1emuLjuK5ubNGmSJKmpqUmS9O/fP88++2ydOX//+983WW4AAAAA6g/FHwDARnrhhRcyceLEzJ07N0nSo0ePnHPOOdltt93W6zplZWUZPnx49t9//xxwwAGZOHFiVqxYkREjRiRJhg0bls6dO6e8vDxJMnjw4EyYMCH77bdf7VKfY8eOzeDBg2sLwPPOOy8HHnhgrrjiivy///f/8thjj+UnP/lJfvKTn2zCvwEAAAAA6gPFHwDARrj77rtz9NFHp3fv3unfv3+S5KGHHsree++d//mf/8lnPvOZj3ytE044Ia+//nrGjRuXhQsXpnfv3pk+fXpKS0uTJPPnz6/zhN+YMWNSVFSUMWPG5LXXXsuOO+6YwYMH5/LLL6+d88lPfjK33357Ro8enUsvvTTdunXLxIkTc/LJJ2+ivwEAAAAA6ouimvfXgWqgli1blnbt2mXp0qVp27ZtoeMAsBU68SczCx2h0br1zH6b5bqb8vPBfvvtl4EDB+bKK6+sM37BBRfkf//3fzN79uyNun594PMUAPDffD746PxdAcDms3LVu+kx7u4kyTOXDkyrkq3nebfN9Rmh+MOnAACwLnPnzs2Xv/zlNcZPO+20PPPMMwVIBAAAAEBjpfgDANgIO+64Y+bMmbPG+Jw5c9KhQ4ctHwgAAACARmvreeYRAKAeOuOMM3LmmWfmxRdfzIEHHpjkvXf8XXXVVSkrKytwOgAAAAAaE0/8AQBshLFjx2bcuHG59tprc8ghh+SQQw7Jj370o1x88cUZM2ZMoeMBALAWkyZNSteuXdOiRYv07ds3jz322Ec679Zbb01RUVGGDBmyeQMCAGwgxR8AwHr6wx/+kNWrVydJioqKct555+Uf//hHli5dmqVLl+Yf//hHzjnnnBQVFRU4KQAA/+22225LWVlZxo8fn9mzZ6dXr14ZOHBgFi9e/IHnvfzyy/nmN7+Zgw8+eAslBQBYf4o/AID1dOyxx+bNN99MkjRp0qT2l0Rt2rRJmzZtCpgMAIAPM2HChJxxxhkZMWJEevTokcmTJ6dVq1aZMmXKOs+pqqrKySefnEsuuSS77rrrFkwLALB+FH8AAOtpxx13zCOPPJIkqamp8WQfAMBWYtWqVZk1a1YGDBhQO1ZcXJwBAwZk5syZ6zzv0ksvTYcOHfLlL3/5Q39GZWVlli1bVmcDANhSmhY6AADA1uass87KMccck6KiohQVFaVjx47rnFtVVbUFkwEA8EGWLFmSqqqqlJaW1hkvLS3NvHnz1nrOgw8+mBtvvDFz5sz5SD+jvLw8l1xyycZGBQDYIIo/AID1dPHFF+fEE0/M888/n6OPPjo33XRTtt1220LHAgBgE3vrrbdyyimn5IYbbkj79u0/0jmjR49OWVlZ7f6yZcvSpUuXzRURAKAOxR8AwAbo3r179txzzwwfPjzHH398WrduXehIAAB8iPbt26dJkyZZtGhRnfFFixatdRWHF154IS+//HIGDx5cO1ZdXZ0kadq0aZ599tnstttudc5p3rx5mjdvvhnSAwB8OO/4AwDYQDU1NbnllluyYMGCQkcBAOAjKCkpSZ8+fVJRUVE7Vl1dnYqKivTr12+N+d27d89TTz2VOXPm1G5HH310DjvssMyZM8eTfABAveOJPwCADVRcXJw99tgj//rXv7LHHnsUOg4AAB9BWVlZhg8fnv333z8HHHBAJk6cmBUrVmTEiBFJkmHDhqVz584pLy9PixYtss8++9Q5//0l3v97HACgPlD8AQBshCuvvDLf+ta3ct111/nlDwDAVuCEE07I66+/nnHjxmXhwoXp3bt3pk+fntLS0iTJ/PnzU1xskSwAYOuk+AMA2AjDhg3LypUr06tXr5SUlKRly5Z1jr/xxhsFSgYAwLqMGjUqo0aNWuuxGTNmfOC5N99886YPBACwiSj+AAA2wsSJEwsdAQAAAACSKP4AADbK8OHDCx0BAAAAAJIkFiwHANhIL7zwQsaMGZOTTjopixcvTpL86U9/yv/3//1/BU4GAAAAQGOi+AMA2Aj3339/evbsmUcffTTTpk3L8uXLkyR//etfM378+AKnAwAAAKAxUfwBAGyECy64IN/5zndyzz33pKSkpHb88MMPzyOPPFLAZAAAAAA0Noo/AICN8NRTT+XYY49dY7xDhw5ZsmRJARIBAAAA0Fgp/gAANsK2226bBQsWrDH+5JNPpnPnzgVIBAAAAEBjpfgDANgIJ554Ys4///wsXLgwRUVFqa6uzkMPPZRvfvObGTZsWKHjAQAAANCIKP4AADbCFVdcke7du6dLly5Zvnx5evTokU9/+tM58MADM2bMmELHAwAAAKARaVroAAAAW7OSkpLccMMNGTduXJ566qksX748++23X/bYY49CRwMAAACgkVH8AQBsgOrq6nzve9/LH/7wh6xatSpHHHFExo8fn5YtWxY6GgAAAACNlKU+AQA2wOWXX54LL7wwrVu3TufOnXPNNddk5MiRhY4FAAAAQCOm+AMA2AA///nP8+Mf/zh333137rjjjvzP//xPbrnlllRXVxc6GgAAAACNlOIPAGADzJ8/P4MGDardHzBgQIqKivLPf/6zgKkAAAAAaMwUfwAAG+Ddd99NixYt6ow1a9Ysq1evLlAiAAAAABq7poUOAACwNaqpqcmpp56a5s2b14698847Oeuss7LNNtvUjk2bNq0Q8QAAAABohBR/AAAbYPjw4WuMfelLXypAEgAAAAB4j+IPAGAD3HTTTYWOAAAAAAB1eMcfAAAAAAAANACKPwAAAAAAAGgAFH8AAAAAAADQACj+AAAAAAAAoAFQ/AEAAAAAAEADoPgDAAAAAACABkDxBwAAAAAAAA2A4g8AAAAAAAAaAMUfAAAAAAAANACKPwAAAAAAAGgAFH8AAAAAAADQACj+AAAAAAAAoAFQ/AEAAAAAAEADoPgDAAAAAACABkDxBwAAAAAAAA2A4g8AAAAAAAAaAMUfAAAAAAAANACKPwAAAAAAAGgAFH8AAAAAAADQACj+AAAAAAAAoAFQ/AEAAAAAAEADoPgDAAAAAACABkDxBwAAAAAAAA2A4g8AAAAAAAAagIIWfw888EAGDx6cnXbaKUVFRbnjjjvqHK+pqcm4cePSqVOntGzZMgMGDMhzzz1XmLAAAAAAAABQjxW0+FuxYkV69eqVSZMmrfX4d7/73fzwhz/M5MmT8+ijj2abbbbJwIED884772zhpAAAAAAAAFC/NS3kDz/qqKNy1FFHrfVYTU1NJk6cmDFjxuSYY45Jkvz85z9PaWlp7rjjjpx44olbMioAAAAAAADUa/X2HX8vvfRSFi5cmAEDBtSOtWvXLn379s3MmTPXeV5lZWWWLVtWZwMA2FpMmjQpXbt2TYsWLdK3b9889thjHzh/4sSJ2XPPPdOyZct06dIl55133jpXR7jyyitTVFSUc889dzMkBwAAAKDQ6m3xt3DhwiRJaWlpnfHS0tLaY2tTXl6edu3a1W5dunTZrDkBADaV2267LWVlZRk/fnxmz56dXr16ZeDAgVm8ePFa50+dOjUXXHBBxo8fn7lz5+bGG2/MbbfdlgsvvHCNuY8//niuv/767Lvvvpv7NgAAAAAokHpb/G2o0aNHZ+nSpbXbq6++WuhIAAAfyYQJE3LGGWdkxIgR6dGjRyZPnpxWrVplypQpa53/8MMPp3///hk6dGi6du2aI488MieddNIaTwkuX748J598cm644YZst912W+JWAAAAACiAelv8dezYMUmyaNGiOuOLFi2qPbY2zZs3T9u2betsAAD13apVqzJr1qw6y5wXFxdnwIAB61zm/MADD8ysWbNqi74XX3wxf/zjHzNo0KA680aOHJnPfe5zda79QSydDgAAALB1alroAOvSrVu3dOzYMRUVFendu3eSZNmyZXn00Udz9tlnFzYcAMAmtmTJklRVVa11mfN58+at9ZyhQ4dmyZIlOeigg1JTU5N33303Z511Vp2lPm+99dbMnj07jz/++EfOUl5enksuuWTDbgQAAACAgilo8bd8+fI8//zztfsvvfRS5syZk+233z677LJLzj333HznO9/JHnvskW7dumXs2LHZaaedMmTIkMKFBmCDnPiTtT+xBGy4GTNm5IorrsiPf/zj9O3bN88//3zOOeecXHbZZRk7dmxeffXVnHPOObnnnnvSokWLj3zd0aNHp6ysrHZ/2bJl3psMAAAAsBUoaPH3xBNP5LDDDqvdf/8XTMOHD8/NN9+cb3/721mxYkXOPPPMvPnmmznooIMyffr09frFFQDA1qB9+/Zp0qTJei1zPnbs2Jxyyik5/fTTkyQ9e/as/ex00UUXZdasWVm8eHE+8YlP1J5TVVWVBx54ID/60Y9SWVmZJk2arHHd5s2bp3nz5pvw7gAAAADYEgpa/B166KGpqalZ5/GioqJceumlufTSS7dgKgCALa+kpCR9+vRJRUVF7eoG1dXVqaioyKhRo9Z6zsqVK1NcXPeVze8XeTU1NTniiCPy1FNP1Tk+YsSIdO/ePeeff/5aSz8AAAAAtl719h1/AACNTVlZWYYPH579998/BxxwQCZOnJgVK1ZkxIgRSZJhw4alc+fOKS8vT5IMHjw4EyZMyH777Ve71OfYsWMzePDgNGnSJG3atMk+++xT52dss8022WGHHdYYBwAAAGDrp/gDAKgnTjjhhLz++usZN25cFi5cmN69e2f69OkpLS1NksyfP7/OE35jxoxJUVFRxowZk9deey077rhjBg8enMsvv7xQtwAAAABAASn+AADqkVGjRq1zac8ZM2bU2W/atGnGjx+f8ePHf+Tr//c1AAAAAGg4ij98CgAAAAAAAFDfKf4AAAAAAACgAVD8AQAAAAAAQAOg+AMAAAAAAIAGQPEHAAAAAAAADYDiDwAAAAAAABoAxR8AAAAAAAA0AIo/AAAAAAAAaAAUfwAAAAAAANAAKP4AAAAAAACgAVD8AQAAAAAAQAOg+AMAAAAAAIAGQPEHAAAAAAAADYDiDwAAAAAAABoAxR8AAAAAAAA0AIo/AAAAAAAAaAAUfwAAAAAAANAAKP4AAAAAAACgAVD8AQAAAAAAQAPQtNABAAAAAAAAYF1qamry9uqqNcZXrlpzrLFT/AEAAAAAAFBvvb26Kj3G3V3oGFsFS30CAAAAAABAA+CJPwAAAAAAALYKT4wZkFYlTdYYb9lszbHGSPEHAAAAAADAVqFVSZO0KlFvrYulPgEAAAAAAKABUPwBAAAAAABAA6D4AwAAAAAAgAZA8QcAAAAAAAANgOIPAAAAAAAAGgDFHwAAAAAAADQAij8AAAAAAABoABR/AAAAAAAA0AAo/gAAAAAAAKABUPwBAAAA0KhMmjQpXbt2TYsWLdK3b9889thj65x7ww035OCDD852222X7bbbLgMGDPjA+QAAhaT4AwAAAKDRuO2221JWVpbx48dn9uzZ6dWrVwYOHJjFixevdf6MGTNy0kkn5b777svMmTPTpUuXHHnkkXnttde2cHIAgA+n+AMAAACg0ZgwYULOOOOMjBgxIj169MjkyZPTqlWrTJkyZa3zb7nllnz1q19N796907179/z0pz9NdXV1KioqtnByAIAPp/gDAAAAoFFYtWpVZs2alQEDBtSOFRcXZ8CAAZk5c+ZHusbKlSuzevXqbL/99ms9XllZmWXLltXZAAC2FMUfAAAAAI3CkiVLUlVVldLS0jrjpaWlWbhw4Ue6xvnnn5+ddtqpTnn4n8rLy9OuXbvarUuXLhudGwDgo1L8AQAAAMBHcOWVV+bWW2/N7bffnhYtWqx1zujRo7N06dLa7dVXX93CKQGAxqxpoQMAAAAAwJbQvn37NGnSJIsWLaozvmjRonTs2PEDz/3+97+fK6+8Mn/+85+z7777rnNe8+bN07x5802SFwBgfXniDwAAAIBGoaSkJH369ElFRUXtWHV1dSoqKtKvX791nvfd7343l112WaZPn579999/S0QFANggnvgDAAAAoNEoKyvL8OHDs//+++eAAw7IxIkTs2LFiowYMSJJMmzYsHTu3Dnl5eVJkquuuirjxo3L1KlT07Vr19p3AbZu3TqtW7cu2H0AAKyN4g8AAACARuOEE07I66+/nnHjxmXhwoXp3bt3pk+fntLS0iTJ/PnzU1z8f4tkXXfddVm1alW+8IUv1LnO+PHjc/HFF2/J6AAAH0rxBwAAAECjMmrUqIwaNWqtx2bMmFFn/+WXX978gQAANhHv+AMAAAAAAIAGQPEHAAAAAAAADYDiDwAAAAAAABoAxR8AAAAAAAA0AIo/AAAAAAAAaAAUfwAAAAAAANAAKP4AAAAAAACgAVD8AQAAAAAAQAOg+AMAAAAAAIAGQPEHAAAAAAAADYDiDwAAAAAAABoAxR8AAAAAAAA0AIo/AAAAAAAAaAAUfwAAAAAAANAAKP4AAAAAAACgAajXxd/FF1+coqKiOlv37t0LHQsAYLOZNGlSunbtmhYtWqRv37557LHHPnD+xIkTs+eee6Zly5bp0qVLzjvvvLzzzju1x8vLy/PJT34ybdq0SYcOHTJkyJA8++yzm/s2AAAAACiAel38Jcnee++dBQsW1G4PPvhgoSMBAGwWt912W8rKyjJ+/PjMnj07vXr1ysCBA7N48eK1zp86dWouuOCCjB8/PnPnzs2NN96Y2267LRdeeGHtnPvvvz8jR47MI488knvuuSerV6/OkUcemRUrVmyp2wIAAABgC2la6AAfpmnTpunYsWOhYwAAbHYTJkzIGWeckREjRiRJJk+enLvuuitTpkzJBRdcsMb8hx9+OP3798/QoUOTJF27ds1JJ52URx99tHbO9OnT65xz8803p0OHDpk1a1Y+/elPb8a7AQAAAGBLq/dP/D333HPZaaedsuuuu+bkk0/O/PnzCx0JAGCTW7VqVWbNmpUBAwbUjhUXF2fAgAGZOXPmWs858MADM2vWrNrlQF988cX88Y9/zKBBg9b5c5YuXZok2X777dc5p7KyMsuWLauzAQAAAFD/1esn/vr27Zubb745e+65ZxYsWJBLLrkkBx98cJ5++um0adNmredUVlamsrKydt8vqgCArcGSJUtSVVWV0tLSOuOlpaWZN2/eWs8ZOnRolixZkoMOOig1NTV59913c9ZZZ9VZ6vM/VVdX59xzz03//v2zzz77rDNLeXl5Lrnkkg2/GQAAAAAKol4/8XfUUUfli1/8Yvbdd98MHDgwf/zjH/Pmm2/m17/+9TrPKS8vT7t27Wq3Ll26bMHEAABbzowZM3LFFVfkxz/+cWbPnp1p06blrrvuymWXXbbW+SNHjszTTz+dW2+99QOvO3r06CxdurR2e/XVVzdHfAAAAAA2sXr9xN9/23bbbfPxj388zz///DrnjB49OmVlZbX7y5YtU/4BAPVe+/bt06RJkyxatKjO+KJFi9b5vuOxY8fmlFNOyemnn54k6dmzZ1asWJEzzzwzF110UYqL/+87XqNGjcqdd96ZBx54IDvvvPMHZmnevHmaN2++kXcEAAAAwJa2VRV/y5cvzwsvvJBTTjllnXP8ogpoyE78ydrf8wVs/UpKStKnT59UVFRkyJAhSd5bmrOioiKjRo1a6zkrV66sU+4lSZMmTZIkNTU1tf/7ta99LbfffntmzJiRbt26bb6bAAAAAKCg6nXx981vfjODBw/Oxz72sfzzn//M+PHj06RJk5x00kmFjgYAsMmVlZVl+PDh2X///XPAAQdk4sSJWbFiRUaMGJEkGTZsWDp37pzy8vIkyeDBgzNhwoTst99+6du3b55//vmMHTs2gwcPri0AR44cmalTp+b3v/992rRpk4ULFyZJ2rVrl5YtWxbmRgEAAADYLOp18fePf/wjJ510Uv71r39lxx13zEEHHZRHHnkkO+64Y6GjAQBscieccEJef/31jBs3LgsXLkzv3r0zffr0lJaWJknmz59f5wm/MWPGpKioKGPGjMlrr72WHXfcMYMHD87ll19eO+e6665Lkhx66KF1ftZNN92UU089dbPfEwAAAABbTr0u/m699dZCRwAA2KJGjRq1zqU9Z8yYUWe/adOmGT9+fMaPH7/O672/5CcAAABAfVdTU5O3V1etMb5y1ZpjrF29Lv4AAAAAAABoHN5eXZUe4+4udIytWvGHTwEAAAAAAADqO0/8AQAAAAAAUK88MWZAWpU0WWO8ZbM1x/g/ij8AAAAAAADqlVYlTdKqRI21viz1CQAAAAAAAA2A4g8AAAAAAAAaAMUfAAAAAAAANACKPwAAAAAAAGgAFH8AAAAAAADQACj+AAAAAAAAoAFQ/AEAAAAAAEADoPgDAAAAAACABkDxBwAAAAAAAA2A4g8AAAAAAAAaAMUfAAAAAAAANACKPwAAAAAAAGgAFH8AAAAAAADQACj+AAAAAAAAoAFQ/AEAAAAAAEADoPgDAAAAAACABkDxBwAAAAAAAA2A4g8AAAAAAAAaAMUfAAAAAAAANACKPwAAAAAAAGgAFH8AAAAAAADQACj+AAAAAAAAoAFQ/AEAAAAAAEADoPgDAAAAAACABkDxBwAAAAAAAA2A4g8AAAAAAAAaAMUfAAAAAAAANACKPwAAAAAAAGgAmhY6QENw4k9mFjrCBrv1zH6FjgBb1Nb8f68AAAAAAA1BTU1N3l5dtcb4ylVrjrF+FH8AAAAAAABsMW+vrkqPcXcXOkaDZKlPAAAAAAAAaAA88QcAAAAAAEBBPDFmQFqVNFljvGWzNcf4cIo/AAAAAAAACqJVSZO0KlFXbSqW+gQAAAAAAIAGQPEHAAAAAAAADYBnJwEAAPhQVVVV+ctf/pIFCxakU6dOOfjgg9OkiXduAAAA1Cee+AMAAOADTZs2LbvvvnsOO+ywDB06NIcddlh23333TJs2rdDRAAAA+A+KPwAAANZp2rRp+cIXvpCePXtm5syZeeuttzJz5sz07NkzX/jCF5R/AAAA9YjiDwAAgLX6/7V37/E1nfkex787ib0ThBBEopG43+8pJXWM4Uy0pNQcDBkNM9VqpVXUnUYpUYOhU/RUXdo50Zi2OCYMx8Skbmm1bq1KtYhLFalqJaVyfc4ffdljN1G57WQn+bxfr7za/ey1nvVbv7Wty/Nba++cnBxNmjRJAwYM0JYtW/TAAw+oevXqeuCBB7RlyxYNGDBAzz//vHJycso6VAAAAACAKPwBAAAAAO5i7969Onv2rGbMmCE3N8fLRzc3N02fPl0pKSnau3dvGUUIAAAAALgThT8AAAAAQL4uXbokSWrbtm2+799uvz0dAAAAAKBsUfgDAAAAAOTL399fknT8+PF837/dfns6AAAAAEDZovAHAAAAAMhXz549FRwcrAULFig3N9fhvdzcXMXExKhRo0bq2bNnGUUIAAAAALgThT8AAAAAQL7c3d21ZMkSxcfHa9CgQUpKSlJ6erqSkpI0aNAgxcfHa/HixXJ3dy/rUAEAAAAAkjzKOgAAAAAAgOsaPHiw3n33XU2aNEk9evSwtzdq1EjvvvuuBg8eXIbRAQAAAADuROEPAAAAAPCLBg8erIEDB2rv3r26dOmS/P391bNnT570AwAAAFyUMUY/ZuWUdRh3dTPTdWMr7yj8AQAAAADuyd3dXb/61a/KOgwAAAAABfBjVo5av7CzrMNAGeA3/gAAAAAAAAAAAIAKgCf+AAAAAAAAAAAAKqiPZ/VVVavrfk2/VxXXja08ovAHAAAAAAAAAABQQVW1uquqlXJQZcFXfQIAAAAAAAAAAAAVAIU/AAAAAAAAAAAAoAKg8AcAAAAAAAAAAABUABT+AAAAAAAAAAAAgAqAwh8AAIALWbFihYKDg+Xp6alu3brp4MGDvzj9smXL1KJFC3l5eSkwMFATJkzQrVu3itUnAAAAAAAAyicKfwAAAC5i48aNmjhxoqKjo3X48GF16NBBYWFhSk1NzXf6DRs2aNq0aYqOjlZycrLWrFmjjRs3asaMGUXuEwAAAAAAAOUXhT8AAAAXsXTpUo0ZM0ajR49W69at9dprr6lq1apau3ZtvtMfOHBAoaGhGjFihIKDg/Wb3/xGw4cPd3iir7B9AgAAAAAAoPyi8AcAAOACMjMzdejQIfXt29fe5ubmpr59+yopKSnfeXr06KFDhw7ZC31nzpzR9u3b9fDDDxe5TwAAAAAAAJRfHmUdAAAAAKSrV68qJydHfn5+Du1+fn76/PPP851nxIgRunr1qh588EEZY5Sdna2xY8fav+qzKH1KUkZGhjIyMuyv09LSirpaAAAAAAAAKEXl4om/FStWKDg4WJ6enurWrZvD11cBAABUVomJiVqwYIFWrlypw4cPa9OmTdq2bZvmzZtXrH5jYmJUs2ZN+19gYGAJRQwAAOAaCjvW9M4776hly5by9PRUu3bttH379lKKFAAAoHBcvvC3ceNGTZw4UdHR0Tp8+LA6dOigsLAwpaamlnVoAAAAJaZOnTpyd3fXlStXHNqvXLmi+vXr5zvP7NmzNXLkSD3++ONq166dHn30US1YsEAxMTHKzc0tUp+SNH36dF2/ft3+d+HCheKvIAAAgIso7FjTgQMHNHz4cP3xj3/UkSNHNGjQIA0aNEjHjx8v5cgBAADuzeULf0uXLtWYMWM0evRotW7dWq+99pqqVq2qtWvXlnVoAAAAJcZqtapLly5KSEiwt+Xm5iohIUHdu3fPd56bN2/Kzc3xdM7d3V2SZIwpUp+SZLPZVKNGDYc/AACAiqKwY03Lly9Xv379NHnyZLVq1Urz5s1T586d9eqrr5Zy5AAAAPfm0r/xl5mZqUOHDmn69On2Njc3N/Xt21dJSUllGBkAAEDJmzhxoiIjIxUSEqKuXbtq2bJlunHjhkaPHi1Jeuyxx9SgQQPFxMRIksLDw7V06VJ16tRJ3bp106lTpzR79myFh4fbC4D36hMAAKAyKcpYU1JSkiZOnOjQFhYWpi1bthRq2Tczs+WRmV3omAEAKIqbmTllHQLKiEsX/q5evaqcnBz5+fk5tPv5+enzzz/Pd56MjAxlZGTYX1+/fl2SlJaW5rQ4s3684bS+nc2ZeQFcUXn+9wpURs46Tt3uNy0tTd7e3rJYLE5ZTmENGzZM33zzjV544QVdvnxZHTt21I4dO+znQufPn3d4wm/WrFmyWCyaNWuWLl68qLp16yo8PFzz588vcJ8FYYyRxHkDAAD4t9vnBbfPE8qLoow1Xb58Od/pL1++nO/0dxubCnnh73KzVS1O+AAAFElaWpqyrS5dDqqUnHU+VeG2dExMjF588cU87YGBgWUQjevb9FxZRwAAwN05+zgVGBio69evu9RXWUZFRSkqKirf9xITEx1ee3h4KDo6WtHR0UXusyDS09MlcT4FAADySk9PV82aNcs6DJdyt7Gpi6tGlX4wAABI8l9W1hHgl3z77bclej7l0oW/OnXqyN3dXVeuXHFov3LliurXr5/vPNOnT3f4+oXc3Fxdu3ZNvr6+9rv509LSFBgYqAsXLrjUQF9ZIid5kZO8yEle5CQvcuKIfOTlCjkxxig9PV3e3t7y9vYukxjKk4CAAF24cMGlno4EUDZcYR8OwDXcPp8KCAgo61AKpShjTfXr1y/W2NT333+voKAgnT9/niKpi+L45trYPq6N7ePa2D6u7fr162rYsKFq165dov26dOHParWqS5cuSkhI0KBBgyT9VMhLSEi4613rNptNNpvNoc3HxyffaWvUqMGH/WfISV7kJC9ykhc5yYucOCIfeZV1Thh0KTg3Nzfdd999ZR0GABdS1vtwAK6hPJ5PFWWsqXv37kpISNBzzz1nb9u1a5e6d++e7/T5jU1JP+WLfadr4/jm2tg+ro3t49rYPq7tzp91KQkuXfiTpIkTJyoyMlIhISHq2rWrli1bphs3bmj06NFlHRoAAAAAAADKmXuNNT322GNq0KCBYmJiJEnjx49Xr169tGTJEvXv319xcXH6+OOP9frrr5flagAAAOTL5Qt/w4YN0zfffKMXXnhBly9fVseOHbVjx448P6oMAAAAAAAA3Mu9xprOnz/vcOd9jx49tGHDBs2aNUszZsxQs2bNtGXLFrVt27asVgEAAOCuXL7wJ0lRUVF3/bqForDZbIqOjs73axcqK3KSFznJi5zkRU7yIieOyEde5AQAyi/24QAqil8aa0pMTMzTNmTIEA0ZMqRIy2Lf6frYRq6N7ePa2D6uje3j2py1fSzGGFOiPQIAAAAAAAAAAAAodSX7i4EAAAAAAAAAAAAAygSFPwAAAAAAAAAAAKACoPAHAAAAACg1Z8+elcVi0dGjR39xupMnT6p+/fpKT08vlbgyMzMVHBysjz/+uFSWBwAAAADOUCEKfytWrFBwcLA8PT3VrVs3HTx48BenX7ZsmVq0aCEvLy8FBgZqwoQJunXrVrH6dDUlnZOYmBjdf//98vb2Vr169TRo0CCdPHnS2atRopzxOblt4cKFslgseu6555wQufM4IycXL17U73//e/n6+srLy0vt2rUrV4MnJZ2TnJwczZ49W40aNZKXl5eaNGmiefPmqTz9vGphcpKVlaW5c+eqSZMm8vT0VIcOHbRjx45i9emKSjonlW0fW9DPyW3ldR8LoHIZNWqULBaLLBaLrFarmjZtqrlz5yo7O7usQ8tXYmKiLBaLvv/+e4f2O9fjzr9+/fqVeozTp0/XM888I29v71JZntVq1fPPP6+pU6eWyvIAVCyFvcZ555131LJlS3l6eqpdu3bavn17KUVaORVm+6xevVo9e/ZUrVq1VKtWLfXt27fcXbOWR0UdJ4iLi5PFYtGgQYOcG2AlV9jt8/3332vcuHHy9/eXzWZT8+bN2c85kTPHvVE8e/bsUXh4uAICAmSxWLRly5Z7zpOYmKjOnTvLZrOpadOmWr9+feEXbMq5uLg4Y7Vazdq1a81nn31mxowZY3x8fMyVK1fynT42NtbYbDYTGxtrUlJSzM6dO42/v7+ZMGFCkft0Nc7ISVhYmFm3bp05fvy4OXr0qHn44YdNw4YNzQ8//FBaq1UszsjJbQcPHjTBwcGmffv2Zvz48U5ek5LjjJxcu3bNBAUFmVGjRpkPP/zQnDlzxuzcudOcOnWqtFarWJyRk/nz5xtfX18THx9vUlJSzDvvvGOqV69uli9fXlqrVSyFzcmUKVNMQECA2bZtmzl9+rRZuXKl8fT0NIcPHy5yn67GGTmpbPvYguTktvK6jwVQ+URGRpp+/fqZS5cumbNnz5qVK1cai8ViFixYkO/0GRkZpRyho3/9619Gkvnuu+8c2u9cjzv/rl27VmLLTklJMZLMkSNH7jrNuXPnTJUqVcxXX31VYsstiGvXrhmr1WqOHz9eqssFUL4V9nx4//79xt3d3SxatMicOHHCzJo1y1SpUsV8+umnpRx55VDY7TNixAizYsUKc+TIEZOcnGxGjRplatasWerHpMqkqOMEKSkppkGDBqZnz55m4MCBpRNsJVTY7ZORkWFCQkLMww8/bPbt22dSUlJMYmKiOXr0aClHXjk4c9wbxbd9+3Yzc+ZMs2nTJiPJbN68+RenP3PmjKlataqZOHGiOXHihPnLX/5i3N3dzY4dOwq13HJf+OvatasZN26c/XVOTo4JCAgwMTEx+U4/btw48+tf/9qhbeLEiSY0NLTIfboaZ+Tk51JTU40k8/7775dM0E7mrJykp6ebZs2amV27dplevXqVq0FpZ+Rk6tSp5sEHH3ROwKXAGTnp37+/+cMf/uAwzeDBg01EREQJRu48hc2Jv7+/efXVVx3afr6+lW0fW5Cc/FxF38cWNCfleR8LoPKJjIzMM+D0n//5n+aBBx5weP+ll14y/v7+Jjg42BhjzCeffGJ69+5tPD09Te3atc2YMWNMenp6nn7nz59v6tWrZ2rWrGlefPFFk5WVZZ5//nlTq1Yt06BBA7N27Vr7PLcLa2+//bbp3r27sdlspk2bNiYxMdHh/Tv/IiMj77oedxo+fLgZOnSoQ1tmZqbx9fU1b775pjHGmH/84x8mNDTU1KxZ09SuXdv079/f4UawghT+/vSnP5mQkJA87fv27TO9evUyXl5exsfHx/zmN7+xFyVv3bplnnnmGVO3bl1js9lMaGioOXjwoDHmp2NTgwYNzMqVKx36O3z4sLFYLObs2bP2tt69e5tZs2bdNTYA+LnCng8PHTrU9O/f36GtW7du5sknn3RqnJVVca9Bs7Ozjbe3t/04h5JXlG2UnZ1tevToYd544417nr+geAq7fVatWmUaN25sMjMzSyvESq00agEoGQUp/E2ZMsW0adPGoW3YsGEmLCysUMsq11/1mZmZqUOHDqlv3772Njc3N/Xt21dJSUn5ztOjRw8dOnTI/rjrmTNntH37dj388MNF7tOVOCMn+bl+/bokqXbt2iUYvXM4Myfjxo1T//79HfouD5yVk61btyokJERDhgxRvXr11KlTJ61evdq5K1NCnJWTHj16KCEhQV988YUk6dixY9q3b58eeughJ65NyShKTjIyMuTp6enQ5uXlpX379hW5T1fijJzkp6LvYwuak/K6jwWA27y8vJSZmWl/nZCQoJMnT2rXrl2Kj4/XjRs3FBYWplq1aumjjz7SO++8o3/+85+Kiopy6Gf37t36+uuvtWfPHi1dulTR0dEaMGCAatWqpQ8//FBjx47Vk08+qa+++sphvsmTJ2vSpEk6cuSIunfvrvDwcH377bcKDAzUe++9J+mn39G7dOmSli9fXqB1ioiI0N///nf98MMP9radO3fq5s2bevTRRyVJN27c0MSJE/Xxxx8rISFBbm5uevTRR5Wbm1vg3O3du1chISEObUePHlWfPn3UunVrJSUlad++fQoPD1dOTo4kacqUKXrvvff05ptv6vDhw2ratKnCwsJ07do1ubm5afjw4dqwYYNDn7GxsQoNDVVQUJC9rWvXrtq7d2+BYwVQuRXlfDgpKSnPOW5YWFi5uCYqb0riGvTmzZvKysoqF9dn5VFRt9HcuXNVr149/fGPfyyNMCutomyfrVu3qnv37ho3bpz8/PzUtm1bLViwwH7OhpJTWrUAlJ6SOkfwKMmgStvVq1eVk5MjPz8/h3Y/Pz99/vnn+c4zYsQIXb16VQ8++KCMMcrOztbYsWM1Y8aMIvfpSpyRk5/Lzc3Vc889p9DQULVt27bE16GkOSsncXFxOnz4sD766COnxu8MzsrJmTNntGrVKk2cOFEzZszQRx99pGeffVZWq1WRkZFOXaficlZOpk2bprS0NLVs2VLu7u7KycnR/PnzFRER4dT1KQlFyUlYWJiWLl2q//iP/1CTJk2UkJCgTZs22U/uKuM+9l45+bnKsI8tSE7K8z4WAIwxSkhI0M6dO/XMM8/Y26tVq6Y33nhDVqtV0k+/IXTr1i299dZbqlatmiTp1VdfVXh4uF5++WX7vrV27dp65ZVX5ObmphYtWmjRokW6efOm/Zxj+vTpWrhwofbt26ff/e539uVFRUXpt7/9rSRp1apV2rFjh9asWaMpU6bYBy/r1asnHx8fh/jj4+NVvXp1h7YZM2ZoxowZCgsLU7Vq1bR582aNHDlSkrRhwwY98sgj9t/iu73M29auXau6devqxIkTBT62nTt3Lk/hb9GiRQoJCdHKlSvtbW3atJH0U7Fx1apVWr9+vf0Gq9WrV2vXrl1as2aNJk+erIiICC1ZskTnz59Xw4YNlZubq7i4OM2aNcthOQEBATp37lyB4gSAopwPX758Od/pL1++7LQ4K6uSuAadOnWqAgICuCHRSYqyjfbt26c1a9bo6NGjpRBh5VaU7XPmzBnt3r1bERER2r59u06dOqWnn35aWVlZio6OLo2wK43SqAWgdN3tHCEtLU0//vijvLy8CtRPuX7irygSExO1YMECrVy5UocPH9amTZu0bds2zZs3r6xDKzOFzcm4ceN0/PhxxcXFlXKkpedeOblw4YLGjx+v2NjYPE+tVFQF+Zzk5uaqc+fOWrBggTp16qQnnnhCY8aM0WuvvVaGkTtPQXLyt7/9TbGxsdqwYYMOHz6sN998U4sXL9abb75ZhpE7z/Lly9WsWTO1bNlSVqtVUVFRGj16tNzcKt3hxq6wOakM+9h75aQy7mMBVAy3C2aenp566KGHNGzYMM2ZM8f+frt27exFP0lKTk5Whw4d7EU/SQoNDVVubq5Onjxpb2vTpo3DccPPz0/t2rWzv3Z3d5evr69SU1Md4unevbv9/z08PBQSEqLk5OR7rkfv3r119OhRh7+xY8fa+xk6dKhiY2Ml/VRw+9///V+Hm5q+/PJLDR8+XI0bN1aNGjUUHBwsSTp//vw9l33bjz/+mOcYcPuJv/ycPn1aWVlZCg0NtbdVqVJFXbt2ta9zx44d1apVK/tTf++//75SU1M1ZMgQh768vLx08+bNAscKAKi4Fi5cqLi4OG3evJlrExeRnp6ukSNHavXq1apTp05Zh4N85Obmql69enr99dfVpUsXDRs2TDNnzqyw44PlDfWRyqFcP/FXp04dubu768qVKw7tV65cUf369fOdZ/bs2Ro5cqQef/xxST9dfN+4cUNPPPGEZs6cWaQ+XYkzcnLnIENUVJTi4+O1Z88e3Xfffc5bkRLkjJwcOnRIqamp6ty5s32enJwc7dmzR6+++qoyMjLk7u7uvJUqJmd9Tvz9/dW6dWuH+Vq1amX/KilX5qycTJ48WdOmTbPffd+uXTudO3dOMTExLv8UZFFyUrduXW3ZskW3bt3St99+q4CAAE2bNk2NGzcucp+uxBk5uVNl2cfeKyflfR8LoPLq3bu3Vq1aJavVqoCAAHl4OF5u3VngK4wqVao4vLZYLPm2FearNH9JtWrV1LRp07u+HxERoV69eik1NVW7du2Sl5eX+vXrZ38/PDxcQUFBWr16tQICApSbm6u2bds6fO3pvdSpU0ffffedQ1tB7279JREREdqwYYOmTZumDRs2qF+/fvL19XWY5tq1a6pbt26xlwWgcijK+XD9+vXL7TVReVOca9DFixdr4cKF+uc//6n27ds7M8xKrbDb6PTp0zp79qzCw8PtbbfPgTw8PHTy5Ek1adLEuUFXIkX5N+Tv768qVao4XLe3atVKly9fVmZmpsONcCgeZ9cCUPrudo5Qo0aNQl0PleutaLVa1aVLFyUkJNjbcnNzlZCQ4HB3651u3ryZ58N7eydkjClSn67EGTm5/d+oqCht3rxZu3fvVqNGjZy0BiXPGTnp06ePPv30U4e7oENCQhQREaGjR4+6/IC0sz4noaGhDnenS9IXX3zh8JsprspZObnbNCU1MOdMxdkfenp6qkGDBsrOztZ7772ngQMHFrtPV+CMnEiVbx97291yUt73sQAqr9sFs4YNG+Yp+uWnVatWOnbsmG7cuGFv279/v/0rPYvrgw8+sP9/dna2Dh06pFatWkmSfcClKL+10qNHDwUGBmrjxo2KjY3VkCFD7IXIb7/9VidPntSsWbPUp08ftWrVKk8BryA6deqkEydOOLS1b9/e4XhzpyZNmshqtWr//v32tqysLH300UcON6aNGDFCx48f16FDh/Tuu+/m+/Xrx48fV6dOnQodM4DKqSjnw927d8+zP9u1a1e5uCYqb4p6vbJo0SLNmzdPO3bsyPPV0yhZhd1GLVu2zHO9+Mgjj9i/sSAwMLA0w6/wivJvKDQ0VKdOnXIY+/riiy/k7+9P0a+EOWs8E2WnxM4RTDkXFxdnbDabWb9+vTlx4oR54oknjI+Pj7l8+bIxxpiRI0eaadOm2aePjo423t7e5u233zZnzpwx//d//2eaNGlihg4dWuA+XZ0zcvLUU0+ZmjVrmsTERHPp0iX7382bN0t9/YrCGTn5uV69epnx48c7e1VKjDNycvDgQePh4WHmz59vvvzySxMbG2uqVq1q/ud//qfU168onJGTyMhI06BBAxMfH29SUlLMpk2bTJ06dcyUKVNKff2KorA5+eCDD8x7771nTp8+bfbs2WN+/etfm0aNGpnvvvuuwH26OmfkpLLtYwuSk58rb/tYAJVPZGSkGThwYKHev3HjhvH39ze//e1vzaeffmp2795tGjdubCIjI39xvvz2iUFBQebPf/6zMcaYlJQUI8k0bNjQbNq0ySQnJ5snnnjCVK9e3XzzzTfGGGO++uorY7FYzPr1601qaqpJT0+3L69fv34Ox6NLly7Z57tt5syZpnXr1sbDw8Ps3bvX3p6Tk2N8fX3N73//e/Pll1+ahIQEc//99xtJZvPmzQ7xHTly5K752rp1q6lXr57Jzs62t508edJYrVbz1FNPmWPHjpnk5GSzcuVKe2zjx483AQEB5h//+If57LPPTGRkpKlVq5a5du2aQ9+hoaGmQ4cOxtvbO99jbVBQkHnrrbfuGhsA/Fxhz4f3799vPDw8zOLFi01ycrKJjo42VapUMZ9++mlZrUKFVtjts3DhQmO1Ws27777rcCy8faxEySvsNvq5e52HoXgKu33Onz9vvL29TVRUlDl58qSJj4839erVMy+99FJZrUKFVhrj3ii69PR0c+TIEXPkyBEjySxdutQcOXLEnDt3zhhjzLRp08zIkSPt0585c8ZUrVrVTJ482SQnJ5sVK1YYd3d3s2PHjkItt9wX/owx5i9/+Ytp2LChsVqtpmvXruaDDz6wv9erVy+HC+esrCwzZ84c06RJE+Pp6WkCAwPN008/nWew8Zf6LA9KOieS8v1bt25d6a1UMTnjc3Kn8jgo7Yyc/P3vfzdt27Y1NpvNtGzZ0rz++uultDYlo6RzkpaWZsaPH28aNmxoPD09TePGjc3MmTNNRkZGKa5V8RQmJ4mJiaZVq1bGZrMZX19fM3LkSHPx4sVC9VkelHROKts+tqCfkzuVx30sgMqlKIU/Y4z55JNPTO/evY2np6epXbu2GTNmjMPAYnEKfxs2bDBdu3Y1VqvVtG7d2uzevdthnrlz55r69esbi8Vi309HRkbme0xq0aKFw7wnTpwwkkxQUJDJzc11eG/Xrl32/Xz79u1NYmJioQt/WVlZJiAgIM8FbmJiounRo4ex2WzGx8fHhIWF2c+9fvzxR/PMM8+YOnXqGJvNZkJDQ83Bgwfz9L1y5UojyTz22GN53jtw4IDx8fEpNzffAHAdhTkfNsaYv/3tb6Z58+bGarWaNm3amG3btpVyxJVLYbZPUFBQvsfC6Ojo0g+8Einsv6E7UfhzvsJunwMHDphu3boZm81mGjdubObPn+9wQxdKlrPHvVF0//rXv/I9ptx5/dWrV68883Ts2NFYrVbTuHHjIo0PWozh+U0AAAAAqCjOnj2rRo0a6ciRI+rYsWNZh1NkK1as0NatW7Vz585SW+awYcPUoUMHzZgxo9SWCQAAAAAl6d4/PAEAAAAAQCl78skn9f333ys9PV3e3t5OX15mZqbatWunCRMmOH1ZAAAAAOAsPPEHAAAAABVIRXniDwAAAABQeBT+AAAAAAAAAAAAgArArawDAAAAAAAAAAAAAFB8FP4AAAAAAAAAAACACoDCHwAAAAAAAAAAAFABUPgDAAAAAAAAAAAAKgAKfwAAAAAAAAAAAEAFQOEPgEubM2eO/Pz8ZLFYtGXLlrIOBwAAAAAAAJUIY1MAyhsKfwBKxKhRo2SxWGSxWGS1WtW0aVPNnTtX2dnZRe4zOTlZL774ov77v/9bly5d0kMPPVSCEQMAAAAAAKCiYGwKAH7iUdYBAKg4+vXrp3Xr1ikjI0Pbt2/XuHHjVKVKFU2fPr1Q/eTk5Mhisej06dOSpIEDB8pisRQ5rqysLFWpUqXI8wMAAAAAAMD1MTYFADzxB6AE2Ww21a9fX0FBQXrqqafUt29fbd26VRkZGXr++efVoEEDVatWTd26dVNiYqJ9vvXr18vHx0dbt25V69atZbPZ9Ic//EHh4eGSJDc3N/vJVW5urubOnav77rtPNptNHTt21I4dO+x9nT17VhaLRRs3blSvXr3k6emp2NhYjRo1SoMGDdKCBQvk5+cnHx8f+11fkydPVu3atXXfffdp3bp1Dus0depUNW/eXFWrVlXjxo01e/ZsZWVl2d+fM2eOOnbsqL/+9a8KDg5WzZo19bvf/U7p6en2aXJzc7Vo0SI1bdpUNptNDRs21Pz58+3vX7hwQUOHDpWPj49q166tgQMH6uzZsyW5aQAAAAAAACo8xqYYmwJA4Q+AE3l5eSkzM1NRUVFKSkpSXFycPvnkEw0ZMkT9+vXTl19+aZ/25s2bevnll/XGG2/os88+0yuvvGI/0bl06ZIuXbokSVq+fLmWLFmixYsX65NPPlFYWJgeeeQRh74kadq0aRo/frySk5MVFhYmSdq9e7e+/vpr7dmzR0uXLlV0dLQGDBigWrVq6cMPP9TYsWP15JNP6quvvrL34+3trfXr1+vEiRNavny5Vq9erT//+c8Oyzp9+rS2bNmi+Ph4xcfH6/3339fChQvt70+fPl0LFy7U7NmzdeLECW3YsEF+fn6SfrrjKywsTN7e3tq7d6/279+v6tWrq1+/fsrMzCzBrQEAAAAAAFC5MDb1E8amgErGAEAJiIyMNAMHDjTGGJObm2t27dplbDabGTVqlHF3dzcXL150mL5Pnz5m+vTpxhhj1q1bZySZo0ePOkyzefNm8/PdVEBAgJk/f75D2/3332+efvppY4wxKSkpRpJZtmxZnviCgoJMTk6Ova1FixamZ8+e9tfZ2dmmWrVq5u23377rev7pT38yXbp0sb+Ojo42VatWNWlpafa2yZMnm27duhljjElLSzM2m82sXr063/7++te/mhYtWpjc3Fx7W0ZGhvHy8jI7d+68axwAAAAAAAD4N8amGJsC8BN+4w9AiYmPj1f16tWVlZWl3NxcjRgxQv/1X/+l9evXq3nz5g7TZmRkyNfX1/7aarWqffv2v9h/Wlqavv76a4WGhjq0h4aG6tixYw5tISEheeZv06aN3Nz+/aCzn5+f2rZta3/t7u4uX19fpaam2ts2btyoV155RadPn9YPP/yg7Oxs1ahRw6Hf4OBgeXt721/7+/vb+0hOTlZGRob69OmT7zodO3ZMp06dcphfkm7dumX/HnkAAAAAAADcG2NTP2FsCqjcKPwBKDG9e/fWqlWrZLVaFRAQIA8PD23cuFHu7u46dOiQ3N3dHaavXr26/f+9vLyK9SPJP1etWrU8bT//EWWLxZJvW25uriQpKSlJERERevHFFxUWFqaaNWsqLi5OS5YsuWe/t/vw8vL6xTh/+OEHdenSRbGxsXneq1u37i/OCwAAAAAAgH9jbCpvH4xNAZUPhT8AJaZatWpq2rSpQ1unTp2Uk5Oj1NRU9ezZs1j916hRQwEBAdq/f7969eplb9+/f7+6du1arL7zc+DAAQUFBWnmzJn2tnPnzhWqj2bNmsnLy0sJCQl6/PHH87zfuXNnbdy4UfXq1ctztxYAAAAAAAAKjrGpvBibAioft3tPAgBF17x5c0VEROixxx7Tpk2blJKSooMHDyomJkbbtm0rdH+TJ0/Wyy+/rI0bN+rkyZOaNm2ajh49qvHjx5d47M2aNdP58+cVFxen06dP65VXXtHmzZsL1Yenp6emTp2qKVOm6K233tLp06f1wQcfaM2aNZKkiIgI1alTRwMHDtTevXuVkpKixMREPfvssw4/5AwAAAAAAIDCY2yKsSmgsuGJPwBOt27dOr300kuaNGmSLl68qDp16uiBBx7QgAEDCt3Xs88+q+vXr2vSpElKTU1V69attXXrVjVr1qzE437kkUc0YcIERUVFKSMjQ/3799fs2bM1Z86cQvUze/ZseXh46IUXXtDXX38tf39/jR07VpJUtWpV7dmzR1OnTtXgwYOVnp6uBg0aqE+fPtxlBQAAAAAAUAIYm2JsCqhMLMYYU9ZBAAAAAAAAAAAAACgevuoTAAAAAAAAAAAAqAAo/AEAAAAAAAAAAAAVAIU/AAAAAAAAAAAAoAKg8AcAAAAAAAAAAABUABT+AAAAAAAAAAAAgAqAwh8AAAAAAAAAAABQAVD4AwAAAAAAAAAAACoACn8AAAAAAAAAAABABUDhDwAAAAAAAAAAAKgAKPwBAAAAAAAAAAAAFQCFPwAAAAAAAAAAAKACoPAHAAAAAAAAAAAAVAD/D5pknegZ7dR2AAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'full_performances': array([0.92496036, 0.9296717 , 0.92268393, 0.91085103, 0.80544397,\n", + " 0.92159231, 0.92195513, 0.92742979, 0.94270547, 0.89704149,\n", + " 0.91374033, 0.91845134, 0.91761411, 0.91542801, 0.90279132,\n", + " 0.88043954, 0.90439775, 0.92869027, 0.89636597, 0.92093057,\n", + " 0.92079846, 0.92634561, 0.93064771, 0.9216495 , 0.93057201,\n", + " 0.92313046, 0.92961422, 0.93589523, 0.93857416, 0.94273073]), 'quantiles': {'5': 0.8876064332443163, '25': 0.9141622483781227, '50': 0.9218023145499467, '75': 0.9293832304634039, '95': 0.9408463788484324}, 'average': 0.9167714158726751, 'std_dev': 0.024776796648712244}\n" + ] + } + ], + "source": [ + "from promptbench.prompteval import efficient_eval\n", + "\n", + "result = efficient_eval(model, prompt_list, dataset, proj_func, \n", + " budget=1200, # The maximum number of examples that can be evaluated during the process. Increasing this value covers more data points, while decreasing it reduces computation.\n", + " visualize=True, # If set to True, the function will generate and display visualizations of the model's performance (combined_result.png), including histograms, boxplots, and cumulative distribution functions (CDFs).\n", + " pca_dim=25, # The number of dimensions retained during PCA on the prompt embeddings. Higher values retain more dimensional information, while lower values reduce dimensionality.\n", + " method='EmbPT') # The evaluation method to use. 'EmbPT' involves embedding the prompts and using these embeddings in model fitting. 'Rasch' does not obtain prompt embeddings; instead, prompts are one-hot encoded in this method.\n", + "\n", + "print(result)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "promptbench", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.9" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/promptbench/prompteval/__init__.py b/promptbench/prompteval/__init__.py new file mode 100644 index 0000000..cfdb5a7 --- /dev/null +++ b/promptbench/prompteval/__init__.py @@ -0,0 +1 @@ +from .efficient_eval import * \ No newline at end of file diff --git a/promptbench/prompteval/efficient_eval.py b/promptbench/prompteval/efficient_eval.py new file mode 100644 index 0000000..1b375d0 --- /dev/null +++ b/promptbench/prompteval/efficient_eval.py @@ -0,0 +1,213 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# Source Attribution: +# The majority of this code is derived from the following sources: +# - PromptEval GitHub Repository: https://github.com/felipemaiapolo/prompteval + +import numpy as np +import matplotlib.pyplot as plt +import pandas as pd +from tqdm import tqdm +from sentence_transformers import SentenceTransformer +from sklearn.decomposition import PCA + +from ..utils import InputProcess, OutputProcess +from .methods import StratSample, ExtendedRaschModel + +def get_prompt_embedding(prompt_list, pca_dim): + """ + Generates prompt embeddings using a pre-trained sentence transformer model and reduces + their dimensionality using PCA (Principal Component Analysis). + + Parameters: + prompt_list (list of str): A list of text prompts for which embeddings are to be generated. + pca_dim (int): The number of principal components to retain during dimensionality reduction. + + Returns: + np.ndarray: A matrix where each row corresponds to the reduced-dimensionality embedding + of a prompt. + """ + + embedder = SentenceTransformer('sentence-transformers/facebook-dpr-question_encoder-multiset-base') + pca = PCA(n_components=pca_dim) + X = pca.fit_transform(embedder.encode(prompt_list)) + + return X + +def get_Y_seen(model, prompt_list, example_list, proj_func, budget=1000): + """ + Generates a matrix of observed (seen) examples and their corresponding labels based on + model predictions. The function randomly samples examples up to the given budget and + evaluates the model's performance on those examples. + + Parameters: + model (promptbench.LLMModel): The model to evaluate. + prompt_list (list of str): A list of prompts used to generate input for the model. + example_list (list): A list of labeled examples used for evaluation. + proj_func (function): A function used to project model outputs into a classification + space or other relevant space. + budget (int, optional): The maximum number of examples to be evaluated. Defaults to 1000. + + Returns: + tuple: + seen_examples (np.ndarray): A boolean matrix indicating which examples were observed + (True) and which were not (False). + Y_seen (np.ndarray): A matrix where each element is 1 if the model's prediction matches + the true label, 0 otherwise, and -99 for unseen examples. + """ + + # create an empty matrix Y, with 'template_num' columns, and 'dataset_size' rows + example_num = len(example_list) + prompt_num = len(prompt_list) + Y_seen = np.zeros((prompt_num, example_num)) + + # 随机抽样 + seen_examples = StratSample(np.zeros(Y_seen.shape).astype(bool), budget, random_seed=0) + + # using np.where to find the indices of all True elements + true_indices = np.where(seen_examples) + + # iterate over all True indices and fill in the corresponding values in Y_seen + for row, col in tqdm(zip(true_indices[0], true_indices[1]), total=len(true_indices[0])): + prompt = prompt_list[row] + data = example_list[col] + # test it! + input_text = InputProcess.basic_format(prompt, data) + label = data['label'] + raw_pred = model(input_text) + # process output + pred = OutputProcess.cls(raw_pred, proj_func) + Y_seen[row, col] = 1 if pred == label else 0 + + # mark the unseen examples + Y_seen[~seen_examples] = -99 #just a placeholder for non-observed + + return seen_examples, Y_seen + +def fit_Y(X, Y_seen, seen_examples): + """ + Fits a model to the seen examples using the Extended Rasch Model and calculates the + predicted scores for each prompt. + + Parameters: + X (np.ndarray): The matrix of prompt embeddings. + Y_seen (np.ndarray): The matrix of observed example results (1 for correct, 0 for incorrect, + -99 for unseen). + seen_examples (np.ndarray): A boolean matrix indicating which examples were observed. + + Returns: + np.ndarray: A vector of predicted scores for each prompt, calculated as the mean score + across all seen examples. + """ + + extended_rasch_cov = ExtendedRaschModel() + extended_rasch_cov.fit(seen_examples, Y_seen, X) + S_hat_cov = extended_rasch_cov.get_Y_hat().mean(1) + + return S_hat_cov + +def visualize_result(data): + """ + Visualizes the distribution of model performance using a histogram, boxplot, and + cumulative distribution function (CDF). + + Parameters: + data (np.ndarray): A vector of performance scores to be visualized. + + Returns: + None: The function displays and saves the plots as 'combined_result.png'. + """ + + fig, axes = plt.subplots(1, 3, figsize=(18, 6)) + + # first subplot - Histogram + axes[0].hist(data, alpha=0.75, density=True, label='PromptEval') + # axes[0].hist(groundtruth, alpha=0.75, density=True, label='Ground Truth') + axes[0].set_xlabel("Performance") + axes[0].set_ylabel("Density") + + # second subplot- Boxplot + axes[1].boxplot([data], labels=['PromptEval (cov)']) + axes[1].set_ylabel("Performance Distribution") + + # third subplot - CDF + bins = np.linspace(0, 1.1, 100) + axes[2].hist(data, density=True, cumulative=True, bins=bins, histtype='step', linewidth=1.5, label='PromptEval') + # axes[2].hist(groundtruth, density=True, cumulative=True, bins=bins, histtype='step', linewidth=1.5, label='Ground Truth') + axes[2].set_xlim(0.0, 1.0) + axes[2].legend(fontsize=10) + axes[2].set_xlabel(f"Performance") + axes[2].set_ylabel("CDF") + + plt.tight_layout() + + plt.savefig('combined_result.png') + plt.show() + + +def efficient_eval(model, prompt_list, example_list, proj_func, budget=1000, visualize=True, pca_dim=25, method='EmbPT'): + """ + Efficient evaluation of a model on a list of prompts and examples. + + Parameters: + model (promptbench.LLMModel): The model to evaluate. This is typically a large language model that + will generate responses based on the provided prompts. + prompt_list (list of str): A list of prompts for which the model's performance will be evaluated. + example_list (list): A list of examples used for evaluation purposes. These examples are used + in conjunction with the prompts to generate model responses. + proj_func (function): A projection function used to map the model's output to a desired space + (e.g., embedding space or scoring space). + budget (int, optional): The maximum number of examples to be used for evaluation. + Defaults to 1000. + visualize (bool, optional): Whether to visualize the results. If True, a visualization of + the model's performance will be generated. Defaults to True. + pca_dim (int, optional): The number of principal components to retain when using PCA + for dimensionality reduction in the EmbPT method. Defaults to 25. + method (str, optional): The evaluation method to be used. Can be 'EmbPT' for embedding-based + prompt tuning or 'Rasch' for Rasch model evaluation. Defaults to 'EmbPT'. + + Returns: + dict: A dictionary containing the following keys: + 'full_performances' (np.ndarray): The complete list of model performance scores + for each prompt after fitting the examples. + 'quantiles' (dict): A dictionary containing the 5th, 25th, 50th, 75th, and 95th + percentiles of the performance scores. + 'average' (float): The average performance score across all prompts. + 'std_dev' (float): The standard deviation of the performance scores. + visual_result: if you set visualize=True, the function will generate combined_result.png for you to see the result. + """ + + # get prompt embedding + if method == 'EmbPT': + X = get_prompt_embedding(prompt_list, pca_dim) + elif method == 'Rasch': + X = None + else: + raise ValueError("Invalid method specified") + + # get Y_seen + seen_examples, Y_seen = get_Y_seen(model, prompt_list, example_list, proj_func, budget) + # fit Y + S_hat_cov = fit_Y(X, Y_seen, seen_examples) # n个prompt最终的scores + + # Calculate quantiles (5th, 25th, 50th, 75th, 95th) + percentile_list = [5, 25, 50, 75, 95] + quantiles = np.percentile(S_hat_cov, percentile_list) + quantiles_dict = {str(k): v for k, v in zip(percentile_list, quantiles)} + + # Calculate the average + average = np.mean(S_hat_cov) + # Calculate the standard deviation + std_dev = np.std(S_hat_cov) + + if visualize: + visualize_result(S_hat_cov) + + # Return the calculated statistics + return { + 'full_performances': S_hat_cov, + 'quantiles': quantiles_dict, + 'average': average, + 'std_dev': std_dev + } \ No newline at end of file diff --git a/promptbench/prompteval/methods.py b/promptbench/prompteval/methods.py new file mode 100644 index 0000000..6c8412e --- /dev/null +++ b/promptbench/prompteval/methods.py @@ -0,0 +1,291 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# Source Attribution: +# The majority of this code is derived from the following sources: +# - PromptEval GitHub Repository: https://github.com/felipemaiapolo/prompteval + +import copy +import numpy as np +from sklearn.linear_model import LogisticRegression as LR # type: ignore +from tqdm import tqdm # type: ignore + +class LogisticRegression: + """ + Logistic regression model. + + Attributes: + reg (float): The regularization parameter for the logistic regression model. This is equivalent to the prior Gaussian covariance scaling in the Bayesian setup with gaussian, ie, prior cov = reg*identity. + """ + + def __init__(self, reg=1e2): + """ + Initializes the logistic regression model with a regularization parameter. + + Parameters: + reg (float): Regularization parameter (default is 100). + """ + self.reg = reg + + def fit(self, X, y): + """ + Fits the logistic regression model to the data. + + Parameters: + X (array-like): Feature matrix. + y (array-like): Target vector of 0s and 1s. + """ + # This block of code is just a trick to run the Scikit-Learn implementation for logistic regression + if np.var(y) == 0: + y_copy = copy.deepcopy(y) + local_state = np.random.RandomState(0) + ind = local_state.choice(len(y_copy)) + y_copy[ind] = 1 - np.median(y_copy) + else: + y_copy = copy.deepcopy(y) + + # Fitting the model + logreg = LR(C=self.reg, random_state=0, solver="liblinear", fit_intercept=False).fit(X, y_copy) + self.mu = logreg.coef_.squeeze() + + +class ExtendedRaschModel: + """ + An extended Rasch model incorporating covariates for both formats and examples. + + Attributes: + seen_examples (array-like): Boolean array indicating seen examples. + Y (array-like): Target matrix of 0s and 1s. + X (array-like): Covariates for formats. + Z (array-like): Covariates for examples. + x_dim (int): Dimension of X. + z_dim (int): Dimension of Z. + n_formats (int): Number of formats. + n_examples (int): Number of examples. + rasch_model (LogisticRegression): The fitted logistic regression model. + gammas (array-like): Coefficients for the format covariates. + thetas (array-like): Format parameters. + psi (array-like): Coefficients for the example covariates. + betas (array-like): Example parameters. + logits (array-like): Logits of the fitted model. + """ + + def __init__(self): + """ + Initializes the extended Rasch model. + """ + pass + + def fit(self, seen_examples, Y, X=None, Z=None): + """ + Fits the extended Rasch model to the data. + + Parameters: + seen_examples (array-like): Boolean array indicating seen examples. + Y (array-like): Target matrix. + X (array-like): Covariates for formats (default is identity matrix). + Z (array-like): Covariates for examples (default is identity matrix). + """ + self.seen_examples = seen_examples + self.Y = Y + + # X (formats covariates) + if type(X) != np.ndarray: + self.X = np.eye(Y.shape[0]) + else: + self.X = X + self.x_dim = self.X.shape[1] + + # Z (examples covariates) + if type(Z) != np.ndarray: + self.Z = np.eye(Y.shape[1]) + else: + self.Z = Z + self.z_dim = self.Z.shape[1] + + # Formatting the data + self.n_formats, self.n_examples = seen_examples.shape + features, labels = GenXY(seen_examples, Y, self.X, self.Z) + + if type(X) != np.ndarray and type(Z) != np.ndarray: # basic Rasch model (no need to include intercept) + features = features[:, :-1] + elif ( + type(X) != np.ndarray or type(Z) != np.ndarray + ): # just one set of covariates (no need to include intercept) + pass + else: # two sets of covariates (need to include intercept) + features = np.hstack((features, np.ones((features.shape[0], 1)))) + + # Fitting the model + self.rasch_model = LogisticRegression() + self.rasch_model.fit(features, labels) + + # Predicted probs + self.gammas = self.rasch_model.mu[: self.x_dim] + self.thetas = self.X @ self.gammas + self.psi = self.rasch_model.mu[self.x_dim :] + + if type(X) != np.ndarray and type(Z) != np.ndarray: # basic Rasch model (no intercept) + self.betas = np.hstack((self.psi, np.array([0]))) + self.logits = self.thetas[:, None] + self.betas[None, :] + elif type(X) != np.ndarray or type(Z) != np.ndarray: # just one set of covariates (no intercept) + self.betas = self.Z @ self.psi + self.logits = self.thetas[:, None] + self.betas[None, :] + else: # two sets of covariates (intercept included) + self.betas = self.Z @ self.psi[:-1] + self.logits = self.thetas[:, None] + self.betas[None, :] + self.psi[-1] + + def get_Y_hat(self): + """ + Computes the predicted probabilities. + + Returns: + array-like: Predicted probabilities. + """ + P_hat = sigmoid(self.logits) + Y_hat = np.zeros(self.seen_examples.shape) + Y_hat[self.seen_examples] = self.Y[self.seen_examples] + Y_hat[~self.seen_examples] = P_hat[~self.seen_examples] + return Y_hat + + +class Baseline: + """ + A baseline model for evaluating prompts. + + Attributes: + seen_examples (array-like): Boolean array indicating seen examples. + quantiles (list): List of quantiles for evaluation. + estimates (dict): Dictionary to store evaluation metrics. + """ + + def __init__(self): + """ + Initializes the baseline model. + """ + pass + + def fit(self, Y, quantiles, rounds_eval, random_seed=None): + """ + Fits the baseline model and evaluates the prompts. + + Parameters: + Y (array-like): Target matrix. + quantiles (list): List of quantiles for evaluation. + rounds_eval (list): List of evaluation rounds. + random_seed (int): Random seed for reproducibility (default is None). + """ + n_formats, n_examples = Y.shape + self.seen_examples = np.zeros(Y.shape).astype(bool) + self.quantiles = quantiles + self.estimates = {"n_seen": [], "estimates": [], "accs_hat": []} + + for num_seen_examples in rounds_eval: + self.seen_examples = StratSample(self.seen_examples, num_seen_examples, random_seed) + eps = 1e-10 + accs = np.array([(Y[i, s].sum() + eps) / (s.sum() + eps) for i, s in enumerate(self.seen_examples)]) + self.estimates["n_seen"].append(self.seen_examples.sum()) + self.estimates["estimates"].append(np.percentile(accs, quantiles).tolist()) + self.estimates["accs_hat"].append(accs.tolist()) + + +def StratSample(seen_examples, max_seen, random_seed, active_arms=None, random_column=False): + """ + Generates a stratified sample from the seen examples matrix until the maximum number of seen examples is reached. + + Parameters: + seen_examples (array-like): The matrix of seen examples. + max_seen (int): The maximum number of seen examples. + random_seed (int): The random seed for reproducibility. + active_arms (list or ndarray, optional): List of active arms. Defaults to None. + random_column (bool, optional): If True, selects a column randomly. Defaults to False. + + Returns: + array-like: The updated matrix of seen examples. + """ + matrix = seen_examples + rows, columns = matrix.shape + + if type(active_arms) == list or type(active_arms) == np.ndarray: + pass + else: + active_arms = list(range(rows)) + + local_state = np.random.RandomState(random_seed) + + # initialize the sums of each row and column + row_sums = matrix.sum(1) + col_sums = matrix.sum(0) + + while True: + + if row_sums.sum() >= max_seen: + return matrix + + min_row_sum = row_sums[active_arms].min() + min_sum_rows = [i for i in active_arms if row_sums[i] == min_row_sum] + next_row = local_state.choice(min_sum_rows) + + avail_columns = [i for i in range(columns) if not matrix[next_row, i]] + + if not avail_columns: # nothing else to see + return matrix + + if random_column: + next_column = local_state.choice(avail_columns) + else: + # the most time-consuming step, so we speed it up by reduct the repeated calculation + min_column_sum = col_sums[avail_columns].min() + min_sum_columns = [i for i in avail_columns if col_sums[i] == min_column_sum] + next_column = local_state.choice(min_sum_columns) + + matrix[next_row, next_column] = True + + row_sums[next_row] += 1 + col_sums[next_column] += 1 + + +def GenXY(seen_items, Y, X, Z): + """ + Generates combined feature and label matrices. + + Parameters: + seen_items (array-like): Matrix indicating which items have been seen. + Y (array-like): Target values matrix. + X (array-like): Feature matrix for the first set of features. + Z (array-like): Feature matrix for the second set of features. + + Returns: + tuple: Combined feature matrix and corresponding labels. + """ + Y_seen = -np.ones(Y.shape) + Y_seen[seen_items] = Y[seen_items] + + W_x = [] + W_z = [] + + labels = [] + for i in range(seen_items.shape[0]): + for j in range(seen_items.shape[1]): + if seen_items[i, j] == True: + W_x.append(X[i]) + W_z.append(Z[j]) + labels.append(Y_seen[i, j]) + + W = np.hstack((np.vstack(W_x), np.vstack(W_z))) + labels = np.array(labels) + return W, labels + + +def sigmoid(x): + """ + Applies the sigmoid function to the input. + + Parameters: + x (array-like): The input data. + + Returns: + array-like: The output of the sigmoid function. + """ + x_clipped = np.clip(x, -30, 30) + return 1 / (1 + np.exp(-x_clipped))