experiments/botnet_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "40c2fa8a-b6e5-4182-9c55-809dba5a32db",
   "metadata": {},
   "source": [
    "# Offline bot classification using ANTE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d1c4592-bc2f-4b9a-9d4c-2e3390d80ad2",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f2f158f-ad38-4c6e-9391-42616d11a283",
   "metadata": {},
   "source": [
    "The goal is to classify traffic between devices into two different groups: bot or non-bot. This task is considered as a binary classification problem, where 0 indicates that traffic is caused by a bot, and 1 otherwise."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f27d1419-a9da-4f77-bf7a-725735f50915",
   "metadata": {},
   "source": [
    "This analysis uses the CTU-13 dataset. This dataset captures both malicious (generated by bots) and normal network traffic recorded at the Czech Technical University. This analysis specifically uses capture 51, characterized by its large volume of data, totalizing 64GB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "054862e9-2c92-4d33-9b56-70dc9bf57f98",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = \"../../data/raw/CTU-datasets/botnet-capture-20110818-bot-2.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e218b213-2892-4759-afc3-9e5aaf115363",
   "metadata": {},
   "source": [
    "The dataset is too large to include in the repository, so a small sample is used to visualize all the dataset columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "dbd16712-bfd0-426a-98f3-523190a8b6e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>frame.time</th>\n",
       "      <th>frame.time_epoch</th>\n",
       "      <th>_ws.col.Source</th>\n",
       "      <th>_ws.col.Destination</th>\n",
       "      <th>_ws.col.Protocol</th>\n",
       "      <th>frame.len</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Aug 18, 2011 10:46:56.445440000 -03</td>\n",
       "      <td>1.313675e+09</td>\n",
       "      <td>Cisco_db:19:c3</td>\n",
       "      <td>147.32.84.191</td>\n",
       "      <td>ARP</td>\n",
       "      <td>60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Aug 18, 2011 10:47:47.832016000 -03</td>\n",
       "      <td>1.313675e+09</td>\n",
       "      <td>PcsCompu_b5:b7:19</td>\n",
       "      <td>147.32.84.191</td>\n",
       "      <td>ARP</td>\n",
       "      <td>60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Aug 18, 2011 10:47:47.832023000 -03</td>\n",
       "      <td>1.313675e+09</td>\n",
       "      <td>147.32.84.255</td>\n",
       "      <td>Broadcast</td>\n",
       "      <td>ARP</td>\n",
       "      <td>60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Aug 18, 2011 10:47:48.820312000 -03</td>\n",
       "      <td>1.313675e+09</td>\n",
       "      <td>147.32.84.255</td>\n",
       "      <td>147.32.84.191</td>\n",
       "      <td>ARP</td>\n",
       "      <td>60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Aug 18, 2011 10:47:48.820320000 -03</td>\n",
       "      <td>1.313675e+09</td>\n",
       "      <td>147.32.84.255</td>\n",
       "      <td>147.32.84.191</td>\n",
       "      <td>ARP</td>\n",
       "      <td>60</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            frame.time  frame.time_epoch     _ws.col.Source  \\\n",
       "0  Aug 18, 2011 10:46:56.445440000 -03      1.313675e+09     Cisco_db:19:c3   \n",
       "1  Aug 18, 2011 10:47:47.832016000 -03      1.313675e+09  PcsCompu_b5:b7:19   \n",
       "2  Aug 18, 2011 10:47:47.832023000 -03      1.313675e+09      147.32.84.255   \n",
       "3  Aug 18, 2011 10:47:48.820312000 -03      1.313675e+09      147.32.84.255   \n",
       "4  Aug 18, 2011 10:47:48.820320000 -03      1.313675e+09      147.32.84.255   \n",
       "\n",
       "  _ws.col.Destination _ws.col.Protocol  frame.len  \n",
       "0       147.32.84.191              ARP         60  \n",
       "1       147.32.84.191              ARP         60  \n",
       "2           Broadcast              ARP         60  \n",
       "3       147.32.84.191              ARP         60  \n",
       "4       147.32.84.191              ARP         60  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "sample = pd.read_csv(\"../../data/raw/CTU-datasets/sample.csv\", sep=\";\")\n",
    "sample.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1293896-9d1a-4f66-bc53-00214efa7a35",
   "metadata": {},
   "source": [
    "Each line is a network packet exchanged between devices. The column <b>\"_ws.col.Source\"</b> represents the source of the packet and <b>\"_ws.col.Destination\"</b>  its destination. The size of the packet can be found in the column <b>\"frame.len\"</b>. If either the source or the destination is a bot, the packet is considered malicious."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fe4ef7a7-dea1-4066-8bca-ab1cdc7e9e12",
   "metadata": {},
   "outputs": [],
   "source": [
    "bots = [\"147.32.84.191\", \"147.32.84.165\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dbaaabe-cd1b-46e1-9d15-0bfeccaed003",
   "metadata": {},
   "source": [
    "For this analysis, the dataset is split into two segments. The first segment is composed of the initial moments of the capture.  In this segment, the network traffic is low, and most of the bots are exchanging just a few messages. In the second segment, the network traffic rises exponentially, caused by an increase in packets sent by bots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a6d67cac-35cb-46a7-a323-258edcc8ed3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dateutil import parser\n",
    "training_segment_bounds = parser.parse(\"2011-08-18T10:48:00\"), parser.parse(\"2011-08-18T10:50:01\")\n",
    "testing_segment_bounds = parser.parse(\"2011-08-18T10:50:01\"), parser.parse(\"2011-08-18T10:55:00\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a206f8f-b41c-410e-b7ea-f1d364ece203",
   "metadata": {},
   "source": [
    "\n",
    "The dramatic change in network behavior between the two segments is characterized by a change in data distribution. ANTE is used to autonomously detect this change and adapt its classification pipeline. In this analysis, the change is done by simulating a streaming environment. The initial classification pipeline is optimized using the first segment. The second segment is consumed in batches."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73112681-7669-4ffd-92e5-c012f6de920f",
   "metadata": {},
   "source": [
    "## ANTE building block"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55529905-f236-4fc7-82db-6328805f7f02",
   "metadata": {},
   "source": [
    "All data operations used by ANTE are build using the same basic building block, illustrated in the figure below.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b397e58-815a-4a9a-bf6e-7688e6e996bc",
   "metadata": {},
   "source": [
    "<img src=\"images/antecomponent.jpg\" width=\"600\" height=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1419e1a-facc-4dfe-9af4-cc4e101b6bb8",
   "metadata": {},
   "source": [
    "This component is used to represent a classifier, a feature preprocessor, or any component responsible for data transformations.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "206eea20-04f1-4cb9-b30e-6138d031be30",
   "metadata": {},
   "source": [
    "## Creating the feature pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b675eb6-0223-49d0-91e5-d9f0833e0227",
   "metadata": {},
   "source": [
    "\n",
    "The network traffic is classified using a bag of features approach. Firstly, all network packets are grouped using a sliding window.  Secondly,  a vector of features is calculated for every window. Subsequently, the feature vector is used as input of a classifier.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb9f5006-c9b4-40b1-bcc3-e11c8f5ad741",
   "metadata": {},
   "source": [
    "The feature pipeline is illustrated in the figure below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00ab2ece-08ef-43c2-9c77-7c2d85f63266",
   "metadata": {},
   "source": [
    "<img src=\"images/featurepipeline.jpg\" width=\"900\" height=\"900\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b18d4c5d-299d-41bd-8421-1761128c6066",
   "metadata": {},
   "outputs": [],
   "source": [
    "#enable caching for feature pipeline results\n",
    "import prefect\n",
    "prefect.context.config.flows.checkpointing = True\n",
    "#ignore warnings\n",
    "import logging\n",
    "logging.getLogger(\"prefect\").setLevel(logging.ERROR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4122f1d5-553c-494b-873e-ff12abd0685d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ante.pipelines.featurepipeline import FeaturesPipeline\n",
    "window_size = 30\n",
    "window_offset = 5\n",
    "feature_pipeline = FeaturesPipeline()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ac736b0c-058a-442a-a97a-860ddeba8199",
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "run_feature_pipeline = partial(\n",
    "    feature_pipeline.run, \n",
    "    dataset=dataset,\n",
    "    bots=bots,\n",
    "    window_size=window_size,\n",
    "    window_offset=window_offset\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d7896636-a174-442b-b5ab-30103831e5e5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "training_segment_data = run_feature_pipeline(\n",
    "    start_date=training_segment_bounds[0].timestamp(),\n",
    "    end_date=training_segment_bounds[1].timestamp()\n",
    ");\n",
    "\n",
    "#removes pyspark output for brevity\n",
    "from IPython.display import clear_output\n",
    "clear_output()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "79e00a28-4c24-42b8-8240-7e2ec7927ca7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IP                         object\n",
       "Spc                       float64\n",
       "Tss                       float64\n",
       "Smin                      float64\n",
       "Smax                      float64\n",
       "Savg                      float64\n",
       "Svar                      float64\n",
       "SITmin                    float64\n",
       "SITmax                    float64\n",
       "SITavg                    float64\n",
       "SITvar                    float64\n",
       "Rpc                       float64\n",
       "Tsr                       float64\n",
       "Ravg                      float64\n",
       "Rvar                      float64\n",
       "RITmin                    float64\n",
       "RITmax                    float64\n",
       "RITavg                    float64\n",
       "RITvar                    float64\n",
       "in_degree                 float64\n",
       "out_degree                float64\n",
       "centrality                float64\n",
       "eigenvector_centrality    float64\n",
       "window_start               object\n",
       "window_end                 object\n",
       "window_index                int64\n",
       "isBot                       int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#features \n",
    "training_segment_data.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "61038cce-811b-4c45-91e6-8a3b188ae140",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "testing_segment = run_feature_pipeline(\n",
    "    start_date=testing_segment_bounds[0].timestamp(),\n",
    "    end_date=testing_segment_bounds[1].timestamp()\n",
    ")\n",
    "testing_segment[\"window_start\"] = pd.to_datetime(testing_segment['window_start'])\n",
    "\n",
    "#removes pyspark output for brevity\n",
    "clear_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85eadd80-09ac-40da-af04-8cc8e4eccc08",
   "metadata": {},
   "source": [
    "## Creating the classification pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46c733c4-c9a9-41e2-96c7-11bfd2d8e541",
   "metadata": {},
   "source": [
    "Ante autonomously optimizes a classification pipeline using AutoML"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3791880-15d9-4406-89a2-1d4ce74edfff",
   "metadata": {},
   "source": [
    "<img src=\"images/automl.jpg\" width=900 height=900 />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a9cde18e-946b-45cd-a319-aaf131b03b3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ante.pipelines.classificationpipeline import MetaClassificationPipeline\n",
    "meta_classification_pipeline = MetaClassificationPipeline(time_limit=60)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8c9111c-fa17-4be4-aa8b-890c4ee9014e",
   "metadata": {},
   "source": [
    "The feature pipeline and the classification are combined into a single final pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14f3fe54-25fa-47da-8ee3-2fe6f1d4cde4",
   "metadata": {},
   "source": [
    "<img src=\"images/finalpipeline.jpg\" width=900 height=900 />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7733ef3e-3589-45ff-9554-8d87fdc5ec3e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def remove_meta_columns(df):\n",
    "    return df.drop(['IP', 'isBot', 'window_start', 'window_end', 'window_index'], axis=1)\n",
    "\n",
    "training_segment_x, training_segment_y = remove_meta_columns(training_segment_data), training_segment_data[\"isBot\"]\n",
    "classification_pipeline = meta_classification_pipeline.run(training_segment_x, training_segment_y)\n",
    "\n",
    "#removes output for brevity\n",
    "clear_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1547773-15ed-4351-9806-b83a697c51c8",
   "metadata": {},
   "source": [
    "## Adapting to changes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1bd2050-f6f8-4fe5-b357-9f9871472426",
   "metadata": {},
   "source": [
    "The final piece of the ANTE framework is the drift detector. This component is responsible for detecting changes in the data distribution and adapting the classification pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "532e25bf-5dd6-406f-9f63-c09fa18aa79b",
   "metadata": {},
   "source": [
    "<img src=\"images/ante.jpg\" width=900 height=900 />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5df7c859-ec9d-455b-91db-f695a6a63c71",
   "metadata": {},
   "source": [
    "In this analysis, a streaming environment is simulated by feeding batches of data from the testing segment. If the nature of the data observed by the drift detector is different from the data in which the current classification pipeline was optimized, the drift detector triggers a change in the classification pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a1f0ab2a-7297-4a3a-a786-751f3029da3c",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from ante.driftdetection.ensemble import ClusterMetricsEnsembleDriftDetector\n",
    "from ante.util import sliding_window\n",
    "\n",
    "\n",
    "drift_detector = ClusterMetricsEnsembleDriftDetector()\n",
    "drift_detector.set_reference_concept(training_segment_x.copy(deep=True))\n",
    "\n",
    "run_statistics = []\n",
    "\n",
    "for batch_start, batch_end, batch_data in sliding_window(testing_segment, \"window_start\", \"60s\", \"1s\"):\n",
    "    if batch_data.shape[0] > 1:\n",
    "        batch_data_y = batch_data['isBot']\n",
    "        batch_data_x = remove_meta_columns(batch_data).copy().reset_index(drop=True)\n",
    "\n",
    "        drift_detection, essemble_predictions = drift_detector.fit_predict([batch_data_x])\n",
    "        #if there was a change in the data distribution\n",
    "        if drift_detection == 1:\n",
    "            #replace the classification pipeline\n",
    "            classification_pipeline = meta_classification_pipeline.run(batch_data_x, batch_data_y)\n",
    "            #set the new 'normal'\n",
    "            drift_detector.set_reference_concept(batch_data_x.copy(deep=True))\n",
    "    \n",
    "        batch_data_y_hat = classification_pipeline.run(batch_data_x)\n",
    "        accuracy = accuracy_score(batch_data_y, batch_data_y_hat)\n",
    "  \n",
    "        run_statistics.append({\n",
    "            \"drift_detection\": drift_detection,\n",
    "            \"batch_start\": batch_start,\n",
    "            \"batch_end\": batch_end,\n",
    "            \"accuracy\": accuracy,\n",
    "        })\n",
    "run_statistics = pd.DataFrame.from_records(run_statistics)\n",
    "\n",
    "#removes output for brevity\n",
    "clear_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "370677fd-be52-40ca-bdfa-e83807c285b0",
   "metadata": {},
   "source": [
    "In the image below, we can see the work of the drift detector. The dashed squares mark the places where the drift detector was triggered. When the accuracy drops the classification pipeline is replaced, resulting in better accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0550bd75-b020-4980-8e60-e4f073985238",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2EAAAE9CAYAAACC32qaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA+JElEQVR4nO3de3zUZ533//eVMySEJByTcOyRQ6ktSY9Waw+2VoGqrdZ6u+rqAp7W9XZ3PXQftwT3du+f+lN3PRNcrcfWvVurQG1rrdV2bWtLsC0UeoAChSTQcgiQhEwyM9f9RyYQYCbMTGbmmmu+r+fjkQdhMt+ZT+abb3J95nNdn8tYawUAAAAAyI0i1wEAAAAAQJCQhAEAAABADpGEAQAAAEAOkYQBAAAAQA6RhAEAAABADpGEAQAAAEAOlWTjQSdOnGhnzZqVjYcGAAAAgLzX1ta2z1o7Kd7XspKEzZo1S+vXr8/GQwMAAABA3jPG7Ez0NaYjAgAAAEAOkYQBAAAAQA6RhAEAAABADpGEAQAAAEAOkYQBAAAAQA6RhAEAAABADpGEAQAAAEAOkYQBAAAAQA6RhAEAAABADpW4DgAAkDt/eH6v9h4OpX28kXTN3CmaNK48c0EBGNHBnn49uHmvIta6DgXISxWlRXrHhdNch5ESkjAEWkNDgzo7OyVJCxcuVFtbm5YtW6bVq1cfu097e7va2tq0ZMmSY7etWrVKy5YtkzHm2G2LFi3S2rVrtXjxYq1bt+7Y7dZatba2avny5cduW7NmjZqamtTY2HjstqVLl6q1tVVNTU3asGGDJKm+vl4dHR1qaWnRypUrj913/fr1kqTm5uZjt61YsUItLS18T3xPCb+noooqTf+HOzVatzR36cs3nz/qx0nXrFmztHPnTn4W+J4C8z09G2nU6ic6BSC+iVXl3iVhxmbhXZXm5mY79IsMyFctLS1qaWlxHQaQM52Hjuqy//MHfe6GOXr7BYOD1M985jP6yle+kvRjtKx5Tut3HtCTt12roiJz+gOyoKGhQR0dHU6eG3Dhi2s3686nXtEf/vFNrkMB8lKRkSZXV7gO4xTGmDZrbXO8r1EJQ2CtXLmSJAyBEo4MvulWV1mmqeMH/1j9ZNU3U3qMGxZM1f3P7dHTu7u0cEZtxmNMBgkYgqYnFFZVecmx67apqUltbW2OowIwGjTmAICACEcHk7DS4uMVrKamppQe403nTFZxkdFDW/ZmNLZU8OYJgqa7fzAJGzI0rRKAv0jCACAgItGoJKm46Piv/lQHc+PHluqiWbX6/eZXMxpbKoav4QGCoCcUVmU5k5eAQkIShsBi3SKC5lglbJRrua6dO0Uv7D2iXQd6MxEWgNMYTMKKj/2/vr7eYTQAMoEkDAACYmhNWPGwJCydwdy1c6dIkn7vcEoiECTdocgJ0xFZFwn4jyQMgTW8BTEQBEOVsJJha8LSGczNmlipsyZX6aEtbqYkUsVG0Jw8HZF1kYD/SMIAICCG1oSVDFsTlu5g7pq5k/XEy/t1uG8gE6EBGMHJSRjrIgH/kYQBQEAMxKYjlgybjpjuYO7Nc6coHLV65MXXMhJbKqhiI2i6Qyd2RwTgP65oBNaKFStchwDkVCR66pqwdF04o1Z1lWX6t3u36GdP7Ezp2FsvnqEbY5tFAxhZOBJVKBxVZRlDNqCQUAlDYDGnHkEzEIlNRywe/a/+4iKjT117tqbVjVXUKumPba/16BsPvihr7ahjAIKgJxSRpBO6I7IuEvAfb6sgsBoaGugwhUAZqoQNn444msHc+y+bpfdfNiulY+548hV9/lcbtbnzsOY3jE/realiI0i6+8OSxHREoMBQCUNgdXZ2ug4ByKl43RFz7fr5U1VcZHTfxj1pPwZVbARJT2gwCRvemIN1kYD/SMIAICDCxxpzHP/Vn+vBXF1lmS49o06/3diZ9pTEhoaGDEcF5K/uEJUwoBCRhCGwFi5c6DoEIKfCsRb1mWjMMRpvXVCvl/f16Pk9R9I6nio2giReJQyA/0jCEFhtbW2uQwByaqgSVupwOqI0OCWxyEj3bSSZAk7neBJ2vDEH6yIB/5GEIbCWLVvmOgQgp+K1qHcxmJtYVa5LZk/QvWlOSaSKjSDpHuqOOKxFPesiAf+RhCGwVq9e7ToEIKeGGnOUDmtR72ow99YFU7XttR699Gp3ysdSxUaQxJuOyLpIwH8kYQAQEPHWhLkazF1/3lQZI937bOpTEqliI0jiNeZgXSTgP1Z5AkBAHO+OeDwJczWYmzyuQhfNqtN/PPSSvvvHrSkde7BjnFqzFBeQb3pCYRUZqaKU982BQkIShsBqb293HQKQU8c2ay7Oj8HcisXzUq6EPb2rS3/uf6O6evtVM7YsS5EB+aO3P6LK8hIZc/zNE9ZFAv4jCUNgtbW1Ma8egTIQm444vBLmcjA3v2G85jeMT+mYZ3Z16cZt+/Xg5r16V/P0LEUG5I/uUPiUPcJYFwn4Lz/eDgUcWLJkiesQgJyKRE7tjujbYO78aeM1ZVyp7t+0x3UoQE70hMKn7BHGukjAfyRhABAQA9FT14T5NpgzxmhuVb8efWmfjvQNuA4HyLruOEkY3X0B/5GEAUBARKJRFReZE9aW+DiYu/Orn1V/JKqHX3jNdShA1vWEwqoatlEzgMJAEobAWrVqlesQgJwKR+0JVTBfhdq3aNK4ct2/iTbdKHw9ocgJGzUDKAwkYQgs36ZhAaMVjhRGEiZZXT9/ih5+/jUd7Y+4DgbIqniNOejuC/iPJAyBNXxKFhAEkag9oSmH5OdgbtWqVbrhvHodHYjoTy8yJRGFraf/1DVhvjXUAXAqkjAACIiBSFSlJ+0R5uNgbtmyZbpkdp1qxpbqvk2dCkeiKX1Ya11/C0DS4nVHpLsv4D8mGQNAQMSrhC1ZssS7pMQYI2utrps3Rf+1frd+83RHSse/btp4/frjr6cajrwXCkc0ELE05gAKEEkYAmvRokWuQwByKhy1p1TCfPapa8/RzAmVikaTTyJffLVba5/p0PN7jmhufXUWowNGryc0uObx5EoYAP9xVSOw1q5d6zoEIKfCkegplTCfNdSM0cevOiulY147EtK6Zzv0wHN7SMKQ93pCYUmnJmF09wX8VzhviQIpWrx4sesQgJyK16Lex8HcaKrYk8aVq3lmre7ftCeDEQHZ0R1Lwk7ujkh3X8B/JGEIrHXr1rkOAcipSNSqpPjEJMzHwdxoq9jXz5+q5/cc0c79PRmKCMiORJUw1jMC/iMJA4CAGIhYFRed+Gvfx8HcaKvY18+fKkl64DmqYchvxythNOYACg1JGAAERCQaLYjNmkdbxZ5eN1bzG6r1wHN7MxQRkB005gAKF0kYAsu3ttzAaIXjTEcMquvnT1XbzoN69XCf61CAhI5NRyw7MQmjuy/gP5IwBFZra6vrEICcCkdObcwR1MHc0JTE322mGob8lagxB919Af+RhCGwli9f7joEIKciUauSk9aE+TiYy0QV+5wpVZo9sZJ1YchriRpz0N0X8B+TjOGlSNRqS+dhDUSiKR03sapc0+vGZikqIL8NRKMqLz11MOdbItba2jrqro7GGF03f4paH3lZl/7bQykdO71ujH6x9NKC2vga+am7P6yy4iKVlZz4s0Z3X8B/JGHw0l1tu/TZuzemdewbz5mkD18xO8MRAfkvErWnbNbs42Bu+fLlGWmt/4HLZqm7L6xwJPnK2v6ekH6/5VU9tf2ALj9r4qhjAEbSEwprLJ0RgYJEEgYv7T0ckiT98IPNKbXY3rT7kH7yxE594IdPauEX1mrJt/87peedUl2h7/6PhbwDDi8NrgnjZ3dIQ80YfekdC1I65mh/RBf+6+90/3N7SMKQdT2hyClNOQAUBq5seKk7FFZ5SZGunjMlpeOuOneyll95ptY926F71u9QSWlZ0sfuORzSg5v3as+hPqY0wkvhAmlR79KYsmJdec4k/e65vWpZPF9FvJ7Iou5Q+JSmHBLdfYFCQBIGL3WHwhpXkd6Pb1lJkd65cJpuapqe0h+yh7bs1Yd/vF77e/pJwuCleC3qfRzMrVmzxunzv+W8qXrgub16ZneXLpxR6zQWFLaeUFiVcaYjZmJdJAC3mJcCLw3+Ycrtewi1lYNVswM9oZw+L5Ap8VrU+7hVQ1NTk9Pnv/rcKSopMmz2jKzr6Y/E/VtHd1/AfyRh8FJ3X/wpGtk0IZaE7e/uz+nzApky2JjjxF/7Pg7mGhsbnT7/+LGluuzMCbp/U6eXlUT4oyfBdEQA/iMJg5e6M1AJW7p0aUr3r4slYQd7ScLgp3A0qtJi1jBlwvXzp2rH/l69uLfbdSgoYC5mfQDIDZIweKmnf/TvDqY6DauqvERlxUXa30MSBj+FI6e2qEd6rps3RcaIzZ6RVYkac7heFwlg9EjC4KVMTEdMdV2JMUZ1lWU6wHREeCocPXVNmI+DuVSr2NkwubpCC2fU6v5NJGHIDmttwsYcrtdFAhg9atzwUnco/mLlVGzYsCHlY2ory3SAShg8FY5EVXLSHnc+DubypZnI9fOn6N9++7xu/t5jSmG7Qo0fU6b/eM8FTDPDiPoGoopaxf05aWxsZD0i4DkqYfBSd2hAVXHeHcy2CZVlOsCaMHgqXiXMdZOLdORL4viOC6fpmjmTVVZSpNLi5D4k6fdb9ureZzsdR4981x0KSxKNOYACxZUN74QjUfUNRFVVXjqqx6mvr0/5mLrKMu062Duq5wVcicTZJ8xH6VSxs2HSuHL95wcvSukYa62u/tqfdNeG3Xr3RdOzFBkKQU8sCassY6gGFCIqYfBOT39EkuLOk09FR0dHysewJgy+stYqHKdFPXLLGKObFjbqye0HtOsAb+ggsaFKWLzpiPmwLhLA6PDXGN7J1BSNlpaWlI+pqyzTkVBYoXBkVM8N5FokOrh+5OTpiD4O5tKpYueTdyycJmOkuzfsdh0K8ljPCH/r8mVdJID0kYTBO8f+MFWMLglbuXJlyscc2yusZ2BUzw3kWngoCTtpOqKPg7l0qtj5pLFmjC47Y4Lu3rBb0SjNFRBfT/9QJYzuiEAhIgmDd0aaopFtE2JJGB0S4ZtwgkqYj4O5dKrY+ebmpmnadeContpxwHUoyFPdocEZF/EqYfmyLhJA+kjC4J3uvsEkbJyDJKyOJAyeikSGkrATf+37OJhLp4qdb95y3lRVlhUzJREJ9Th8wxFA9nFlwzuZ+sO0fv36lI+ZUDWYhO3vCY3quYFcG4hGJZ06HRFujC0r0Q0L6vXbjXv0yWvOVllJ8u+JlpcUa/yY0XWHRf4b6W+d7+siAZCEwUMu906pHUslDH4aasxRfNJ0RAZz7tzcNE13te3WFV9+OKXjjJHu/ujlWjijNkuRIR8cm3pfduqaMN/XRQIgCYOHMpWENTc3y9rUFsXXjC2TMdJBkjB4ZmhNWOlJ0xF9HMylU8XOR5fMrtN33rtQB1PYAN5K+tK9m/Wbv7aThBW4nlBYFaVFKik+tUra0tJSEGsjgSAjCYN3XM6TLy4yqh1bpv0kYfBMODI4HfHkShiDOXeMMXrb+alXIv/80j7d/9werVg8X0VFTC8tVN2hSMI3G1euXMl1C3iOxhzwzpFQWGXFRSmtocikusoypiPCO4la1PvY5KK5udl1CE7dsGCq9h4O6a+7DroOBVnUEwrTlAMoYFzd8E5PKDzqPcIkacWKFWkdV0clDB4KJ+iOCP9cPWeyyoqL9NuNe9Q0s851OBiF7lBYt/1qow73nbr35Kb2Q5o8rsJBVABygb/G8E5PKBJ388pUpTuVo66yjDVh8E44Gn86IvwzrqJUbzh7ou7ftCflda3IL//11C6teaZD+7v7dbDnxI/GmjG68YKGuMcVyrpIIMiohME7R/rCqiwb/Y9uQ0NDWk0J6qrK9NQOkjD4Zag7YulJ0xF9HMylW8UuJG85b6oeev5VPbv7kF43vcZ1OEhDNGr10yd26sIZNbrnY693HQ6AHKMSBu/0hMIal4HpiJ2dnWkdN6GyTAd7+xWN8g40/DEQid+i3kc0JJDePG+KSoqM7tu0x3UoSNMjL72m7ft69MHLZ6V8bNDXRQKFgCQM3unpd7tYua6yTFErdR09dQ4/kK+GKmEnrwnzcTDX0BB/ilaQ1Iwt02VnTtB9mzqZkuipHz+2Q5PGleuG89irDwgipiPCO919YU2vGzvqx1m4cGFax9VVHt+weehzIN8NrQk7uTuij9KtYheaty6o1+d/tVG/3bhHjbVjkj6upMhoztRxcfefQm7s2NejP774mj559dnOOv0CcIskDN7pDoU1LgOVsLa2trSOG56EAb443h3R/yQMg66bN0Vf+M0mffwXG1I+9sxJlfqn687VW86bKmP4mci1nz6xU8XG6L2XzEjreNZFAv4jCYN3MrV3yrJly9Ta2prycceTsNCoYwByZWg64slrwnwczKVbxS40E6rK9ZuPX6G9h/tSOm5/T7++/6dt+ujPN+j8aeN15TmTlOs0bFJ1hebVj9O5U6sTbkjsiwM9/ertDyd9/4GI1X+t36UbFtRrSnV6LehZFwn4z+/ffAicaNSqpz+SkT/aq1evTisJm1BZLknsFQavDEQGpyOWnjQFzcfBXLpV7EI0r6Fa8xqqUz7uHRc26lcbduubf3hJ3354axYiS+zkJWw1Y0tTTgKbZtbp/7tpgSZWlWcsrnT8fvNefeRnbcc2Q0/FBy+fmfbzptvdF0D+IAmDV3pi7za6fOe0trJUktgrDF5JVAnzcTCXbhUbxxUXGb2rebre1Tw9589trVXHoT5t6TisLZ2H9Vp3arMK+sNR/eqv7XrbNx/Vt25dqItnu9mweiAS1Zd+u0UzJ4zV8ivPTOnYCZVlo9pom3WRgP9IwuCVnlBEkpx2RywvKVZVeQmVMHglnGCfMB8Hc+lWsZEfjDFqrBmjxpoxunbelLQe428um6mP/3yDbl39hP7m0pmaWJVak6T5jeN11bmT03ruIXc+tUvb9/XoPz/QrGvmpvd9AAgukjB4pTs02Ba+KgP7hLW3t6d9bF1lGY054JWh7ojFRXRig//mN4zXmr+/Qrf9aqNuf2xHWo/xf965QLdenF5jjJ5QWP/x+5d08ew6XT1ndMlcOlgXCfiPJAxe6Y5VwqrKi0f9WG1tbWnvN0QSBt8k6o7IYA6+qq4o1bffu1DfuCWa0nEDkag+/vMNuu2ejSqJTctM1epHX9a+7pBWv7/JSXdJ1kUC/iMJg1e6+wbXhFWWjf5Hd8mSJWlvcjqhskx7UuxIBrg0NB3x5H3CfBzMjaaKjcJzcrOZZO7/vfc1aelP1uszdz+r/khUC2fUJn18b39ErY+8rBvOm6oLUzguk1gXCfiPJAxe6Q7FGnNkYDriaNRWlmlz52GnMQCpCCdozOHjYG40VWxAkipKi9X6N8360O1P6V/u2ZTy8SVFRv98/blZiCw5rIsE/EcSBq/0hNx3R5QGK2H7e/plrWWjU3ghMtSi/qQ1YT4O5kZTxQaGjCkr1o/+9iL9eeu+Y1s4JOvMSVU6Y1JVliIDEAQkYfDKUCUsE90RV61alfaxdZVl6g9HM7ZnGZBtxyphxbxpAAypKC2msyEAJ2iTBa90Z7AStmzZsrSPrascbIfMXmHwxbE1YUUkYYDvWBcJ+I+38OGVnlBYJUVG5SWjf//AGJN+Y47YnjR7Dvdp0rjypI8rKTIqSXEROZAJkWNJ2Ik/fz4O5kZTxQYKAesiAf+RhMEr3aGwqipKnK/Dmlg1mHi96/uPp3RcRWmRfv/pKzWtdmw2wgISGlrzcnIlzMfB3Giq2EAhYF0k4D+SMHilOxTOSHv60TqvYbz+99vP0+G+gaSPefVwSLc/tkNbOo+QhCHnIlErY6Sik5IwHwdzo6liAwCQD9yPZoEU9ITCGWuEsWjRorSPLSoyet+lM1M65rUjg0lY+8HetJ8XSNdAxJ7SGREAALjBX2R4ZWg6YiasXbs2I4+TrIlVZSovKVJ719GcPi8gSZFo9JQ9wgD4iXWRgP9IwuCV7lAkI+3pJWnx4sUZeZxkGWPUWDOGJAxOhKNWJXHa0/s4mBtNFRsoBKyLBPxHEgavdPcNqKq8OCOPtW7duow8Tioaa8eo/SBJGHIvHLFx29P7OJjLdRUbyDeum1MBGD2SMHilJ+T35shUwuBKOGpVHGdNmI+DuVxXsQEAyDSSMHilJxTO2HREFxprxmhfd7/6BiKuQ0HAhCNRlcaZjugjF1VsAAAyiSQM3rDWqrs/c90RXbS4bqwdI0lUw5BzkailMQdQIFgXCfiPJAze6O2PyFplLAlrbW3NyOOkorEmloSxLgw5Fo5alRaf+iufwRzgH9ZFAv4jCYM3ekJhScrYdMTly5dn5HFSQSUMroQTtKj3cTDHRs0IOtZFAv4jCYM3jsSSsHEZ2ifMhanVFSouMlTCkHOJuiP6OJhzUcUG8gnrIgH/kYTBG8cqYWX+JmElxUWaWl1BJQw5F0mwT5iPgzkXVWwAADKJJAze6M7wdMQ1a9Zk5HFS1VjDXmHIvYEELeoBAEDu8RcZ3ujuy+x0xKampow8Tqoaa9krDLkXiUbjTkcE4B/WRQL+IwmDN3r6M1sJa2xszMjjpPy8NWO053CfwpGok+dHMA0kWBPm42DOVRUbyBesiwT8RxIGbwxVwirLix1HMjqNtWMUiVrtOdznOhQESKI1YT4O5lxVsYF8wbpIwH8kYfBGdygiSRpXXuo4ktFhrzC4EI5alcRZE+bjYM5VFRsAgEwhCYM3ekJhFRmpojQzP7ZLly7NyOOkir3C4EI4wpowAADyBUkYvNEdCquyvETGZGYg6WoaFpUwuBCJ2ribNQPwD+siAf/5u+ESvNbRdVT3bdqTUlOAp3d1aVyGmnJIg+tK2traMvZ4yaooLdbEqjIqYcipgUhUpcWnvu/m42DOVRUbyBesiwT8d9oRrTFmsaR7rbW0ckPG/ODR7frhn7enfNwbzp6YsRg2bNiQscdKVUMNbeqRW4kqYT4O5nxsJgJkUmNjo5edTQEcl0xZ4RZJ/26MuVvSD621z2c5JgTA/p6QpteN0b2ffENKx1WWFUbxtrFmjF7Ye8R1GAiQcILuiD4O5lxVsQEAyJTTjmitte8zxlRLulXS7cYYK+lHku6w1jKKRFoO9g6orrJc1RXuOh3W19c7e+7GmjF6+IVXZa3N2Bo3YCThBPuE+chlFRsAgExIqjGHtfawpLsk3SmpXtI7JG0wxvx9FmNDAevq7VftWLet5js6Opw9d2PtGPUNRLW/p99ZDAiWcNSqOE6LegD+YV0k4L/T/kU2xiwxxtwj6Y+SSiVdbK29QdLrJP1jdsNDoerqHVDNGLdJWEtLi7PnpkMici0Sjao0znREHwdzLqvYQD5gXSTgv2QW2Nwk6RvW2keG32it7TXGfDg7YaHQHeztV83YMqcxrFy50lkiNrRX2K827NYLe5Kf1VtWUqQbFkxVeUlxtkJDgQpH4jfm8HEw57KKDeQD1kUC/ksmCWuR1Dn0H2PMGElTrLU7rLUPZSswFK5wJKojfWHVOk7CXJo5oVKVZcX68eM7Uz62q3eePvj62VmICoUsHI2/JszHwVxLS4vTSjbgGusiAf8lk4T9X0mXD/t/JHbbRVmJCAWv6+iAJKm20u10RJeqykv0+G3X6EhfOKXjlv54ve7e0E4ShpSFo1GVxNknzMfBnMsqNgAAmZBMElZirT3WPcBa22+MCW4JA6PW1TuYhI13vCZs/fr1Tp+/uqI05e6QNzdN0xfXbdYLe47o3KnjshQZClGiShgA/7AuEvBfMq2yXjPGLBn6jzHmRkn7shcSCl1X72BOH+TpiOm68YIGlRQZ3b1ht+tQ4JFo1MpaqSROd0QGc4B/WBcJ+C+ZJOwjkm4zxrxijNkl6bOSlmc3LBSyg7FKmOskrLm52enzp2NCVbmumjNZ9/y1XeFI1HU48MRAdPBnJd5mzT4O5lxXsQHXmI4L+O+0SZi1dpu19lJJ8yTNtdZebq3dmv3QUKiGKmE1jvcJ89XNTdP02pGQHn2JgjSSE4laSYrbHZHBHOCflStXug4BwCgltXOnMeZtkj4m6dPGmC8YY76Q3bBQyIbWhJGEpeeqcyerdmyp7mJKIpIUjiVh8daE+TiY87GKDQDAcMls1vx9SbdI+ntJRtK7JM3MclwoYAd7+1VSZFRVnkxfmOxZsWKF0+dPV1lJkW68oFEPPrdXh2IJLTCScCRxEgYAAHIvmVHw5dba840xz1prVxpjvibpvmwHhsJ1sHdANWNLZYzbAaHP07Bubpqm2x/boVtXP5FSq/+y4iL943Xn6rzG8VmMDvkmfGxNWFKTHwDkOdZFAv5L5i9yX+zfXmNMg6QBSbTTQtoOHe1XTR50RmxoaHAdQtrmN1TrvZfM0NiyYoUGokl/bHilS/9817OKxqanIRhGqoT5OJjztYoNAMCQZCpha40xNZK+KmmDJCtpdTaDQmE72DOg2jxYD9bZ2ek6hLQZY/Rv71iQ8nG/ebpd/3Dn07rnr+26qWlaFiJDPhqpMYePfK5iA5nQ3Nwsa3kzDfDZiJUwY0yRpIestV3W2rs1uBZsjrWWxhxI28He/KiEBdHi8xu0oHG8vva7F9Q3EHEdDnJkqDFHaZzpiD42ufC5ig0AgHSaJMxaG5X0nWH/D1lrD2U9KhS0Q0cHVDPGfSVs4cKFrkPIuaIio8+/dY46DvXp9sd2uA4HOTK0p1yhVMJ8rmIDACAltybsIWPMTcZ1FwUUjIO9/aqtdF8Ja2trcx2CE5efOVFXz5ms7zy8VQd7+l2HgxwYqUU9AP+wLhLwXzJrwpZL+rSksDGmT4Nt6q21tjqrkaEg9Q1E1DcQzYs9wpYtW6bW1lbXYTjx2bfM0Q3/8Yiu+/dHUtoqwBhp+RvP0C0XzchidMi0Y4054kxH9HEwF8QqNjAc6yIB/5129GWtHZeLQBAMB3sHKy81Y9xXwlavXh3YJOzcqeP0lZtfp0defC2l47bv69Ft92zSmZOq1DyrLkvRIdOOtaiPUwnzcTAX1Co2MKShoUEdHR2uwwAwCqdNwowxb4x3u7X2kcyHg0LXFdtcOB+6IwbdzU3TdHOKHRIP9w1o8bf+W5/4xV917yev0ISq8ixFh0wa6o5YUnxqEubjYC7IVWxAYl0kUAiSWRP2z8M+/pektZJashgTCtixShjdEb1UXVGq77x3oQ709ut//tcz7DfmiYFI4hb1Pg7mVq9mlxQAgN+SmY64ePj/jTHTJf17tgJCYRuqhOXDmrD29nbXIXjpvMbxalk8X7fds1FLf7JeU8dXpHT8tXOn6Ko5k7MUHeI5VgkrSuZ9NwD5jnWRgP+SX5F/3G5JczMdCILh+HRE95WwtrY29htK060XT9fWV7u15pl2PbM7+eNCA1Hd8eQr+sYtF+jGCxqzFyBOcGxNWJzpiAzmAP+wLhLwXzJrwr4laWjOUZGkCyRtyGJMKGDHpyO6r4QtWbJE1jKdLh3GGH1h8Tx9YfG8lI7r7Q/rQ7c/pf/5y6cliUQsR451R4wzHdHHwRxVbAQd6yIB/yVTCVs/7POwpDustX/OUjwocF29/RpTWqyK0mLXocCBsWUl+uEHLzqWiO0+eFT1KU5nnFtfrbn17JCRivAI0xF9HMxRxUbQBbm7L1AokknC7pLUZ62NSJIxptgYM9Za25vd0FCIDvYO5EUVDO4MJWIfvn29vvrACykfX1xkdNtb5+pDr58l9pBPzkjTEX0czFHFBgD4Lpkk7CFJ10rqjv1/jKTfSbo8W0GhcHX1DuRNZ8RVq1a5DiGwxpaV6Od/d4l2Hzwqq+QH0+Go1Zfve17/um6znt7VpS/ftEBjy9JZ2hosQ4054nVHBAAAuZfM6KXCWjuUgMla222MGZvFmFDAunr782aPsGXLlrkOIdCKioxmTEj9V8n339ek7/1pm772uxf0+Lb9Kf88LZg2Xl9/9wUpP6/PhtaEldIdESgIrIsE/JdMEtZjjFlord0gScaYJklHsxsWCtXB3n6dO3Wc6zAkDTaXYEqTf4qKjD5+1Vm6YHqN7njyFUVTOIevHOjVrza06wuL5uVNRTYXhqYjFseZjujjYI4qNoKOdZGA/5JJwj4l6f8aYzokGUlTJd2SzaBQuPJpOiL89vqzJur1Z01M6ZjHtu3Te1f/Rc/sPqQrz5mUpcjyz/HGHPG7I/o2mKOKjaBjXSTgv2Q2a37KGDNH0rmxm16w1g5kNywUImutuo4O5M10RATPgsbxMkZ6ZldXsJKwEVrU+ziYo4oNAPDdaRcIGGM+LqnSWrvJWrtJUpUx5mPZDw2F5kgorEjUqmZMflTCFi1a5DoE5Ni4ilKdNalKz+zqch1KTo3Uoh4AAOReMn+Rl1pru4b+Y609KGlp1iJCwerqGSyg5kuL+rVr17oOAQ68bnqNntndFahKSmSEFvUA/MO6SMB/ySRhxWbYZjzGmGJJ+VHKgFe6jvZLkmrzZE3Y4sWLXYcAB143vUb7uvvV3hWc/kIDkcQt6n0czFHFRtCxLhLwXzJJ2P2SfmmMucYYc42kOyTdl92wUIgO9uZXJWzdunWuQ4ADF0yrkSQ9s+uQ20ByKDJCYw4fB3NUsRF0bFQP+C+ZJOyzkv4g6SOxj40a3LAZSElX72AljO6IcOncqeNUVlKkZ3Z3uQ4lZ8IjbNbs42COKjYAwHenTcKstVFJf5G0Q9LFkq6WtCW7YaEQHewZmo6YH5UwBFNZSZHmN1Tr6QA15whHoiopMl4mXPFQxQYA+C5hi3pjzDmSbo197JP0S0my1l6Vm9BQaLqODk5HHD8mP5KwIDVmwIleN61Gv3xq12ByUlz4HQMjURu3CgbAT6yLBPw30ujjeQ1WvRZZa6+w1n5LUiQ3YaEQdfUOaFxFSd4MeltbW12HAEcumF6jowMRbX2t23UoOTEQsSpNcN0xmAP8w7pIwH8jjYbfKalT0sPGmNWxphy8lYq0Heztz5vOiJK0fPly1yHAkddNr5GkwOwXFolGE1bCfBzMUcVG0LEuEvBfwumI1tpfS/q1MaZS0o2SPiVpsjHme5Lusdb+LicRIi+9erhPG17pSumYba91sx4MeWHWhLGqrijR07sO6ZaLXEeTfeGoVWmCPcIWL17sXSLW2trqZVdHIFNYFwn4L2ESNsRa2yPpF5J+YYyplfQuDXZMJAkLsNvu2aTfb9mb8nFvW1CfhWiA1BhjBjdtDkglLBxJvCbMx8Hc8uXLScIAAF47bRI2nLX2oKTW2AcCylqrp3d16fr5U/QP15yT0rGzJ1ZmKarUrVmzxnUIcOiC6TX67h+36X+v26xUmgY21ozRB18/O3uBZUE4alVSlB9rMQEAQIpJGCBJew+HtK87pMvPnKh5DdWuw0lbU1OT6xDg0FVzJuunT+zUL558JeljwlGr/nBUb54/VY01/myXGI5GVZJgOiIA/7AuEvAfSRhS9mxsk9vzGse7DWSUGhsb+UMWYAtn1OrpL1yX0jFtOw/opu89ruc7D3uWhCWejujjNUAVG0HHukjAf8xPQco2tR9SkZHm1ftbBQPSce7UwZ/5LZ2HHUeSmkjEqjTBdEQft2qgio2go7sv4D+SMKRsY/shnT15nMaUFbsOBcipqvISzagbqy2dR1yHkpLwCC3qfRzMNTY2ug4BAIBRIQlDSqy12th+SAum+T0VUZKWLl3qOgR4aM7Ucdqyx69KWDhqWRMGAEAeIQlDSvYc7tO+7n4t8Hw9mOTnNCy4N7e+Wjv29ehof8R1KEmLRK1KElTCAPiHdZGA/0jCkJKNuw9J8r8ph8S6EqRnbn21olZ6ca8/UxIHItGELep9HMxRxUbQ8fcL8B9JGFKysYCacmzYsMF1CPDQ3PpxkvxqzhEZoTuij4M5qtgIOtZFAv4jCUNKNrYf0jlTaMqB4JpeO1aVZcV6fo9PlbDEa8J8HMz5mDgCADAcSRiSZq3VpvZDBTEVUZLq6+tdhwAPFRUZnTt1nDZ7VgkrpDVhVLEBAL4jCUPSOg8VTlMOSero6HAdAjw1t75aWzoPe7PR8WB3RH7dA4WCdZGA//irjKRtbC+cphyS1NLS4joEeGpOfbWO9IXVcajPdShJCUeiCSthPg7mqGIj6FgXCfiPJAxJ29R+SMVFpiCackjSypUrXYcAT80bas7R4ceUxJEac/g4mKOKjaBjXSTgvxLXAcCN/d0h7djfm9IxT7y8X2dPrqIpBwLv3KmDb0Q8v+ewrp03xXE0pzcQjao0wXTEpqYmtbW15Tii0WlpaaGSjUBjXSTgP5KwALLW6pbWJ7T11e6Uj7314ulZiAjwS1V5iWbUjdWWTj86JEYiiSthPg7mVq5cSRIGAPAaSVgAPb5tv7a+2q1PXnO2mmbWJn2ckXTBjJqsxZVr69evdx0CPDa3fpy27PFjOmI4alWaoEU9AP+wLhLwH0lYAP3sLztVM7ZUH3vTmaooZWohkI45U6v14Oa9OtofyfspuuER1oQxmAP8w7pIwH8kYQHz6uE+/e65vfrb188KfALW3NzsTYtx5J+59dWKWult33pUZSm0f68eU6rV72/W+DGlWYzuRIPdEePH6ONgjio2go51kYD/6I4YML98apfCUav3XjLTdSiA115/1gS988JGnTWpSjPqxib1MWlcuZ7cfkB/3rovp7GOtFkzAznAP3T3BfxHJSxAIlGrO558RVecNVGzJ1a6Dgfw2riKUn39lgtSOqY/HNX5Kx/Qk9sP6K0LcjcNcCBqVZxgTZiPTS6oYgMAfEclLEAefv5VdRzq0/suneE6lLywYsUK1yEgYMpKinTh9Fqt33kgp887UiUMAADkHpUwT219tVsv7EmtPfbtj23X5HHlumZu/u9rlAu+vfuPwnDRrFp9++GtOtI3oHEV2V8XZq2NJWG85wYUCtZFAv4jCfPQrgO9ets3H1UoHE352E+/+ZyEm7YGTUNDg5dNCeC3i2bXKfoHacMrXbrynElZf75wdHDaXqJKmI+DOarYAADfkYR56F/XbVZxkdHdH70spXfSi4w0e2JVFiPzS2dnp+sQEEALZ9SquMho/Y4DOUnCIkNJWAG9+UIVG0HHukjAfyRhnvnjC6/qd5v36rNvmaOmmXWuwwGQosryEs1vqNaT23OzLmwgMlgxT1QJ83EwRxUbAOC7wnlrNABC4YhWrt2sMyZW6kNXzHIdjvcWLlzoOgQEVPPMOj29q0uhcCTrzzVUCUu0WbOPqGIDAHxHEuaR//zv7dq+r0crlsxXeUmwN1rOhLa2NtchIKAunl2rUDiqTe2Hs/5cQ2vCShO0qAfgH9ZFAv5jOqIDPaGwPnT7U3p5X09Kxx3s6deb503JyTqSIFi2bJlaW1tdh4EAap41OJX4qR0H1DSzNqvPFY4MVcLiv+fm42COKjaCjnWRgP9Iwhz4waPb9ZftB3TTwmkqK0m+GDm2rFjLrzwji5EFy+rVq0nC4MTEqnKdMalST20/oI9ceWZWnyscja0JS1AJ83EwRxUbQce6SMB/JGE5tr87pNZHtukt86fqa+9+netwADhy0cw63f/cHkWjVkVZXK81VAlL1JjDx8EcVWwEHesiAf+RhOXYtx/eqqMDEf3T9ee6DgWAQxfNrtMv1+/Suo2daqypSPq48pJizW+oljHJJW7h0zTm8HEwRxUbAOA7krAc2nWgVz97Yqfe3TxdZ01mvy7X2tvbXYeAALtkdp2MkT55x19TPvY7712ot51fn9R9I8cac9CHCSgUrIsE/EcSlkNff/BFFRmjT117jutQoMF1JQ0NDa7DQEBNrxurtZ+4Qgd6+lM67rZ7Nurnf9mZdBI2tE9YokoYgznAP6yLBPxHEpaGgz39unX1Ezp0dCCl4zoP9Wn5lWdo6vjkpx4he5YsWeLdJrUoLOc1jk/5mFsvnqGvPvCCtu/r0eyJlae9/1AlLNGaMB8Hc1SxEXSsiwT8RxKWhkdeek3P7zmit51fr8qy5Pfrqq0s0yeuOiuLkQEodO9qmqavP/ii7nzqFX3+hrmnvf/x7ojxpyP6OJijio2gY10k4D+SsDQ8vm2/qitK9M33XJhwig8AZMPk6gpdO3ey7lq/W//45nNPu83F6boj+jiYo4oNAPAdK7XT8Ni2/brkjAkkYJ5btWqV6xCAtNx68Qzt7+nXg5v3nva+p5uOCAAAco8kLEW7D/bqlQO9uvzMCa5DwSgtW7bMdQhAWt5w9iQ11ozRHU++ctr7DgwlYQk2awbgH9ZFAv4LzHTESNTqhT1HNK+helSP8/i2/ZKky8+cmImw4JAxhilN8FJxkdF7Lpqurz34op7acUCTx5UnvG9n19HYMfHfc/NxMEcVG0HHukjAf4FJwr507xbd+dQreuxzV6tmbFnaj/P4tv2aUFmmc6awzxcAd9590XT9+0Mv6V3ffzyp+ydqIuTjYI4qNoKOdZGA/wKThL37omn64Z+368eP7dQ/XHt2Wo9hrdVj2/br0jMnyBim9gBwZ0p1he5cdql2Heg97X2rK0oTbhDv42COKjYAwHeBScLmTK3WtXMn60ePbdffvWG2KstT/9Z37O/VnsN9rAcrEIsWLXIdAjAqF82q00Wz6lyHAQAAUhSoxhwffdNZ6uodSGoxezyPbdsnSbrsDJKwQrB27VrXIQAAkDLWRQL+C1QS1jSzVpeeUacfPLpdoXAk5eMf27ZfU6srNHtiZRaiQ64tXrzYdQiAcz4O5qhiI+hYFwn4L1BJmCR97E1nac/hPv36r6l1BLPW6olt+3U568EKxrp161yHADjn42COKjaCjnEI4L/ArAkb8oazJ+q8xmp974/bVF1RmvRx+7pD2t/Tr0tZDwaggPjY5GLx4sUkYgAArwUuCTPG6O+vPlvLf9qmj/58Q0rHlhQZXXEW+4MBgEtUsQEAvgtcEiZJ18+fqof/6U0prwsbP6ZU9ePHZCkq5Jpv7/4DACCxLhIoBIFMwiTRXANqbW31cj0MkEkM5gD/MB0X8F/gGnMAQ5YvX+46BMA5HwdzVLERdHT3BfxHEgYAAebjYK61tdV1CIBTrIsE/Gey8Y5ic3OzXb9+fcYfF8gkH7vCAZnm43XgY8xAJnENAH4wxrRZa5vjfY1KGAJrzZo1rkMAAABAAJGEIbCamppchwAAQMqoggH+IwlDYDU2NroOAXDOx8EcVWwEHesiAf+RhAFAgPk4mKOKjaCjuy/gP5IwAAgwHwdzVLEBAL4jCUNgLV261HUIAAAACCCSMASWj9OwAABgXSTgP5IwBBbrSgA/B3NUsRF0/P0C/MdmzQgsNrsEpI6ODjU0NLgOA0AK+PsF+IHNmgEAcfnY5IIqAADAdyRhCKz6+nrXIQBIw4YNG1yHAADAqJCEIbA6OjpchwAAQMpYFwn4jyQMgdXS0uI6BMA5HwdzVLERdHT3BfxHYw4EFgubAQA+ampqUltbm+swAJwGjTkAAHH52OSCKjaCjnWRgP+ohCGwqIQBfl4HPsYMZBLXAOAHKmFAHLxRAADwEesiAf+RhAFAgDGYA/xDd1/AfyRhCKzm5rjVYSBQfBzMUcVG0LEuEvAfSRgABBiDOcA/K1eudB0CgFGiMQcCi4XNgJ/XgY8xA5nENQD4gcYcQBwrVqxwHQIAAAACiCQMgcU0LACAj5htBPiPJAyB1dDQ4DoEwDkfB3NUsQEAviMJQ2B1dna6DgFAGqhiI+jo7gv4jyQMAALMx8EcVWwAgO9IwhBYCxcudB0CgDRQxQYA+I4kDIHV1tbmOgQAAFLGukjAfyRhCKxly5a5DgFwzsfBHFVsBB3rIgH/sVkzAovNLgEAPmpoaFBHR4frMACcBps1AwDi8rHJBVVsBB3rIgH/kYQBQID5OJhbvXq16xAAABgVkjAEVnt7u+sQAABIGesiAf+RhCGw6I4IMJgDfMTfL8B/JGEIrCVLlrgOAXDOx8EcVWwEHesiAf+RhAFAgPk4mPMxcQQyiXWRgP9oUY/AokU94Od14GPMQCZxDQB+oEU9EMeqVatchwAAAIAAIglDYPk4DQsAANZFAv4jCUNgGWNchwA45+Ngjio2go51kYD/SMIAIMB8HMxRxUbQ0d0X8B9JGAAEmI+DOarYAADfkYQhsBYtWuQ6BAAAAAQQSRgCa+3ata5DAAAgZayLBPxHEobAWrx4sesQAOd8HMxRxUbQsS4S8B+bNSOw2OwSAOAj/n4BfmCzZiABY8yxj7a2NrW1tZ1wW0tLiySpoaHh2G1NTU2SBt+JHH7fjo4OrV279oTbWltbT3meoQrc4sWLT7hdklpbW0+4be3atero6DjhtqF3QJuamo7d1tDQIElqaWnhe+J7Sut78snJ8efD61eIPxN8T/n7Pc2cOTONKwdAPqESBgAAAAAZZqiEAQAAAEB+IAkDAAAAgBwiCQMAAACAHCIJAwAAAIAcIgkDAAAAgBwiCQMAAACAHCIJAwAAAIAcIgkDAAAAgBwiCQMAAACAHCIJAwAAAIAcIgkDAAAAgBwy1trMP6gxr0namfEHTs9ESftcBxFwnAO3eP3d4xy4xzlwj3PgFq+/e5wD93J9DmZaayfF+0JWkrB8YoxZb61tdh1HkHEO3OL1d49z4B7nwD3OgVu8/u5xDtzLp3PAdEQAAAAAyCGSMAAAAADIoSAkYa2uAwDnwDFef/c4B+5xDtzjHLjF6+8e58C9vDkHBb8mDAAAAADySRAqYQAAAACQN/IqCTPG/NAY86oxZtOw2+qMMQ8aY16K/Vub4NiIMebp2MeaYbd/whiz1RhjjTETR3juD8Se4yVjzAeG3d5kjNkYe4xvGmNMpr7ffOTqHBhjZhpjNsSOfc4Y85FhXwvMOcjS6/9zY8wLxphNsccvTXA814DcnQOugeOydA7+0xjzjDHmWWPMXcaYqgTHfz72Or9gjLl+2O1vid221RjzuUx+v/nI1Tkwxlw87NhnjDHvGPa1wJyDbLz+w77+TWNM9wjPzTUgd+eAa+C4LP0eut0Ys33Y1y5IcHz2x0TW2rz5kPRGSQslbRp221ckfS72+eckfTnBsd0Jbr9Q0ixJOyRNTHCfOkkvx/6tjX1eG/vak5IulWQk3SfpBtevU4GegzJJ5bHPq2L3bQjaOcjS6//W2GtnJN0h6aNx7sM14P4ccA1k9xxUD/v860OPddJ95kl6RlK5pNmStkkqjn1sk3RG7Dw9I2me69epQM/BWEklsc/rJb0qqSRo5yAbr3/sa82SfjrCOeIacH8OuAayeA4k3S7p5tM8b07GRHlVCbPWPiLpwEk33yjpx7HPfyzp7Sk+5l+ttTtOc7frJT1orT1grT0o6UFJbzHG1Gvwj8YTdvCV/0mqz+8bV+fAWttvrQ3F/luuWJU2aOcgS6//b22MBn95TItzN66BGFfngGvguCydg8OSFHvXcoykeAuib5R0p7U2ZK3dLmmrpItjH1uttS9ba/sl3Rm7b8FydQ6stb3W2nDsvxXD7hOoc5CN198YUyzpq5I+M8LduAZiXJ0DroHjsnEOkpSTMVFeJWEJTLHWdsY+3yNpiiQZY5qNMT8Ydr8KY8x6Y8wTxpi3n+5BTzq+UdKuYV/eHbutMfb5ybcHTS7OgYwx040xz2rwXHzZWtshzoGUodffDE6B+xtJ98c5nmtgZLk4B1wDIxv1OTDG/Ch27BxJ34rdtsQY88XYXUa6DuLdHjS5OAcyxlxijHlO0kZJH4kNSDkHo3/9PyFpzbDHUOx4roHk5eIccA2MLBN/j79kBqdFf8MYUx7n+JyMiUrSPdAFa601xtjY5+sl/d2wL8+01rYbY86Q9AdjzEZr7bYRHuvk45GEbJ4Da+0uSecbYxok/doYc1d2vgt/jfL1/66kR6y1jyY4HknI5jngGkhOuufAWvu3sXeivyXpFkk/staukXTKmg2MLJvnwFr7F0nzjTFzJf3YGHNfjr4tb6T6+ks6Kuldkt4U57G4BtKQzXPANZCcNH8PfV6DyVuZBtvVf1bSF12MiXyohO2Nlf+GpuW8Gu9O1tr22L8vS/qjBtchJatd0vRh/58Wu61dJ04bGro9aHJxDoY/ToekTZLeIM6BlIHX3xizQtIkSZ9O8BxcAyPLxTkY/jhcA6fKyO8ha21Eg9N4bopz+EjXQbzbgyYX52D4/bZI6pZ0njgH0uhe/wslnSVpqzFmh6SxxpitcQ7nGhhZLs7B8MfhGjjVqH4PWWs77aCQpB9pcJrnyXIyJvIhCVsjaagryQck/ebkOxhjaoeVEydKer2kzSk8xwOSros9Tq2k6yQ9ECt3HjbGXGqMMZLeH+/5AyDr58AYM80YM2bosSRdIekFzoGkUb7+xpi/0+D85luttdEEz8E1MLKsnwOugdNK+xyYQWfFbjeSlkh6PsFzvMcYU26MmS3pbA2u4XtK0tnGmNnGmDJJ71EwKwdZPwex17gk9vlMDU5b3CHOgTSK199ae6+1dqq1dpa1dpakXmvtWQmeg2sgsayfA66B0xrt3+OhBM5ocD3XppOPV67GRDYPup8MfWiwa1inpAENzrP8sKQJkh6S9JKk30uqi923WdIPYp9frsF5s8/E/v3wsMf8ZOyxwpI6hh1z7PjY/z+kwQWoWyX97bDbm2MnaJukb0uDG1wX6oercyDpzZKejR3/rKRlQTwHWXr9w7HX7unYxxe4BvLvHHANZO8caPANxz/Hbtsk6eeKderTYDLwxWHP/S+x1/kFDet6pcEOly/GvvYvrl+jQj0HGlwz+VzsOtkg6e1BPAfZ+D100uN3D/ucayCPzgHXQHbPgaQ/DPs99DNJVScfH/t/1sdEJvaAAAAAAIAc8GE6IgAAAAAUDJIwAAAAAMghkjAAAAAAyCGSMAAAAADIIZIwAAAAAMghkjAAQN4zxkwwxjwd+9hjjGmPfd5tjPmu6/gAAEgFLeoBAF4xxrRocI+d/991LAAApINKGADAW8aYNxlj1sU+bzHG/NgY86gxZqcx5p3GmK8YYzYaY+43xpTG7tdkjPmTMabNGPOAMabe7XcBAAgakjAAQCE5U9LVkpZI+pmkh621CyQdlfS2WCL2LUk3W2ubJP1Q0pdcBQsACKYS1wEAAJBB91lrB4wxGyUVS7o/dvtGSbMknSvpPEkPGmMUu0+ngzgBAAFGEgYAKCQhSbLWRo0xA/b4wueoBv/mGUnPWWsvcxUgAABMRwQABMkLkiYZYy6TJGNMqTFmvuOYAAABQxIGAAgMa22/pJslfdkY84ykpyVd7jQoAEDg0KIeAAAAAHKIShgAAAAA5BBJGAAAAADkEEkYAAAAAOQQSRgAAAAA5BBJGAAAAADkEEkYAAAAAOQQSRgAAAAA5BBJGAAAAADk0P8D7Xk9uTCvKHkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1080x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.patches as patches\n",
    "from datetime import timedelta\n",
    "fig, ax = plt.subplots(figsize=(15, 5))\n",
    "ax.yaxis.set_ticks([])\n",
    "plt.plot(run_statistics[\"batch_end\"], run_statistics[\"accuracy\"])\n",
    "\n",
    "\n",
    "drift_regions = run_statistics[run_statistics[\"drift_detection\"] == 1]\n",
    "for region in drift_regions.to_dict('records'):\n",
    "    ax.add_patch(\n",
    "     patches.Rectangle(\n",
    "        (region['batch_start'], 0.8),\n",
    "        timedelta(seconds=60),\n",
    "        0.2,\n",
    "        facecolor = 'gray',\n",
    "        linestyle='--',\n",
    "        fill=False \n",
    "     ) ) \n",
    "            \n",
    "plt.ylabel('Accuracy')\n",
    "plt.xlabel('Time')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b9f0fd4-4dcb-4e9a-af2a-84be12232308",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}