retrospective_fairnessviolation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retrospective analysis of a fairness violation in an income classification pipeline\n",
    "\n",
    "When screening our [ML pipeline for income classification](https://github.com/schelterlabs/arguseyes-example/blob/main/pipelines/openml-classification-incomelevel.py) on Github with this [configuration](https://github.com/schelterlabs/arguseyes-example/blob/main/openml-classification-incomelevel-fairness.yaml), ArgusEyes detects a **fairness violation** in the pipeline. \n",
    "\n",
    "The screenshot shows the result of the [screening during the build triggered by a Github action](https://github.com/schelterlabs/arguseyes-example/actions/runs/3523396218/jobs/5907507086): The **classifier treats data from non-male person disparately unfair**, with respect to given threshold on the equal opportunity metric.\n",
    "\n",
    "In the following, we show how to **leverage ArgusEyes to retrospectively analyze the pipeline run** (based on metadata and captured data artifacts), and **figure out the root cause of the fairness violation issue**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![data-leakage-screening-via-a-github-action](github-action-fairness-screening.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the metadata and artifacts from the original run of the pipeline\n",
    "\n",
    "ArgusEyes needs the run id from the mlflow run where ArgusEyes stored the metadata and artifacts. (Note we use a local run here for demo purposes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ebaaac70",
   "metadata": {},
   "outputs": [],
   "source": [
    "from arguseyes.retrospective import PipelineRun, FairnessRetrospective\n",
    "\n",
    "run = PipelineRun(run_id='d0f8dbbe5ba340e89b869bd023e8c0c6')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interactively explore the dataflow plan and data of the pipeline run\n",
    "\n",
    "We can view a dataflow plan of the pipeline, which highlights the input datasets, as well as the features and labels for the train and test data computed by the pipeline. We can interactively explore the pipeline data. Clicking on the pink data vertices provides us with details about the corresponding data.\n",
    "\n",
    "<span style='color: red;'>[Note that this interactive widget is only shown during the actual execution of the notebook with jupyter and not rendered in the offline view on Github.]</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Pipeline Data Explorer"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "874c87617cdb4e318ac009d4ce58d396",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(CytoscapeWidget(cytoscape_layout={'name': 'dagre'}, cytoscape_style=[{'selector': 'node', 'css'…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "run.explore_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrospective analysis of the label errors issue\n",
    "\n",
    "ArgusEyes allows us to instantiate a special `FairnessRetrospective`, which helps us understand the predictions of the pipeline with respect to different demographic groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "retrospective = FairnessRetrospective(run)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fairness in ML is typically defined with respect to a privileged group and a disadvantaged group. These groups are identified in the training data by examining a privileged value (e.g., 'male') for a sensitive attribute (e.g., 'sex'). The retrospective tells us for which group definitions we computed fairness metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6af91c93",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{('race', 'white'), ('sex', 'male')}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retrospective.fairness_criteria()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Group-wise confusion matrices\n",
    "\n",
    "ArgusEyes allows data scientists to conduct a deep dive into the fairness of the pipeline. We can for example compute and inspect the confusion matrices for the test predictions of the pipeline, separately for the protected and disadvantaged group. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <div style=\"display:flex; margin-left:-5px; margin-right:-5px; width: 85%;\">\n",
       "          <div style=\"flex: 50%; padding: 5px;\">\n",
       "            <h5>Confusion matrix for the privileged group (sex=male)</h5>\n",
       "            <table>\n",
       "              <tr>\n",
       "                <td></td><td><b>positive prediction</b></td><td><b>negative prediction</b></td><td></td>\n",
       "              </tr>\n",
       "              <tr>\n",
       "                <td><b>positive label</b></td>\n",
       "                <td style=\"border-left: 2px solid black;border-top: 2px solid black;\">TP=179</td>\n",
       "                <td style=\"border-right: 2px solid black;border-top: 2px solid black;\">FN=206</td>\n",
       "                <td>385</td>\n",
       "              </tr>    \n",
       "              <tr>\n",
       "                <td><b>negative label</b></td>\n",
       "                <td style=\"border-left: 2px solid black;border-bottom: 2px solid black;\">FP=166</td>\n",
       "                <td style=\"border-right: 2px solid black;border-bottom: 2px solid black;\">TN=813</td>\n",
       "                <td>979</td>\n",
       "              </tr>  \n",
       "              <tr>\n",
       "                <td></td><td>345</td><td>1019</td><td></td>\n",
       "              </tr>    \n",
       "            </table>\n",
       "          </div>\n",
       "          <div style=\"flex: 50%; padding: 5px;\">\n",
       "            <h5>Confusion matrix for the disadvantaged group (sex!=male)</h5>  \n",
       "            <table>\n",
       "              <tr>\n",
       "                <td></td><td><b>positive prediction</b></td><td><b>negative prediction</b></td><td></td>\n",
       "              </tr>\n",
       "              <tr>\n",
       "                <td><b>positive label</b></td>\n",
       "                <td style=\"border-left: 2px solid black;border-top: 2px solid black;\">TP=141</td>\n",
       "                <td style=\"border-right: 2px solid black;border-top: 2px solid black;\">FN=151</td>\n",
       "                <td>292</td>\n",
       "              </tr>    \n",
       "              <tr>\n",
       "                <td><b>negative label</b></td>\n",
       "                <td style=\"border-left: 2px solid black;border-bottom: 2px solid black;\">FP=121</td>\n",
       "                <td style=\"border-right: 2px solid black;border-bottom: 2px solid black;\">TN=421</td>\n",
       "                <td>542</td>\n",
       "              </tr>  \n",
       "              <tr>\n",
       "                <td></td><td>262</td><td>572</td><td></td>\n",
       "              </tr>    \n",
       "            </table>\n",
       "          </div>  \n",
       "        </div>  \n",
       "        "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "retrospective.show_confusion_matrix('sex', 'male')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Common fairness metrics\n",
    "\n",
    "Additionally, ArgusEyes can compute and plot common fairness metrics for a given group definition. Negatively values in these metrics indicate unfairness, and we see that our pipeline is unfair with respect to non-male persons in the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "08e6b7bc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'predictive_parity': -0.019327359221152696,\n",
       " 'equal_opportunity': -0.017941647393702187,\n",
       " 'statistical_parity': -0.061216129735507774}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retrospective.fairness_metrics('sex', 'male')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fe24e0d1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAGdCAYAAAAfTAk2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA0y0lEQVR4nO3de1hVVeL/8Q8IHFQ8kIgcSZQuKGjeBgRRJ5ugMK2JySb1cRKNtBqli+aoaTpZk019TSs1v/Zt9Gvp6Nduv9EcJ8PKHiUvqDXeyBoVRwU0FURTUNbvjx5OHbkojidk+X49z37yrL3W2msd9j77w2bvk48xxggAAMAivnU9AAAAgMuNgAMAAKxDwAEAANYh4AAAAOsQcAAAgHUIOAAAwDoEHAAAYB0CDgAAsI5fXQ+gLpSXl+vgwYNq0qSJfHx86no4AADgIhhjdOLECUVERMjXt+ZrNFdlwDl48KAiIyPrehgAAOAS7N+/Xy1btqyxzlUZcJo0aSLphzfI6XTW8WgAAMDFKC4uVmRkpPs8XpOrMuBU/FnK6XQScAAAqGcu5vYSbjIGAADWIeAAAADrEHAAAIB1CDgAAMA6BBwAAGAdAg4AALAOAQcAAFiHgAMAAKxDwAEAANYh4AAAAOsQcAAAgHUIOAAAwDoEHAAAYB0CDgAAsA4BBwAAWIeAAwAArEPAAQAA1iHgAAAA6xBwAACAdQg4AADAOgQcAABgHQIOAACwDgEHAABYh4ADAACsQ8ABAADWIeAAAADrEHAAAIB1CDgAAMA6BBwAAGAdAg4AALAOAQcAAFiHgAMAAKzzswScWbNmKSoqSoGBgUpMTNSGDRtqrL906VLFxMQoMDBQHTp00IoVKyrV2blzp379618rODhYjRs3VteuXZWXl+etKQAAgHrE6wFnyZIlGjVqlCZPnqzNmzerU6dOSk1NVWFhYZX1161bp4EDByojI0NbtmxRWlqa0tLStG3bNnedb7/9Vj179lRMTIw+/fRTffXVV3r66acVGBjo7ekAAIB6wMcYY7y5gcTERHXt2lUzZ86UJJWXlysyMlKZmZkaN25cpfr9+/fXyZMntXz5cndZt27d1LlzZ82ZM0eSNGDAAPn7++utt966pDEVFxcrODhYRUVFcjqdl9QHAAD4edXm/O3VKzilpaXKyclRSkrKjxv09VVKSoqys7OrbJOdne1RX5JSU1Pd9cvLy/Xhhx+qTZs2Sk1NVfPmzZWYmKgPPvjAa/MAAAD1i1cDzpEjR3Tu3DmFh4d7lIeHhys/P7/KNvn5+TXWLywsVElJiV544QX17t1bH330kX7zm9/onnvu0WeffVZln2fOnFFxcbHHAgAA7OVX1wOorfLycknS3XffrSeeeEKS1LlzZ61bt05z5sxRr169KrWZOnWqnnnmmZ91nAAAoO549QpOs2bN1KBBAxUUFHiUFxQUyOVyVdnG5XLVWL9Zs2by8/NTu3btPOrExsZW+xTV+PHjVVRU5F72799/qVMCAAD1gFcDTkBAgOLi4pSVleUuKy8vV1ZWlpKSkqpsk5SU5FFfklatWuWuHxAQoK5duyo3N9ejztdff63WrVtX2afD4ZDT6fRYAACAvbz+J6pRo0YpPT1d8fHxSkhI0IwZM3Ty5EkNHTpUkjR48GBde+21mjp1qiTpscceU69evTRt2jT17dtXixcv1qZNmzR37lx3n2PGjFH//v11880361e/+pVWrlypZcuW6dNPP/X2dAAAQD3g9YDTv39/HT58WJMmTVJ+fr46d+6slStXum8kzsvLk6/vjxeSunfvrkWLFmnixIl66qmnFB0drQ8++EA33XSTu85vfvMbzZkzR1OnTtWjjz6qtm3b6t1331XPnj29PR0AAFAPeP17cK5EfA8OAAD1zxXzPTgAAAB1gYADAACsQ8ABAADWIeAAAADrEHAAAIB1CDgAAMA6BBwAAGAdAg4AALAOAQcAAFiHgAMAAKxDwAEAANYh4AAAAOsQcAAAgHUIOAAAwDoEHAAAYB0CDgAAsA4BBwAAWIeAAwAArEPAAQAA1iHgAAAA6xBwAACAdQg4AADAOgQcAABgHQIOAACwDgEHAABYh4ADAACsQ8ABAADWIeAAAADrEHAAAIB1CDgAAMA6BBwAAGAdAg4AALAOAQcAAFiHgAMAAKxDwAEAANYh4AAAAOsQcAAAgHUIOAAAwDo/S8CZNWuWoqKiFBgYqMTERG3YsKHG+kuXLlVMTIwCAwPVoUMHrVixotq6Dz/8sHx8fDRjxozLPGoAAFBfeT3gLFmyRKNGjdLkyZO1efNmderUSampqSosLKyy/rp16zRw4EBlZGRoy5YtSktLU1pamrZt21ap7vvvv68vvvhCERER3p4GAACoR7wecF5++WUNGzZMQ4cOVbt27TRnzhw1atRIf/nLX6qs/8orr6h3794aM2aMYmNj9eyzz+oXv/iFZs6c6VHvwIEDyszM1MKFC+Xv7+/taQAAgHrEqwGntLRUOTk5SklJ+XGDvr5KSUlRdnZ2lW2ys7M96ktSamqqR/3y8nLdf//9GjNmjNq3b3/BcZw5c0bFxcUeCwAAsJdXA86RI0d07tw5hYeHe5SHh4crPz+/yjb5+fkXrP/nP/9Zfn5+evTRRy9qHFOnTlVwcLB7iYyMrOVMAABAfVLvnqLKycnRK6+8ovnz58vHx+ei2owfP15FRUXuZf/+/V4eJQAAqEteDTjNmjVTgwYNVFBQ4FFeUFAgl8tVZRuXy1Vj/c8//1yFhYVq1aqV/Pz85Ofnp3379mn06NGKioqqsk+HwyGn0+mxAAAAe3k14AQEBCguLk5ZWVnusvLycmVlZSkpKanKNklJSR71JWnVqlXu+vfff7+++uorbd261b1ERERozJgx+sc//uG9yQAAgHrDz9sbGDVqlNLT0xUfH6+EhATNmDFDJ0+e1NChQyVJgwcP1rXXXqupU6dKkh577DH16tVL06ZNU9++fbV48WJt2rRJc+fOlSSFhoYqNDTUYxv+/v5yuVxq27att6cDAADqAa8HnP79++vw4cOaNGmS8vPz1blzZ61cudJ9I3FeXp58fX+8kNS9e3ctWrRIEydO1FNPPaXo6Gh98MEHuummm7w9VAAAYAkfY4yp60H83IqLixUcHKyioiLuxwEAoJ6ozfm73j1FBQAAcCEEHAAAYB0CDgAAsA4BBwAAWIeAAwAArEPAAQAA1iHgAAAA6xBwAACAdQg4AADAOgQcAABgHQIOAACwDgEHAABYh4ADAACsQ8ABAADWIeAAAADrEHAAAIB1CDgAAMA6BBwAAGAdAg4AALAOAQcAAFiHgAMAAKxDwAEAANYh4AAAAOsQcAAAgHUIOAAAwDoEHAAAYB0CDgAAsA4BBwAAWIeAAwAArEPAAQAA1iHgAAAA6xBwAACAdQg4AADAOgQcAABgHQIOAACwDgEHAABYh4ADAACs87MEnFmzZikqKkqBgYFKTEzUhg0baqy/dOlSxcTEKDAwUB06dNCKFSvc68rKyjR27Fh16NBBjRs3VkREhAYPHqyDBw96exoAAKCe8HrAWbJkiUaNGqXJkydr8+bN6tSpk1JTU1VYWFhl/XXr1mngwIHKyMjQli1blJaWprS0NG3btk2SdOrUKW3evFlPP/20Nm/erPfee0+5ubn69a9/7e2pAACAesLHGGO8uYHExER17dpVM2fOlCSVl5crMjJSmZmZGjduXKX6/fv318mTJ7V8+XJ3Wbdu3dS5c2fNmTOnym1s3LhRCQkJ2rdvn1q1anXBMRUXFys4OFhFRUVyOp2XODMAAPBzqs3526tXcEpLS5WTk6OUlJQfN+jrq5SUFGVnZ1fZJjs726O+JKWmplZbX5KKiork4+OjkJCQKtefOXNGxcXFHgsAALCXVwPOkSNHdO7cOYWHh3uUh4eHKz8/v8o2+fn5tap/+vRpjR07VgMHDqw2zU2dOlXBwcHuJTIy8hJmAwAA6ot6/RRVWVmZ7rvvPhlj9Prrr1dbb/z48SoqKnIv+/fv/xlHCQAAfm5+3uy8WbNmatCggQoKCjzKCwoK5HK5qmzjcrkuqn5FuNm3b59Wr15d49/iHA6HHA7HJc4CAADUN169ghMQEKC4uDhlZWW5y8rLy5WVlaWkpKQq2yQlJXnUl6RVq1Z51K8IN7t379bHH3+s0NBQ70wAAADUS169giNJo0aNUnp6uuLj45WQkKAZM2bo5MmTGjp0qCRp8ODBuvbaazV16lRJ0mOPPaZevXpp2rRp6tu3rxYvXqxNmzZp7ty5kn4IN/fee682b96s5cuX69y5c+77c5o2baqAgABvTwkAAFzhvB5w+vfvr8OHD2vSpEnKz89X586dtXLlSveNxHl5efL1/fFCUvfu3bVo0SJNnDhRTz31lKKjo/XBBx/opptukiQdOHBAf/vb3yRJnTt39tjWJ598oltuucXbUwIAAFc4r38PzpWI78EBAKD+uWK+BwcAAKAuEHAAAIB1CDgAAMA6BBwAAGAdAg4AALAOAQcAAFiHgAMAAKxDwAEAANYh4AAAAOsQcAAAgHUIOAAAwDoEHAAAYB0CDgAAsA4BBwAAWIeAAwAArEPAAQAA1iHgAAAA6xBwAACAdQg4AADAOgQcAABgHQIOAACwDgEHAABYh4ADAACsQ8ABAADWIeAAAADrEHAAAIB1CDgAAMA6BBwAAGAdAg4AALAOAQcAAFiHgAMAAKxDwAEAANYh4AAAAOsQcAAAgHUIOAAAwDoEHAAAYB0CDgAAsM7PEnBmzZqlqKgoBQYGKjExURs2bKix/tKlSxUTE6PAwEB16NBBK1as8FhvjNGkSZPUokULNWzYUCkpKdq9e7c3pwAAAOoRrwecJUuWaNSoUZo8ebI2b96sTp06KTU1VYWFhVXWX7dunQYOHKiMjAxt2bJFaWlpSktL07Zt29x1XnzxRb366quaM2eO1q9fr8aNGys1NVWnT5/29nQAAEA94GOMMd7cQGJiorp27aqZM2dKksrLyxUZGanMzEyNGzeuUv3+/fvr5MmTWr58ubusW7du6ty5s+bMmSNjjCIiIjR69Gg9+eSTkqSioiKFh4dr/vz5GjBgwAXHVFxcrODgYBUVFcnpdF6mmQIAAG+qzfnbq1dwSktLlZOTo5SUlB836OurlJQUZWdnV9kmOzvbo74kpaamuuvv2bNH+fn5HnWCg4OVmJhYbZ9nzpxRcXGxxwIAAOzl583Ojxw5onPnzik8PNyjPDw8XLt27aqyTX5+fpX18/Pz3esryqqrc76pU6fqmWeeuaQ5XIqocR/+bNvClWfvC33regjsg6jz/ZB9EHW9D14VT1GNHz9eRUVF7mX//v11PSQAAOBFXg04zZo1U4MGDVRQUOBRXlBQIJfLVWUbl8tVY/2K/9amT4fDIafT6bEAAAB7eTXgBAQEKC4uTllZWe6y8vJyZWVlKSkpqco2SUlJHvUladWqVe761113nVwul0ed4uJirV+/vto+AQDA1cWr9+BI0qhRo5Senq74+HglJCRoxowZOnnypIYOHSpJGjx4sK699lpNnTpVkvTYY4+pV69emjZtmvr27avFixdr06ZNmjt3riTJx8dHjz/+uJ577jlFR0fruuuu09NPP62IiAilpaV5ezoAAKAe8HrA6d+/vw4fPqxJkyYpPz9fnTt31sqVK903Cefl5cnX98cLSd27d9eiRYs0ceJEPfXUU4qOjtYHH3ygm266yV3nD3/4g06ePKnhw4fr+PHj6tmzp1auXKnAwEBvTwcAANQDXv8enCuRt78Hh6cHrm51/eSAxD6Iut8P2QfhjX3wivkeHAAAgLpAwAEAANYh4AAAAOsQcAAAgHUIOAAAwDoEHAAAYB0CDgAAsA4BBwAAWIeAAwAArEPAAQAA1iHgAAAA6xBwAACAdQg4AADAOgQcAABgHQIOAACwDgEHAABYh4ADAACsQ8ABAADWIeAAAADrEHAAAIB1CDgAAMA6BBwAAGAdAg4AALAOAQcAAFiHgAMAAKxDwAEAANYh4AAAAOsQcAAAgHUIOAAAwDoEHAAAYB0CDgAAsA4BBwAAWIeAAwAArEPAAQAA1iHgAAAA6xBwAACAdQg4AADAOn51PQAAl9/eF/rW9RAAoE559QrO0aNHNWjQIDmdToWEhCgjI0MlJSU1tjl9+rRGjBih0NBQBQUFqV+/fiooKHCv//LLLzVw4EBFRkaqYcOGio2N1SuvvOLNaQAAgHrGqwFn0KBB2r59u1atWqXly5drzZo1Gj58eI1tnnjiCS1btkxLly7VZ599poMHD+qee+5xr8/JyVHz5s319ttva/v27ZowYYLGjx+vmTNnenMqAACgHvExxhhvdLxz5061a9dOGzduVHx8vCRp5cqV6tOnj/79738rIiKiUpuioiKFhYVp0aJFuvfeeyVJu3btUmxsrLKzs9WtW7cqtzVixAjt3LlTq1evvqixFRcXKzg4WEVFRXI6nZc4QwBAdaLGfVjXQ0Ad88afymtz/vbaFZzs7GyFhIS4w40kpaSkyNfXV+vXr6+yTU5OjsrKypSSkuIui4mJUatWrZSdnV3ttoqKitS0adNq1585c0bFxcUeCwAAsJfXAk5+fr6aN2/uUebn56emTZsqPz+/2jYBAQEKCQnxKA8PD6+2zbp167RkyZIa//Q1depUBQcHu5fIyMjaTQYAANQrtQ4448aNk4+PT43Lrl27vDHWSrZt26a7775bkydP1u23315tvfHjx6uoqMi97N+//2cZHwAAqBu1fkx89OjRGjJkSI11rr/+erlcLhUWFnqUnz17VkePHpXL5aqyncvlUmlpqY4fP+5xFaegoKBSmx07dig5OVnDhw/XxIkTaxyPw+GQw+GosQ4AALBHrQNOWFiYwsLCLlgvKSlJx48fV05OjuLi4iRJq1evVnl5uRITE6tsExcXJ39/f2VlZalfv36SpNzcXOXl5SkpKcldb/v27br11luVnp6uP/3pT7WdAgAAsJzX7sGJjY1V7969NWzYMG3YsEFr167VyJEjNWDAAPcTVAcOHFBMTIw2bNggSQoODlZGRoZGjRqlTz75RDk5ORo6dKiSkpLcT1Bt27ZNv/rVr3T77bdr1KhRys/PV35+vg4fPuytqQAAgHrGq99kvHDhQo0cOVLJycny9fVVv3799Oqrr7rXl5WVKTc3V6dOnXKXTZ8+3V33zJkzSk1N1ezZs93r33nnHR0+fFhvv/223n77bXd569attXfvXm9OBwAA1BNe+x6cKxnfgwMA3sX34MDa78EBAACoKwQcAABgHQIOAACwDgEHAABYh4ADAACsQ8ABAADWIeAAAADrEHAAAIB1CDgAAMA6BBwAAGAdAg4AALAOAQcAAFiHgAMAAKxDwAEAANYh4AAAAOsQcAAAgHUIOAAAwDoEHAAAYB0CDgAAsA4BBwAAWIeAAwAArEPAAQAA1iHgAAAA6xBwAACAdQg4AADAOgQcAABgHQIOAACwDgEHAABYh4ADAACsQ8ABAADWIeAAAADrEHAAAIB1CDgAAMA6BBwAAGAdAg4AALAOAQcAAFiHgAMAAKzj1YBz9OhRDRo0SE6nUyEhIcrIyFBJSUmNbU6fPq0RI0YoNDRUQUFB6tevnwoKCqqs+91336lly5by8fHR8ePHvTADAABQH3k14AwaNEjbt2/XqlWrtHz5cq1Zs0bDhw+vsc0TTzyhZcuWaenSpfrss8908OBB3XPPPVXWzcjIUMeOHb0xdAAAUI95LeDs3LlTK1eu1P/8z/8oMTFRPXv21GuvvabFixfr4MGDVbYpKirSm2++qZdfflm33nqr4uLiNG/ePK1bt05ffPGFR93XX39dx48f15NPPumtKQAAgHrKawEnOztbISEhio+Pd5elpKTI19dX69evr7JNTk6OysrKlJKS4i6LiYlRq1atlJ2d7S7bsWOHpkyZogULFsjX98JTOHPmjIqLiz0WAABgL68FnPz8fDVv3tyjzM/PT02bNlV+fn61bQICAhQSEuJRHh4e7m5z5swZDRw4UC+99JJatWp1UWOZOnWqgoOD3UtkZGTtJwQAAOqNWgeccePGycfHp8Zl165d3hirJGn8+PGKjY3V7373u1q1KSoqci/79+/32vgAAEDd86ttg9GjR2vIkCE11rn++uvlcrlUWFjoUX727FkdPXpULperynYul0ulpaU6fvy4x1WcgoICd5vVq1frn//8p9555x1JkjFGktSsWTNNmDBBzzzzTKV+HQ6HHA7HxU4RAADUc7UOOGFhYQoLC7tgvaSkJB0/flw5OTmKi4uT9EM4KS8vV2JiYpVt4uLi5O/vr6ysLPXr10+SlJubq7y8PCUlJUmS3n33XX3//ffuNhs3btQDDzygzz//XDfccENtpwMAACxU64BzsWJjY9W7d28NGzZMc+bMUVlZmUaOHKkBAwYoIiJCknTgwAElJydrwYIFSkhIUHBwsDIyMjRq1Cg1bdpUTqdTmZmZSkpKUrdu3SSpUog5cuSIe3vn37sDAACuTl4LOJK0cOFCjRw5UsnJyfL19VW/fv306quvuteXlZUpNzdXp06dcpdNnz7dXffMmTNKTU3V7NmzvTlMAABgGR9TcRPLVaS4uFjBwcEqKiqS0+ms6+EAgHWixn1Y10NAHdv7Qt/L3mdtzt/8v6gAAIB1CDgAAMA6BBwAAGAdAg4AALAOAQcAAFiHgAMAAKxDwAEAANYh4AAAAOsQcAAAgHUIOAAAwDoEHAAAYB0CDgAAsA4BBwAAWIeAAwAArEPAAQAA1iHgAAAA6xBwAACAdQg4AADAOgQcAABgHQIOAACwDgEHAABYh4ADAACsQ8ABAADWIeAAAADrEHAAAIB1CDgAAMA6BBwAAGAdAg4AALAOAQcAAFiHgAMAAKxDwAEAANYh4AAAAOsQcAAAgHUIOAAAwDoEHAAAYB0CDgAAsA4BBwAAWMdrAefo0aMaNGiQnE6nQkJClJGRoZKSkhrbnD59WiNGjFBoaKiCgoLUr18/FRQUVKo3f/58dezYUYGBgWrevLlGjBjhrWkAAIB6yGsBZ9CgQdq+fbtWrVql5cuXa82aNRo+fHiNbZ544gktW7ZMS5cu1WeffaaDBw/qnnvu8ajz8ssva8KECRo3bpy2b9+ujz/+WKmpqd6aBgAAqId8jDHmcne6c+dOtWvXThs3blR8fLwkaeXKlerTp4/+/e9/KyIiolKboqIihYWFadGiRbr33nslSbt27VJsbKyys7PVrVs3HTt2TNdee62WLVum5OTkSx5fcXGxgoODVVRUJKfTecn9AACqFjXuw7oeAurY3hf6XvY+a3P+9soVnOzsbIWEhLjDjSSlpKTI19dX69evr7JNTk6OysrKlJKS4i6LiYlRq1atlJ2dLUlatWqVysvLdeDAAcXGxqply5a67777tH///hrHc+bMGRUXF3ssAADAXl4JOPn5+WrevLlHmZ+fn5o2bar8/Pxq2wQEBCgkJMSjPDw83N3mX//6l8rLy/X8889rxowZeuedd3T06FHddtttKi0trXY8U6dOVXBwsHuJjIz8zyYIAACuaLUKOOPGjZOPj0+Ny65du7w1VpWXl6usrEyvvvqqUlNT1a1bN/31r3/V7t279cknn1Tbbvz48SoqKnIvF7riAwAA6je/2lQePXq0hgwZUmOd66+/Xi6XS4WFhR7lZ8+e1dGjR+Vyuaps53K5VFpaquPHj3tcxSkoKHC3adGihSSpXbt27vVhYWFq1qyZ8vLyqh2Tw+GQw+GocdwAAMAetQo4YWFhCgsLu2C9pKQkHT9+XDk5OYqLi5MkrV69WuXl5UpMTKyyTVxcnPz9/ZWVlaV+/fpJknJzc5WXl6ekpCRJUo8ePdzlLVu2lPTD4+hHjhxR69atazMVAIAXeeMGU6A2vPIUlSTdcccdKigo0Jw5c1RWVqahQ4cqPj5eixYtkiQdOHBAycnJWrBggRISEiRJjzzyiFasWKH58+fL6XQqMzNTkrRu3Tp3v2lpafrmm280d+5cOZ1OjR8/Xv/617+0detW+fv7X9TYeIoKAID6p86fopKkhQsXKiYmRsnJyerTp4969uypuXPnuteXlZUpNzdXp06dcpdNnz5dd955p/r166ebb75ZLpdL7733nke/CxYsUGJiovr27atevXrJ399fK1euvOhwAwAA7Oe1KzhXMq7gAABQ/1wRV3AAAADqCgEHAABYh4ADAACsQ8ABAADWIeAAAADrEHAAAIB1CDgAAMA6BBwAAGAdAg4AALAOAQcAAFiHgAMAAKxDwAEAANYh4AAAAOsQcAAAgHUIOAAAwDoEHAAAYB0CDgAAsA4BBwAAWIeAAwAArEPAAQAA1iHgAAAA6xBwAACAdQg4AADAOgQcAABgHQIOAACwDgEHAABYh4ADAACsQ8ABAADWIeAAAADrEHAAAIB1CDgAAMA6BBwAAGAdv7oeQF0wxkiSiouL63gkAADgYlWctyvO4zW5KgPOiRMnJEmRkZF1PBIAAFBbJ06cUHBwcI11fMzFxCDLlJeX6+DBg2rSpIl8fHzqejhWKS4uVmRkpPbv3y+n01nXw8FViH0QdY190HuMMTpx4oQiIiLk61vzXTZX5RUcX19ftWzZsq6HYTWn08mBjTrFPoi6xj7oHRe6clOBm4wBAIB1CDgAAMA6BBxcVg6HQ5MnT5bD4ajroeAqxT6IusY+eGW4Km8yBgAAduMKDgAAsA4BBwAAWIeAAwAArEPAuQpERUVpxowZ7tc+Pj764IMP/qM+L0cfdemWW27R448/XtfDuOrU9/3Gmz799FP5+Pjo+PHjdT0UK/2n+94f//hHde7c+bKNZ/78+QoJCbls/dXW+ecFGxFwrkKHDh3SHXfccVF1qzuoa9PHlei9997Ts88+6359NRzs+M95M4R0795dhw4dcn+JWV2fAOvakCFDlJaWVut2l+Mzq6ow9OSTTyorK6vW47lSbdy4UcOHD3e/tvGXj6vym4zro9LSUgUEBFyWvlwu1xXRR12oeB+bNm1a10NBPVNWVubV/gMCAurtcVUf/KfvbVBQkIKCgi7TaOpOxWdgWFhYXQ/F+wx+dr169TIjRowwI0aMME6n04SGhpqJEyea8vJyd53WrVubKVOmmPvvv980adLEpKenG2OM+fzzz03Pnj1NYGCgadmypcnMzDQlJSXudgUFBebOO+80gYGBJioqyrz99tumdevWZvr06e46ksz777/vfr1//34zYMAAc80115hGjRqZuLg488UXX5h58+YZSR7LvHnzKvWRlJRk/vCHP3jMsbCw0Pj5+ZnPPvvMGGPM6dOnzejRo01ERIRp1KiRSUhIMJ988kmN75MkM3v2bNO7d28TGBhorrvuOrN06VKPOn/4wx9MdHS0adiwobnuuuvMxIkTTWlpqXv95MmTTadOncwbb7xhoqKijI+Pj/tn8Nhjj7n/ff48S0pKTJMmTSpt7/333zeNGjUyxcXFNY79SnLu3Dnz/PPPm6ioKBMYGGg6duxYaV4ffvihiY6ONoGBgeaWW25x/+yPHTtmjPnxffyp6dOnm9atW7tfb9iwwaSkpJjQ0FDjdDrNzTffbHJycjzanL/vne/06dMmMzPThIWFGYfDYXr06GE2bNjgXv/JJ58YSWb58uWmQ4cOxuFwmMTERPPPf/7TXWfevHkmODjYvP/+++bGG280DofD3H777SYvL89jW7NnzzbXX3+98ff3N23atDELFiyoNNbZs2ebu+66yzRq1Mikp6dX2k8qjsvzjzFjjOnUqZOZPHmyR39vvPGGSUtLMw0bNjQ33nij+X//7/9VmtuxY8fc//7pMnnyZPPMM8+Y9u3bV3rfOnXqZCZOnFjt+3qlWrp0qbnppptMYGCgadq0qUlOTjYlJSVm8uTJleZf8XlR0zF/sZ9ZZ86cMSNGjDAul8s4HA7TqlUr8/zzzxtjfvhZ/rR9xT5e1THw5ptvmnbt2pmAgADjcrnMiBEj3OumTZtmbrrpJtOoUSPTsmVL88gjj5gTJ06411fsp9XZs2ePkWT++te/mqSkJONwOEz79u3Np59+6q5z9uxZ88ADD7iP7TZt2pgZM2Z49JOenm7uvvtu89xzz5kWLVqYqKgo9zwr9tmq5rxnzx7j4+NjNm7c6NHf9OnTTatWrcy5c+eqHfuVgoBTB3r16mWCgoLMY489Znbt2mXefvtt06hRIzN37lx3ndatWxun02n+67/+y3zzzTfupXHjxmb69Onm66+/NmvXrjVdunQxQ4YMcbe74447TKdOnUx2drbZtGmT6d69u2nYsGG1AefEiRPm+uuvN7/85S/N559/bnbv3m2WLFli1q1bZ06dOmVGjx5t2rdvbw4dOmQOHTpkTp06VamPmTNnmlatWnkEtNdee82j7MEHHzTdu3c3a9asMd9884156aWXjMPhMF9//XW175MkExoaat544w2Tm5trJk6caBo0aGB27NjhrvPss8+atWvXmj179pi//e1vJjw83Pz5z392r588ebJp3Lix6d27t9m8ebP58ssv3T+DioDz3XffmZYtW5opU6a452mMMcOGDTN9+vTxGNOvf/1rM3jw4Bp/vlea5557zsTExJiVK1eab7/91sybN884HA73B2VeXp5xOBxm1KhR7v0xPDy81gEnKyvLvPXWW2bnzp1mx44dJiMjw4SHh3uEwQsFnEcffdRERESYFStWmO3bt5v09HRzzTXXmO+++84Y82MIiI2NNR999JH56quvzJ133mmioqI8TnL+/v4mPj7erFu3zmzatMkkJCSY7t27u7fz3nvvGX9/fzNr1iyTm5trpk2bZho0aGBWr17tMdbmzZubv/zlL+bbb781e/fuNe+++66RZHJzc82hQ4fM8ePHjTEXH3BatmxpFi1aZHbv3m0effRRExQUVGlux44dM2fOnDEzZswwTqfTvU+eOHHC7N+/3/j6+nqEvs2bNxsfHx/z7bffVvu+XokOHjxo/Pz8zMsvv2z27NljvvrqKzNr1ixz4sQJc+LECXPfffeZ3r17u+d/5swZY0zNx/zFfma99NJLJjIy0qxZs8bs3bvXfP7552bRokXGmB9+OasIRocOHTKFhYXGmMrHwOzZs01gYKCZMWOGyc3NNRs2bPDYB6ZPn25Wr15t9uzZY7Kyskzbtm3NI4884l5/sQGnZcuW5p133jE7duwwDz74oGnSpIk5cuSIMcaY0tJSM2nSJLNx40bzr3/9y30uWbJkibuf9PR0ExQUZO6//36zbds2s23bNmOM5z5b3Zxvu+028/vf/95jXB07djSTJk264M/3SkDAqQO9evUysbGxHoFg7NixJjY21v26devWJi0tzaNdRkaGGT58uEfZ559/bnx9fc33339vcnNzjSSPD7+dO3caSdUGnP/+7/82TZo0cX/Inq+qE9v5fVRcrVmzZo17fVJSkhk7dqwxxph9+/aZBg0amAMHDnj0kZycbMaPH1/ldiu28fDDD3uUJSYmenxInO+ll14ycXFxHuP39/d3H7AVfhpwjKn6BLV+/XrToEEDc/DgQWPMD1fH/Pz8PH6DutKdPn3aNGrUyKxbt86jPCMjwwwcONAYY8z48eNNu3btPNaPHTu21gHnfOfOnTNNmjQxy5Ytc5fVFHBKSkqMv7+/WbhwobustLTUREREmBdffNEY82MIWLx4sbvOd999Zxo2bOj+UK/4Lf6LL75w16k4DtavX2+MMaZ79+5m2LBhHtv/7W9/6xFoJZnHH3/co85PQ8hPXWzA+elVlpKSEiPJ/P3vf6+y7+pOgHfccYfHMZCZmWluueWWSvWudDk5OUaS2bt3b5XrK648XEhVx/yFPrMyMzPNrbfe6vEZXF3d6vqNiIgwEyZMuOD4KixdutSEhoa6X19swHnhhRfcZWVlZaZly5Yev8Sdb8SIEaZfv37u1+np6SY8PNwdECtc6Mq+McYsWbLEXHPNNeb06dPGmB9+Zj4+PmbPnj01zPTKwU3GdaRbt27y8fFxv05KStLu3bt17tw5d1l8fLxHmy+//FLz5893/y04KChIqampKi8v1549e7Rz5075+fkpLi7O3SYmJqbGGxW3bt2qLl26/Ef3pISFhen222/XwoULJUl79uxRdna2Bg0aJEn65z//qXPnzqlNmzYeY//ss8/07bff1th3UlJSpdc7d+50v16yZIl69Oghl8uloKAgTZw4UXl5eR5tWrdufUl/b05ISFD79u31v//7v5Kkt99+W61bt9bNN99c677qyjfffKNTp07ptttu83jvFyxY4H7vd+7cqcTERI9257/vF6OgoEDDhg1TdHS0goOD5XQ6VVJSUunnUZ1vv/1WZWVl6tGjh7vM399fCQkJHj/z88fXtGlTtW3b1qOOn5+funbt6n5dcRxU1Nm5c6fHdiSpR48elbZz/jH4n+rYsaP7340bN5bT6VRhYWGt+hg2bJj++te/6vTp0yotLdWiRYv0wAMPXNZx/hw6deqk5ORkdejQQb/97W/1xhtv6NixYxdsdzHH/IUMGTJEW7duVdu2bfXoo4/qo48+qlX7wsJCHTx4UMnJydXW+fjjj5WcnKxrr71WTZo00f3336/vvvtOp06dqtW2frqv+/n5KT4+3mM/nTVrluLi4hQWFqagoCDNnTu30vvRoUOHS7qHMy0tTQ0aNND7778v6Ycb33/1q18pKiqq1n3VBQLOFaxx48Yer0tKSvTQQw9p69at7uXLL7/U7t27dcMNN1zSNho2bHg5hqpBgwbpnXfeUVlZmRYtWqQOHTqoQ4cO7nE3aNBAOTk5HmPfuXOnXnnllUveZkWI6tOnj5YvX64tW7ZowoQJKi0t9ah3/vtYGw8++KDmz58vSZo3b56GDh3qEUyvdCUlJZKkDz/80OO937Fjh955552L7sfX11fmvP+ry/k33aanp2vr1q165ZVXtG7dOm3dulWhoaGVfh71ycXuOxfz/kg/BLaf8vHxUXl5ea3GdNddd8nhcOj999/XsmXLVFZWpnvvvbdWfVwJGjRooFWrVunvf/+72rVrp9dee01t27bVnj17qm1zscf8hfziF7/Qnj179Oyzz+r777/XfffdV6v38EKfm3v37tWdd96pjh076t1331VOTo5mzZolSZf1eFi8eLGefPJJZWRk6KOPPtLWrVs1dOjQy/YZGBAQoMGDB2vevHn1MkwTcOrI+vXrPV5/8cUXio6OVoMGDapt84tf/EI7duzQjTfeWGkJCAhQTEyMzp49q5ycHHeb3NzcGh9p7dixo7Zu3aqjR49WuT4gIMDjqlJ17r77bp0+fVorV67UokWL3FdvJKlLly46d+6cCgsLK437Qk82fPHFF5Vex8bGSpLWrVun1q1ba8KECYqPj1d0dLT27dt3wbFWpbp5/u53v9O+ffv06quvaseOHUpPT7+k/utKu3bt5HA4lJeXV+m9j4yMlCTFxsZqw4YNHu3Of9/DwsKUn5/vcRLfunWrR521a9fq0UcfVZ8+fdS+fXs5HA4dOXLkosd6ww03KCAgQGvXrnWXlZWVaePGjWrXrl214zt27Ji+/vpr934hSWfPntWmTZvcryuOg4o6sbGxHtupGP/52zlfxW/B5+8rYWFhOnTokPt1cXFxjSfqi1HdPunn56f09HTNmzdP8+bN04ABAy7bLyo/Nx8fH/Xo0UPPPPOMtmzZooCAAPfVgqrmfzHH/MV+ZjmdTvXv319vvPGGlixZonfffdf9Oejv719jH02aNFFUVFS1j43n5OSovLxc06ZNU7du3dSmTRsdPHjwgmOqyk/39YrP94r9eO3aterevbt+//vfq0uXLrrxxhsveFW8OtXN+cEHH9THH3+s2bNn6+zZs7rnnnsuqf+6wGPidSQvL0+jRo3SQw89pM2bN+u1117TtGnTamwzduxYdevWTSNHjtSDDz6oxo0ba8eOHVq1apVmzpyptm3bqnfv3nrooYf0+uuvy8/PT48//niNH34DBw7U888/r7S0NE2dOlUtWrTQli1bFBERoaSkJEVFRWnPnj3aunWrWrZsqSZNmlT5f8ht3Lix0tLS9PTTT2vnzp0aOHCge12bNm00aNAgDR48WNOmTVOXLl10+PBhZWVlqWPHjurbt2+141u6dKni4+PVs2dPLVy4UBs2bNCbb74pSYqOjlZeXp4WL16srl276sMPP3R/ONZWVFSU1qxZowEDBsjhcKhZs2aSpGuuuUb33HOPxowZo9tvv10tW7a8pP7rSpMmTfTkk0/qiSeeUHl5uXr27KmioiKtXbtWTqdT6enpevjhhzVt2jSNGTNGDz74oHJyctxXrSrccsstOnz4sF588UXde++9Wrlypf7+97/L6XS660RHR+utt95SfHy8iouLNWbMmFqdeBs3bqxHHnlEY8aMUdOmTdWqVSu9+OKLOnXqlDIyMjzqTpkyRaGhoQoPD9eECRPUrFkzj+9M8ff3V2Zmpl599VX5+flp5MiR6tatmxISEiRJY8aM0X333acuXbooJSVFy5Yt03vvvaePP/64xjG2bt1aPj4+Wr58ufr06aOGDRsqKChIt956q+bPn6+77rpLISEhmjRpUo2/rFyMqKgolZSUKCsrS506dVKjRo3UqFEjST+cdH56kquP1q9fr6ysLN1+++1q3ry51q9fr8OHD7vnFRUVpX/84x/Kzc1VaGiogoODL+qYv5jPrJdfflktWrRQly5d5Ovrq6VLl8rlcrn/nF8RXnr06CGHw6Frrrmm0vj/+Mc/6uGHH1bz5s11xx136MSJE1q7dq0yMzN14403qqysTK+99pruuusurV27VnPmzLmk92nWrFmKjo5WbGyspk+frmPHjrmvokRHR2vBggX6xz/+oeuuu05vvfWWNm7cqOuuu67W26luzrGxserWrZvGjh2rBx54oH6F6Tq+B+iq1KtXL/P73//ePPzww8bpdJprrrnGPPXUU5UeEz//pkVjfngU97bbbjNBQUGmcePGpmPHjuZPf/qTe/2hQ4dM37593Y8+Lliw4II3k+3du9f069fPOJ1O06hRIxMfH+++GfP06dOmX79+JiQkpNpHLiusWLHCSDI333xzpXFX3O0fFRVl/P39TYsWLcxvfvMb89VXX1X7Pkkys2bNMrfddptxOBwmKirK4+kAY4wZM2aMCQ0NNUFBQaZ///5m+vTpHjfuVXfD4fk3GWdnZ5uOHTsah8Nhzj8ssrKyjCTzf//3f9WO9UpWXl5uZsyYYdq2bWv8/f1NWFiYSU1NdT/Cb4wxy5Ytcz9S/ctf/tL85S9/qXQz7euvv24iIyNN48aNzeDBg82f/vQnj5uMN2/ebOLj401gYKCJjo42S5cuvagbGX/q+++/N5mZmaZZs2Y1Pia+bNky0759exMQEGASEhLcT8cZ8+PNm++++665/vrrjcPhMCkpKWbfvn0e27qYx8SrGuuUKVOMy+UyPj4+7sfEi4qKTP/+/Y3T6TSRkZFm/vz5Vd5kfH5/wcHB7mOqqhuYH374YRMaGup+TPynfvnLX1b5yHh9sWPHDpOamur+SoA2bdqY1157zb2+sLDQ/VmnnzwmfqFj/mI+s+bOnWs6d+5sGjdubJxOp0lOTjabN2929/G3v/3N3HjjjcbPz6/Gx8TnzJnjPq5atGhhMjMz3etefvll06JFC9OwYUOTmppqFixYcFE3kVeouMl40aJFJiEhwQQEBJh27dp5POl3+vRpM2TIEBMcHGxCQkLMI488YsaNG+cxzupu1j7/2KxqzhXefPPNSg+w1AcEnDpw/skVVbvQyfDnsmDBAhMaGlrpKQSbVfe0UF27mHFd6MRhg/LycnPDDTeYadOm1fVQ4CUVAWfLli11PRQzZcoU06FDh7oeRq3xJyqgGqdOndKhQ4f0wgsv6KGHHrps3yQN/CcOHz6sxYsXKz8/X0OHDq3r4cBiJSUl2rt3r2bOnKnnnnuurodTa9xkDFTjxRdfVExMjFwul8aPH1/XwwEkSc2bN9eUKVM0d+7cKu8NAS6XkSNHKi4uTrfccku9enqqgo8x5z3bCAAAUM9xBQcAAFiHgAMAAKxDwAEAANYh4AAAAOsQcAAAgHUIOAAAwDoEHAAAYB0CDgAAsA4BBwAAWOf/A6yQi50YxYCgAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "retrospective.plot_fairness_metrics('sex', 'male')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fixing the pipeline\n",
    "\n",
    "In contrast to 'technical' issues like data leakage, fairness issues are much harder to fix, as they might not have a technical root cause, but might be a result of bias in the data, bias in the world or an inappropriate formalisation of a problem. In general fairness issues require consultation of legal experts and domain experts.\n",
    "\n",
    "If the root cause of unfairness is non-representative training data, then a pipeline can be fixed by increasing the training data to include more diverse and representative samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}