notebooks/tutorials/Tutorial-2_Add_contributivity_measurement.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CWSFYwBkp6x7"
   },
   "source": [
    "Tutorials table of content:\n",
    "\n",
    "- [Tutorial 1: Run a first scenario](./Tutorial-1_Run_your_first_scenario.ipynb)\n",
    "- Tutorial 2: Add contributivity measurements methods\n",
    "- [Tutorial 3: Use a custom dataset](./Tutorial-3_Use_homemade_dataset.ipynb)\n",
    "\n",
    "\n",
    "# Tutorial 2 : Exploring contributivity \n",
    "\n",
    "With this example, we dive deeper into the potential of the library, and the notion of contributivity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VotO3fz-VY8c"
   },
   "source": [
    "## 1 - Prerequisites\n",
    "\n",
    "In order to run this example, you'll need to:\n",
    "\n",
    "* use python 3.7 +\n",
    "* install this package https://pypi.org/project/mplc/\n",
    "\n",
    "If you did not follow our first tutorial, it is recommended to [take a look at it !](https://github.com/SubstraFoundation/distributed-learning-contributivity/blob/master/notebooks/examples/1%20_INTRO_MNIST.ipynb) \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "3ErUs18-VY8d",
    "outputId": "d44c15ab-4484-477b-e3be-4e052931a0ce",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting librosa==0.8.0\n",
      "\u001B[?25l  Downloading https://files.pythonhosted.org/packages/26/4d/c22d8ca74ca2c13cd4ac430fa353954886104321877b65fa871939e78591/librosa-0.8.0.tar.gz (183kB)\n",
      "\r",
      "\u001B[K     |█▉                              | 10kB 18.9MB/s eta 0:00:01\r",
      "\u001B[K     |███▋                            | 20kB 6.1MB/s eta 0:00:01\r",
      "\u001B[K     |█████▍                          | 30kB 7.2MB/s eta 0:00:01\r",
      "\u001B[K     |███████▏                        | 40kB 7.9MB/s eta 0:00:01\r",
      "\u001B[K     |█████████                       | 51kB 6.6MB/s eta 0:00:01\r",
      "\u001B[K     |██████████▊                     | 61kB 7.4MB/s eta 0:00:01\r",
      "\u001B[K     |████████████▌                   | 71kB 7.9MB/s eta 0:00:01\r",
      "\u001B[K     |██████████████▎                 | 81kB 8.5MB/s eta 0:00:01\r",
      "\u001B[K     |████████████████                | 92kB 8.0MB/s eta 0:00:01\r",
      "\u001B[K     |█████████████████▉              | 102kB 8.3MB/s eta 0:00:01\r",
      "\u001B[K     |███████████████████▋            | 112kB 8.3MB/s eta 0:00:01\r",
      "\u001B[K     |█████████████████████▍          | 122kB 8.3MB/s eta 0:00:01\r",
      "\u001B[K     |███████████████████████▏        | 133kB 8.3MB/s eta 0:00:01\r",
      "\u001B[K     |█████████████████████████       | 143kB 8.3MB/s eta 0:00:01\r",
      "\u001B[K     |██████████████████████████▊     | 153kB 8.3MB/s eta 0:00:01\r",
      "\u001B[K     |████████████████████████████▌   | 163kB 8.3MB/s eta 0:00:01\r",
      "\u001B[K     |██████████████████████████████▎ | 174kB 8.3MB/s eta 0:00:01\r",
      "\u001B[K     |████████████████████████████████| 184kB 8.3MB/s \n",
      "\u001B[?25hRequirement already satisfied: audioread>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from librosa==0.8.0) (2.1.8)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from librosa==0.8.0) (1.18.5)\n",
      "Requirement already satisfied: scipy>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from librosa==0.8.0) (1.4.1)\n",
      "Requirement already satisfied: scikit-learn!=0.19.0,>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from librosa==0.8.0) (0.22.2.post1)\n",
      "Requirement already satisfied: joblib>=0.14 in /usr/local/lib/python3.6/dist-packages (from librosa==0.8.0) (0.16.0)\n",
      "Requirement already satisfied: decorator>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from librosa==0.8.0) (4.4.2)\n",
      "Requirement already satisfied: resampy>=0.2.2 in /usr/local/lib/python3.6/dist-packages (from librosa==0.8.0) (0.2.2)\n",
      "Requirement already satisfied: numba>=0.43.0 in /usr/local/lib/python3.6/dist-packages (from librosa==0.8.0) (0.48.0)\n",
      "Collecting soundfile>=0.9.0\n",
      "  Downloading https://files.pythonhosted.org/packages/eb/f2/3cbbbf3b96fb9fa91582c438b574cff3f45b29c772f94c400e2c99ef5db9/SoundFile-0.10.3.post1-py2.py3-none-any.whl\n",
      "Collecting pooch>=1.0\n",
      "\u001B[?25l  Downloading https://files.pythonhosted.org/packages/ce/11/d7a1dc8173a4085759710e69aae6e070d0d432db84013c7c343e4e522b76/pooch-1.2.0-py3-none-any.whl (47kB)\n",
      "\u001B[K     |████████████████████████████████| 51kB 6.7MB/s \n",
      "\u001B[?25hRequirement already satisfied: six>=1.3 in /usr/local/lib/python3.6/dist-packages (from resampy>=0.2.2->librosa==0.8.0) (1.15.0)\n",
      "Requirement already satisfied: llvmlite<0.32.0,>=0.31.0dev0 in /usr/local/lib/python3.6/dist-packages (from numba>=0.43.0->librosa==0.8.0) (0.31.0)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from numba>=0.43.0->librosa==0.8.0) (50.3.0)\n",
      "Requirement already satisfied: cffi>=1.0 in /usr/local/lib/python3.6/dist-packages (from soundfile>=0.9.0->librosa==0.8.0) (1.14.2)\n",
      "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from pooch>=1.0->librosa==0.8.0) (20.4)\n",
      "Collecting appdirs\n",
      "  Downloading https://files.pythonhosted.org/packages/3b/00/2344469e2084fb287c2e0b57b72910309874c3245463acd6cf5e3db69324/appdirs-1.4.4-py2.py3-none-any.whl\n",
      "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from pooch>=1.0->librosa==0.8.0) (2.23.0)\n",
      "Requirement already satisfied: pycparser in /usr/local/lib/python3.6/dist-packages (from cffi>=1.0->soundfile>=0.9.0->librosa==0.8.0) (2.20)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->pooch>=1.0->librosa==0.8.0) (2.4.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->pooch>=1.0->librosa==0.8.0) (2020.6.20)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->pooch>=1.0->librosa==0.8.0) (3.0.4)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->pooch>=1.0->librosa==0.8.0) (2.10)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->pooch>=1.0->librosa==0.8.0) (1.24.3)\n",
      "Building wheels for collected packages: librosa\n",
      "  Building wheel for librosa (setup.py) ... \u001B[?25l\u001B[?25hdone\n",
      "  Created wheel for librosa: filename=librosa-0.8.0-cp36-none-any.whl size=201376 sha256=8f4aa6eb1bd3bea8bcff8f15745686943b29715612f17de8d0dc17593e1d4194\n",
      "  Stored in directory: /root/.cache/pip/wheels/ee/10/1e/382bb4369e189938d5c02e06d10c651817da8d485bfd1647c9\n",
      "Successfully built librosa\n",
      "Installing collected packages: soundfile, appdirs, pooch, librosa\n",
      "  Found existing installation: librosa 0.6.3\n",
      "    Uninstalling librosa-0.6.3:\n",
      "      Successfully uninstalled librosa-0.6.3\n",
      "Successfully installed appdirs-1.4.4 librosa-0.8.0 pooch-1.2.0 soundfile-0.10.3.post1\n",
      "Collecting Keras==2.3.1\n",
      "\u001B[?25l  Downloading https://files.pythonhosted.org/packages/ad/fd/6bfe87920d7f4fd475acd28500a42482b6b84479832bdc0fe9e589a60ceb/Keras-2.3.1-py2.py3-none-any.whl (377kB)\n",
      "\u001B[K     |████████████████████████████████| 378kB 9.3MB/s \n",
      "\u001B[?25hRequirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from Keras==2.3.1) (2.10.0)\n",
      "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from Keras==2.3.1) (3.13)\n",
      "Collecting keras-applications>=1.0.6\n",
      "\u001B[?25l  Downloading https://files.pythonhosted.org/packages/71/e3/19762fdfc62877ae9102edf6342d71b28fbfd9dea3d2f96a882ce099b03f/Keras_Applications-1.0.8-py3-none-any.whl (50kB)\n",
      "\u001B[K     |████████████████████████████████| 51kB 8.4MB/s \n",
      "\u001B[?25hRequirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from Keras==2.3.1) (1.4.1)\n",
      "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from Keras==2.3.1) (1.15.0)\n",
      "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from Keras==2.3.1) (1.1.2)\n",
      "Requirement already satisfied: numpy>=1.9.1 in /usr/local/lib/python3.6/dist-packages (from Keras==2.3.1) (1.18.5)\n",
      "Installing collected packages: keras-applications, Keras\n",
      "  Found existing installation: Keras 2.4.3\n",
      "    Uninstalling Keras-2.4.3:\n",
      "      Successfully uninstalled Keras-2.4.3\n",
      "Successfully installed Keras-2.3.1 keras-applications-1.0.8\n",
      "Collecting matplotlib==3.1.3\n",
      "\u001B[?25l  Downloading https://files.pythonhosted.org/packages/7e/07/4b361d6d0f4e08942575f83a11d33f36897e1aae4279046606dd1808778a/matplotlib-3.1.3-cp36-cp36m-manylinux1_x86_64.whl (13.1MB)\n",
      "\u001B[K     |████████████████████████████████| 13.1MB 240kB/s \n",
      "\u001B[?25hRequirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.1.3) (1.18.5)\n",
      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.1.3) (0.10.0)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.1.3) (1.2.0)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.1.3) (2.8.1)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.1.3) (2.4.7)\n",
      "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from cycler>=0.10->matplotlib==3.1.3) (1.15.0)\n",
      "\u001B[31mERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.\u001B[0m\n",
      "Installing collected packages: matplotlib\n",
      "  Found existing installation: matplotlib 3.2.2\n",
      "    Uninstalling matplotlib-3.2.2:\n",
      "      Successfully uninstalled matplotlib-3.2.2\n",
      "Successfully installed matplotlib-3.1.3\n"
     ]
    },
    {
     "data": {
      "application/vnd.colab-display-data+json": {
       "pip_warning": {
        "packages": [
         "matplotlib",
         "mpl_toolkits"
        ]
       }
      }
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting numpy==1.19.0\n",
      "\u001B[?25l  Downloading https://files.pythonhosted.org/packages/93/0b/71ae818646c1a80fbe6776d41f480649523ed31243f1f34d9d7e41d70195/numpy-1.19.0-cp36-cp36m-manylinux2010_x86_64.whl (14.6MB)\n",
      "\u001B[K     |████████████████████████████████| 14.6MB 211kB/s \n",
      "\u001B[31mERROR: tensorflow 2.3.0 has requirement numpy<1.19.0,>=1.16.0, but you'll have numpy 1.19.0 which is incompatible.\u001B[0m\n",
      "\u001B[31mERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.\u001B[0m\n",
      "\u001B[31mERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.\u001B[0m\n",
      "\u001B[?25hInstalling collected packages: numpy\n",
      "  Found existing installation: numpy 1.18.5\n",
      "    Uninstalling numpy-1.18.5:\n",
      "      Successfully uninstalled numpy-1.18.5\n",
      "Successfully installed numpy-1.19.0\n"
     ]
    },
    {
     "data": {
      "application/vnd.colab-display-data+json": {
       "pip_warning": {
        "packages": [
         "numpy"
        ]
       }
      }
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: scipy==1.4.1 in /usr/local/lib/python3.6/dist-packages (1.4.1)\n",
      "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from scipy==1.4.1) (1.19.0)\n",
      "Collecting scikit-learn==0.22.1\n",
      "\u001B[?25l  Downloading https://files.pythonhosted.org/packages/d1/48/e9fa9e252abcd1447eff6f9257636af31758a6e46fd5ce5d3c879f6907cb/scikit_learn-0.22.1-cp36-cp36m-manylinux1_x86_64.whl (7.0MB)\n",
      "\u001B[K     |████████████████████████████████| 7.1MB 8.8MB/s \n",
      "\u001B[?25hRequirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn==0.22.1) (0.16.0)\n",
      "Requirement already satisfied: numpy>=1.11.0 in /usr/local/lib/python3.6/dist-packages (from scikit-learn==0.22.1) (1.19.0)\n",
      "Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.6/dist-packages (from scikit-learn==0.22.1) (1.4.1)\n",
      "Installing collected packages: scikit-learn\n",
      "  Found existing installation: scikit-learn 0.22.2.post1\n",
      "    Uninstalling scikit-learn-0.22.2.post1:\n",
      "      Successfully uninstalled scikit-learn-0.22.2.post1\n",
      "Successfully installed scikit-learn-0.22.1\n",
      "Requirement already satisfied: pandas==1.0.5 in /usr/local/lib/python3.6/dist-packages (1.0.5)\n",
      "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas==1.0.5) (2018.9)\n",
      "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from pandas==1.0.5) (1.19.0)\n",
      "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas==1.0.5) (2.8.1)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.6.1->pandas==1.0.5) (1.15.0)\n",
      "Collecting seaborn==0.10.0\n",
      "\u001B[?25l  Downloading https://files.pythonhosted.org/packages/70/bd/5e6bf595fe6ee0f257ae49336dd180768c1ed3d7c7155b2fdf894c1c808a/seaborn-0.10.0-py3-none-any.whl (215kB)\n",
      "\u001B[K     |████████████████████████████████| 225kB 9.5MB/s \n",
      "\u001B[?25hRequirement already satisfied: matplotlib>=2.1.2 in /usr/local/lib/python3.6/dist-packages (from seaborn==0.10.0) (3.1.3)\n",
      "Requirement already satisfied: scipy>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from seaborn==0.10.0) (1.4.1)\n",
      "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from seaborn==0.10.0) (1.19.0)\n",
      "Requirement already satisfied: pandas>=0.22.0 in /usr/local/lib/python3.6/dist-packages (from seaborn==0.10.0) (1.0.5)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.1.2->seaborn==0.10.0) (2.4.7)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.1.2->seaborn==0.10.0) (2.8.1)\n",
      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.1.2->seaborn==0.10.0) (0.10.0)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=2.1.2->seaborn==0.10.0) (1.2.0)\n",
      "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.22.0->seaborn==0.10.0) (2018.9)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.1->matplotlib>=2.1.2->seaborn==0.10.0) (1.15.0)\n",
      "Installing collected packages: seaborn\n",
      "  Found existing installation: seaborn 0.10.1\n",
      "    Uninstalling seaborn-0.10.1:\n",
      "      Successfully uninstalled seaborn-0.10.1\n",
      "Successfully installed seaborn-0.10.0\n",
      "Collecting loguru==0.4.1\n",
      "\u001B[?25l  Downloading https://files.pythonhosted.org/packages/b2/f4/2c8b94025c6e30bdb331c7ee628dc152771845aedff35f0365c2a4dacd42/loguru-0.4.1-py3-none-any.whl (54kB)\n",
      "\u001B[K     |████████████████████████████████| 61kB 4.6MB/s \n",
      "\u001B[?25hCollecting aiocontextvars>=0.2.0; python_version < \"3.7\"\n",
      "  Downloading https://files.pythonhosted.org/packages/db/c1/7a723e8d988de0a2e623927396e54b6831b68cb80dce468c945b849a9385/aiocontextvars-0.2.2-py2.py3-none-any.whl\n",
      "Collecting contextvars==2.4; python_version < \"3.7\"\n",
      "  Downloading https://files.pythonhosted.org/packages/83/96/55b82d9f13763be9d672622e1b8106c85acb83edd7cc2fa5bc67cd9877e9/contextvars-2.4.tar.gz\n",
      "Collecting immutables>=0.9\n",
      "\u001B[?25l  Downloading https://files.pythonhosted.org/packages/99/e0/ea6fd4697120327d26773b5a84853f897a68e33d3f9376b00a8ff96e4f63/immutables-0.14-cp36-cp36m-manylinux1_x86_64.whl (98kB)\n",
      "\u001B[K     |████████████████████████████████| 102kB 5.6MB/s \n",
      "\u001B[?25hBuilding wheels for collected packages: contextvars\n",
      "  Building wheel for contextvars (setup.py) ... \u001B[?25l\u001B[?25hdone\n",
      "  Created wheel for contextvars: filename=contextvars-2.4-cp36-none-any.whl size=7666 sha256=64d75b5476a68babd01f8df2f968984448bab5004fe4ded35569fc03471f1f03\n",
      "  Stored in directory: /root/.cache/pip/wheels/a5/7d/68/1ebae2668bda2228686e3c1cf16f2c2384cea6e9334ad5f6de\n",
      "Successfully built contextvars\n",
      "Installing collected packages: immutables, contextvars, aiocontextvars, loguru\n",
      "Successfully installed aiocontextvars-0.2.2 contextvars-2.4 immutables-0.14 loguru-0.4.1\n",
      "Collecting tensorflow==2.2.0\n",
      "\u001B[?25l  Downloading https://files.pythonhosted.org/packages/3d/be/679ce5254a8c8d07470efb4a4c00345fae91f766e64f1c2aece8796d7218/tensorflow-2.2.0-cp36-cp36m-manylinux2010_x86_64.whl (516.2MB)\n",
      "\u001B[K     |████████████████████████████████| 516.2MB 31kB/s \n",
      "\u001B[?25hRequirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0) (1.12.1)\n",
      "Requirement already satisfied: numpy<2.0,>=1.16.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0) (1.19.0)\n",
      "Collecting tensorflow-estimator<2.3.0,>=2.2.0\n",
      "\u001B[?25l  Downloading https://files.pythonhosted.org/packages/a4/f5/926ae53d6a226ec0fda5208e0e581cffed895ccc89e36ba76a8e60895b78/tensorflow_estimator-2.2.0-py2.py3-none-any.whl (454kB)\n",
      "\u001B[K     |████████████████████████████████| 460kB 50.5MB/s \n",
      "\u001B[?25hRequirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0) (1.1.2)\n",
      "Requirement already satisfied: astunparse==1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0) (1.6.3)\n",
      "Requirement already satisfied: gast==0.3.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0) (0.3.3)\n",
      "Collecting tensorboard<2.3.0,>=2.2.0\n",
      "\u001B[?25l  Downloading https://files.pythonhosted.org/packages/1d/74/0a6fcb206dcc72a6da9a62dd81784bfdbff5fedb099982861dc2219014fb/tensorboard-2.2.2-py3-none-any.whl (3.0MB)\n",
      "\u001B[K     |████████████████████████████████| 3.0MB 44.9MB/s \n",
      "\u001B[?25hRequirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0) (1.1.0)\n",
      "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0) (1.15.0)\n",
      "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0) (0.35.1)\n",
      "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0) (1.32.0)\n",
      "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0) (0.10.0)\n",
      "Requirement already satisfied: google-pasta>=0.1.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0) (0.2.0)\n",
      "Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0) (2.10.0)\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0) (3.3.0)\n",
      "Requirement already satisfied: protobuf>=3.8.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0) (3.12.4)\n",
      "Requirement already satisfied: scipy==1.4.1; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0) (1.4.1)\n",
      "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (2.23.0)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.7.0)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (0.4.1)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (3.2.2)\n",
      "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (50.3.0)\n",
      "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.17.2)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.0.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (2020.6.20)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (2.10)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (3.0.4)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.24.3)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.3.0)\n",
      "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.7.0)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (0.2.8)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (4.1.1)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (4.6)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (3.1.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (3.1.0)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.6/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (0.4.8)\n",
      "Installing collected packages: tensorflow-estimator, tensorboard, tensorflow\n",
      "  Found existing installation: tensorflow-estimator 2.3.0\n",
      "    Uninstalling tensorflow-estimator-2.3.0:\n",
      "      Successfully uninstalled tensorflow-estimator-2.3.0\n",
      "  Found existing installation: tensorboard 2.3.0\n",
      "    Uninstalling tensorboard-2.3.0:\n",
      "      Successfully uninstalled tensorboard-2.3.0\n",
      "  Found existing installation: tensorflow 2.3.0\n",
      "    Uninstalling tensorflow-2.3.0:\n",
      "      Successfully uninstalled tensorflow-2.3.0\n",
      "Successfully installed tensorboard-2.2.2 tensorflow-2.2.0 tensorflow-estimator-2.2.0\n",
      "Collecting ruamel.yaml==0.16.10\n",
      "\u001B[?25l  Downloading https://files.pythonhosted.org/packages/a6/92/59af3e38227b9cc14520bf1e59516d99ceca53e3b8448094248171e9432b/ruamel.yaml-0.16.10-py2.py3-none-any.whl (111kB)\n",
      "\u001B[K     |████████████████████████████████| 112kB 8.0MB/s \n",
      "\u001B[?25hCollecting ruamel.yaml.clib>=0.1.2; platform_python_implementation == \"CPython\" and python_version < \"3.9\"\n",
      "\u001B[?25l  Downloading https://files.pythonhosted.org/packages/88/ff/ec25dc01ef04232a9e68ff18492e37dfa01f1f58172e702ad4f38536d41b/ruamel.yaml.clib-0.2.2-cp36-cp36m-manylinux1_x86_64.whl (549kB)\n",
      "\u001B[K     |████████████████████████████████| 552kB 16.2MB/s \n",
      "\u001B[?25hInstalling collected packages: ruamel.yaml.clib, ruamel.yaml\n",
      "Successfully installed ruamel.yaml-0.16.10 ruamel.yaml.clib-0.2.2\n",
      "Looking in indexes: https://test.pypi.org/simple/\n",
      "Collecting mplc\n",
      "\u001B[?25l  Downloading https://test-files.pythonhosted.org/packages/d0/8e/29062a6f0e974f0a4eca3d8e3e9094a66b635a1f4c4399266a9c6628ef15/mplc-0.0.0.1-py3-none-any.whl (79kB)\n",
      "\u001B[K     |████████████████████████████████| 81kB 3.2MB/s \n",
      "\u001B[?25hRequirement already satisfied: loguru==0.4.1 in /usr/local/lib/python3.6/dist-packages (from mplc) (0.4.1)\n",
      "Requirement already satisfied: scipy==1.4.1 in /usr/local/lib/python3.6/dist-packages (from mplc) (1.4.1)\n",
      "Requirement already satisfied: scikit-learn==0.22.1 in /usr/local/lib/python3.6/dist-packages (from mplc) (0.22.1)\n",
      "Requirement already satisfied: pandas==1.0.5 in /usr/local/lib/python3.6/dist-packages (from mplc) (1.0.5)\n",
      "Requirement already satisfied: Keras==2.3.1 in /usr/local/lib/python3.6/dist-packages (from mplc) (2.3.1)\n",
      "Requirement already satisfied: ruamel.yaml==0.16.10 in /usr/local/lib/python3.6/dist-packages (from mplc) (0.16.10)\n",
      "Requirement already satisfied: matplotlib==3.1.3 in /usr/local/lib/python3.6/dist-packages (from mplc) (3.1.3)\n",
      "Requirement already satisfied: seaborn==0.10.0 in /usr/local/lib/python3.6/dist-packages (from mplc) (0.10.0)\n",
      "Requirement already satisfied: numpy==1.19.0 in /usr/local/lib/python3.6/dist-packages (from mplc) (1.19.0)\n",
      "Requirement already satisfied: tensorflow==2.2.0 in /usr/local/lib/python3.6/dist-packages (from mplc) (2.2.0)\n",
      "Requirement already satisfied: librosa==0.8.0 in /usr/local/lib/python3.6/dist-packages (from mplc) (0.8.0)\n",
      "Requirement already satisfied: aiocontextvars>=0.2.0; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from loguru==0.4.1->mplc) (0.2.2)\n",
      "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn==0.22.1->mplc) (0.16.0)\n",
      "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas==1.0.5->mplc) (2018.9)\n",
      "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas==1.0.5->mplc) (2.8.1)\n",
      "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from Keras==2.3.1->mplc) (3.13)\n",
      "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from Keras==2.3.1->mplc) (1.0.8)\n",
      "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from Keras==2.3.1->mplc) (2.10.0)\n",
      "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from Keras==2.3.1->mplc) (1.1.2)\n",
      "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from Keras==2.3.1->mplc) (1.15.0)\n",
      "Requirement already satisfied: ruamel.yaml.clib>=0.1.2; platform_python_implementation == \"CPython\" and python_version < \"3.9\" in /usr/local/lib/python3.6/dist-packages (from ruamel.yaml==0.16.10->mplc) (0.2.2)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.1.3->mplc) (2.4.7)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.1.3->mplc) (1.2.0)\n",
      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.1.3->mplc) (0.10.0)\n",
      "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0->mplc) (1.32.0)\n",
      "Requirement already satisfied: google-pasta>=0.1.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0->mplc) (0.2.0)\n",
      "Requirement already satisfied: tensorboard<2.3.0,>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0->mplc) (2.2.2)\n",
      "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0->mplc) (0.35.1)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0->mplc) (1.1.0)\n",
      "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0->mplc) (0.10.0)\n",
      "Requirement already satisfied: astunparse==1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0->mplc) (1.6.3)\n",
      "Requirement already satisfied: protobuf>=3.8.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0->mplc) (3.12.4)\n",
      "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0->mplc) (1.12.1)\n",
      "Requirement already satisfied: tensorflow-estimator<2.3.0,>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0->mplc) (2.2.0)\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0->mplc) (3.3.0)\n",
      "Requirement already satisfied: gast==0.3.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow==2.2.0->mplc) (0.3.3)\n",
      "Requirement already satisfied: pooch>=1.0 in /usr/local/lib/python3.6/dist-packages (from librosa==0.8.0->mplc) (1.2.0)\n",
      "Requirement already satisfied: numba>=0.43.0 in /usr/local/lib/python3.6/dist-packages (from librosa==0.8.0->mplc) (0.48.0)\n",
      "Requirement already satisfied: soundfile>=0.9.0 in /usr/local/lib/python3.6/dist-packages (from librosa==0.8.0->mplc) (0.10.3.post1)\n",
      "Requirement already satisfied: decorator>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from librosa==0.8.0->mplc) (4.4.2)\n",
      "Requirement already satisfied: audioread>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from librosa==0.8.0->mplc) (2.1.8)\n",
      "Requirement already satisfied: resampy>=0.2.2 in /usr/local/lib/python3.6/dist-packages (from librosa==0.8.0->mplc) (0.2.2)\n",
      "Requirement already satisfied: contextvars==2.4; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from aiocontextvars>=0.2.0; python_version < \"3.7\"->loguru==0.4.1->mplc) (2.4)\n",
      "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (1.17.2)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (1.7.0)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (1.0.1)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (0.4.1)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (3.2.2)\n",
      "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (2.23.0)\n",
      "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (50.3.0)\n",
      "Requirement already satisfied: appdirs in /usr/local/lib/python3.6/dist-packages (from pooch>=1.0->librosa==0.8.0->mplc) (1.4.4)\n",
      "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from pooch>=1.0->librosa==0.8.0->mplc) (20.4)\n",
      "Requirement already satisfied: llvmlite<0.32.0,>=0.31.0dev0 in /usr/local/lib/python3.6/dist-packages (from numba>=0.43.0->librosa==0.8.0->mplc) (0.31.0)\n",
      "Requirement already satisfied: cffi>=1.0 in /usr/local/lib/python3.6/dist-packages (from soundfile>=0.9.0->librosa==0.8.0->mplc) (1.14.2)\n",
      "Requirement already satisfied: immutables>=0.9 in /usr/local/lib/python3.6/dist-packages (from contextvars==2.4; python_version < \"3.7\"->aiocontextvars>=0.2.0; python_version < \"3.7\"->loguru==0.4.1->mplc) (0.14)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (4.1.1)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4; python_version >= \"3\" in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (4.6)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (0.2.8)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (1.3.0)\n",
      "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (1.7.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (2020.6.20)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (2.10)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (1.24.3)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (3.0.4)\n",
      "Requirement already satisfied: pycparser in /usr/local/lib/python3.6/dist-packages (from cffi>=1.0->soundfile>=0.9.0->librosa==0.8.0->mplc) (2.20)\n",
      "Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.6/dist-packages (from rsa<5,>=3.1.4; python_version >= \"3\"->google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (0.4.8)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.6/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (3.1.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->mplc) (3.1.0)\n",
      "Installing collected packages: mplc\n",
      "Successfully installed mplc-0.0.0.1\n"
     ]
    }
   ],
   "source": [
    "!pip install mplc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R1t8vwsxVY8g"
   },
   "source": [
    "## 2 - Context \n",
    "\n",
    "In collaborative data science projects partners sometimes need to train a model on multiple datasets, contributed by different data providing partners. In such cases the partners might have to measure how much each dataset involved contributed to the performance of the model. This is useful for example as a basis to agree on how to share the reward of the ML challenge or the future revenues derived from the predictive model, or to detect possible corrupted datasets or partners not playing by the rules. The library explores this question and the opportunity to implement some mechanisms helping partners in such scenarios to measure each dataset's *contributivity* (as *contribution to the performance of the model*).\n",
    "\n",
    "In the [first tutorial](./Tutorial-1_Run_your_first_scenario.ipynb), you learned how to parametrize and run a scenario.\n",
    "In this tutorial, you will learn how to add one of the contributivity measurement implemented to your scenario run.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 70
    },
    "colab_type": "code",
    "id": "S2CY2M0Hp6x8",
    "outputId": "aee525c3-e70b-4e46-84bf-207d8006258e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
      "  import pandas.util.testing as tm\n"
     ]
    }
   ],
   "source": [
    "# imports\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "sns.set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "esSTt9Qrp6yC"
   },
   "source": [
    "## 2 -  Setup and run the scenario\n",
    "\n",
    "We will use the same dataset, and overall setup for the scenario. The main change relies on the contributivity parameter, which is a list of the contributivity methods that will be tested. As these methods are time-consuming, the parameter is set empty. \n",
    "\n",
    "All methods available are:\n",
    "\n",
    "```python\n",
    "- \"Shapley values\"\n",
    "- \"Independent scores\"\n",
    "- \"TMCS\"\n",
    "- \"ITMCS\"\n",
    "- \"IS_lin_S\"\n",
    "- \"IS_reg_S\"\n",
    "- \"AIS_Kriging_S\"\n",
    "- \"SMCS\"\n",
    "- \"WR_SMC\"\n",
    "```\n",
    "\n",
    "See in the documentation the [dedicated section](https://github.com/SubstraFoundation/distributed-learning-contributivity/blob/master/subtest/docs/documentation.md#contributivity-measurement-approaches-studied-and-implemented) for explanation of the different methods.  \n",
    "\n",
    "Here we will use the Shapley value, a contributivity measurement which came from cooperative game theory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 306
    },
    "colab_type": "code",
    "id": "M4ZdwGV3p6yD",
    "outputId": "e78c8050-5cac-47b5-f51f-e8987378fce3"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "/usr/local/lib/python3.6/dist-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.\n",
      "  warnings.warn(msg, category=FutureWarning)\n",
      "2020-09-22 07:02:56.095 | INFO     | mplc.utils:init_gpu_config:136 - Found GPU: /physical_device:GPU:0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz\n",
      "11493376/11490434 [==============================] - 1s 0us/step\n",
      "2020-09-22 07:02:57.866 | INFO     | mplc.scenario:__init__:299 - ### Description of data scenario configured:\n",
      "2020-09-22 07:02:57.866 | INFO     | mplc.scenario:__init__:300 -    Number of partners defined: 3\n",
      "2020-09-22 07:02:57.868 | INFO     | mplc.scenario:__init__:301 -    Data distribution scenario chosen: random\n",
      "2020-09-22 07:02:57.869 | INFO     | mplc.scenario:__init__:302 -    Multi-partner learning approach: fedavg\n",
      "2020-09-22 07:02:57.869 | INFO     | mplc.scenario:__init__:303 -    Weighting option: uniform\n",
      "2020-09-22 07:02:57.869 | INFO     | mplc.scenario:__init__:304 -    Iterations parameters: 10 epochs > 3 mini-batches > 8 gradient updates per pass\n",
      "2020-09-22 07:02:57.870 | INFO     | mplc.scenario:__init__:310 - ### Data loaded: mnist\n",
      "2020-09-22 07:02:57.870 | INFO     | mplc.scenario:__init__:311 -    54000 train data with 54000 labels\n",
      "2020-09-22 07:02:57.870 | INFO     | mplc.scenario:__init__:312 -    6000 val data with 6000 labels\n",
      "2020-09-22 07:02:57.870 | INFO     | mplc.scenario:__init__:313 -    10000 test data with 10000 labels\n"
     ]
    }
   ],
   "source": [
    "from mplc.scenario import Scenario\n",
    "\n",
    "my_scenario = Scenario(partners_count=3,\n",
    "                            amounts_per_partner=[0.001, 0.699, 0.3],\n",
    "                            epoch_count=10,\n",
    "                            minibatch_count=3,\n",
    "                            dataset='mnist',\n",
    "                            methods=[\"Shapley values\"])  # <- Here is the difference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "tr2Bf8vSp6yP",
    "outputId": "c2ea5301-846c-4748-fe58-94bf65305504",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-09-22 07:03:05.810 | INFO     | mplc.scenario:split_data:548 - ### Splitting data among partners:\n",
      "2020-09-22 07:03:05.810 | INFO     | mplc.scenario:split_data:549 -    Simple split performed.\n",
      "2020-09-22 07:03:05.810 | INFO     | mplc.scenario:split_data:550 -    Nb of samples split amongst partners: 43738\n",
      "2020-09-22 07:03:05.811 | INFO     | mplc.scenario:split_data:552 -    Partner #0: 43 samples with labels [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n",
      "2020-09-22 07:03:05.811 | INFO     | mplc.scenario:split_data:552 -    Partner #1: 30573 samples with labels [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n",
      "2020-09-22 07:03:05.811 | INFO     | mplc.scenario:split_data:552 -    Partner #2: 13122 samples with labels [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n",
      "2020-09-22 07:03:20.830 | INFO     | mplc.multi_partner_learning:compute_test_score:136 - ## Training and evaluating model on partners with ids: ['#0', '#1', '#2']\n",
      "2020-09-22 07:03:20.877 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:301 - (fedavg) Very first minibatch of epoch n°0, init new models for each partner\n",
      "2020-09-22 07:03:43.604 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:03:58.802 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:04:14.484 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.583', '0.820']\n",
      "2020-09-22 07:04:14.584 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:04:29.763 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:04:45.241 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:05:00.832 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.303', '0.914']\n",
      "2020-09-22 07:05:00.881 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:05:16.134 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:05:31.180 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:05:46.987 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.192', '0.942']\n",
      "2020-09-22 07:05:47.134 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:06:03.114 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:06:18.727 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:06:34.492 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.118', '0.964']\n",
      "2020-09-22 07:06:34.541 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:06:49.785 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:07:04.960 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:07:20.835 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.081', '0.976']\n",
      "2020-09-22 07:07:20.883 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:07:36.067 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:07:51.531 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:08:07.098 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.079', '0.977']\n",
      "2020-09-22 07:08:07.143 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:08:22.943 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:08:38.094 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:08:53.706 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.073', '0.979']\n",
      "2020-09-22 07:08:53.755 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:09:09.147 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:09:24.213 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:09:39.655 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.065', '0.981']\n",
      "2020-09-22 07:09:39.699 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:09:55.285 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:10:10.466 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:10:25.890 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.063', '0.982']\n",
      "2020-09-22 07:10:25.937 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:10:41.593 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:10:56.713 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:11:12.752 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.064', '0.982']\n",
      "2020-09-22 07:11:12.752 | INFO     | mplc.multi_partner_learning:compute_test_score:204 - ### Evaluating model on test data:\n",
      "2020-09-22 07:11:12.908 | INFO     | mplc.multi_partner_learning:compute_test_score:206 -    Model metrics names: ['loss', 'accuracy']\n",
      "2020-09-22 07:11:12.908 | INFO     | mplc.multi_partner_learning:compute_test_score:207 -    Model metrics values: ['0.059', '0.982']\n",
      "2020-09-22 07:11:13.667 | INFO     | mplc.multi_partner_learning:compute_test_score:218 - Training and evaluation on multiple partners: done.\n",
      "2020-09-22 07:11:13.668 | INFO     | mplc.scenario:run:700 - Shapley values\n",
      "2020-09-22 07:11:13.668 | INFO     | mplc.contributivity:compute_SV:143 - # Launching computation of Shapley Value of all partners\n",
      "2020-09-22 07:11:13.668 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:84 - ## Training and evaluating model on partner with id #0\n",
      "2020-09-22 07:11:13.754 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:97 -    Training model...\n",
      "2020-09-22 07:13:22.322 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:107 -    Model evaluation on test data: [('loss', '3.803'), ('accuracy', '0.623')]\n",
      "2020-09-22 07:13:22.323 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:84 - ## Training and evaluating model on partner with id #1\n",
      "2020-09-22 07:13:22.406 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:97 -    Training model...\n",
      "2020-09-22 07:13:28.013 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:107 -    Model evaluation on test data: [('loss', '0.084'), ('accuracy', '0.973')]\n",
      "2020-09-22 07:13:28.015 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:84 - ## Training and evaluating model on partner with id #2\n",
      "2020-09-22 07:13:28.087 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:97 -    Training model...\n",
      "2020-09-22 07:13:31.776 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:107 -    Model evaluation on test data: [('loss', '0.075'), ('accuracy', '0.976')]\n",
      "2020-09-22 07:13:31.777 | INFO     | mplc.multi_partner_learning:compute_test_score:136 - ## Training and evaluating model on partners with ids: ['#0', '#1']\n",
      "2020-09-22 07:13:31.809 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:301 - (fedavg) Very first minibatch of epoch n°0, init new models for each partner\n",
      "2020-09-22 07:13:46.096 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:14:00.299 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:14:15.126 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.889', '0.705']\n",
      "2020-09-22 07:14:15.224 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:14:29.699 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:14:43.829 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:14:58.275 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.399', '0.874']\n",
      "2020-09-22 07:14:58.308 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:15:12.726 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:15:27.362 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:15:41.984 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.247', '0.926']\n",
      "2020-09-22 07:15:42.022 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:15:56.024 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:16:10.328 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:16:26.132 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.164', '0.946']\n",
      "2020-09-22 07:16:26.165 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:16:40.755 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:16:55.092 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:17:09.782 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.147', '0.957']\n",
      "2020-09-22 07:17:09.818 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:17:24.067 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:17:38.502 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:17:53.193 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.114', '0.966']\n",
      "2020-09-22 07:17:53.233 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:18:07.538 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:18:22.044 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:18:36.681 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.120', '0.964']\n",
      "2020-09-22 07:18:36.718 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:18:51.513 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:19:05.913 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:19:20.457 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.139', '0.961']\n",
      "2020-09-22 07:19:20.488 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:19:34.777 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:19:49.295 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:20:03.959 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.085', '0.976']\n",
      "2020-09-22 07:20:03.994 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:20:18.117 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:20:32.366 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:20:46.914 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.107', '0.974']\n",
      "2020-09-22 07:20:46.915 | INFO     | mplc.multi_partner_learning:compute_test_score:204 - ### Evaluating model on test data:\n",
      "2020-09-22 07:20:47.044 | INFO     | mplc.multi_partner_learning:compute_test_score:206 -    Model metrics names: ['loss', 'accuracy']\n",
      "2020-09-22 07:20:47.044 | INFO     | mplc.multi_partner_learning:compute_test_score:207 -    Model metrics values: ['0.098', '0.972']\n",
      "2020-09-22 07:20:47.083 | INFO     | mplc.multi_partner_learning:compute_test_score:218 - Training and evaluation on multiple partners: done.\n",
      "2020-09-22 07:20:47.086 | INFO     | mplc.multi_partner_learning:compute_test_score:136 - ## Training and evaluating model on partners with ids: ['#0', '#2']\n",
      "2020-09-22 07:20:47.107 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:301 - (fedavg) Very first minibatch of epoch n°0, init new models for each partner\n",
      "2020-09-22 07:21:01.674 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:21:15.845 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:21:30.295 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.743', '0.742']\n",
      "2020-09-22 07:21:30.313 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:21:45.691 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:21:59.971 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:22:14.338 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.560', '0.817']\n",
      "2020-09-22 07:22:14.356 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:22:28.496 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:22:42.690 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:22:57.390 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.299', '0.901']\n",
      "2020-09-22 07:22:57.409 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:23:11.618 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:23:25.891 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:23:40.376 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.188', '0.940']\n",
      "2020-09-22 07:23:40.394 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:23:54.447 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:24:09.010 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:24:23.522 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.222', '0.933']\n",
      "2020-09-22 07:24:23.538 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:24:37.820 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:24:52.149 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:25:06.560 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.174', '0.951']\n",
      "2020-09-22 07:25:06.579 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:25:21.207 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:25:35.486 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:25:49.912 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.138', '0.959']\n",
      "2020-09-22 07:25:49.930 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:26:04.155 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:26:18.466 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:26:32.886 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.169', '0.954']\n",
      "2020-09-22 07:26:32.905 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:26:47.851 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:27:02.095 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:27:16.447 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.109', '0.971']\n",
      "2020-09-22 07:27:16.465 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:27:30.794 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:27:45.160 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:27:59.394 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.131', '0.963']\n",
      "2020-09-22 07:27:59.395 | INFO     | mplc.multi_partner_learning:compute_test_score:204 - ### Evaluating model on test data:\n",
      "2020-09-22 07:27:59.537 | INFO     | mplc.multi_partner_learning:compute_test_score:206 -    Model metrics names: ['loss', 'accuracy']\n",
      "2020-09-22 07:27:59.538 | INFO     | mplc.multi_partner_learning:compute_test_score:207 -    Model metrics values: ['0.138', '0.963']\n",
      "2020-09-22 07:27:59.576 | INFO     | mplc.multi_partner_learning:compute_test_score:218 - Training and evaluation on multiple partners: done.\n",
      "2020-09-22 07:27:59.578 | INFO     | mplc.multi_partner_learning:compute_test_score:136 - ## Training and evaluating model on partners with ids: ['#1', '#2']\n",
      "2020-09-22 07:27:59.623 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:301 - (fedavg) Very first minibatch of epoch n°0, init new models for each partner\n",
      "2020-09-22 07:28:01.529 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:03.783 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:05.939 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.345', '0.894']\n",
      "2020-09-22 07:28:05.987 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:07.874 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:09.802 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:12.230 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.178', '0.948']\n",
      "2020-09-22 07:28:12.353 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:14.258 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:16.154 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:18.297 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.108', '0.967']\n",
      "2020-09-22 07:28:18.344 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:20.200 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:22.513 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:24.656 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.072', '0.979']\n",
      "2020-09-22 07:28:24.702 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:26.605 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:28.494 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:31.003 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.060', '0.983']\n",
      "2020-09-22 07:28:31.051 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:32.963 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:34.844 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:36.979 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.052', '0.984']\n",
      "2020-09-22 07:28:37.023 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:38.902 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:41.100 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:43.290 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.047', '0.987']\n",
      "2020-09-22 07:28:43.336 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:45.239 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:47.128 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:49.261 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.046', '0.986']\n",
      "2020-09-22 07:28:49.304 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:51.554 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:53.432 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:55.574 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.051', '0.986']\n",
      "2020-09-22 07:28:55.622 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:57.504 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:28:59.698 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:29:01.858 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.043', '0.988']\n",
      "2020-09-22 07:29:01.858 | INFO     | mplc.multi_partner_learning:compute_test_score:204 - ### Evaluating model on test data:\n",
      "2020-09-22 07:29:01.988 | INFO     | mplc.multi_partner_learning:compute_test_score:206 -    Model metrics names: ['loss', 'accuracy']\n",
      "2020-09-22 07:29:01.989 | INFO     | mplc.multi_partner_learning:compute_test_score:207 -    Model metrics values: ['0.039', '0.988']\n",
      "2020-09-22 07:29:02.023 | INFO     | mplc.multi_partner_learning:compute_test_score:218 - Training and evaluation on multiple partners: done.\n",
      "2020-09-22 07:29:02.025 | INFO     | mplc.multi_partner_learning:compute_test_score:136 - ## Training and evaluating model on partners with ids: ['#0', '#1', '#2']\n",
      "2020-09-22 07:29:02.070 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:301 - (fedavg) Very first minibatch of epoch n°0, init new models for each partner\n",
      "2020-09-22 07:29:16.992 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:29:32.036 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:29:47.524 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.663', '0.807']\n",
      "2020-09-22 07:29:47.571 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:30:02.547 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:30:17.497 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:30:32.944 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.254', '0.936']\n",
      "2020-09-22 07:30:32.991 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:30:48.069 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:31:03.174 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:31:18.820 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.165', '0.951']\n",
      "2020-09-22 07:31:18.871 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:31:33.852 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:31:48.713 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:32:06.045 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.108', '0.964']\n",
      "2020-09-22 07:32:06.089 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:32:21.069 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:32:35.922 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:32:51.110 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.089', '0.974']\n",
      "2020-09-22 07:32:51.157 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:33:06.738 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:33:21.800 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:33:37.076 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.081', '0.975']\n",
      "2020-09-22 07:33:37.120 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:33:52.438 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:34:07.508 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:34:22.574 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.067', '0.980']\n",
      "2020-09-22 07:34:22.623 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:34:38.042 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:34:53.090 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:35:08.259 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.068', '0.980']\n",
      "2020-09-22 07:35:08.307 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:35:23.535 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:35:38.415 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:35:53.684 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.061', '0.983']\n",
      "2020-09-22 07:35:53.732 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:36:09.310 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:36:24.216 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 07:36:39.284 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.078', '0.978']\n",
      "2020-09-22 07:36:39.285 | INFO     | mplc.multi_partner_learning:compute_test_score:204 - ### Evaluating model on test data:\n",
      "2020-09-22 07:36:39.421 | INFO     | mplc.multi_partner_learning:compute_test_score:206 -    Model metrics names: ['loss', 'accuracy']\n",
      "2020-09-22 07:36:39.421 | INFO     | mplc.multi_partner_learning:compute_test_score:207 -    Model metrics values: ['0.069', '0.979']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 4,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-09-22 07:36:39.457 | INFO     | mplc.multi_partner_learning:compute_test_score:218 - Training and evaluation on multiple partners: done.\n",
      "2020-09-22 07:36:39.460 | INFO     | mplc.scenario:run:704 - ## Evaluating contributivity with Shapley values: \n",
      "Shapley\n",
      "Computation time: 0:25:25.791579\n",
      "Number of characteristic function computed: 7\n",
      "Contributivity scores: [0.20238334 0.38973331 0.38658333]\n",
      "Std of the contributivity scores: [0. 0. 0.]\n",
      "Normalized contributivity scores: [0.20678793 0.39821531 0.39499677]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "my_scenario.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Xx9DUyR6p6yR"
   },
   "source": [
    "## 3 - Accuracy score between each partner and comparison with aggregated model performance\n",
    "\n",
    "Like in the first tutorial, we take a look at the scores, local and global."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 343
    },
    "colab_type": "code",
    "id": "XZwNSGhyVY8o",
    "outputId": "fa7b068b-772f-4a72-afca-6c57cd04324a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>partner 0</th>\n",
       "      <th>partner 1</th>\n",
       "      <th>partner 2</th>\n",
       "      <th>collective model</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.344889</td>\n",
       "      <td>0.815500</td>\n",
       "      <td>0.800389</td>\n",
       "      <td>0.513000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.658889</td>\n",
       "      <td>0.923333</td>\n",
       "      <td>0.909667</td>\n",
       "      <td>0.867167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.682722</td>\n",
       "      <td>0.951444</td>\n",
       "      <td>0.946667</td>\n",
       "      <td>0.923542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.878556</td>\n",
       "      <td>0.965444</td>\n",
       "      <td>0.960111</td>\n",
       "      <td>0.954667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.925000</td>\n",
       "      <td>0.970722</td>\n",
       "      <td>0.970000</td>\n",
       "      <td>0.968000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.856111</td>\n",
       "      <td>0.976444</td>\n",
       "      <td>0.974722</td>\n",
       "      <td>0.975250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.862944</td>\n",
       "      <td>0.981056</td>\n",
       "      <td>0.976000</td>\n",
       "      <td>0.977792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.938778</td>\n",
       "      <td>0.979444</td>\n",
       "      <td>0.978167</td>\n",
       "      <td>0.980125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.932000</td>\n",
       "      <td>0.981389</td>\n",
       "      <td>0.980556</td>\n",
       "      <td>0.981333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.934444</td>\n",
       "      <td>0.984833</td>\n",
       "      <td>0.980833</td>\n",
       "      <td>0.981625</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   partner 0  partner 1  partner 2  collective model\n",
       "0   0.344889   0.815500   0.800389          0.513000\n",
       "1   0.658889   0.923333   0.909667          0.867167\n",
       "2   0.682722   0.951444   0.946667          0.923542\n",
       "3   0.878556   0.965444   0.960111          0.954667\n",
       "4   0.925000   0.970722   0.970000          0.968000\n",
       "5   0.856111   0.976444   0.974722          0.975250\n",
       "6   0.862944   0.981056   0.976000          0.977792\n",
       "7   0.938778   0.979444   0.978167          0.980125\n",
       "8   0.932000   0.981389   0.980556          0.981333\n",
       "9   0.934444   0.984833   0.980833          0.981625"
      ]
     },
     "execution_count": 5,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history_df = my_scenario.mpl.history.partners_to_dataframe()\n",
    "accuracy_df = history_df[history_df.Minibatch==2].pivot(index='Epoch', columns='Model', values='val_acc')\n",
    "score_mpl_model = my_scenario.mpl.history.history['mpl_model']['val_accuracy'][:,-1]\n",
    "accuracy_df['collective model'] = score_mpl_model\n",
    "accuracy_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KsDY6LcYp6yU"
   },
   "source": [
    "We can plot the evolution of the accuracy through the epochs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 415
    },
    "colab_type": "code",
    "id": "bz5Iw4YOVY8r",
    "outputId": "81021e0c-0e47-4a49-ab53-b5144df9adb3",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.FacetGrid at 0x7f23e032f6a0>"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAF9CAYAAAAgOZDCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3hUVfrA8e/09Ep6AqEFQguBQECKUkQQBBGsa8OKgMiKIq6uyq6oqCu7gmX9qbsqNrCAIE0QQQUC0gIktBAS0ia9TTL1nt8f0SwREBImmUk4n+fJk8nMnXPfc+9k3nvOPfdclRBCIEmSJEmS21G7OgBJkiRJks5NJmlJkiRJclMySUuSJEmSm5JJWpIkSZLclEzSkiRJkuSmZJKWJEmSJDclk7TkdpYsWcJjjz3W5Pffd999fP31106M6PxGjhzJ9u3bW2RdF9KtWzeysrKa/P477riDFStWODGilpOTk0O3bt2w2+2uDkWSnEom6WZ0xx13MGDAAKxWq6tDabPOldDfffddJk+e7PR1zZ8/n8WLFzu9XFe41AMhSZJahkzSzSQnJ4dffvkFlUrF5s2bW3TdsjXhnuR+kSSpsWSSbiYrV64kISGByZMns3Llygav5efnM2vWLAYNGkRycjJ/+9vf6l9bvnw548aNIzExkWuvvZbDhw8DZ3dlntmqS0lJYfjw4bzzzjsMGTKEJ598koqKCh588EEGDRrEgAEDePDBBykoKKh/f3l5OU8++SRDhw5lwIABzJgxA4AJEybw/fff1y9ns9lITk4mLS3tnPXcsmULkyZNIikpiVtuuYUjR44A8M477zB79uwGyz7//PM8//zzABiNRqZPn87AgQO5+uqrWb58+TnL/61uZ/qti3nbtm38+9//Zt26dSQmJjJx4kSgYbetoii8+eabjBgxgsGDBzNv3jyqqqqA/3WRfv3111x11VUkJyfz1ltvnTOOzz//nNWrV/Pee++RmJjI9OnT619LT0/nuuuuo3///syZMweLxXLe/WK1Wlm4cCFDhw5l6NChLFy4sL6n5auvvuLWW29tsN4z93tZWRnTp0+nX79+TJkyhcWLF5+1/Pbt2xkzZgxJSUksWLCAc00oeL7tBpCbm8stt9xCYmIi99xzD6WlpfWv7d+/n1tuuYWkpCQmTpxISkrKObcV1O3fhx9+mEGDBjFy5Eg+/PDD+teWLFnC7NmzmTNnDomJiUyePLn+cwOQkZHBHXfcQVJSEuPHj29wkGs2m3nppZcYMWIE/fv359Zbb8VsNte/vnr16nPuy9TUVG644Qb69evHFVdcwYsvvnje2CXJrQipWYwePVosW7ZMHDx4UPTo0UMUFRUJIYSw2+3iuuuuEwsXLhQmk0mYzWaxe/duIYQQa9euFUOHDhUHDhwQiqKIU6dOiZycHCGEEHFxceLUqVP15T/xxBPitddeE0IIsXPnThEfHy9efvllYbFYRG1trSgtLRXr168XNTU1oqqqSjz88MPioYceqn///fffLx555BFRXl4urFarSElJEUII8c4774hHHnmkfrnvvvtOTJgw4Zx1PHz4sBg0aJDYv3+/sNvt4quvvhIjRowQFotF5OTkiD59+oiqqqr6eg8ZMkTs27dPCCHEbbfdJp599llhNptFWlqaSE5OFtu3bxdCCPH666+LuXPn1tdt2LBhDdY7YsQI8fPPP5+17G9uv/12sXz5ciGEECtWrBCjR48W2dnZorq6WsycOVM89thjQgghTp8+LeLi4sRTTz0lamtrRXp6uujZs6c4ceLEOet75jY/M5YpU6aIgoICUVZWJsaOHSs++eST8+6Xf/7zn+LGG28UxcXFoqSkRNx8881i8eLFQgghvvzyS3HLLbc0KP/M/T5nzhwxZ84cUVNTI44fPy6GDx/eYPm4uDjxwAMPiIqKCpGbmyuSk5PF1q1bz1mX8223UaNGiZMnT4ra2lpx++23i1deeUUIIURBQYEYOHCg+OGHH4TD4RA//fSTGDhwoCgpKTmrbIfDISZPniyWLFkiLBaLyM7OFiNHjhTbtm2rX3ePHj3EunXrhNVqFe+++64YMWKEsFqtwmq1itGjR4u33npLWCwWsX37dtG3b1+RkZEhhBDiueeeE7fffrsoKCgQdrtd7NmzR1gslgvuy5tuukl8/fXXQgghqqur6z+HkuTuZEu6Gfzyyy/k5eUxbtw4evXqRUxMDGvWrAHqjugLCwuZN28eXl5eGAwGkpKSAPjiiy+477776NOnDyqVig4dOhAVFXVR61Sr1cyePRu9Xo+HhweBgYFcc801eHp64uPjw0MPPcTu3bsBKCwsZNu2bSxYsAB/f390Oh0DBw4EYOLEiWzdupXq6moAvvnmmwYtrTN9/vnn3HzzzSQkJKDRaJg8eTI6nY79+/cTFRVFjx492LRpEwA7d+7Ew8ODvn37kp+fz969e3nssccwGAzEx8dz4403smrVqqZv9PNYvXo1d999NzExMXh7e/Poo4+ydu3aBl3Ps2bNwsPDg+7du9O9e/cGrbqLcccddxAWFkZAQAAjRowgPT29/rXf75fVq1czc+ZMgoODCQoKYubMmXzzzTcXXIfD4WDjxo08/PDDeHp60qVLF66//vqzlrv//vvx8/MjMjKS5OTkRtflhhtuoGPHjnh4eDB27Nj6uqxatYrhw4dz5ZVXolarGTJkCL169WLr1q1nlXHw4EFKS0uZNWsWer2emJgYbrrpJtauXVu/TM+ePRk7diw6nY5p06ZhtVo5cOAABw4coKamhgceeAC9Xs/gwYMZMWIE3377LYqi8OWXX/LUU08RFhaGRqOhX79+6PX6+nLPty+1Wi3Z2dmUlpbi7e1N3759G7VdJMlVtK4OoC1auXIlQ4YMISgoCKjrQv7666+5++67yc/PJzIyEq327E2fn59P+/btm7TOwMBADAZD/d+1tbW8+OKL/Pjjj1RUVABgMplwOBwUFBTg7++Pv7//WeWEhYXRr18/NmzYwNVXX822bdt46qmnzrnOvLw8Vq5cybJly+qfs9lsFBYW1td7zZo1XH/99axZs4YJEyYAdQcJ/v7++Pj41L8vMjKSQ4cONanuf6SwsLDBgU5UVBR2u52SkpL659q1a1f/2NPTk5qamkatIyQkpMH7f6s/nL1fCgsLiYyMrP87MjKywfLnU1pait1uJyIiov65Mx+fLxaTyXTxFTnH+3/bFnl5eaxfv54tW7bUv26320lOTj6rjNzcXAoLC+sPPqHuIOPMv8PDw+sfq9VqwsLC6rdDeHg4avX/2g+RkZEYjUbKysqwWCzExMScN/7z7cuFCxfy+uuvM27cOKKjo5k1axYjRoy48AaRJBeTSdrJzGYz69atQ1EUhgwZAoDVaqWyspIjR44QERFBfn4+drv9rEQdERFBdnb2Ocv19PSktra2/u+ioiLCwsLq/1apVA2Wf//998nMzGT58uWEhISQnp7O9ddfjxCC8PBwKioqqKysxM/P76x1TZ48mRUrVuBwOOjbt2+D9fw+3unTp/PQQw+d8/Vx48axaNEiCgoK+O677/j8888BCA0NpaKigurq6vpEnZ+ff871eHp6Njjn6HA4Gpwn/X29fy80NJTc3Nz6v/Py8tBqtQQHBzc4R38xLrSui3lPaGgoeXl5dO3aFaird2hoKHB2XYuKiuofBwUFodVqKSgooGPHjvXvbarG1iUiIoJJkybVjym40LLR0dFs3LjxvMucue0VRcFoNNZvh4KCAhRFqU/U+fn5xMbG1h/wnD59mu7duzcq/tjYWF577TUURWHjxo3Mnj2blJQUvLy8GlWOJLU02d3tZJs2bUKj0fDtt9+ycuVKVq5cydq1a0lKSmLlypX06dOHkJAQ/vGPf1BTU4PFYmHPnj0ATJ06lffff59Dhw4hhCArK6s+wXTv3p01a9bgcDjYtm1bfdf1+ZhMJgwGA35+fpSXl7N06dL610JDQxk+fDgLFiygoqICm83WoLzRo0eTlpbGhx9+eM4u1d/ceOONfPbZZxw4cAAhBDU1Nfzwww/1XeVBQUEMHDiQJ598kujoaDp37gzUfYknJiby2muvYbFYOHLkCF988cU5u9U7duyIxWLhhx9+wGaz8dZbbzW4pC04OJjc3FwURTlnjBMmTOCDDz7g9OnTmEwmFi9ezLhx487Zk3EhwcHB5OTkNPp9Zxo/fjxvvfUWpaWllJaW8sYbb3DdddcBdfv4+PHjpKenY7FYWLJkSf37NBoNV199NUuXLqW2tpaMjIxLOj1woe32exMnTmTLli38+OOPOBwOLBYLKSkp5zzQ6dOnD97e3rzzzjuYzWYcDgfHjh0jNTW1fpnDhw+zceNG7HY7H3zwAXq9noSEBPr06YOHhwfvvvsuNpuNlJQUvv/+e6699lrUajVTpkzhxRdfxGg04nA42Ldv30Vd4rhq1SpKS0tRq9X1B6ZnttYlyV3JT6mTff3119xwww1ERkYSEhJS//OnP/2J1atXI4Tg7bffJisrixEjRjB8+HDWrVsH1LU8p0+fzty5c+nXrx8zZ86s76p+6qmn2LJlC0lJSaxevZrRo0f/YRx33XUXFouFQYMGcfPNNzNs2LAGr7/88stotVrGjRvHFVdcwQcffFD/moeHB2PGjCEnJ4err776vOvo3bs3f//73/nb3/7GgAEDGDNmDF999VWDZSZMmMD27dvru7p/89prr5Gbm8uwYcOYNWsWDz/8MFdcccVZ6/D19eXZZ5/l6aefZvjw4Xh6ejboKh07diwAycnJ57w2esqUKUycOJHbb7+dUaNGodfr+etf//oHW+78pk6dyokTJ0hKSqofDd9YM2bMoFevXkycOJGJEyfSs2fP+rI6duzIzJkzufvuuxkzZgz9+/dv8N5nnnmGqqoqhgwZwrx58xg/fnyD87GNcaHt9nsRERG8+eab/Pvf/2bw4MFceeWVvPfee+dM8hqNhrfffpsjR44watQoBg0axNNPP11/8AYwatQo1q5dy4ABA1i1ahVLlixBp9Oh1+t5++232bZtG4MGDWLBggW8/PLL9Qd4TzzxBHFxcUydOpWBAwfy6quvXtSBxo8//sj48eNJTExk4cKFLF68GA8Pj4vdXJLkMiohznGNhnTZW7p0KadOneLVV191dSjSebzyyisUFxezaNEiV4fSKEuWLCErK0t+tiTpIsiWtHSW8vJyvvzyS26++WZXhyKdISMjgyNHjiCEIDU1lS+++OIPezokSWr95MAxqYHly5fzwgsvMHHiRAYMGODqcKQzmEwm5s6dS2FhIcHBwdxzzz2MGjXK1WFJktSMZHe3JEmSJLkp2d0tSZIkSW5KJmlJkiRJclNt7px0SUk1itK4HvzAQC/Kyho3y1Rrd7nV+XKrL8g6twYhIb6uDkFyc7IlDWi1GleH0OIutzpfbvUFWWdJagtkkpYkSZIkNyWTtCRJkiS5qRZJ0osWLWLkyJF069aNY8eOnXMZh8PBggULGD16NFdffTUrVqxoidAkSZIkyW21SJIeNWoUH3/88R/eG3n16tVkZ2ezceNGPv/8c5YsWXLJNzOQJEmSpNasRZJ0UlLSOe99e6a1a9dy4403olarCQoKYvTo0axfv74lwpMkSZIkt+Q2l2Dl5+cTGRlZ/3dERESj7/cLEBzs06T1X46XQlxudb7c6guyzpLU2rlNknaWplwnHRLiS1FRVTNF5J4utzpfbvUFWefWQB5QSBfiNqO7IyIiyMvLq/87Pz+/wX2DJUmSJOly4zZJeuzYsaxYsQJFUSgtLWXTpk1cc801rg5LkiRJklymRZL0888/z/DhwykoKGDatGmMHz8egPvvv5+DBw8CMGnSJKKjoxkzZgw33XQTM2fOJCYmpiXCkyRJkiS31OZuVSnPSV+cy63Ol1t9Qda5NZDnpKULaXMDxyRJkppCCAF2K8JmBpsZ8euP2icYtW87lMpC7Kf2IGyWumWsv74eFI0hcYKrw5faKJmkJakNEori6hCcSggBioJKo8FRW4Mw19YlS0VB7eGDxt8PS8ZByjLsmC0WNB5arLl5KHYbuk4DQIDj9C7UtjKshaUoVgtaDwXsFlTdJqDxC8NyeDP2EynoPAE12Ex16/YZOQVNh8HUHtyJdddXCACNFo8QTxx2PaKwBm03ExovbxduIamtkklaktyYcNgRpjKE1YQwm+p+W2pQ6TzQdRmEEArmTW8iLKYzfmrAbibkyeUA1Kx9FUf+MdBoUKm1oNGCWovniAfQhHfFdmQb1qPbADWoNKj0ehSrQBXWA01EPI4yI/bMXRhCg7AUVWKvtqAL8sOzSzeqM6tQzLXYCjLAoeARE4V3jy4Urd6MsFoJnnoHai9vjO8uQVjNCIeC4nAQmNQZi0ahalcOGrUWw9RhVP68FW1aCQ4hEEKgUcA8bRTW0yZ8N+zA5qvm2BBP+u00YSn/3zayhAex46YErvlwC9nlcLKzgSHBFnL2gMYB8C0AGXE+DIvxIG9rCRq7IHhQe3b46Yh7670G2zx04lX8pC0h7qu68TJbR1aifP0aCTvzzljKTuqY4Th27iFxcyqeg4xoOnZqvg+CdNmSSVqSmlF9F6qlGmGpqU+kmtBOqL0DseelYz+5+6wkq4sbgj7xOpTiU1R/8TxCof5HpQV9VCyWGk8Ucy2iyIhaq6Um34ZQeQG+CKHGlJGBza6l9GAVwuxP8FW9qc0yUrn/FMJRi9izFByCgLvHU5lViS21EJWfHtuVYfikFGEuTK2vh+KjJneID112VWEpE5S219LHXoBxwynUZguoQNFATsVxkvM2UpSlwi5UBNTW8F3hTtpVGAm32ig2aKjWqvHLS+f7UA+CPSz0DuvF95XHKfG3MiXchwytg1KNwkCVH7uVcgo0+dwxZjT7Kg9zIFjHVYN7k1qWS7GwMKJdL9J9VGjVaoImTuFY5Wlq9Ha8kiZT0f47Si3VDGs/hPTS41gNVnyG3klV1BeUmkrodMUtBJcc5GRAOsOiBpNeeowScxmdBt1Au5JDnPIIZmj0YMq8KilO8iDmymmklR2jtLaModGDqPSooDhZT/vRvdFHnn/KY0m6FHLgGK1vsIkzXG51dlZ9hRDgsKHS6gGwn05FmMpRzJWI2ipEbSX2yjI03a+jqqyEyj3r8C7OpzZaTU2NisB8BzUGFTVTrycmS03Jrh+wVZfj6VBh8dKSf2UUcbsrMJ0upbpfHP3umsXRx/6M2myrj6E4oT2DH3qKo48+grrGTMSDM0jVFBPyZl3LWWjU2LVqej/6KOsrjhD06UZC/MLZN6k3xrS9TCgK4bS5kBJ7JUnRA/ghwYPTJ/YzTTeIPZZMNodV8GzITfyYvokM02nu6ns73xft5idtDs/3mMWaY2s5UHOSZ6/6C6uPr+dw+Qme6HUXG3O3c6Iqm4c6TeSnksOcqjFyZ587+MW4n9yqPCZ1GcfhkqMU1ZYwImYoJytOUW6ppF9oH/JNRky2GroEdKTcUoHNYSfEKxiLwwqAQaNv0f3cUuTAMelCZJKm9f1jO8PlVuc/qq+wW+oTrDBXofLwQRPaGaW2krKdn1BtqaRdbS25lgpKTDXEmT3I7zyY8sLT9KjJYo+HlYCDNgJrFPaPDKayxMrQXxquK/XWBEqzshl1wEK5v57PRvvzRGkC2ft/pkwx0T2sB5nqClb1tPNX3RgOHPyBA17lzLr9FTZ/tYTsitNM6XEDu0sPcRgjs8c9xaadyzlelcWDw2axOX8HxwuP8dCAB9iau4Pj5Rn8ZeRMvjnwPZmV2dzWfQr7Cg+SU53HdZ2uIb30GIU1xVwZfQWZFdmUWcrpF9qHApORKquJroGdKLdUYLFbCPMOxWw3owiBl84TIQQqlaoldlujtbbPtUzS0oXIJE3r+8d2hrZa598+ziqVCqXCiFKRj722CjtV6MqrKagpoTi8A/3jr+VQ2rccPbKBMcWV7DUYyFTrGJ9XTZpfKGm+8ODUv5Ky6DHUFXY6xUXyU4yenl9nNFhfu8kTORRrQPvVJjq1703RsF7kVeYxVNOJYr2NKg/oGZVAkbWcapuJTv4dKDOXU2s3E+kTTrXVhFWxEuQRiNluQREKXjpPHIoDlUqFWtX0qQza6j7+I62tzjJJSxciz0lLbksIUXcpjKUaYa5GmKsQVjO6zgMBsOxdhVJy+tfXqqmyVpEvzPQZ+xcOiip2HF7Fn44d5ztfL46o9Nx3qoJMgydfJGTSucQHx4ZNqL1saK+ZQvCKjUQUVVEGRFCIT2woKoMPodF9MPuV4XnFdYzoGovJN4WA0Bh0gUFoAwNR6/UMB+hZN0FPOND71/jPHOsbqm1HKO0ACPQIIPDX53303vVLemgN9ctr1Jpm2qqSJLUmMklLLUY47KDYUek8UGorceSm1SXe3xKwuRq1XwiGgTciHHaq/zMdFHv9+xVApVJTHtGRo2UZJBSf4lBNPht94BFDBPv1AayzGnn6+AlE1nHK2+sw+Q+n93fb6E0NFUAkNcy/6wk4fIogvzCiusbh2e86YrUx4HCgDQhEGxiIxs8flUpF53tnNqhDwPAxLbvRJEm6rMkkLTULoThQyvJwFJ1EKTqFoygTpfQ0+j7jMAyciqgwYv7+7V+XVqEyeIOHD2j1VFqr8NJ6UtrrSvYoFYwMiOe4qObjwp083XMaGRVZfHrkSzr3vJ+Ak8cYnP4L5bU2EoZcQSc/LWUvv4k3MPfFl3GEV2HwDye0ZxwmjSfagEDUnp6ohnUgYNiV9fH69Onrku0kSZL0R2SSli6ZEAqiwoijKBN1cAc0QVHY0n/A8vNHdQvoPNGExKLvfQ2a9n0AsAeGUzR+NmH+MZQpNjZmb2FCp2vINxl556e/My/pYaq7DmBL6n/pHz2JSLWWcTmFmH/YRVzXOJ7rO4eyJ55CA8SqVBARgY+iIbxLErWPz8fQvgMaT08ICcWzU2cCQ3yxt6JzlZIkSSCTtNREjsKT2DN/wVGUiaP4FFhrAdD3n4wmKAptTB9UIx5ACW5PjtpGqFcIZdYqPjnyJTd46BEIFh/9jBkJ9+Cr9+FU5WmqrNV08Ivm5pARGA6ewDPXyLzsAFQHlhM1Zy4cr8F0cCceBm9CEgeiv/te9BERGKJjUBv+dz7Xq1t3F20VSZIk55JJWvpDSk0FSlFmXTIuykSfcC3ayO44CjOwHtyAOigGXedBaEI6YguOwuEfTpW1mq9yNjMoPAl/g57XUl7n9vib6BHUDb2mLkFH+0QwveedtPeJQl9SxazsGBwH1xLw4Azi9uZT+eM2VFot+ugYPLvU3Q0t7I67UHt5odbVXTPrP3SYKzeNJElSs5NJWqonLCbQ6lFpdFgPbsSauh5hKq17UaVCHRAFtroWszZuGKpuQ9mWv5ton0hifKOY9+NzTOo8jiujh3C87CTdA7vSNbATD/S+i87+sXjUWLnb1B3ztz9RmHUKj5zTaObMRVGpqdi2FUNMe4TdTtCYsQSMGIUhMgqV9n8fUa1/gCs2iyRJksvIJH2ZUqxm7PlHf20l1w3sEpVGPK99HG10T1QGbzThcWhCOqIOiSVD60Cv96aDXwwLU16jZ3B3JnUex7pTm7kicgDdgrpwQ5fxdAnshF6j4/khf6lfV1yFHo29AocQGP/7PmoPDwztO+B/5Qg0Pr7ow8PpsvRtVOq6a4L1EZGu2iySJEluRSbpy4gwV4PeC5VaTdG3b1Kb9jMAKu+gumTcbRi1Xj7U1BQRFjeEL4QRL63CpIhufLbjZSJ8wnmg953EB8cR5ROBSqXi71fMx1PrCcDI9sPPWmfBf9+n8qdt+A0ZRvi0e+m46FW0gUH1CVmSJEk6P5mk2zghBErRSayHv8d+MgXPqx9G2z4B/wHXUhuViGdYV3aWH6XSWs3Y2JEs+eUN1Co1j/Z/CEVRUETdLQ/v630H/no/AG7o8r975/6WoH+j2KxUbt9O5U/biJ77OD59EzG0b4//kLrzx7rgdi1Uc0mSpNZPJuk2Stgt2E7sxJb2PUpxFug80HUbTqWnF0HA+3m7SDdm8EzHJE6cyqS4toSxsSOZ0GkMGlXdbFd/ip9aX16UT8Qfrs9RW0v55u8o37wJR1Ulhg6x2MvK8Omb2JzVlCRJatNkkm5jhKKgUquxZ+zCsu0/qAOjMQy9E12XwWwp2M3qw//l2cHzSIrsQ7C2HUIIbo+/sX6O6O5BXRu1PltxEUptLRr/AErXrsEzrjtBY8fh2a27296EQZIkqbWQSboNEIoDe9Y+bGnfo/YNxWP43Wg7D8TTL5Ta4Ci+PLmBIZYSerfriVWx4an1pGt4XzoaOgOgomnJtHTDOoq/XIFnl67EzHuSji++LEdgS5IkOZFM0q2YUlOOLX0rtiM/IExlqLyD0MYkAGBBUOrrT6Baw8HiNKJ9oxgWNYixsaOavD4hBDWHD1H23QbC7pyGR8dOBF49hoBRdfNZywQtAVSarOi0ajwN8utFki6V/C9qpYS1FtNn88BuRRPdC/2QO9G074Pq17snvZ36Xyqt1Tyd/CjPDX4CvUbf9HXZ7VTtTqF0w3qsOafRBARgKzTiFd8Dr7huzqqS1AZ8u+MUX249CYBBryHAW0+AjwF/n7rfdT8Nn5PJXJLOT/53tBLCWoPt2M/YT+3F89q5qPSeeAy7G01oZ9T+YQAcLzvJyoy1zEy4lwmdrkGn1qJWqZucoB21tdiKCtFHRFC0/HM0vr6ETbsXv+TBDSYZkSSAH1Pz+HLrSfrHhdApyo/yKisVJgvlVRZOFVRRXl2M1aac9T6DTkOAjx7/MxJ4w8Re99tDr3H5OAdFEVjtDqx2BZtNqXtsUwj0M+Dn1fQDYUk6H/lN6+YcxVnY0rZgO7Ed7FbUIZ0QNRWofILRdb0CgONlGQR6BOCh9cCm2Ci3VNAloOMlrbdq7x6M/3kXjY8vsQtfIuYvT6MLbievb5bOKTWjmA/WHaVnxyAenNQTrebsz4kQArPVQXm1hfJqK+XVFip+/f3bc1kFVRyoLsFic5z1fr1OXZe0vfUE+Brw9zYQ4KsnwNtQn+StqCgwVmGzK1jtClab49fHZyfWM5//33IKNtuvz/3+ebsDu6/OgssAACAASURBVEOcs/5R7bz5+33JTt+ukiSTtBsz/7wM2+FNoNGh6zIIXY+RaEL+l3wdigOLw8KbB95nYHg/bu0+hScHzGlya8OSl0vZhvUEjBiJISICr569CLpmHCq1Gn1IqLOqJbUxJ/MqeXPlIWLCfJhxfa9zJmgAlUqFp0GLp0FLRLD3H5ZZa7HXJ+6KM5L6b4k9q6CK8vMk84ul16rR6zTotGr0WjU6rQaDTo1Oq8bPW1///P+W0dQtpzv7cXTIH9dHkppKJmk3olQWYUvfgrZjfzShndG274PaNwRdt6F191s+w+qM9Rwty2Bu/xnM7Hsf7X2jARqdoIUQ1B4/Rtn6tZhSD6DS6/HsGof/0GFETp/ptLpJbVNBaQ3/XHEAf289c25McNr55cYk8wqTlfIqC+UmC74+Hphrreh1mvrE2yCx6uoSr1ajdnnXuSRdDJmkXUwoCo6cVKxpW3Bkp4IKVF7+dUk6pg/E9KlfttZey9ac7YyMGU6YdyhWxYZdsTepa1s4HFhyc9FHRlK47EMclZUET5pMwFUj0fj6OrOKUhtVUW3htc/3o1LBozf3xd+75c/J/pbMw4O8AAgJ8aVI3jdcakNkknYhR1EmtZveQFQVo/L0R9/vOnTdr0TtE9xgObtix6bYOV2Vx+qTG4j0DmdgeD8Ghvdr0nrNmSfZ+/Q7WCsr6fTyP4icMQttYFCDezK3FTVmO599f5zjORXMnNyL6BAfV4fUJtRa7CxecYCqGhvzbkskLNDL1SFJUpskk7QLqf1CUfuHoxt4E9rYfqg0Z+8Ou2LnhV2L6RrYmVu73cCzgx4n1CukSetz1NTgqKxE264d+nbBBE6egkpvQB/+x1N+tlbpp0p5f206pVUWfDx1vPLpPubd1o+odvL84aWwOxSWfnWQ3CITs6f2oWOEn6tDkqQ2SyWEOPdwxVaqpKQaRWlclVq6i8yetQ+VTzs0wTHnXeZ42UlSiw8zpet1bMreSpR3BPHBcU1ep9VoJG/JPxFCIXbBQkIjAttst6DF5uCLHzLYvCeHsCAv7psQT3SEP/OX/oQAnrgt8YLnOtuC5vhcK0Lw7uo0dqYZuXd8PEN6u9cBXmvr7g4JkaeWpD8mr6dpYUpNBbU/vItlxyfnfD2vugCH4iC7Kod9hQepslYzuv2Vl5Sga9LTyF74N+xVlYTdcXebvsY5I7eC5/6zm817chjdP5rnpg2gc6Q/0aG+PH5r3c0+Xv50HwWlNS6OtHVaseUEO9OMTL2qs9slaElqi2SSbmGWHZ+CzYJh6B1nvZZdmcMLuxaTUrCH4dFX8Oygx/HVX9o5VHtFObmvL0Yb4E/7p5/Fq3v8JZXnruwOhS+3ZvDCsj3Y7Q4ev6Uvt10dh0GnqV8msp03j9+aiKIIXv5kL0aZqBtlw65sNuw6zaj+0YxLbu/qcCTpsiCTdAuyn07FnrETfd/xaAIigbpLoFZlrGNd5iZifKOY0vU6EkJ6oVNr0Wl0TV6XsNup+PknNH7+RDw0k5gn/9pmr3U+XVjN3z/4hW93ZDGkVwQL7kkmPjbonMtG/Zqo7Q7By5/uo7BMJuqLsTOtgM+/P0FS91BuHdVVXr4kSS1EJukWIuwWzD99hNo/HH3iBGyKnYPFaahUKsrM5ZRbKlCpVIyIGYq37tJGyipWK7n/eg3jf96l9kg6Pn36ovH0dFJN3IdDUfh2xyn+9t/dVJisPDylN/eMj8fL44+786NDfHjslr5YbQ5e/nQfReW1LRNwK5V2qpT31qTTLSaA+yfEo1bLBC1JLUUm6RailJxGWKoxDJ+GSqNjzckNvJ36X4ymQu7scTO3dp/inPWYa1HpdOhCQwmbdi9e8T2cUq67MZbW8NKyvXy59SSJXdvx93sHktj14ke9tw+rO0dtsTp4+ZN9FMtEfU5ZBVUs/eogEcFePDylNzqt5sJvkiTJaVpsdHdmZibz58+nvLycgIAAFi1aRGxsbINlioqKeOaZZ8jJycFutzN9+nQmTZrUqPW48+huYa1FpfdEEQpCCDIrsy95ju0zmQ6lkv9//ybigYfw7tnrD5dtbaNgf6MIwZa9uazYcgKdVs2fxsSRHB92we7X89U3q6CKVz7dh5eHlidu60ewv0dzhd7iLnUfF5XXsvCjPeg0Kv5yRxKBvu5/HX1r+1zL0d3ShbRYS/rZZ5/ltttuY8OGDdx2220888wzZy3z0ksv0atXL1avXs3HH3/M4sWLyc/Pb6kQm4VQFKwH1tUnaLPdzIu7/smB4sNOS9BCCMo2bST3X4vRBQWhDw93SrnuprTSzD8+28/H3x0jrn0Af7s3mUE9wi/p/GiHcF/m3tIXk9nOy5/upbTS7MSIW6/KGiuvfb4fh0Phzzf1bRUJWpLaohZJ0iUlJaSlpTFhwgQAJkyYQFpaGqWlpQ2WO3LkCMOGDQMgKCiI7t27s27dupYIsdnY0jZjSfkce84hAMwOC356XwIMzpsAouSblRR99gnefROJeeIpdMHtnFa2OxBC8PPBfP76Xgon8yq5c2w3/nxjgtMSR8cIPx69OYHqWhsvf7qPsiqLU8ptrSxWB/9akUpplYVHpiYQKSd/kSSXaZEknZ+fT1hYGBpN3fksjUZDaGjoWa3knj17snbtWoQQnD59mn379pGXl9cSITYLpboEy+4v0cT0RtsxiXyTEYvdwsOJ99PJP/aSy3dUVWGvqsR3QDLBE68n8qFZqD3aTnctQIXJytKvDvLet+nEhPiw4N6BXNU3yumjiztH+vPnm/pSYbLy8qf7KK++PBO13aHw1qpDnCqoZPrEnnSJ9nd1SJJ0WXOrWS3mz5/PCy+8wKRJk4iMjGTw4MH1if1iBQc37bpiZ58bEkJg3LIUFYLIiQ+hC/Dj/7b+l9xKI6+P/xta9aUNwDFlZZP+0ot4RkTQc8EzkNCt0WW4+/mwn1PzePOLA9Ra7NxzXU8mDu+M5hJGFl+oviEhvizw8+S5/9vBa8sP8MJDQwj0a90HPY3Zx0IIlizfT2pGCTOnJjBmcGzzBdaM3P1zLUmN0SJJOiIiAqPRiMPhQKPR4HA4KCwsJCKi4YxFQUFBvPrqq/V/33///XTp0qVR63KXgWO2k7sxH/8FQ/LNlNu8oKiKW7pMpbCmmLKSS782t+rISewWK77XTmpS7O48wMZktvHJd8fYcdhIh19HYUe186a0pLrJZV5sfUN99TwytQ+LVxxg/hs/Me/WRPxccHcnZ2jsPv5q20m+25XNxCGx9O8S7Lafjz/izp/rc5EHFNKFtEh3d3BwMPHx8axZswaANWvWEB8fT1BQwwknysrKsNvtAOzYsYNjx47Vn8dubRzGE6iD26PrPYZySwWv7XmLKmv1JQ0WE0JQun4thZ99gm+//nRcuAjPTp2cGLXrHTpZwjPv7SIlrZCJQ2J56s7+LX5DjG7tA5kzNYHi8lpe+WwflTXWFl2/K3y/N4c1208xPCGSSUOdd8WBJEmXpsW6u5977jnmz5/Pm2++iZ+fH4sWLQLqWsuzZ8+md+/epKamsnDhQtRqNYGBgbz99tt4ttJJODwG34qwmVGpNZSZy6m0VqJTN30GMcVmpfDDD6jc8TM+SQMRDkeburWk2Wpn+ZYMftiXS2Q7bx6e0pvYcNfdXal7h0BmT+3Dv75I5dVP9/P4rX3x9WqdLeoL2XO0kI83HqNvl3bccU2cnE1MktyIvAsWzu0icxRmoFSXoO04AJVKRV51AUEegeg1OtSqpnVcOKqryV3yT8wZJwieNJmgCRMv+YvUnboFj50u571v0yguNzNmYAw3DO/k9Ekzmlrfw5ml/OuLVCKDvXjs1kR8PJt+oNXSLqbOx06X8+pn++kQ7sNjtyQ2mOu8NXKnz/XFkN3d0oXIGcecSDjsmLf+B8uOz8BhxaE4eDv1v7x76KMmJ2jFZkVlMKD28CBi+kyCr5vUZlo6NruD5VtOsOjjvQgB825L5OaRXd1qVqueHYOYPaU3eSUm/vHZfkxmm6tDcprcompe/yKVkAAPHpma0OoTtCS1RW41uru1s6auQynLwfOaR1BpDWiAO3vcjEbVtC+/qj2/UPTZx0Q/+jhRc+a2meQMdTN9vbsmjdxiE1f2jeSmEV3wNLjnx7FXp2Bm3dCbpV8d5B+f7eexW/ri5dF6WtTnUlpp5rXlB9Dp1Pz5poRW1UMgSZcT2ZJ2EqXCiHXvKrQdk9B2SCSvuoAP0z4n3DuUjv6Nv62fYrFQ+OkytIGBqL282kyCdigK3/ycyfMf/kK12cacGxO4a2x3t03Qv+nTuR0zJvfmdGE1//j8ADVmu6tDajKT2cZryw9gttp59Ka+tPNvneM+JOlyIJO0EwghMP/0Aah1GK74EwBZlac5UnocGnnGX7FaKfpyBUJRiHnsCaIfn4/WP6AZom55+SUmXvhoDyt/zGRA91D+fm8yfToHuzqsi9a3SztmXN+LbGMVi1fsp9bS+hK11eZgyRepFJbVMOuGPsSEXtr9yiVJal7u3XxpJURtJaK6FEPyjai9AymuLWVQRBL9wxLQay5+RLC9vIzcpa9jyTqFR2wsvv0HNGPULUsRgsXLD2C2OphxfS+SurfOe1snxoUwfVJP3lp5mMUrDvDnGxPcvhfgN4oieGd1GsdzKnhwUk/iOwS6OiRJki5AtqSdQO3lj9fUv6OLv4oqazUv7f4X35xc36gEbT6VSdbzC7Dm5xM5c3abStAAJ3IqKK4wc+vorq02Qf+mf7dQpk/qycncSv614gAWq8PVIV2QEIKPNx1j77EibhndlYHxYa4OSZKkiyCT9CWypm5AqSxEpdGhUqnx1nkxodMYksP7N6qcmrTDqLRa2j/5FD59E5spWtdJSTOi16pJ7No2bv6R1D2UByb24HhuBf/64gAWm3sn6jU7stiyN5dxye25OinG1eFIknSRWkc/nZuy56Zh2fkpwlKNYcAUjpaeINeUz5VRV6C5iLm5haJQ8s1KtIFBBI4bj/9VI9B4tb07DjkUhd1HCkno0g4Pfdv5yA2MD0NRBP+3Jo3Xv0hl9tQ+bnkZ04+peXy97SSDe4Yz5arOrg5HkqRGkC3pJhJ2K+YfP0DlF4Y+8ToA9hUdZFvOdhShXFQZZd9toHTNN1iyT6FSqdpkggZIP1VGda2N5B5tr4t1UM9w7hvfgyNZZSz5MhWrm7WoD5wo5oN1R+nVMYhp13ZH3UauEpCky0Xbada0MOvebxCVRjzHz0Ol1WOy1XBz3PVU20zoNBd3zWnAlVeh8fbBb8jQZo7WtVLSjHgatPTu1HpGcjfG4F7hKELw/rfpLP3qIA9P6e0WE7IczSrlrVWHiAnzYcbkXmg18phcklob+V/bBI7SHKwH1qGNG4I2qgcFJiNPb3+BfUUH8dVf3CUteW8uofz7zfgPHdZmroE+F5vdwd7jRfSPC0GnbbsftyG9I7h7XHcOZZbyxteHsNkvrjfF2ewOhdJKM+mnSlnwbgoB3gbm3JjQpk4zSNLlRP7nNoFSko3Kyx/DoFsA8NH5kBzen64BF3dHKmG3U31gP7rQttf9+3upGSXUWhxtsqv794YlRKIIwQfrj/Lm1weZeUNvp7VeHYpCpclGebXl1x8r5VUWKkz/e1xebaGqxlZ/ab6/j54/35yAfyu91aYkSTJJN4mu6xVoOw1ApdGRWnQYL50Xt3SbfNHvt+TlgsOBR/sOzRile0hJM+LnpaN7h7YxIcuFXNk3CkXARxuO8tbKQzx0/R93M/+WfCtMFsqrrA2T8K+PK6qtVJqsZ82Lo1KBn7eeAB8Dgb4GOkb6EeBjwN+n7rkBvSKxmdv+bTYlqS2TSboRFFMZtqPb0Cdci0qjQwjBulOb0aq1PNrvoYvutrZkZwFgaONJutZi50BGCcP6RKBRt92u7t8bkRiFogg+/u4Yb686zNA+EVScmXirLJSb6h5Xmqz8/j50KsDXW0/Ar8k2Ntyv/nGAj4EAXz3+3gb8vHV/uF0DfA0UySQtSa2aTNKNYNn+MfbsA+i6DEblF4pdOJjTbzo1tppGnVfWeHvj3bsPutDWPanHhew7XoTNrlwWXd2/N6p/NIoQfLrpOHuPFdU/7+el+7W1a6B9qM+vSddAgLe+7rfPhZOvJEmXD5mkL5L91D7smb+gHzAVtV8ox8sy+G/aZ8xIuIcon4hGleWT2B+fxMZNdtIapaQVEuxnoHOUv6tDcYmrk2LoFhOA3SEI8NHj562XI6wlSWoU+Y1xEYS1FvPPH6EOjEafMBYAg9ZAB99oQjwbd1mRUBROzptL2ebvmiNUt1FVYyXtVCkD48Mu62tz24f50inSjyA/D5mgJUlqNPmtcREsv3yFMJXhMfxuVGotB4oO4a/344E+dzVqfm4AW6ERe2kJaoNHM0XrHvYcLcKhiMuyq1uSJMlZZJK+AKEoiKpidD1GoAnrgtlu5qP05azKWNek8jS+foTfez9ePXo4OVL3kpJmJCLYS94KUZIk6RLIc9IXoFKr8RgzG4QDIQQGjYHHkx7GQ2NoUnn2igo8u8WjCwpycqTuo6zKwrHT5Uwa2rFNT9QiSZLU3GRL+g/Yjv6IPecQKpUKlVrLL8b9vL7///DWeeFv8GtSmUWffUze0n85OVL3sivdiAAGyq5uSZKkSyKT9HkolUWYf/4I2+HN9c8JBBqVGi+tZ5PKFEJgyc7G0L69s8J0SylpRjqE+RIe5OXqUCRJklo12d19DkIIzD9/CCo1hiG3A5Becoz+oQkMCEtscheuvawMR3VVm57ExFhaw6mCKm4a0cXVoUiSJLV6siV9DvaMFBynD2IYMAW1TzD5JiNLD7zL5tPbLukc628zjXnEtN0knZJuBGBgfNueqEWSJKklyJb07whzNZYdn6AO6YiuxygAIrzDmN7nbroGdL6ksj3j4oia82ib7e4WQpCSZiQu2p8gv7Z9iZkkSVJLkEn6d5TqYtAa8Bg+DZVazebsbVRYKrm+y7WoVZfW8WDOzEQbFIza0LSR4e4up8hEfkkNo8fEuToUSZKkNkF2d/+Opl0s3je/hCa4rrVbZi6n2Fx6yQkawPjBfyhd880ll+OuUtKMqFUq+neXXd2SJEnOIFvSvxIOG5aU5ej7jkftFYAQgpzqPKbGTcShOC65fMViQdhtGNro+WghBLvSjfToGIifl7x/sSRJkjPIlvSvrPvWYDv0HUrJaQBSi9N4afe/OFxyBI1ac8nlqw0GOr/2OoFjrrnkstxRRl4lxRVmkuPltdGSJEnOIpM0YC3Owbr/W7RdBqGN6Q1AfFAcU7tOpHtgV6eso2pXCoWfLIM2OgNXSpoRrUZNv7gQV4ciSZLUZlz2SVoIheK1b4POgGHwbQB8m/kdB4sPMyJmqFNa0QDV+/dRvX8vqjZ4n2CHorD7SCEJnYPxNMgzKJIkSc7S9jJGI9mO/oj5dDoeyTej9vTDrthJKzlKRsUpp67Hkp3VZicxOZJdTqXJKu94JUmS5GSXfbNH1FTg2bEPmm7DcCgOqm0m5vafgd0Jg8V+o1gsWI0F+AwY6LQy3UlKmhEPvYY+nRt3b21JkiTpj132LWlDv4mE3/I0KpWKrbnb+dvOVyipLUOv0TltHfaKCvSRUXjEdnRame7CZlfYc7SIxK4h6HXOOTUgSZIk1bnsW9IAql/PO/dp1xOz3Uw7T+feRlIfGkrsguedWqa7OJRZQq3FLru6JUmSmsFl35L+zZqTG7E6rFzb8Wqn3wO55JuVGJd96NQy3UVKmhEfTx09YgNdHYokSVKb02It6czMTObPn095eTkBAQEsWrSI2NjYBsuUlJTw5JNPkp+fj91uJzk5maeffhqttnnDLK0pZ1vOdgwaPZE+4U4vv/rAfjRe3k4v19UsVgf7TxRzRa8ItBp5vCdJkuRsLfbN+uyzz3LbbbexYcMGbrvtNp555pmzlnn77bfp3Lkzq1ev5ptvvuHw4cNs3LixWeOyOqz4GXx4ZtDjjIwZ1izr8LtiCH5Dm6dsV9p3ogirTSFZ3vFKkiSpWbRIki4pKSEtLY0JEyYAMGHCBNLS0igtLW2wnEqlwmQyoSgKVqsVm81GWFjznutcfXIDT2x8Ab1G77Rros+kWCz4Dx2OX/Igp5ftarvSCgn0NdA1JsDVoUiSJLVJLZKk8/PzCQsLQ6OpS4IajYbQ0FDy8/MbLDdjxgwyMzMZOnRo/U///v2bNbZugV1Ijunn1NHcZ6r6ZRcnZk3HWlTYLOW7SnWtjYMnSxjQPRR1G51FTZIkydXcanT3+vXr6datGx988AEmk4n777+f9evXM3bs2IsuIzjYp1HrHBHSvNcuVxXlo9briewWi0rjXpcohYT4Nvm9e3dm4VAEY4d0vKRyWlJridOZZJ0lqXVrkSQdERGB0WjE4XCg0WhwOBwUFhYSERHRYLlly5bxwgsvoFar8fX1ZeTIkaSkpDQqSZeUVKMoolHxhYT4UlRU1aj3XKzyoyfQR8dQXFrTLOU31aXWefOuLEIDPfE3aJpt2zlTc+5jdyXr7P7kAYV0IS3S3R0cHEx8fDxr1qwBYM2aNcTHxxMU1PB65OjoaLZt2waA1Wplx44ddO3qnBtcuIJQFCynszG0b+/qUJyqvNrCkawykuPDnH65miRJkvQ/LTa6+7nnnmPZsmVcc801LFu2jAULFgBw//33c/DgQQD+8pe/sGfPHq677jquv/56YmNjuemmm1oqROdTFEJuvR2/QVe4OhKn2n2kEAFyAhNJkqRmphJCNK5v2M25U3e3vbwcIQTagAC3a3FeSp0XfvgLVrvCgntaz1zkra0b1Blknd2f7O6WLkTOQNGMyjZ/R+b8xxB2u6tDcZqi8loy8iplK1qSJKkFyCTdjKw5pzFERqLWNc/lXa6wK90IwMDucgITSZKk5uZWl2C1NZEzZ2OvrHB1GE6Vkmakc5Qf7QI8XR2KJElSmydb0s3EXlVJ6fq14FBcHYrT5BZVk1NkIjlednVLkiS1BJmkm4k58yQlK7/CXl7m6lCcJiXdiEoFA2SSliRJahEySTcTS1YWqFQYYmJcHYpTCCFISTMS3yEQf2+9q8ORJEm6LMgk3Uws2dnoQkNRe7SNc7eZ+VUUlZtlV7ckSVILkkm6mdjKSjHEdHB1GE6zK92IVqOif7cQV4ciSZJ02ZCju5tJ+6eeQdhsrg7DKRRFsCvdSO9OwXh5tJ3LySRJktydbEk3A3N2FkWff4JSY3J1KE5x7HQ55dVWOYGJJElSC5NJuhnUHkmnfNN3oHavW1M2VUq6Eb1OTULndq4ORZIk6bIik3Qz0AQE4JM0EK2fn6tDuWR2h8IvRwpJ7BqCQd82DjokSZJaC3lOuhn4DkjGb+AgV4fhFIczSzGZ7XJUtyRJkgvIlrSTKVYrJ/88m4qftrk6FKdISTfi7aGlV6egCy8sSZIkOZVM0k5mycnBUV2F2svb1aFcMovNwb5jxfTvFoJWIz8qkiRJLU1+8zqZJfsUAB7t27s2ECc4cKIYi80hu7olSZJcRCZpJ7NkZ6P28kYb3PpHQu9KL8TfR0+39oGuDkWSJOmyJAeOOZlnXBzaoCBUKpWrQ7kkNWY7qRklXJUYiVrduusiSZLUWskk7URCUfDpl4Ra3/pvQLH3WBF2hyInMJEkSXIh2d3tRNb8PE7Mmk71/n2uDuWSpaQbaefvQaeI1n+ttyRJUmslk7QTWbKyQFHQhYa6OpRLUmmykn6qjOQeYa2+216SJKk1k0naibwT+hI1Zy768AhXh3JJdh8pRBFCjuqWJElyMZmknaj2xHG0QcGo1K17s6akG4lq5010qI+rQ5EkSbqste5s4kaEEBS8+2/KN3/n6lAuSUmFmRM5FQyUA8YkSZJcTiZpJ7EVF6HU1mJo38HVoVySXelGAJLjW/d5dUmSpLZAJmknsWRnAa1/prGUdCMdI/wIDfRydSiSJEmXPZmkncSSnQ1qNfroaFeH0mT5JSayjdXy2mhJkiQ3ISczcZKgCRPxHTgIta71TmSSkmZEBQzoLru6JUmS3IFsSTtJxZbvQVFcHUaTCSFISS+kW/sAAn0Nrg5HkiRJQiZpp7BXlFO0/FNqjqS5OpQmyzZWYyytkaO6JUmS3IhM0k7gqKpCHxWNR2xHV4fSZClpRjRqFUndZFe3JEmSu7joJD1z5kw2bdqEzWZrznhaJUN0DLELnseza5yrQ2kSRQhS0o307BiEj6fO1eFIkiRJv7roJJ2UlMQbb7zB0KFDefbZZ9m7d29zxtWqFH7+KUWff+rqMJrsRE4FZVUWOapbkiTJzVx0kp42bRpff/01y5Ytw8/Pj7lz5zJmzBiWLl1KdnZ2c8bo9kz792IrLXF1GE2WkmZEr1WT2LWdq0ORJEmSztDoc9Jdu3Zl7ty5vPLKK3h4ePDGG28wefJk7r77bo4cOdIcMbo1R00NtqKiVjvTmENR2H2kkIQu7fDQyyvyJEmS3EmjvpVPnjzJN998w5o1a9DpdEyaNIlJkyYRFBTEJ598wowZM/j++++bK1a3ZDld14vg0UqTdPqpMqprbbKrW5IkyQ1ddJK+4YYbyM3N5dprr+Uf//gHCQkJDV6fNm0aH3300Xnfn5mZyfz58ykvLycgIIBFixYRGxvbYJl58+Zx9OjR+r+PHj3KG2+8wahRoy42zJanUuHZPR5DK50ONCXNiKdBQ+9OQa4ORZIkSfodlRBCXMyC69evZ+TIkej1TZtR684772TKlClMmjSJVatW8eWXX/Lhhx+ed/kjR45w11138eOPP/5/e3ce1tSZ7wH8m4R9U3aCgCxeEeqCimvttKJ1aUG02KLO1Gm1Oup1m45etbZqaavFttYpdbnVLjq2esetVMa2Vm1rbUes+xJXwp4QEISACIQk9w80IyoSQjbI9/M88zwQct7zO+k8fnPec877a9E+S0uroNHodUg6vr7uUs2hxAAAIABJREFUKCmpbNE2bZ2vrztk8nLMTzuKPl19MfXZaEuXZFK2+t+Yx2zdfH3dLV2CyRw8eBB+fn7o2bOnyfd1+PBhrF69Gmq1Go899hhWrVoFZ2dnk+/XHPS+Ju3m5obCwsJGr0mlUvz666/NbltaWgqJRIL4+HgAQHx8PCQSCcrKyprcZteuXUhISDD4S4G55L/3Lm7s2WXpMgxyLqsUt2vVnOomIqNSq9U4ePAgzp07Z5Kx73Xr1i288cYb2LhxI3744Qe4urri008/Nfp+LUXv6e6UlBRs27at0Wuurq5ISUnB999//8ht5XI5/P39IRKJAAAikQh+fn6Qy+Xw8npwmrWurg779u3DF198oW95Ot7ebi3eBjDsG62mrg5Xr12Fd8/H2uQ34jNZZejo5og/9A2BSNT+17Vpi/+NWovHTC0VGRmJ//7v/8ahQ4dQU1ODV199FSNHjgQA/O1vf0N2djZUKhVCQkKwcuVKdOjQAZmZmXj77bfRvXt3SCQSzJkzB4cPH8Zvv/2GnTt34uWXX4ZYLMbKlSvRq1cvnD59GgKBAB9++CEiIiIAAHv37sVXX30FtVoNNzc3rFixAuHh4dizZw+++eYbuLq6Ijc3F++99x6ioqJ09R45cgTdu3fXXT6dMGECFi9ejNmzZ5v9szMFvUO6tLQUfn6NV6Py8/NDSUmJ0Ys6ePAgAgMDG/2H0Jc5p7s1dXUIeHkqREHBbWqKDQBc3Z1wXFKEJ3qKUVZ2y9LlmFxbmwY1Bh6z9bv/C8XhE3n44bhpHml9un8I4mL1u3dGKBQiPT0dUqkUEydORGxsLLy9vbF06VLdidWHH36ITZs2YcGCBQCA69evIyUlBb179wbQ8O949+7d8ac//QkAkJmZievXr2PVqlVISUnBhg0bsH79enzwwQc4ceIEvv32W3z55ZdwcHDAzz//jNdeew07duwAAJw9exbp6ekIeci9P3K5HIGBgbrfAwMDIZfLDf+grIzeIR0cHIx///vfGDRokO61zMxMBOnRmlEsFkOhUECtVkMkEkGtVqO4uBhisfih79+9ezeSkpL0Lc1i1JWVcI6Mgp2np6VLabFjF4qgqtdwqpuIHvD8888DAMLDwxEdHY0zZ85g2LBhSE9Px759+6BSqVBdXd3o5t/OnTvrAropYWFhiI5uuP8lJiYGP/74I4CGa8qXL1/W7Ver1UKpVOq269Onz0MD2hboHdKzZ8/GnDlzMH78eAQHByM/Px979uzBypUrm93W29sbUVFRyMjIQGJiIjIyMhAVFfXQqe6ioiKcPHkSa9asadmRWEDZ/gxUHj+GiI/WW7qUFjtyugDeHo6I6NTB0qUQ0R1xsfqf7ZrbiRMnsH37duzYsQNeXl7Yt28f/vnPf+r+7uLi0uwY995jJBQKUV9fD6AhlJOSkjBv3ryHbufq6trkmGKxGJmZmbrfZTJZkyeAbZHeFyKHDx+Ozz77DNXV1fj5559RXV2NzZs3Y/jw4Xptv2LFCmzbtg0jR47Etm3b8OabbwIApk2bhvPnz+vet3fvXgwdOhQdOlh/eNTm58IxOAQCgcDSpbRIZXUdzlwtQf8ofwjbWO1EZHq7d+8GAOTk5EAikSAmJgZKpRJubm7o2LEj6urqdO9pipubGyor9bv0EBcXh/T0dBQVFQFouDnswoULem37xBNP4Pz588jJyQEA7NixA6NHj9Zr27agRYuZ9OzZ0+Db6SMiIrBz584HXt+0aVOj32fOnGnQ+Oam1WhQW1CADn94ytKltNjJKyVQa7Sc6iaih1Kr1Rg7dixu376NlJQUeHt744knnsA333yDkSNHwtPTE7GxsY1OsO43ZswYLFmyBN99953uxrGm9OvXD/Pnz8fMmTOhVquhUqkwatQodO/evdla3dzckJKSgr/85S/QaDSIiorC0qVLDTpua6T3c9IAcOnSJZw4cQI3b97EvZs1NUVhCea6caxWVojcZUsRMGUaPAY/3qJtLS31y1OoqqlHypR+bW4WwFBt7YYiY+AxWz9rvBM9MjISp06deuQUM5mP3tPd//d//4eJEyfi2LFj2LRpE65evYrPP//cZptr2Hv7oNP8V+Hy2GOWLkVvyuo6HDpZgKv55XiydyebCWgiorZK7+nuzZs3Y/PmzYiNjUW/fv2wbt06/Pzzz9i/f78p67NatYWFsOvQEXYdOlq6lEe6XVuPM9du4JhEgYvZZdBotQj2c8OIgZ2hqau3dHlEZGXuXZqZLK9Fz0nHxsYCaLgrT6PR4Mknn8TChQtNVpw1u7FnJzQ1Nej8+nJLl/IAVb0GF6SlOCZR4Oz1G6ir18DbwwmjBoRgYLQ/gvzc4N3BuU1NCxIR2SK9QzogIAAFBQUICgpCaGgoDh06BE9PT9jb25uyPqulLi+Hc9dIS5eho9FocSXvJo5JFDh5pQTVtfVwd7HHkJ5iDIwOQEQnD05vExG1MXqH9CuvvIKsrCwEBQVh1qxZmDdvHlQqVbu6i64lOr+1Etp6y04Xa7Va5BRV4thFBY5fVqCiqg6ODiL07eqLAdH+iOrsCTsbWO6TiKi90iuktVot+vXrp7uF/sknn8Tx48ehUqls8g7A6qtXUHXyBLwTEgELzCTIS28hU6LAMYkCxTdvw04kQM8IHwyI9kevCG842IvMXhMRERmfXiEtEAiQkJCAU6dO6V5zcHCw+g5VplItuYjyHw/BJ+l5s+2zTFmD45eKcUxShDxFFQQCoFuIJ54d2Bl9I33h4mSblx2IiNozvae7o6KikJ2dretYYstq83LhECCG0MRfUqpuq3DicjGOSRS4ll8OLYAwsQcmDvsv9IvyQ0c3R5Pun4jIUObqJ11XV4eZM2fqVii7d4nQ9kDvkO7fvz+mTZuGcePGISAgoNFNSOPHjzdJcdaqJi8XLt1a3qFLr7HrGh6ZypQocCG7DGqNFmJvF4x9Igz9o/3h79n8+rhERJZ0t5909+7djR7Sdxs13SUUCjF16lR4enripZdeMuq+rIHeIX3q1Cl06tQJx48fb/S6QCCwqZDW1NZC5OoGp9Bwo41Zr9bgQnYZMiUKnL5WgjqVBl4ejhjRLxgDov0R7OfGO7OJyCzaWj9pOzs7DB48GAUFBRb5vExN75D+xz/+Yco62gyhoyNC33y71eNotFpcyy/HMYkCJy4X41ZNPdyc7TG4uxgDo/3RJagDm18Q2SDZP5Y99PXAF1MAADcOfIY6Rc4Df/d++mU4BoSh8uxhVJ77qcnt9dGW+km3d3qHtEajafJvQqHtPOZTcfQIqiUSBEydBoHI8Luov/4lGxm/5cDRXoTeXX0wMNof0aFefGSKiCyO/aSth94hHR0d3eSU66VLl4xWkLW7deECanKkrQpoADhz7Qb+K6gDXn0hBo4OfGSKiBo0d8brM2LKI//u3isO7r3ijFkSAOvtJ93e6X3adujQIRw8eFD3v+3bt2Po0KFISdF/CqU9cO3RA57Dnm7VGDV19Si8UYWozp4MaCKyOm2pn3R7p/eZdKdOnR74PTU1FePHj9dNUbR32vp6uMf2h9CxdY8+5cgrodUC4YEeRqqMiMh42lI/aQBISkqCQqGAUqnEH/7wBzzxxBN45513Wnzc1qhF/aTvJ5fLMWbMGPz+++/GrKlVTNlP+va1q8hfvQpBf/ufVj2Ctf9YLnb9lIWP5j0BN2fLLELS1vrutpatHS/AY24L2E+amqP3mfTChQsbXZOuqanB77//jjFjxpikMGtUk5sLaLVwCAho1ThZhRXw83S2WEATEVHboHdId+7cudHvzs7OmDBhAgYPHmz0oqxVbV4uRO4eELWih7RWq4VUpkRUqKcRKyMiMg72k7Yueof07NmzTVlHm1CbnwvHkJBWLSxSpqxFxa06RAR2MGJlRETUHul9d/fbb7/dqMEG0LAKWXu5OK8Pz1HPoGPc8FaNIZU3PPvHm8aIiKg5eod0RkbGA3fade/eHRkZGUYvyhqpq2/BuWs3uPbs1apxsgorYCcSItjPzUiVERFRe6V3SAsEAtx/I7harX7kSmTtSdWJE8he+FeoSkpaNY5UrkTnADeuLEZERM3SOyliY2Oxdu1aXShrNBqkpaUhNjbWZMVZk5r8PAidnGDv42PwGPVqDXKLKnk9mojavYMHD+LcuXMm38+pU6cwYcIEPPPMM3jmmWeQmpr6wAllW6b3jWNLly7FX/7yFwwZMgSBgYGQy+Xw9fXFxo0bTVmf1fBLngjP4U9D0Ip1ygtKqqCq1/B6NBG1a+ZsVenm5oZ3330XoaGhqKurw5///Gekp6dj7NixRt2vpegd0gEBAdi7dy/OnTsHuVwOsViMnj172kRzDa1Gg5sHvoNrr0cvHt8cqYw3jRGRdWtrrSq7du2q+9nBwQHR0dGQyWTm/dBMSO+EvXTpEhQKBWJiYjB69GjExMRAoVDg8uXLpqzPKqgURbixZxdqcqStGierUAkPVwd4ezgZqTIiam9WHF6Dn7L/bfSfW+Juq8oNGzZg2bJlKC0tBdAwo7pnzx7s27cPXbp0waZNm3TbXL9+HS+88ALS09MxfPhwxMXFYfr06Y3Oaq9fv44JEyZg3759GD16NNavXw8AjVpV7tmzB1OnTsVrr72mG/vs2bNYtGgRMjIyGgX0/UpLS/H999/jqaeeatHxWrMWrTi2YcOGRq+pVCosXLgQ+/btM3ph1qQmLw8A4BTSuZl3PppUrkS42KNVz1kTEZlaW2xVWVVVhZkzZ2LKlCm6fbQHeoe0TCZDcHBwo9dCQkJQWFho9KKsTW1eLgR2dnAQBxo8RtVtFRRl1RjSo3VLihJR+7Yi7lWT/mwoa25Vefv2bcyYMQOPP/44pkx5dCvPtkbv6e6AgABcvHix0WsXL16En5+f0YuyNqqSYjh0CoLATu/vNA/IvruIiZjXo4nIurWlVpW1tbWYMWMGevXq1WTIt2V6p85LL72EWbNm4ZVXXkFISAjy8vLw2WefYcaMGaaszyqIZ86GpqamVWNIZUoIAIQypInIyrWlVpW7du3C8ePHUV5ejqNHjwIARo0ahZkzZ7b8wK1Qi1pVfvvtt9i1axeKioogFosxfvx4jBo1ypT1tZixW1XWl5ej7Lv96PjkU62a7l7zzzO4WVmLt6YOMHgMY2prLf1ay9aOF+AxtwVsVUnNadH8bb9+/eDg4ICbN28CaLhQv2vXLowfP94kxVmDmmwpyg8egHu//gaPodVqkS1Tom+krxErIyKi9k7vkD548CAWLlyIzp074/r16+jSpQuuXbuGPn36tOuQFjg4wCXqMTgGBTf/5iYobt7GrZp6hHOlMSKycmxVaV30Dum1a9di5cqVGD16NPr164evv/4au3fvxvXr101Zn8W5PtYdro81f13kUaSyCgC8aYyIiFpG77u7ZTIZRo8e3ei1cePG4euvv9Zr++zsbCQnJ2PkyJFITk5GTk7OQ9+3f/9+JCQkID4+HgkJCbhx44a+JZpEzoo3UJrxTavGyJIp4eggQqAPr/EQEZH+9D6T9vb2xo0bN+Dj44NOnTrh9OnT8PT01LsL1vLlyzFp0iQkJiYiPT0dy5Ytw9atWxu95/z58/j444+xZcsW+Pr6orKystFzdeamrqxEXUE+BIMGt2ocqUyJsAB3CIVcxISIiPSn95n0888/j5MnTwJoeBxr8uTJSExMxMSJE5vdtrS0FBKJBPHx8QCA+Ph4SCQSlJWVNXrfF198gSlTpsDXt+EGK3d3dzg6Oup9MMZWk5cLoHUrjdWp1CgorkJEJ16PJiKiltH7THr69Om6n8eOHYv+/fvj9u3busXRH0Uul8Pf31/XuUQkEsHPzw9yuRxeXl6692VlZSEoKAh//OMfUV1djaeffhozZ8602DKatXeWA3UMfvRydI+Sq6iEWqPl9WgiImoxg5fQCgw0/JnhpqjValy5cgWff/456urq8MorryAwMLBFLce8vd0M2vfDnlcURUXAbvQoBIQ1/RB+c36VKAAA/XoEwtPKGmtY4zOapmRrxwvwmMn0CgoKkJSUhMzMTACtf876iy++QEJCAry9vQEA27dvR21tLV566SVjlWwS+hz3/Z+VPgxf57IFxGIxFAqFrg+oWq1GcXHxAyvQBAYGYtSoUXBwcICDgwOGDRuGc+fOtSikjbmYiaZTODxCI1u1OMLZqyXw9nBCfa0KJSUqg8cxtra26ENr2drxAjzmtoBfKB60detWDB48WBfS+lxSbc/M0gza29sbUVFRyMjIAABdu7F7p7qBhmvVR48ehVarhUqlwrFjx9CtWzdzlPgATU0Nrs+egfLDB1s1Trasgv2jiajNOH36NCZOnIgxY8ZgzJgxuqU2z507h+TkZCQkJCA5ORnnzp1rdiypVIpXXnkFSUlJGDNmTKP1vh+2nw0bNqC4uBhz585FYmIirl+/jrS0NKSmpgIARowY0ag98rZt27BkyZJm93WvtLQ0/PWvf8W0adPw9NNPY/78+ZBIJJg8eTKGDx+u2xcA5Obm4s9//jMSEhIwbtw4HDlyRPe3AwcOYNSoUUhMTMS6desa7ePs2bN48cUX8dxzz+G5557DTz/91Oxn1RSznEkDwIoVK7B48WKsX78eHh4eug9i2rRpmDt3Lnr06IFnn30WFy5cwDPPPAOhUIghQ4ZYbKGU2oJ8QK2GnZe3wWOUV9WiVFmLp2MZ0kSkn/NLlzX63S/uKfgPi9O9Hjb1ZQBA9qefN3rf/a/3eCcFikOHUXz4J93vzSkvL8fs2bORlpaGPn36QK1Wo6qqCnV1dZg7dy5WrVqFQYMG4bfffsPcuXNx4MCBJseqr6/HggUL8N577yEiIgJVVVVISkpCTEwMvL29H7qfIUOGYOfOnfjoo4/QtWvXB8YcO3Ys9u7dqwvmPXv2YMmSJY/c18Pum7p48SJ2794NFxcXjBs3Dh988AE2b96M+vp6DBs2DMnJyQgNDcWCBQvwwgsv4Pnnn8f169fxxz/+Ed9++y00Gg3eeOMNbN++HeHh4Y36aiuVSixfvhyffPIJ/Pz8UFxcjPHjx+tOUlvKbCEdERGBnTt3PvD6vQcnFAqxZMkS3X8AS3IMCkKn+X+DU1i4wWNIZXc6X3GlMSJqA86cOYOIiAj06dMHQMNNvh06dMCVK1dgb2+PQYMGAQAGDx4Me3t7ZGdnN3kNNicnB1lZWXj11f+0yVSpVJBKpcjPz3/ofpozduxYvPDCC1i4cCGysrKgVCoRGxuLrKysJvf1sJAeMmQI3N0bLjVERkaiW7duususYWFhyMvLg4+PDy5duoSkpCQAQJcuXRAVFYUzZ85Aq9UiOjoa4eEN+ZCcnIz3338fQMMMQUFBAaZNm6bbn0AgQG5uLjw9PZs9xvuZLaTbmpq8PNh16AhRKxaZl8qUEAkFCPE37GY2IrI9TZ3x3v+6Pu/zHxYH/2FxxiuuBbRaLTw9PZGenv7A3wyd/g0MDESXLl1w5MgRHD9+HOPGjYNAIHjkvh7m3kd7RSLRA7+r1WqD6gMajjsyMhJffvnlA38rKCho8XhmuSbdFpXs+AolO3e0agyprALBfm5wsBcZqSoiItOJiYlBVlYWTp8+DaDhiZuKigqEhYXp7hMCgH//+9+or69HWFhYk2OFhYXBycmp0aqUWVlZqKqqanI/AODq6vrIPtTjxo3Dzp07kZGRgXHjxjW7L0O5ubkhKioKe/fu1Y13+fJlxMTEICYmBhKJRLdy5r2zxL1790Zubq7uswIarue3oOFkIzyTfghtfT1qCwvg+fRIg8fQaLTIlldiSA/DH98iIjKnjh07Ii0tDe+++y6qq6shFAqxaNEiDB48GB999BHeeecdVFdXw8XFBX//+98fuSKknZ0dNm7ciJUrV+LTTz+FRqOBt7c31q5dCy8vryb3M3nyZLz22mtwcnLCBx988MC4I0aMQEpKCnr06KF7FPhR+2qN999/H8uWLcMXX3wBOzs7rF69WnfD81tvvYUZM2bAyckJI0aM0G3ToUMHrF+/Hu+99x5WrlwJlUqF4OBgbNy40aAaWtRPui0wxiNYNXm5yEtZjoDpM+DRf6BBdeQXV2H5Z8cxLT4ag7oHGDSGKbW1R1Vay9aOF+AxtwV8BIuaw+nuh7i70lhrlgPVdb7i41dERGQgTnc/hPuAgXAMDoa9n7/BY2TJlHB1soOfp7MRKyMiIlvCM+mHqPr9OLRqNQRCwz+ebJkS4YEdLLbuOBERtX0M6ftoNRoovvwHKo/9ZvAYt2vrIbtxCxGc6iYiolZgSN9HXVUFex8fOIUavohJtlwJLXg9moiIWofXpO9j5+GB0DffbtUYd1caC2NIExFRK/BM+j43f/geRZ9uMvjBc6AhpAO8XODqZG/EyoiIyNbwTPo+t86fg7qqyuAbvrRaLaSyCvQIN7wxBxEREcAz6Ua0Wi1q8/LgGBJi8Bg3KmqgrFbxejQREbUaQ/oe9TdvQl1VCcdWLWLCzldERGQcDOl7aGpq4NwtqlV3dktlSjjYCdHJ1/DuWURERACvSTfiGBiI4AWLWjWGVFaBzgHusBPx+w8REbUOk+QeRZ9/iqIvPjV4e1W9BrmKKkRwqpuIiIyAIX2P6ssSaGtrDd4+v7gK9WoNbxojIiKj4HT3HVqtFj5jk2DXsaPBY7DzFRERGRND+g5NVRWcIyNh5+ll8BhSmRId3Rzg5eFkxMqIiMhWcbr7jorfjiL7f/4GTVWVwWNI73S+IiIiMgaG9B21eXmw8/SCyN3doO0rq+tQXH6bna+IiMhoGNJ31Obltmqlsf8sYsKQJiIi42BIA1DX1qKuSN7qlcYEAiA0gCFNRETGwRvHAAiEQnSa91fY+/gaPIZUVoEgXzc4OoiMWBkREdkynkkDqFEoIPLoAHv/AIO212i1kMoreT2aiIiMiiENQJa+DwUfrDZ4+6LSatyurUcYQ5qIiIyIIQ2gtrgETiGdDe4hzc5XRERkCrwmDSB6xRsoLiw1eHupXAlnRxHE3i5GrIqIiGydzZ9J1ymKkL35M6iVSoPHkBZWIEzsAaGBZ+JEREQPY/Mhffv6dcgz9kOjUhm0fW2dGgUltzjVTURERmfzIV2bnwuhoyMcAgy7szunSAmNVstFTIiIyOgY0nl5cA0NhUBo2EchlXOlMSIiMg2bD2mtWg23/4oweHtpoRK+HZ3g4eJgxKqIiIh4dzdClrwOHx833LhhWPcrqVyJyGDDe1ATERE1xebPpAEY/Hx0mbIGNytruYgJERGZhNnOpLOzs7F48WKUl5ejY8eOSE1NRWhoaKP3pKWl4auvvoKfnx8AoE+fPli+fLm5Smwxdr4iIiJTMltIL1++HJMmTUJiYiLS09OxbNkybN269YH3jR07FosWLTJXWa0ilSthJxIgxM+wHtRERESPYpbp7tLSUkgkEsTHxwMA4uPjIZFIUFZWZo7dm4y0sAIh/u6wt+NVAyIiMj6zpItcLoe/vz9EooY2jiKRCH5+fpDL5Q+891//+hcSEhIwZcoUnD592hzlGUSt0SCnqJJT3UREZDJWdXf3hAkTMGPGDNjb2+PXX3/FrFmzsH//fnh6euo9hre3m0H79vVt2ZR1VkE56uo1iIn0b/G21qKt1m0oWztegMdM1NaZJaTFYjEUCgXUajVEIhHUajWKi4shFosbvc/X11f38+OPPw6xWIxr166hf//+eu+rtLQKGo22RfX5+rqjpKSyRduclBQ1bOvu0OJtrYEhx9yW2drxAjzmtoBfKKg5Zpnu9vb2RlRUFDIyMgAAGRkZiIqKgpeXV6P3KRQK3c+XLl1CYWEhwsLCzFFii0llFXB3sYdPBydLl0JERO2U2aa7V6xYgcWLF2P9+vXw8PBAamoqAGDatGmYO3cuevTogTVr1uDixYsQCoWwt7fH6tWrG51dWxOpTIlwsYfBz1gTERE1x2whHRERgZ07dz7w+qZNm3Q/3w1ua1ddo4K8tBoDHzOsKQcREZE++OyQAdhUg4iIzIEhbQCpTAkBgLAAhjQREZkOQ9oAUpkSYh9XuDhZ1RNsRETUzjCkW0ir1TbcNMapbiIiMjGGdAuVlN9G1W0VQ5qIiEyOId1CWXc7X4kZ0kREZFoM6RaSypRwtBehk6+rpUshIqJ2jiHdQlJZBUID3CES8qMjIiLTYtK0gKpejTxFFcI7caqbiIhMjyHdArmKKqg1WoSLO1i6FCIisgEM6RaQyrjSGBERmQ9DugWksgp4eTjC093R0qUQEZENYEi3wN3OV0RERObAkNZTxa063KioQXggr0cTEZF5MKT1JJVVAOD1aCIiMh+GtJ6kMiVEQgE6B7hbuhQiIrIRDGk9SWVKBPm6wdFeZOlSiIjIRjCk9aDRaJEtV3IREyIiMiuGtB7kpbdQU6fmnd1ERGRWDGk93O18FdGJd3YTEZH5MKT1IJUp4epkB39PZ0uXQkRENoQhrQeprAJhYg8IBAJLl0JERDaEId2M27X1KLxxi89HExGR2TGkm5FTVAmtFlxpjIiIzI4h3QyuNEZERJbCkG6GVKaEv6cz3JztLV0KERHZGIb0I2i12obOV5zqJiIiC2BIP0KZshYVt+o41U1ERBbBkH6ELF6PJiIiC2JIP4JUpoS9nRDBfm6WLoWIiGwQQ/oRpDIlOvu7w07Ej4mIiMyP6dOEerUGuYpKTnUTEZHFMKSbUFBSBVW9hiFNREQWw5BuQlbhnc5XfPyKiIgshCHdBKlMiQ6uDvDycLR0KUREZKMY0k2QyioQHsjOV0REZDlmC+ns7GwkJydj5MiRSE5ORk5OTpPvlUql6NWrF1JTU81VXiNVt1VQ3LzN69FERGRRZgvp5cuXY9KkSfj+++8xadIkLFu27KHvU6vVWL58OYYPH26u0h4glTVcj+ZyoEREZElmCenS0lJIJBLEx8cDAOLj4yGRSFBWVvbqucDAAAALKElEQVTAez/55BM89dRTCA0NNUdpDyWVVUAgAEID3C1WAxERkZ05diKXy+Hv7w+RSAQAEIlE8PPzg1wuh5eXl+59ly9fxtGjR7F161asX7/eoH15exu2Opiv738CuaC0Gp0DPBAS5GnQWG3FvcdsC2zteAEeM1FbZ5aQ1odKpcIbb7yBVatW6cLcEKWlVdBotC3axtfXHSUllQAaOl9dySlD30g/3Wvt0b3HbAts7XgBHnNbwC8U1ByzhLRYLIZCoYBarYZIJIJarUZxcTHEYrHuPSUlJcjLy8P06dMBAEqlElqtFlVVVXjrrbfMUSYAQHHzNm7V1POmMSIisjizhLS3tzeioqKQkZGBxMREZGRkICoqqtFUd2BgIDIzM3W/p6Wlobq6GosWLTJHiTpZhQ2dryIY0kREZGFmu7t7xYoV2LZtG0aOHIlt27bhzTffBABMmzYN58+fN1cZzZLKlXByEEHs7WrpUoiIyMaZ7Zp0REQEdu7c+cDrmzZteuj758yZY+qSHkpaqESY2ANCIRcxISIiy+KKY/eoU6lRUFLF69FERGQVGNL3yFVUQq3RMqSJiMgqMKTvcbfzFVcaIyIia8CQvodUroRPByd0cHWwdClEREQM6Xvd7XxFRERkDRjSd9ysrEWZspZT3UREZDUY0nf8p/MVz6SJiMg6MKTvkMorIBIK0NnfsAYdRERExsaQviNbpkSIvxvs7Qxv7kFERGRMDGkAao0W2fJKXo8mIiKrwpAGkFekRK1KzevRRERkVRjSAK7k3gTAzldERGRdGNIArubdhJuzPXw7Olu6FCIiIh2GNIDLuTcRHugBgYCdr4iIyHrYfEjfrq1HQXElr0cTEZHVsfmQzpYrodVyERMiIrI+Nh/SWXdXGhMzpImIyLrYfEhny5QI8nODi5O9pUshIiJqxOZDWipXIrKzp6XLICIieoCdpQuwtMe7B2D4wFBLl0FERPQAmz+Tfn5oF3QN4Zk0ERFZH5sPaSIiImvFkCYiIrJSDGkiIiIrxZAmIiKyUgxpIiIiK8WQJiIislIMaSIiIivFkCYiIrJSDGkiIiIrxZAmIiKyUgxpIiIiK9XuGmwIhQKzbteW2dox29rxAjxmorZOoNVqtZYugoiIiB7E6W4iIiIrxZAmIiKyUgxpIiIiK8WQJiIislIMaSIiIivFkCYiIrJSDGkiIiIrxZAmIiKyUgxpIiIiK2XTIZ2dnY3k5GSMHDkSycnJyMnJsXRJJnXz5k1MmzYNI0eOREJCAmbPno2ysjJLl2UWH3/8MSIjI3H16lVLl2JytbW1WL58OUaMGIGEhAS88cYbli7J5H788UeMHTsWiYmJGDNmDA4cOGDpkoiMwqaXBZ08eTKSkpKQmJiI9PR07N69G1u3brV0WSZTXl6OK1euYMCAAQCA1NRUVFRUYOXKlRauzLQuXryIDz/8EFKpFBs3bkTXrl0tXZJJvf322xAKhViyZAkEAgFu3LgBHx8fS5dlMlqtFv3798eXX36Jrl274vLly5g4cSJOnjwJodCmz0OoHbDZ/weXlpZCIpEgPj4eABAfHw+JRNKuzyw7duyoC2gAiImJgUwms2BFpldXV4eUlBSsWLHC0qWYxa1bt/D1119j3rx5EAgaGk2054C+SygUorKyEgBQWVkJPz8/BjS1C+2uC5a+5HI5/P39IRKJAAAikQh+fn6Qy+Xw8vKycHWmp9FosH37dsTFxVm6FJP6+9//jjFjxiAoKMjSpZhFfn4+OnbsiI8//hiZmZlwdXXFvHnzEBsba+nSTEYgEGDt2rWYNWsWXFxccOvWLXzyySeWLovIKPhV00a99dZbcHFxwZ/+9CdLl2Iyp0+fxoULFzBp0iRLl2I2arUa+fn5iI6Oxp49e7BgwQLMmTMHVVVVli7NZOrr6/G///u/WL9+PX788Uds2LAB8+fPx61btyxdGlGr2WxIi8ViKBQKqNVqAA3/uBUXF0MsFlu4MtNLTU1Fbm4u1q5d266nBH///XdkZWVh2LBhiIuLQ1FREaZOnYqjR49aujSTEYvFsLOz013G6dWrFzw9PZGdnW3hykzn0qVLKC4uRt++fQEAffv2hbOzM7KysixcGVHrtd9/oZvh7e2NqKgoZGRkAAAyMjIQFRXV7qe616xZgwsXLmDdunVwcHCwdDkmNX36dBw9ehSHDx/G4cOHERAQgE8//RRDhgyxdGkm4+XlhQEDBuDXX38F0PAEQ2lpKTp37mzhykwnICAARUVFkEqlAICsrCyUlpYiJCTEwpURtZ5N392dlZWFxYsXQ6lUwsPDA6mpqQgPD7d0WSZz7do1xMfHIzQ0FE5OTgCAoKAgrFu3zsKVmUdcXJxN3N2dn5+P1157DeXl5bCzs8P8+fPx5JNPWrosk/rmm2+wadMm3c1yc+fOxfDhwy1cFVHr2XRIExERWTObne4mIiKydgxpIiIiK8WQJiIislIMaSIiIivFkCYiIrJSDGmiViooKEBkZCTq6+stXQoRtTMMaSIiIivFkCYiIrJSDGlqlxQKBebMmYOBAwciLi5O1yc8LS0Nc+fOxfz589G7d2+MGzcOly9f1m2XlZWFF198EbGxsXj22Wdx6NAh3d9qamrw7rvvYujQoejbty8mTpyImpoa3d/37duHp556CgMGDMCGDRt0r587dw7PPfcc+vTpg8GDB2PVqlVm+ASIqD1gSFO7o9FoMHPmTERGRuLIkSPYsmULtmzZgl9++QUAcOjQIYwaNQrHjx9HfHw8Zs2aBZVKBZVKhRkzZuDxxx/Hb7/9htdffx0LFizQrQmdmpqKixcvYseOHTh+/DgWLlzYqEHJyZMn8d1332HLli1Yt26drsHDO++8g8mTJ+PUqVP44YcfMHr0aPN/KETUJjGkqd05f/48ysrKMHv2bDg4OCA4OBgvvPAC9u/fDwB47LHHMGrUKNjb2+Pll19GXV0dzp49i7Nnz6K6uhrTp0+Hg4MDBg0ahKFDh+Jf//oXNBoNdu/ejaVLl+r6kPfp06dRk5LZs2fDyckJ3bp1Q7du3XRn6HZ2dsjLy0NZWRlcXV0RExNjkc+FiNoeO0sXQGRshYWFKC4uRmxsrO41tVqN2NhYBAYGIiAgQPe6UCiEv78/iouLATR0VLr37DgwMBAKhQI3b95EbW0tgoODm9yvj4+P7mdnZ2dUV1cDaDiT/uijjzB69GgEBQVh9uzZGDp0qNGOl4jaL4Y0tTtisRhBQUE4cODAA39LS0tDUVGR7neNRgOFQgE/Pz8AQFFRETQajS6o5XI5QkND4enpCUdHR+Tn56Nbt24tqic0NBRr1qyBRqPBgQMHMHfuXGRmZsLFxaUVR0lEtoDT3dTu9OzZE66urvjkk09QU1MDtVqNq1ev4ty5cwCAixcv4sCBA6ivr8eWLVvg4OCAXr16oWfPnnBycsLmzZuhUqmQmZmJw4cP45lnnoFQKERSUhJWrVoFhUIBtVqN06dPo66urtl60tPTUVZWBqFQCA8PDwBodLZORNQU/ktB7Y5IJMLGjRtx+fJlDBs2DAMHDsTrr7+OqqoqAMCwYcOwf/9+9OvXD+np6UhLS4O9vT0cHBywceNGHDlyBAMHDsSbb76J1atXIyIiAgCwaNEidO3aFePHj0f//v3x/vvvQ6PRNFvPL7/8gmeffRa9e/fGO++8gw8//FDXz5uI6FHYT5psSlpaGnJzc/H+++9buhQiombxTJqIiMhKMaSJiIisFKe7iYiIrBTPpImIiKwUQ5qIiMhKMaSJiIisFEOaiIjISjGkiYiIrBRDmoiIyEr9P7OMY08Ft1dmAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 490.6x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = sns.relplot(data = accuracy_df, kind=\"line\")\n",
    "ax.set(xlabel='epochs', ylabel='accuracy', title='Accuracy evolution through the epochs')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D9E5AIUQVY8t",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 4 - Contributivity scores\n",
    "\n",
    "We have set our scenario with Shapley values as a contributivity measurement method.\n",
    "\n",
    "While being quite heavy on computing resources, it provides a great measuring tool. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aqoho7oFp6yU"
   },
   "outputs": [],
   "source": [
    "contributivity_score = my_scenario.contributivity_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 151
    },
    "colab_type": "code",
    "id": "7IpNbZDqp6yW",
    "outputId": "d5db4313-e16c-49b6-e803-3d6f4b8138cc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Shapley\n",
      "Computation time: 0:25:25.791579\n",
      "Number of characteristic function computed: 7\n",
      "Contributivity scores: [0.20238334 0.38973331 0.38658333]\n",
      "Std of the contributivity scores: [0. 0. 0.]\n",
      "Normalized contributivity scores: [0.20678793 0.39821531 0.39499677]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(contributivity_score[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "goGzd4ePp6yY"
   },
   "source": [
    "Since we have artificially set our first partner to only have .1% of the total data set, it obviously contributes less in the final model. The poor accuracy of the local model of the first partner spotlights greatly this effect. However, as we use the MNIST dataset, even with .1% of the total data, the model is still able to perform reasonably well according to our Accuracy values\n",
    "\n",
    "There is other way to artificially generate poor contributors. For instance, we can shuffle the labels of one partner, which will mislabel the whole dataset of this particular partner.  \n",
    "To do so, we use the `corruption_parameters`of the scenario object. It must be a list, with size equal to the number of partner. For each partner , a string identifier indicates the method of corruption to apply on its dataset.\n",
    "\n",
    "``` python\n",
    "corrupted_partner=['random-uniform', 'not-corrupted', 'not-corrupted']\n",
    "\n",
    "```\n",
    "Here, with 3 partners, the first one will see its labels shuffled randomly, and the two others will have their dataset untouched.\n",
    "\n",
    "More parameters (like the proportion of the data to be corrupted) can be used, but in that case the string identifiers will not be enough.\n",
    "You will have to instantiate the corresponding corruption object before, and pass it to the scenario, instead of the string identifier.\n",
    "``` python\n",
    "from mplc.corruption import Permutation, Random$\n",
    "\n",
    "partner0_corruption = Permutation(proportion=0.4)\n",
    "partner1_corruption = Random(proportion=0.9)\n",
    "corrupted_partner=[partner9_corruption, partner1_corruption, 'not-corrupted']\n",
    "\n",
    "```\n",
    "Here the first one will see its labels permuted, by a permutation matrix randomly generated (and accessible via partner.corruption.matrix). The second one will see its labels randomly changed, according to a Dirichlet distribution, which you can access to via partner.corruption.matrix attribute. Finally, the last partner will be its data untouched.\n",
    "Please denote that the partner.corruption.matrix will be equal to identity matrix in that case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "b0XHWBPxp6yY",
    "outputId": "77a0bfe5-8298-4449-c066-4b9475d14a59"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-09-22 08:17:17.569 | INFO     | mplc.scenario:__init__:299 - ### Description of data scenario configured:\n",
      "2020-09-22 08:17:17.569 | INFO     | mplc.scenario:__init__:300 -    Number of partners defined: 3\n",
      "2020-09-22 08:17:17.569 | INFO     | mplc.scenario:__init__:301 -    Data distribution scenario chosen: random\n",
      "2020-09-22 08:17:17.569 | INFO     | mplc.scenario:__init__:302 -    Multi-partner learning approach: fedavg\n",
      "2020-09-22 08:17:17.570 | INFO     | mplc.scenario:__init__:303 -    Weighting option: uniform\n",
      "2020-09-22 08:17:17.570 | INFO     | mplc.scenario:__init__:304 -    Iterations parameters: 10 epochs > 3 mini-batches > 8 gradient updates per pass\n",
      "2020-09-22 08:17:17.570 | INFO     | mplc.scenario:__init__:310 - ### Data loaded: mnist\n",
      "2020-09-22 08:17:17.570 | INFO     | mplc.scenario:__init__:311 -    54000 train data with 54000 labels\n",
      "2020-09-22 08:17:17.570 | INFO     | mplc.scenario:__init__:312 -    6000 val data with 6000 labels\n",
      "2020-09-22 08:17:17.570 | INFO     | mplc.scenario:__init__:313 -    10000 test data with 10000 labels\n",
      "2020-09-22 08:17:26.218 | INFO     | mplc.scenario:split_data:548 - ### Splitting data among partners:\n",
      "2020-09-22 08:17:26.219 | INFO     | mplc.scenario:split_data:549 -    Simple split performed.\n",
      "2020-09-22 08:17:26.219 | INFO     | mplc.scenario:split_data:550 -    Nb of samples split amongst partners: 43740\n",
      "2020-09-22 08:17:26.219 | INFO     | mplc.scenario:split_data:552 -    Partner #0: 8748 samples with labels [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n",
      "2020-09-22 08:17:26.219 | INFO     | mplc.scenario:split_data:552 -    Partner #1: 21870 samples with labels [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n",
      "2020-09-22 08:17:26.220 | INFO     | mplc.scenario:split_data:552 -    Partner #2: 13122 samples with labels [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n",
      "2020-09-22 08:17:40.840 | INFO     | mplc.multi_partner_learning:compute_test_score:136 - ## Training and evaluating model on partners with ids: ['#0', '#1', '#2']\n",
      "2020-09-22 08:17:40.934 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:301 - (fedavg) Very first minibatch of epoch n°0, init new models for each partner\n",
      "2020-09-22 08:17:44.326 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:17:47.096 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:17:50.044 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.853', '0.799']\n",
      "2020-09-22 08:17:50.149 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:17:53.148 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:17:55.897 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:17:58.878 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.622', '0.908']\n",
      "2020-09-22 08:17:58.969 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:02.036 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:04.750 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:07.769 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.382', '0.934']\n",
      "2020-09-22 08:18:07.815 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:10.889 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:13.594 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:16.577 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.504', '0.953']\n",
      "2020-09-22 08:18:16.622 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:19.679 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:22.393 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:25.398 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.231', '0.961']\n",
      "2020-09-22 08:18:25.444 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:28.548 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:31.284 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:34.380 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.295', '0.970']\n",
      "2020-09-22 08:18:34.427 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:37.684 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:40.543 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:43.641 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.129', '0.977']\n",
      "2020-09-22 08:18:43.686 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:46.897 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:49.663 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:52.671 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.244', '0.976']\n",
      "2020-09-22 08:18:52.717 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:55.866 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:18:58.641 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:01.679 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.131', '0.979']\n",
      "2020-09-22 08:19:01.723 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:04.788 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:07.539 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:10.587 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.192', '0.979']\n",
      "2020-09-22 08:19:10.587 | INFO     | mplc.multi_partner_learning:compute_test_score:204 - ### Evaluating model on test data:\n",
      "2020-09-22 08:19:10.718 | INFO     | mplc.multi_partner_learning:compute_test_score:206 -    Model metrics names: ['loss', 'accuracy']\n",
      "2020-09-22 08:19:10.718 | INFO     | mplc.multi_partner_learning:compute_test_score:207 -    Model metrics values: ['0.186', '0.978']\n",
      "2020-09-22 08:19:11.434 | INFO     | mplc.multi_partner_learning:compute_test_score:218 - Training and evaluation on multiple partners: done.\n",
      "2020-09-22 08:19:11.434 | INFO     | mplc.scenario:run:700 - Shapley values\n",
      "2020-09-22 08:19:11.434 | INFO     | mplc.contributivity:compute_SV:143 - # Launching computation of Shapley Value of all partners\n",
      "2020-09-22 08:19:11.435 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:84 - ## Training and evaluating model on partner with id #0\n",
      "2020-09-22 08:19:11.513 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:97 -    Training model...\n",
      "2020-09-22 08:19:14.819 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:107 -    Model evaluation on test data: [('loss', '2.397'), ('accuracy', '0.135')]\n",
      "2020-09-22 08:19:14.820 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:84 - ## Training and evaluating model on partner with id #1\n",
      "2020-09-22 08:19:14.894 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:97 -    Training model...\n",
      "2020-09-22 08:19:19.461 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:107 -    Model evaluation on test data: [('loss', '0.056'), ('accuracy', '0.982')]\n",
      "2020-09-22 08:19:19.462 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:84 - ## Training and evaluating model on partner with id #2\n",
      "2020-09-22 08:19:19.550 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:97 -    Training model...\n",
      "2020-09-22 08:19:23.173 | INFO     | mplc.multi_partner_learning:compute_test_score_for_single_partner:107 -    Model evaluation on test data: [('loss', '0.066'), ('accuracy', '0.980')]\n",
      "2020-09-22 08:19:23.175 | INFO     | mplc.multi_partner_learning:compute_test_score:136 - ## Training and evaluating model on partners with ids: ['#0', '#1']\n",
      "2020-09-22 08:19:23.211 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:301 - (fedavg) Very first minibatch of epoch n°0, init new models for each partner\n",
      "2020-09-22 08:19:25.034 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:26.899 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:29.355 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['1.286', '0.798']\n",
      "2020-09-22 08:19:29.444 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:31.305 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:33.153 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:35.271 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['1.159', '0.840']\n",
      "2020-09-22 08:19:35.304 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:37.149 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:39.354 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:41.490 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['1.078', '0.895']\n",
      "2020-09-22 08:19:41.524 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:43.388 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:45.252 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:47.687 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.962', '0.921']\n",
      "2020-09-22 08:19:47.722 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:49.590 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:51.439 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:53.535 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.918', '0.897']\n",
      "2020-09-22 08:19:53.566 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:55.422 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:57.607 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:19:59.714 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.758', '0.933']\n",
      "2020-09-22 08:19:59.747 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:01.582 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:03.424 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:05.513 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.793', '0.938']\n",
      "2020-09-22 08:20:05.546 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:07.790 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:09.656 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:11.753 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.752', '0.942']\n",
      "2020-09-22 08:20:11.786 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:13.636 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:15.479 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:17.901 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.594', '0.962']\n",
      "2020-09-22 08:20:17.934 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:19.797 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:21.652 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:23.759 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.679', '0.952']\n",
      "2020-09-22 08:20:23.759 | INFO     | mplc.multi_partner_learning:compute_test_score:204 - ### Evaluating model on test data:\n",
      "2020-09-22 08:20:23.890 | INFO     | mplc.multi_partner_learning:compute_test_score:206 -    Model metrics names: ['loss', 'accuracy']\n",
      "2020-09-22 08:20:23.890 | INFO     | mplc.multi_partner_learning:compute_test_score:207 -    Model metrics values: ['0.663', '0.953']\n",
      "2020-09-22 08:20:23.925 | INFO     | mplc.multi_partner_learning:compute_test_score:218 - Training and evaluation on multiple partners: done.\n",
      "2020-09-22 08:20:23.927 | INFO     | mplc.multi_partner_learning:compute_test_score:136 - ## Training and evaluating model on partners with ids: ['#0', '#2']\n",
      "2020-09-22 08:20:23.952 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:301 - (fedavg) Very first minibatch of epoch n°0, init new models for each partner\n",
      "2020-09-22 08:20:26.111 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:27.958 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:30.052 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['1.417', '0.768']\n",
      "2020-09-22 08:20:30.077 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:31.912 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:33.725 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:36.096 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['1.130', '0.867']\n",
      "2020-09-22 08:20:36.122 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:37.936 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:39.750 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:41.822 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['1.058', '0.854']\n",
      "2020-09-22 08:20:41.847 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:44.009 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:45.873 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:47.937 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['1.040', '0.916']\n",
      "2020-09-22 08:20:47.962 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:49.789 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:51.604 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:54.017 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.994', '0.933']\n",
      "2020-09-22 08:20:54.042 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:55.886 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:57.715 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:20:59.808 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.883', '0.933']\n",
      "2020-09-22 08:20:59.834 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:01.679 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:03.901 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:05.987 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.848', '0.940']\n",
      "2020-09-22 08:21:06.013 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:07.832 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:09.664 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:11.743 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.758', '0.942']\n",
      "2020-09-22 08:21:11.768 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:13.969 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:15.801 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:17.879 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.719', '0.943']\n",
      "2020-09-22 08:21:17.906 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:19.760 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:21.931 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:24.046 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.777', '0.952']\n",
      "2020-09-22 08:21:24.046 | INFO     | mplc.multi_partner_learning:compute_test_score:204 - ### Evaluating model on test data:\n",
      "2020-09-22 08:21:24.174 | INFO     | mplc.multi_partner_learning:compute_test_score:206 -    Model metrics names: ['loss', 'accuracy']\n",
      "2020-09-22 08:21:24.174 | INFO     | mplc.multi_partner_learning:compute_test_score:207 -    Model metrics values: ['0.765', '0.956']\n",
      "2020-09-22 08:21:24.215 | INFO     | mplc.multi_partner_learning:compute_test_score:218 - Training and evaluation on multiple partners: done.\n",
      "2020-09-22 08:21:24.217 | INFO     | mplc.multi_partner_learning:compute_test_score:136 - ## Training and evaluating model on partners with ids: ['#1', '#2']\n",
      "2020-09-22 08:21:24.254 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:301 - (fedavg) Very first minibatch of epoch n°0, init new models for each partner\n",
      "2020-09-22 08:21:26.073 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:27.920 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:30.026 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.360', '0.892']\n",
      "2020-09-22 08:21:30.065 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:32.283 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:34.164 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:36.277 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.211', '0.936']\n",
      "2020-09-22 08:21:36.314 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:38.167 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:40.349 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:42.468 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.129', '0.963']\n",
      "2020-09-22 08:21:42.506 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:44.399 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:46.273 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:48.378 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.088', '0.976']\n",
      "2020-09-22 08:21:48.415 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:50.649 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:52.538 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:54.666 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.069', '0.979']\n",
      "2020-09-22 08:21:54.704 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:56.543 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:21:58.687 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:00.816 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.068', '0.979']\n",
      "2020-09-22 08:22:00.853 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:02.693 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:04.540 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:06.669 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.071', '0.979']\n",
      "2020-09-22 08:22:06.706 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:08.943 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:10.803 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:12.905 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.053', '0.983']\n",
      "2020-09-22 08:22:12.944 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:14.787 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:16.950 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:19.084 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.049', '0.985']\n",
      "2020-09-22 08:22:19.121 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:21.014 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:22.847 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:24.946 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.047', '0.987']\n",
      "2020-09-22 08:22:24.946 | INFO     | mplc.multi_partner_learning:compute_test_score:204 - ### Evaluating model on test data:\n",
      "2020-09-22 08:22:25.094 | INFO     | mplc.multi_partner_learning:compute_test_score:206 -    Model metrics names: ['loss', 'accuracy']\n",
      "2020-09-22 08:22:25.094 | INFO     | mplc.multi_partner_learning:compute_test_score:207 -    Model metrics values: ['0.047', '0.984']\n",
      "2020-09-22 08:22:25.130 | INFO     | mplc.multi_partner_learning:compute_test_score:218 - Training and evaluation on multiple partners: done.\n",
      "2020-09-22 08:22:25.132 | INFO     | mplc.multi_partner_learning:compute_test_score:136 - ## Training and evaluating model on partners with ids: ['#0', '#1', '#2']\n",
      "2020-09-22 08:22:25.176 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:301 - (fedavg) Very first minibatch of epoch n°0, init new models for each partner\n",
      "2020-09-22 08:22:28.262 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:30.995 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°0, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:33.989 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.740', '0.834']\n",
      "2020-09-22 08:22:34.034 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:37.073 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:39.805 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°1, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:42.809 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.572', '0.873']\n",
      "2020-09-22 08:22:42.897 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:45.990 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:48.719 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°2, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:51.704 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.401', '0.944']\n",
      "2020-09-22 08:22:51.749 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:54.835 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:22:57.625 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°3, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:00.640 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.210', '0.954']\n",
      "2020-09-22 08:23:00.685 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:03.770 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:06.548 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°4, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:09.557 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.300', '0.969']\n",
      "2020-09-22 08:23:09.603 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:12.324 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:15.442 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°5, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:18.458 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.134', '0.975']\n",
      "2020-09-22 08:23:18.501 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:21.244 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:24.321 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°6, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:27.380 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.199', '0.975']\n",
      "2020-09-22 08:23:27.425 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:30.159 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:33.240 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°7, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:36.257 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.115', '0.981']\n",
      "2020-09-22 08:23:36.302 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:39.079 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:41.825 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°8, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:45.336 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.218', '0.978']\n",
      "2020-09-22 08:23:45.380 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°0 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:48.233 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°1 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:51.050 | INFO     | mplc.multi_partner_learning:compute_collaborative_round_fedavg:304 - (fedavg) Minibatch n°2 of epoch n°9, init aggregated model for each partner with models from previous round\n",
      "2020-09-22 08:23:54.474 | INFO     | mplc.multi_partner_learning:compute_test_score:188 -    Model evaluation at the end of the epoch: ['0.083', '0.982']\n",
      "2020-09-22 08:23:54.475 | INFO     | mplc.multi_partner_learning:compute_test_score:204 - ### Evaluating model on test data:\n",
      "2020-09-22 08:23:54.608 | INFO     | mplc.multi_partner_learning:compute_test_score:206 -    Model metrics names: ['loss', 'accuracy']\n",
      "2020-09-22 08:23:54.608 | INFO     | mplc.multi_partner_learning:compute_test_score:207 -    Model metrics values: ['0.078', '0.982']\n",
      "2020-09-22 08:23:54.644 | INFO     | mplc.multi_partner_learning:compute_test_score:218 - Training and evaluation on multiple partners: done.\n",
      "2020-09-22 08:23:54.646 | INFO     | mplc.scenario:run:704 - ## Evaluating contributivity with Shapley values: \n",
      "Shapley\n",
      "Computation time: 0:04:43.211376\n",
      "Number of characteristic function computed: 7\n",
      "Contributivity scores: [0.03558331 0.47293334 0.47378333]\n",
      "Std of the contributivity scores: [0. 0. 0.]\n",
      "Normalized contributivity scores: [0.03622449 0.4814551  0.48232041]\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 11,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_second_scenario = Scenario(partners_count=3,\n",
    "                        amounts_per_partner=[0.2, 0.5, 0.3],                             # <- The repartition is more regular\n",
    "                        corruption_parameters = ['random-uniform', 'not-corrupted', 'not-corrupted'], # <- Here is the true difference\n",
    "                        epoch_count=10,\n",
    "                        minibatch_count=3,\n",
    "                        dataset='mnist',\n",
    "                        methods=[\"Shapley values\"])  \n",
    "\n",
    "my_second_scenario.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 151
    },
    "colab_type": "code",
    "id": "T0qGMB53VY84",
    "outputId": "d6f51659-f31d-4806-a846-837cb83e6e4c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Shapley\n",
      "Computation time: 0:04:43.211376\n",
      "Number of characteristic function computed: 7\n",
      "Contributivity scores: [0.03558331 0.47293334 0.47378333]\n",
      "Std of the contributivity scores: [0. 0. 0.]\n",
      "Normalized contributivity scores: [0.03622449 0.4814551  0.48232041]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "contributivity_score = my_second_scenario.contributivity_list\n",
    "print(contributivity_score[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bNBE7h41VY87"
   },
   "source": [
    "In this case, one can denotes that the contributivity measure of the first partner is really low, and lower that in the previous case. We successfully detect a partner with corrupted labels, without any look at these data !\n",
    "\n",
    "In the first case, if we thought about the repartition of a reward, or the money earned by the collaborative model, this Shapley value gives us a great idea of a fair repartition. It is important to denote that the Shapley value is not strongly related to the size of the dataset.  \n",
    "In the second case, you were able to spotlight a corrupted dataset.It could be a partner who missed something in the preprocessing step ( human error, not a problem, but need to be detected !), or worth. Some partners could not play by the rules, and did not shared their precious data...\n",
    "\n",
    "# That's it!\n",
    "\n",
    "Now you can explore our other tutorials for a better overview of what can be done with `mplc`!\n",
    "\n",
    "This work is collaborative, enthusiasts are welcome to comment open issues and PRs or open new ones.\n",
    "\n",
    "Should you be interested in this open effort and would like to share any question, suggestion or input, you can use the following channels:\n",
    "\n",
    "- This Github repository (issues or PRs)\n",
    "- Substra Foundation's [Slack workspace](https://substra-workspace.slack.com/join/shared_invite/zt-cpyedcab-FHYgpy08efKJ2FCadE2yCA), channel `#workgroup-mpl-contributivity`\n",
    "- Email: hello@substra.org\n",
    "\n",
    "![logo Substra Foundation](../../img/substra_logo_couleur_rvb_w150px.png)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "Tutorial-2_Add_contributivity_measurement.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}