flair_cadec_v3_Bio_ClinicalBERT.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "xONJpNLKtGie"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import os\n",
        "import pickle"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nw6RM8uotGih",
        "outputId": "c60bae34-c09e-484b-8e56-e487682be44e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ]
        }
      ],
      "source": [
        "from google.colab import drive\n",
        "drive.mount(\"/content/drive\")\n",
        "\n",
        "os.chdir('/content/drive/MyDrive/SI 630 project')\n",
        "# os.listdir('.')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P9LwpBadtGih",
        "outputId": "db4ef048-90bd-4a0e-8668-5d81eaae6e20"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['sct',\n",
              " 'text',\n",
              " 'original - unedited version of edited files',\n",
              " 'meddra - unedited version of edited files',\n",
              " 'sct - unedited version of edited files',\n",
              " 'original',\n",
              " 'meddra']"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "os.listdir('datasets/CADEC/CADEC.v2/cadec')\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "VNZtmxiPb5uV"
      },
      "outputs": [],
      "source": [
        "suf = '_v3'\n",
        "\n",
        "data_save_dir = 'data'+suf\n",
        "\n",
        "suf += '_Bio_ClinicalBERT'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "BIUl2-W9tGii",
        "outputId": "abdcc251-4970-4a2a-83d9-8702eeb7dd86"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-732d4755-af89-4446-91dc-054d16900bb6\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ner</th>\n",
              "      <th>meddra</th>\n",
              "      <th>counts_exploded</th>\n",
              "      <th>counts</th>\n",
              "      <th>counts_exploded_rank</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Drug</td>\n",
              "      <td>Lipitor</td>\n",
              "      <td>1081</td>\n",
              "      <td>1073</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ADR</td>\n",
              "      <td>CONCEPT_LESS</td>\n",
              "      <td>1016</td>\n",
              "      <td>266</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ADR</td>\n",
              "      <td>Severe_pain</td>\n",
              "      <td>722</td>\n",
              "      <td>247</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ADR</td>\n",
              "      <td>Myalgia</td>\n",
              "      <td>619</td>\n",
              "      <td>305</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ADR</td>\n",
              "      <td>Pain</td>\n",
              "      <td>292</td>\n",
              "      <td>260</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1294</th>\n",
              "      <td>Finding</td>\n",
              "      <td>Senility</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1233</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1295</th>\n",
              "      <td>Drug</td>\n",
              "      <td>Seroquel</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1233</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1296</th>\n",
              "      <td>Drug</td>\n",
              "      <td>Solaraze</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1233</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1297</th>\n",
              "      <td>Drug</td>\n",
              "      <td>Tagamet</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1233</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1298</th>\n",
              "      <td>Drug</td>\n",
              "      <td>testosterone</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1233</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>1299 rows × 5 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-732d4755-af89-4446-91dc-054d16900bb6')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-732d4755-af89-4446-91dc-054d16900bb6 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-732d4755-af89-4446-91dc-054d16900bb6');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "          ner        meddra  counts_exploded  counts  counts_exploded_rank\n",
              "0        Drug       Lipitor             1081    1073                     1\n",
              "1         ADR  CONCEPT_LESS             1016     266                     2\n",
              "2         ADR   Severe_pain              722     247                     3\n",
              "3         ADR       Myalgia              619     305                     4\n",
              "4         ADR          Pain              292     260                     5\n",
              "...       ...           ...              ...     ...                   ...\n",
              "1294  Finding      Senility                1       1                  1233\n",
              "1295     Drug      Seroquel                1       1                  1233\n",
              "1296     Drug      Solaraze                1       1                  1233\n",
              "1297     Drug       Tagamet                1       1                  1233\n",
              "1298     Drug  testosterone                1       1                  1233\n",
              "\n",
              "[1299 rows x 5 columns]"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "meddra_counts = pd.read_csv(data_save_dir+'/meddra_counts.csv')\n",
        "meddra_counts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "xzc4Zo0PtGii",
        "outputId": "e5499477-2575-4917-ef88-647c7a05c22b"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-c27ddcfd-8268-4f4b-a1dc-8885c4a02b26\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ner</th>\n",
              "      <th>meddra</th>\n",
              "      <th>counts_exploded</th>\n",
              "      <th>counts</th>\n",
              "      <th>counts_exploded_rank</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ADR</td>\n",
              "      <td>CONCEPT_LESS</td>\n",
              "      <td>1016</td>\n",
              "      <td>266</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>ADR</td>\n",
              "      <td>Severe_pain</td>\n",
              "      <td>722</td>\n",
              "      <td>247</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ADR</td>\n",
              "      <td>Myalgia</td>\n",
              "      <td>619</td>\n",
              "      <td>305</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ADR</td>\n",
              "      <td>Pain</td>\n",
              "      <td>292</td>\n",
              "      <td>260</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>ADR</td>\n",
              "      <td>Arthralgia</td>\n",
              "      <td>280</td>\n",
              "      <td>163</td>\n",
              "      <td>6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1268</th>\n",
              "      <td>ADR</td>\n",
              "      <td>Labyrinthitis</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1233</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1269</th>\n",
              "      <td>ADR</td>\n",
              "      <td>Dysphagia</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1233</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1275</th>\n",
              "      <td>ADR</td>\n",
              "      <td>Lupus_erythematosus</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1233</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1278</th>\n",
              "      <td>ADR</td>\n",
              "      <td>Cataract</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1233</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1283</th>\n",
              "      <td>ADR</td>\n",
              "      <td>Crying_associated_with_mood</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1233</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>800 rows × 5 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c27ddcfd-8268-4f4b-a1dc-8885c4a02b26')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-c27ddcfd-8268-4f4b-a1dc-8885c4a02b26 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-c27ddcfd-8268-4f4b-a1dc-8885c4a02b26');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "      ner                       meddra  counts_exploded  counts  \\\n",
              "1     ADR                 CONCEPT_LESS             1016     266   \n",
              "2     ADR                  Severe_pain              722     247   \n",
              "3     ADR                      Myalgia              619     305   \n",
              "4     ADR                         Pain              292     260   \n",
              "5     ADR                   Arthralgia              280     163   \n",
              "...   ...                          ...              ...     ...   \n",
              "1268  ADR                Labyrinthitis                1       1   \n",
              "1269  ADR                    Dysphagia                1       1   \n",
              "1275  ADR          Lupus_erythematosus                1       1   \n",
              "1278  ADR                     Cataract                1       1   \n",
              "1283  ADR  Crying_associated_with_mood                1       1   \n",
              "\n",
              "      counts_exploded_rank  \n",
              "1                        2  \n",
              "2                        3  \n",
              "3                        4  \n",
              "4                        5  \n",
              "5                        6  \n",
              "...                    ...  \n",
              "1268                  1233  \n",
              "1269                  1233  \n",
              "1275                  1233  \n",
              "1278                  1233  \n",
              "1283                  1233  \n",
              "\n",
              "[800 rows x 5 columns]"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "meddra_counts.loc[meddra_counts.ner == 'ADR',:]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hAGPini7tGii",
        "outputId": "20b3e9f0-6d9e-4de1-d941-49d2785bb23e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[Sentence[117]: \"i feel a bit drowsy & have a little blurred vision , so far no gastric problems . ive been on arthrotec 50 for over 10 years on and off , only taking it when i needed it . due to my arthritis getting progressively worse , to the point where i am in tears with the agony , gp ' s started me on 75 twice a day and i have to take it . every day for the next month to see how i get on , here goes . so far its been very good , pains almost gone , but i feel a bit weird , didnt have that when on 50 .\" → [\"i\"/O/O, \"feel\"/O/O, \"a\"/O/O, \"bit\"/B-ADR/B-Drowsy, \"drowsy\"/I-ADR/I-Drowsy, \"&\"/O/O, \"have\"/O/O, \"a\"/O/O, \"little\"/B-ADR/B-Blurred_vision_-_hazy, \"blurred\"/I-ADR/I-Blurred_vision_-_hazy, \"vision\"/I-ADR/I-Blurred_vision_-_hazy, \",\"/O/O, \"so\"/O/O, \"far\"/O/O, \"no\"/O/O, \"gastric\"/B-ADR/B-Excessive_upper_gastrointestinal_gas, \"problems\"/I-ADR/I-Excessive_upper_gastrointestinal_gas, \".\"/O/O, \"ive\"/O/O, \"been\"/O/O, \"on\"/O/O, \"arthrotec\"/B-Drug/B-Arthrotec, \"50\"/O/O, \"for\"/O/O, \"over\"/O/O, \"10\"/O/O, \"years\"/O/O, \"on\"/O/O, \"and\"/O/O, \"off\"/O/O, \",\"/O/O, \"only\"/O/O, \"taking\"/O/O, \"it\"/O/O, \"when\"/O/O, \"i\"/O/O, \"needed\"/O/O, \"it\"/O/O, \".\"/O/O, \"due\"/O/O, \"to\"/O/O, \"my\"/O/O, \"arthritis\"/B-Disease/B-Arthritis, \"getting\"/O/O, \"progressively\"/O/O, \"worse\"/O/O, \",\"/O/O, \"to\"/O/O, \"the\"/O/O, \"point\"/O/O, \"where\"/O/O, \"i\"/O/O, \"am\"/O/O, \"in\"/O/O, \"tears\"/O/O, \"with\"/O/O, \"the\"/O/O, \"agony\"/B-Symptom/B-Agony, \",\"/O/O, \"gp\"/O/O, \"'\"/O/O, \"s\"/O/O, \"started\"/O/O, \"me\"/O/O, \"on\"/O/O, \"75\"/O/O, \"twice\"/O/O, \"a\"/O/O, \"day\"/O/O, \"and\"/O/O, \"i\"/O/O, \"have\"/O/O, \"to\"/O/O, \"take\"/O/O, \"it\"/O/O, \".\"/O/O, \"every\"/O/O, \"day\"/O/O, \"for\"/O/O, \"the\"/O/O, \"next\"/O/O, \"month\"/O/O, \"to\"/O/O, \"see\"/O/O, \"how\"/O/O, \"i\"/O/O, \"get\"/O/O, \"on\"/O/O, \",\"/O/O, \"here\"/O/O, \"goes\"/O/O, \".\"/O/O, \"so\"/O/O, \"far\"/O/O, \"its\"/O/O, \"been\"/O/O, \"very\"/O/O, \"good\"/O/O, \",\"/O/O, \"pains\"/B-Symptom/B-Pain, \"almost\"/O/O, \"gone\"/O/O, \",\"/O/O, \"but\"/O/O, \"i\"/O/O, \"feel\"/B-ADR/B-Malaise, \"a\"/I-ADR/I-Malaise, \"bit\"/I-ADR/I-Malaise, \"weird\"/I-ADR/I-Malaise, \",\"/O/O, \"didnt\"/O/O, \"have\"/O/O, \"that\"/O/O, \"when\"/O/O, \"on\"/O/O, \"50\"/O/O, \".\"/O/O],\n",
              " Sentence[39]: \"hunger pangs . brilliant , i have a new lease of life , i walk up & down steps properly , no longer sideways like a toddler , hip pain as gone other than if i jar it .\" → [\"hunger\"/B-ADR/B-Hunger_pain, \"pangs\"/I-ADR/I-Hunger_pain, \".\"/O/O, \"brilliant\"/O/O, \",\"/O/O, \"i\"/O/O, \"have\"/O/O, \"a\"/O/O, \"new\"/O/O, \"lease\"/O/O, \"of\"/O/O, \"life\"/O/O, \",\"/O/O, \"i\"/O/O, \"walk\"/B-Symptom/B-CONCEPT_LESS, \"up\"/I-Symptom/I-CONCEPT_LESS, \"&\"/I-Symptom/I-CONCEPT_LESS, \"down\"/I-Symptom/I-CONCEPT_LESS, \"steps\"/I-Symptom/I-CONCEPT_LESS, \"properly\"/O/O, \",\"/O/O, \"no\"/O/O, \"longer\"/O/O, \"sideways\"/I-Symptom/I-CONCEPT_LESS, \"like\"/O/O, \"a\"/O/O, \"toddler\"/O/O, \",\"/O/O, \"hip\"/B-Symptom/B-Hip_pain, \"pain\"/I-Symptom/I-Hip_pain, \"as\"/O/O, \"gone\"/O/O, \"other\"/O/O, \"than\"/O/O, \"if\"/O/O, \"i\"/O/O, \"jar\"/O/O, \"it\"/O/O, \".\"/O/O],\n",
              " Sentence[51]: \"no side effects for the first two months . then vaginal bleeding 2 wks after menstral cycle . stomach pain . canker sores in my mouth . headache . been off for 1 week still have bleeding . helped my pain alot . too scared to take this drug again .\" → [\"no\"/O/O, \"side\"/O/O, \"effects\"/O/O, \"for\"/O/O, \"the\"/O/O, \"first\"/O/O, \"two\"/O/O, \"months\"/O/O, \".\"/O/O, \"then\"/O/O, \"vaginal\"/B-ADR/B-Vaginal_bleeding, \"bleeding\"/I-ADR/I-Vaginal_bleeding, \"2\"/O/O, \"wks\"/O/O, \"after\"/O/O, \"menstral\"/O/O, \"cycle\"/O/O, \".\"/O/O, \"stomach\"/B-ADR/B-Stomach_ache, \"pain\"/I-ADR/I-Stomach_ache, \".\"/O/O, \"canker\"/B-ADR/B-Aphthous_ulcer, \"sores\"/I-ADR/I-Aphthous_ulcer, \"in\"/O/O, \"my\"/O/O, \"mouth\"/O/O, \".\"/O/O, \"headache\"/B-ADR/B-Headache, \".\"/O/O, \"been\"/O/O, \"off\"/O/O, \"for\"/O/O, \"1\"/O/O, \"week\"/O/O, \"still\"/O/O, \"have\"/O/O, \"bleeding\"/O/O, \".\"/O/O, \"helped\"/O/O, \"my\"/O/O, \"pain\"/B-Symptom/B-Pain, \"alot\"/O/O, \".\"/O/O, \"too\"/O/O, \"scared\"/O/O, \"to\"/O/O, \"take\"/O/O, \"this\"/O/O, \"drug\"/O/O, \"again\"/O/O, \".\"/O/O],\n",
              " Sentence[68]: \"1st pill taken with food , a few hours after i experienced shortness of breath , a sense of depression , cramping , upset stomache will stop taking immediately . honestly can not recommend this drug , i am still experiencing side effects just after 1 pill and im supposed to take twice / day 75mg . i just cant keep going have to try something else .\" → [\"1st\"/O/O, \"pill\"/O/O, \"taken\"/O/O, \"with\"/O/O, \"food\"/O/O, \",\"/O/O, \"a\"/O/O, \"few\"/O/O, \"hours\"/O/O, \"after\"/O/O, \"i\"/O/O, \"experienced\"/O/O, \"shortness\"/B-ADR/B-Dyspnoea, \"of\"/I-ADR/I-Dyspnoea, \"breath\"/I-ADR/I-Dyspnoea, \",\"/O/O, \"a\"/O/O, \"sense\"/O/O, \"of\"/O/O, \"depression\"/B-ADR/B-Depression, \",\"/O/O, \"cramping\"/B-ADR/B-Muscle_cramp, \",\"/O/O, \"upset\"/B-ADR/B-Upset_stomach, \"stomache\"/I-ADR/I-Upset_stomach, \"will\"/O/O, \"stop\"/O/O, \"taking\"/O/O, \"immediately\"/O/O, \".\"/O/O, \"honestly\"/O/O, \"can\"/O/O, \"not\"/O/O, \"recommend\"/O/O, \"this\"/O/O, \"drug\"/O/O, \",\"/O/O, \"i\"/O/O, \"am\"/O/O, \"still\"/O/O, \"experiencing\"/O/O, \"side\"/O/O, \"effects\"/O/O, \"just\"/O/O, \"after\"/O/O, \"1\"/O/O, \"pill\"/O/O, \"and\"/O/O, \"im\"/O/O, \"supposed\"/O/O, \"to\"/O/O, \"take\"/O/O, \"twice\"/O/O, \"/\"/O/O, \"day\"/O/O, \"75mg\"/O/O, \".\"/O/O, \"i\"/O/O, \"just\"/O/O, \"cant\"/O/O, \"keep\"/O/O, \"going\"/O/O, \"have\"/O/O, \"to\"/O/O, \"try\"/O/O, \"something\"/O/O, \"else\"/O/O, \".\"/O/O],\n",
              " Sentence[50]: \"i have had no side effects been taking arthrotec a little over a year , have not noticed any side effects . it does help alot i noticed that when there are times when i forget to take it i cant stand or walk for any lengths of time .\" → [\"i\"/O/O, \"have\"/O/O, \"had\"/O/O, \"no\"/O/O, \"side\"/O/O, \"effects\"/O/O, \"been\"/O/O, \"taking\"/O/O, \"arthrotec\"/B-Drug/B-Arthrotec, \"a\"/O/O, \"little\"/O/O, \"over\"/O/O, \"a\"/O/O, \"year\"/O/O, \",\"/O/O, \"have\"/O/O, \"not\"/O/O, \"noticed\"/O/O, \"any\"/O/O, \"side\"/O/O, \"effects\"/O/O, \".\"/O/O, \"it\"/O/O, \"does\"/O/O, \"help\"/O/O, \"alot\"/O/O, \"i\"/O/O, \"noticed\"/O/O, \"that\"/O/O, \"when\"/O/O, \"there\"/O/O, \"are\"/O/O, \"times\"/O/O, \"when\"/O/O, \"i\"/O/O, \"forget\"/O/O, \"to\"/O/O, \"take\"/O/O, \"it\"/O/O, \"i\"/B-Symptom/B-Reduced_mobility, \"cant\"/I-Symptom/I-Reduced_mobility, \"stand\"/I-Symptom/I-Reduced_mobility, \"or\"/I-Symptom/I-Reduced_mobility, \"walk\"/I-Symptom/I-Reduced_mobility, \"for\"/I-Symptom/I-Reduced_mobility, \"any\"/I-Symptom/I-Reduced_mobility, \"lengths\"/I-Symptom/I-Reduced_mobility, \"of\"/I-Symptom/I-Reduced_mobility, \"time\"/I-Symptom/I-Reduced_mobility, \".\"/O/O],\n",
              " Sentence[89]: \"it seems as if bleeding forever is a side effect but doctors tell you that its not the drug . i have been on arthotec for three months i still have pain and stiffness . at first week into this drug i was in so much pain i sat on the floor at work and thought i was gaving a baby . i always feel weak and almost fainted a bunch of times . consider somthing that works i am switching today as this isnt in my head .\" → [\"it\"/O/O, \"seems\"/O/O, \"as\"/O/O, \"if\"/O/O, \"bleeding\"/B-ADR/B-Bleeding, \"forever\"/O/O, \"is\"/O/O, \"a\"/O/O, \"side\"/O/O, \"effect\"/O/O, \"but\"/O/O, \"doctors\"/O/O, \"tell\"/O/O, \"you\"/O/O, \"that\"/O/O, \"its\"/O/O, \"not\"/O/O, \"the\"/O/O, \"drug\"/O/O, \".\"/O/O, \"i\"/O/O, \"have\"/O/O, \"been\"/O/O, \"on\"/O/O, \"arthotec\"/B-Drug/B-Arthrotec, \"for\"/O/O, \"three\"/O/O, \"months\"/O/O, \"i\"/O/O, \"still\"/O/O, \"have\"/O/O, \"pain\"/B-Symptom/B-Pain, \"and\"/I-Symptom/I-Pain, \"stiffness\"/I-Symptom/I-Pain, \".\"/O/O, \"at\"/O/O, \"first\"/O/O, \"week\"/O/O, \"into\"/O/O, \"this\"/O/O, \"drug\"/O/O, \"i\"/O/O, \"was\"/O/O, \"in\"/O/O, \"so\"/B-ADR/B-Pain, \"much\"/I-ADR/I-Pain, \"pain\"/I-ADR/I-Pain, \"i\"/O/O, \"sat\"/O/O, \"on\"/O/O, \"the\"/O/O, \"floor\"/O/O, \"at\"/O/O, \"work\"/O/O, \"and\"/O/O, \"thought\"/O/O, \"i\"/O/O, \"was\"/O/O, \"gaving\"/O/O, \"a\"/O/O, \"baby\"/O/O, \".\"/O/O, \"i\"/O/O, \"always\"/O/O, \"feel\"/B-ADR/B-Malaise_and_fatigue, \"weak\"/I-ADR/I-Malaise_and_fatigue, \"and\"/I-ADR/I-Malaise_and_fatigue, \"almost\"/I-ADR/I-Malaise_and_fatigue, \"fainted\"/I-ADR/I-Malaise_and_fatigue, \"a\"/O/O, \"bunch\"/O/O, \"of\"/O/O, \"times\"/O/O, \".\"/O/O, \"consider\"/O/O, \"somthing\"/O/O, \"that\"/O/O, \"works\"/O/O, \"i\"/O/O, \"am\"/O/O, \"switching\"/O/O, \"today\"/O/O, \"as\"/O/O, \"this\"/O/O, \"isnt\"/O/O, \"in\"/O/O, \"my\"/O/O, \"head\"/O/O, \".\"/O/O],\n",
              " Sentence[56]: \"some diarrhea , some constipation , fatigue . it works very well for the heel ; i couldnt walk on it very well before taking it . i take 75mg . twice a day . also have severe osteoarthritis in the knees and hands , but it doesnt seem to relieve that pain at all .\" → [\"some\"/O/O, \"diarrhea\"/B-ADR/B-Diarrhoea, \",\"/O/O, \"some\"/O/O, \"constipation\"/B-ADR/B-Constipation, \",\"/O/O, \"fatigue\"/B-ADR/B-Fatigue, \".\"/O/O, \"it\"/O/O, \"works\"/O/O, \"very\"/O/O, \"well\"/O/O, \"for\"/O/O, \"the\"/B-Symptom/B-Walking_disability, \"heel\"/I-Symptom/I-Walking_disability, \";\"/O/O, \"i\"/I-Symptom/I-Walking_disability, \"couldnt\"/I-Symptom/I-Walking_disability, \"walk\"/I-Symptom/I-Walking_disability, \"on\"/I-Symptom/I-Walking_disability, \"it\"/O/O, \"very\"/I-Symptom/I-Walking_disability, \"well\"/I-Symptom/I-Walking_disability, \"before\"/O/O, \"taking\"/O/O, \"it\"/O/O, \".\"/O/O, \"i\"/O/O, \"take\"/O/O, \"75mg\"/O/O, \".\"/O/O, \"twice\"/O/O, \"a\"/O/O, \"day\"/O/O, \".\"/O/O, \"also\"/O/O, \"have\"/O/O, \"severe\"/B-Disease/B-Osteoarthritis_of_knee/B-Osteoarthrosis_of_hand, \"osteoarthritis\"/I-Disease/I-Osteoarthritis_of_knee/I-Osteoarthrosis_of_hand, \"in\"/I-Disease/I-Osteoarthritis_of_knee/I-Osteoarthrosis_of_hand, \"the\"/I-Disease/I-Osteoarthritis_of_knee/I-Osteoarthrosis_of_hand, \"knees\"/I-Disease/I-Osteoarthritis_of_knee, \"and\"/O/O, \"hands\"/I-Disease/I-Osteoarthrosis_of_hand, \",\"/O/O, \"but\"/O/O, \"it\"/O/O, \"doesnt\"/O/O, \"seem\"/O/O, \"to\"/O/O, \"relieve\"/O/O, \"that\"/O/O, \"pain\"/B-Symptom/B-Pain, \"at\"/O/O, \"all\"/O/O, \".\"/O/O],\n",
              " Sentence[84]: \"ive been on 75mg twice daily for 13 days , with recent body swelling , face , wrists , abdomen , thighs - clothes fit tight . insomina is much worse . i have forgetfulnes and confussion - really bad - almost dangerous . have felt \" empty stomach \" feeling , but not bothersome . have had more loose stools , but tolerable . im looking for a more natural therapy , checking into ha ( hyaluronic acid ) to replace arthrotec .\" → [\"ive\"/O/O, \"been\"/O/O, \"on\"/O/O, \"75mg\"/O/O, \"twice\"/O/O, \"daily\"/O/O, \"for\"/O/O, \"13\"/O/O, \"days\"/O/O, \",\"/O/O, \"with\"/O/O, \"recent\"/O/O, \"body\"/B-ADR/B-Swelling_of_body_region, \"swelling\"/B-ADR/I-ADR/B-Abdominal_swelling/B-Facial_swelling/I-Swelling_of_body_region/B-Swelling_of_lower_limb/B-Swelling_of_wrist_joint, \",\"/O/O, \"face\"/I-ADR/I-Facial_swelling, \",\"/O/O, \"wrists\"/I-ADR/I-Swelling_of_wrist_joint, \",\"/O/O, \"abdomen\"/I-ADR/I-Abdominal_swelling, \",\"/O/O, \"thighs\"/I-ADR/I-Swelling_of_lower_limb, \"-\"/O/O, \"clothes\"/O/O, \"fit\"/O/O, \"tight\"/O/O, \".\"/O/O, \"insomina\"/B-ADR/B-Insomnia, \"is\"/O/O, \"much\"/O/O, \"worse\"/O/O, \".\"/O/O, \"i\"/O/O, \"have\"/O/O, \"forgetfulnes\"/B-ADR/B-Forgetful, \"and\"/I-ADR/I-Forgetful, \"confussion\"/I-ADR/I-Forgetful, \"-\"/O/O, \"really\"/O/O, \"bad\"/O/O, \"-\"/O/O, \"almost\"/O/O, \"dangerous\"/O/O, \".\"/O/O, \"have\"/O/O, \"felt\"/O/O, \"\"\"/B-ADR/B-Feeling_empty, \"empty\"/I-ADR/I-Feeling_empty, \"stomach\"/I-ADR/I-Feeling_empty, \"\"\"/I-ADR/I-Feeling_empty, \"feeling\"/I-ADR/I-Feeling_empty, \",\"/O/O, \"but\"/O/O, \"not\"/O/O, \"bothersome\"/O/O, \".\"/O/O, \"have\"/O/O, \"had\"/O/O, \"more\"/O/O, \"loose\"/B-ADR/B-Loose_stool, \"stools\"/I-ADR/I-Loose_stool, \",\"/O/O, \"but\"/O/O, \"tolerable\"/O/O, \".\"/O/O, \"im\"/O/O, \"looking\"/O/O, \"for\"/O/O, \"a\"/O/O, \"more\"/O/O, \"natural\"/O/O, \"therapy\"/O/O, \",\"/O/O, \"checking\"/O/O, \"into\"/O/O, \"ha\"/B-Drug/B-CONCEPT_LESS, \"(\"/I-Drug/I-CONCEPT_LESS, \"hyaluronic\"/I-Drug/I-CONCEPT_LESS, \"acid\"/I-Drug/I-CONCEPT_LESS, \")\"/I-Drug/I-CONCEPT_LESS, \"to\"/O/O, \"replace\"/O/O, \"arthrotec\"/B-Drug/B-Arthrotec, \".\"/O/O],\n",
              " Sentence[68]: \"none that i can think of . i recently had bilateral knee replacement . prior to that , arthrotec was the only drug that helped . i tried every drug on the market , but , to no avail . i took 50 mg . 3 times a day of arthrotec and was almost pain free . i highly recommend it to anyone that suffers from arthritis .\" → [\"none\"/O/O, \"that\"/O/O, \"i\"/O/O, \"can\"/O/O, \"think\"/O/O, \"of\"/O/O, \".\"/O/O, \"i\"/O/O, \"recently\"/O/O, \"had\"/O/O, \"bilateral\"/O/O, \"knee\"/O/O, \"replacement\"/O/O, \".\"/O/O, \"prior\"/O/O, \"to\"/O/O, \"that\"/O/O, \",\"/O/O, \"arthrotec\"/B-Drug/B-Arthrotec, \"was\"/O/O, \"the\"/O/O, \"only\"/O/O, \"drug\"/O/O, \"that\"/O/O, \"helped\"/O/O, \".\"/O/O, \"i\"/O/O, \"tried\"/O/O, \"every\"/O/O, \"drug\"/O/O, \"on\"/O/O, \"the\"/O/O, \"market\"/O/O, \",\"/O/O, \"but\"/O/O, \",\"/O/O, \"to\"/O/O, \"no\"/O/O, \"avail\"/O/O, \".\"/O/O, \"i\"/O/O, \"took\"/O/O, \"50\"/O/O, \"mg\"/O/O, \".\"/O/O, \"3\"/O/O, \"times\"/O/O, \"a\"/O/O, \"day\"/O/O, \"of\"/O/O, \"arthrotec\"/B-Drug/B-Arthrotec, \"and\"/O/O, \"was\"/O/O, \"almost\"/O/O, \"pain\"/B-Symptom/B-Pain, \"free\"/O/O, \".\"/O/O, \"i\"/O/O, \"highly\"/O/O, \"recommend\"/O/O, \"it\"/O/O, \"to\"/O/O, \"anyone\"/O/O, \"that\"/O/O, \"suffers\"/O/O, \"from\"/O/O, \"arthritis\"/B-Finding/B-Arthritis, \".\"/O/O],\n",
              " Sentence[28]: \"hospitalization due extreme rectal bleed that required blood transfusion . i was extremely sick and initially felt i might not survive . should be removed from market .\" → [\"hospitalization\"/O/O, \"due\"/O/O, \"extreme\"/O/O, \"rectal\"/B-ADR/B-Rectal_haemorrhage, \"bleed\"/I-ADR/I-Rectal_haemorrhage, \"that\"/O/O, \"required\"/O/O, \"blood\"/O/O, \"transfusion\"/O/O, \".\"/O/O, \"i\"/O/O, \"was\"/O/O, \"extremely\"/B-ADR/B-Generally_unwell, \"sick\"/I-ADR/I-Generally_unwell, \"and\"/O/O, \"initially\"/O/O, \"felt\"/B-ADR/B-CONCEPT_LESS, \"i\"/I-ADR/I-CONCEPT_LESS, \"might\"/I-ADR/I-CONCEPT_LESS, \"not\"/I-ADR/I-CONCEPT_LESS, \"survive\"/I-ADR/I-CONCEPT_LESS, \".\"/O/O, \"should\"/O/O, \"be\"/O/O, \"removed\"/O/O, \"from\"/O/O, \"market\"/O/O, \".\"/O/O]]"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "with open(data_save_dir+'/cadec_flair_objects.pkl', 'rb') as f:\n",
        "    docs = pickle.load(f)\n",
        "docs[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sW10pPAPtGij",
        "outputId": "eed90337-68d5-493f-84f3-9ad1df9a41c2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "1186"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "len(docs)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EMT5TsNStGij",
        "outputId": "0f1ee279-8ef9-49cd-9669-56768d8b132d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Sentence[117]: \"i feel a bit drowsy & have a little blurred vision , so far no gastric problems . ive been on arthrotec 50 for over 10 years on and off , only taking it when i needed it . due to my arthritis getting progressively worse , to the point where i am in tears with the agony , gp ' s started me on 75 twice a day and i have to take it . every day for the next month to see how i get on , here goes . so far its been very good , pains almost gone , but i feel a bit weird , didnt have that when on 50 .\" → [\"i\"/O/O, \"feel\"/O/O, \"a\"/O/O, \"bit\"/B-ADR/B-Drowsy, \"drowsy\"/I-ADR/I-Drowsy, \"&\"/O/O, \"have\"/O/O, \"a\"/O/O, \"little\"/B-ADR/B-Blurred_vision_-_hazy, \"blurred\"/I-ADR/I-Blurred_vision_-_hazy, \"vision\"/I-ADR/I-Blurred_vision_-_hazy, \",\"/O/O, \"so\"/O/O, \"far\"/O/O, \"no\"/O/O, \"gastric\"/B-ADR/B-Excessive_upper_gastrointestinal_gas, \"problems\"/I-ADR/I-Excessive_upper_gastrointestinal_gas, \".\"/O/O, \"ive\"/O/O, \"been\"/O/O, \"on\"/O/O, \"arthrotec\"/B-Drug/B-Arthrotec, \"50\"/O/O, \"for\"/O/O, \"over\"/O/O, \"10\"/O/O, \"years\"/O/O, \"on\"/O/O, \"and\"/O/O, \"off\"/O/O, \",\"/O/O, \"only\"/O/O, \"taking\"/O/O, \"it\"/O/O, \"when\"/O/O, \"i\"/O/O, \"needed\"/O/O, \"it\"/O/O, \".\"/O/O, \"due\"/O/O, \"to\"/O/O, \"my\"/O/O, \"arthritis\"/B-Disease/B-Arthritis, \"getting\"/O/O, \"progressively\"/O/O, \"worse\"/O/O, \",\"/O/O, \"to\"/O/O, \"the\"/O/O, \"point\"/O/O, \"where\"/O/O, \"i\"/O/O, \"am\"/O/O, \"in\"/O/O, \"tears\"/O/O, \"with\"/O/O, \"the\"/O/O, \"agony\"/B-Symptom/B-Agony, \",\"/O/O, \"gp\"/O/O, \"'\"/O/O, \"s\"/O/O, \"started\"/O/O, \"me\"/O/O, \"on\"/O/O, \"75\"/O/O, \"twice\"/O/O, \"a\"/O/O, \"day\"/O/O, \"and\"/O/O, \"i\"/O/O, \"have\"/O/O, \"to\"/O/O, \"take\"/O/O, \"it\"/O/O, \".\"/O/O, \"every\"/O/O, \"day\"/O/O, \"for\"/O/O, \"the\"/O/O, \"next\"/O/O, \"month\"/O/O, \"to\"/O/O, \"see\"/O/O, \"how\"/O/O, \"i\"/O/O, \"get\"/O/O, \"on\"/O/O, \",\"/O/O, \"here\"/O/O, \"goes\"/O/O, \".\"/O/O, \"so\"/O/O, \"far\"/O/O, \"its\"/O/O, \"been\"/O/O, \"very\"/O/O, \"good\"/O/O, \",\"/O/O, \"pains\"/B-Symptom/B-Pain, \"almost\"/O/O, \"gone\"/O/O, \",\"/O/O, \"but\"/O/O, \"i\"/O/O, \"feel\"/B-ADR/B-Malaise, \"a\"/I-ADR/I-Malaise, \"bit\"/I-ADR/I-Malaise, \"weird\"/I-ADR/I-Malaise, \",\"/O/O, \"didnt\"/O/O, \"have\"/O/O, \"that\"/O/O, \"when\"/O/O, \"on\"/O/O, \"50\"/O/O, \".\"/O/O]"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "temp = docs[0]\n",
        "temp"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2EeTdfHstGij",
        "outputId": "981ae4fc-ce4d-4967-dafb-c62de60775c3"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'ner': ['Token[0]: \"i\"'/'O' (1.0),\n",
              "  'Token[1]: \"feel\"'/'O' (1.0),\n",
              "  'Token[2]: \"a\"'/'O' (1.0),\n",
              "  'Token[3]: \"bit\"'/'B-ADR' (1.0),\n",
              "  'Token[4]: \"drowsy\"'/'I-ADR' (1.0),\n",
              "  'Token[5]: \"&\"'/'O' (1.0),\n",
              "  'Token[6]: \"have\"'/'O' (1.0),\n",
              "  'Token[7]: \"a\"'/'O' (1.0),\n",
              "  'Token[8]: \"little\"'/'B-ADR' (1.0),\n",
              "  'Token[9]: \"blurred\"'/'I-ADR' (1.0),\n",
              "  'Token[10]: \"vision\"'/'I-ADR' (1.0),\n",
              "  'Token[11]: \",\"'/'O' (1.0),\n",
              "  'Token[12]: \"so\"'/'O' (1.0),\n",
              "  'Token[13]: \"far\"'/'O' (1.0),\n",
              "  'Token[14]: \"no\"'/'O' (1.0),\n",
              "  'Token[15]: \"gastric\"'/'B-ADR' (1.0),\n",
              "  'Token[16]: \"problems\"'/'I-ADR' (1.0),\n",
              "  'Token[17]: \".\"'/'O' (1.0),\n",
              "  'Token[18]: \"ive\"'/'O' (1.0),\n",
              "  'Token[19]: \"been\"'/'O' (1.0),\n",
              "  'Token[20]: \"on\"'/'O' (1.0),\n",
              "  'Token[21]: \"arthrotec\"'/'B-Drug' (1.0),\n",
              "  'Token[22]: \"50\"'/'O' (1.0),\n",
              "  'Token[23]: \"for\"'/'O' (1.0),\n",
              "  'Token[24]: \"over\"'/'O' (1.0),\n",
              "  'Token[25]: \"10\"'/'O' (1.0),\n",
              "  'Token[26]: \"years\"'/'O' (1.0),\n",
              "  'Token[27]: \"on\"'/'O' (1.0),\n",
              "  'Token[28]: \"and\"'/'O' (1.0),\n",
              "  'Token[29]: \"off\"'/'O' (1.0),\n",
              "  'Token[30]: \",\"'/'O' (1.0),\n",
              "  'Token[31]: \"only\"'/'O' (1.0),\n",
              "  'Token[32]: \"taking\"'/'O' (1.0),\n",
              "  'Token[33]: \"it\"'/'O' (1.0),\n",
              "  'Token[34]: \"when\"'/'O' (1.0),\n",
              "  'Token[35]: \"i\"'/'O' (1.0),\n",
              "  'Token[36]: \"needed\"'/'O' (1.0),\n",
              "  'Token[37]: \"it\"'/'O' (1.0),\n",
              "  'Token[38]: \".\"'/'O' (1.0),\n",
              "  'Token[39]: \"due\"'/'O' (1.0),\n",
              "  'Token[40]: \"to\"'/'O' (1.0),\n",
              "  'Token[41]: \"my\"'/'O' (1.0),\n",
              "  'Token[42]: \"arthritis\"'/'B-Disease' (1.0),\n",
              "  'Token[43]: \"getting\"'/'O' (1.0),\n",
              "  'Token[44]: \"progressively\"'/'O' (1.0),\n",
              "  'Token[45]: \"worse\"'/'O' (1.0),\n",
              "  'Token[46]: \",\"'/'O' (1.0),\n",
              "  'Token[47]: \"to\"'/'O' (1.0),\n",
              "  'Token[48]: \"the\"'/'O' (1.0),\n",
              "  'Token[49]: \"point\"'/'O' (1.0),\n",
              "  'Token[50]: \"where\"'/'O' (1.0),\n",
              "  'Token[51]: \"i\"'/'O' (1.0),\n",
              "  'Token[52]: \"am\"'/'O' (1.0),\n",
              "  'Token[53]: \"in\"'/'O' (1.0),\n",
              "  'Token[54]: \"tears\"'/'O' (1.0),\n",
              "  'Token[55]: \"with\"'/'O' (1.0),\n",
              "  'Token[56]: \"the\"'/'O' (1.0),\n",
              "  'Token[57]: \"agony\"'/'B-Symptom' (1.0),\n",
              "  'Token[58]: \",\"'/'O' (1.0),\n",
              "  'Token[59]: \"gp\"'/'O' (1.0),\n",
              "  'Token[60]: \"'\"'/'O' (1.0),\n",
              "  'Token[61]: \"s\"'/'O' (1.0),\n",
              "  'Token[62]: \"started\"'/'O' (1.0),\n",
              "  'Token[63]: \"me\"'/'O' (1.0),\n",
              "  'Token[64]: \"on\"'/'O' (1.0),\n",
              "  'Token[65]: \"75\"'/'O' (1.0),\n",
              "  'Token[66]: \"twice\"'/'O' (1.0),\n",
              "  'Token[67]: \"a\"'/'O' (1.0),\n",
              "  'Token[68]: \"day\"'/'O' (1.0),\n",
              "  'Token[69]: \"and\"'/'O' (1.0),\n",
              "  'Token[70]: \"i\"'/'O' (1.0),\n",
              "  'Token[71]: \"have\"'/'O' (1.0),\n",
              "  'Token[72]: \"to\"'/'O' (1.0),\n",
              "  'Token[73]: \"take\"'/'O' (1.0),\n",
              "  'Token[74]: \"it\"'/'O' (1.0),\n",
              "  'Token[75]: \".\"'/'O' (1.0),\n",
              "  'Token[76]: \"every\"'/'O' (1.0),\n",
              "  'Token[77]: \"day\"'/'O' (1.0),\n",
              "  'Token[78]: \"for\"'/'O' (1.0),\n",
              "  'Token[79]: \"the\"'/'O' (1.0),\n",
              "  'Token[80]: \"next\"'/'O' (1.0),\n",
              "  'Token[81]: \"month\"'/'O' (1.0),\n",
              "  'Token[82]: \"to\"'/'O' (1.0),\n",
              "  'Token[83]: \"see\"'/'O' (1.0),\n",
              "  'Token[84]: \"how\"'/'O' (1.0),\n",
              "  'Token[85]: \"i\"'/'O' (1.0),\n",
              "  'Token[86]: \"get\"'/'O' (1.0),\n",
              "  'Token[87]: \"on\"'/'O' (1.0),\n",
              "  'Token[88]: \",\"'/'O' (1.0),\n",
              "  'Token[89]: \"here\"'/'O' (1.0),\n",
              "  'Token[90]: \"goes\"'/'O' (1.0),\n",
              "  'Token[91]: \".\"'/'O' (1.0),\n",
              "  'Token[92]: \"so\"'/'O' (1.0),\n",
              "  'Token[93]: \"far\"'/'O' (1.0),\n",
              "  'Token[94]: \"its\"'/'O' (1.0),\n",
              "  'Token[95]: \"been\"'/'O' (1.0),\n",
              "  'Token[96]: \"very\"'/'O' (1.0),\n",
              "  'Token[97]: \"good\"'/'O' (1.0),\n",
              "  'Token[98]: \",\"'/'O' (1.0),\n",
              "  'Token[99]: \"pains\"'/'B-Symptom' (1.0),\n",
              "  'Token[100]: \"almost\"'/'O' (1.0),\n",
              "  'Token[101]: \"gone\"'/'O' (1.0),\n",
              "  'Token[102]: \",\"'/'O' (1.0),\n",
              "  'Token[103]: \"but\"'/'O' (1.0),\n",
              "  'Token[104]: \"i\"'/'O' (1.0),\n",
              "  'Token[105]: \"feel\"'/'B-ADR' (1.0),\n",
              "  'Token[106]: \"a\"'/'I-ADR' (1.0),\n",
              "  'Token[107]: \"bit\"'/'I-ADR' (1.0),\n",
              "  'Token[108]: \"weird\"'/'I-ADR' (1.0),\n",
              "  'Token[109]: \",\"'/'O' (1.0),\n",
              "  'Token[110]: \"didnt\"'/'O' (1.0),\n",
              "  'Token[111]: \"have\"'/'O' (1.0),\n",
              "  'Token[112]: \"that\"'/'O' (1.0),\n",
              "  'Token[113]: \"when\"'/'O' (1.0),\n",
              "  'Token[114]: \"on\"'/'O' (1.0),\n",
              "  'Token[115]: \"50\"'/'O' (1.0),\n",
              "  'Token[116]: \".\"'/'O' (1.0)],\n",
              " 'meddra': ['Token[0]: \"i\"'/'O' (1.0),\n",
              "  'Token[1]: \"feel\"'/'O' (1.0),\n",
              "  'Token[2]: \"a\"'/'O' (1.0),\n",
              "  'Token[3]: \"bit\"'/'B-Drowsy' (1.0),\n",
              "  'Token[4]: \"drowsy\"'/'I-Drowsy' (1.0),\n",
              "  'Token[5]: \"&\"'/'O' (1.0),\n",
              "  'Token[6]: \"have\"'/'O' (1.0),\n",
              "  'Token[7]: \"a\"'/'O' (1.0),\n",
              "  'Token[8]: \"little\"'/'B-Blurred_vision_-_hazy' (1.0),\n",
              "  'Token[9]: \"blurred\"'/'I-Blurred_vision_-_hazy' (1.0),\n",
              "  'Token[10]: \"vision\"'/'I-Blurred_vision_-_hazy' (1.0),\n",
              "  'Token[11]: \",\"'/'O' (1.0),\n",
              "  'Token[12]: \"so\"'/'O' (1.0),\n",
              "  'Token[13]: \"far\"'/'O' (1.0),\n",
              "  'Token[14]: \"no\"'/'O' (1.0),\n",
              "  'Token[15]: \"gastric\"'/'B-Excessive_upper_gastrointestinal_gas' (1.0),\n",
              "  'Token[16]: \"problems\"'/'I-Excessive_upper_gastrointestinal_gas' (1.0),\n",
              "  'Token[17]: \".\"'/'O' (1.0),\n",
              "  'Token[18]: \"ive\"'/'O' (1.0),\n",
              "  'Token[19]: \"been\"'/'O' (1.0),\n",
              "  'Token[20]: \"on\"'/'O' (1.0),\n",
              "  'Token[21]: \"arthrotec\"'/'B-Arthrotec' (1.0),\n",
              "  'Token[22]: \"50\"'/'O' (1.0),\n",
              "  'Token[23]: \"for\"'/'O' (1.0),\n",
              "  'Token[24]: \"over\"'/'O' (1.0),\n",
              "  'Token[25]: \"10\"'/'O' (1.0),\n",
              "  'Token[26]: \"years\"'/'O' (1.0),\n",
              "  'Token[27]: \"on\"'/'O' (1.0),\n",
              "  'Token[28]: \"and\"'/'O' (1.0),\n",
              "  'Token[29]: \"off\"'/'O' (1.0),\n",
              "  'Token[30]: \",\"'/'O' (1.0),\n",
              "  'Token[31]: \"only\"'/'O' (1.0),\n",
              "  'Token[32]: \"taking\"'/'O' (1.0),\n",
              "  'Token[33]: \"it\"'/'O' (1.0),\n",
              "  'Token[34]: \"when\"'/'O' (1.0),\n",
              "  'Token[35]: \"i\"'/'O' (1.0),\n",
              "  'Token[36]: \"needed\"'/'O' (1.0),\n",
              "  'Token[37]: \"it\"'/'O' (1.0),\n",
              "  'Token[38]: \".\"'/'O' (1.0),\n",
              "  'Token[39]: \"due\"'/'O' (1.0),\n",
              "  'Token[40]: \"to\"'/'O' (1.0),\n",
              "  'Token[41]: \"my\"'/'O' (1.0),\n",
              "  'Token[42]: \"arthritis\"'/'B-Arthritis' (1.0),\n",
              "  'Token[43]: \"getting\"'/'O' (1.0),\n",
              "  'Token[44]: \"progressively\"'/'O' (1.0),\n",
              "  'Token[45]: \"worse\"'/'O' (1.0),\n",
              "  'Token[46]: \",\"'/'O' (1.0),\n",
              "  'Token[47]: \"to\"'/'O' (1.0),\n",
              "  'Token[48]: \"the\"'/'O' (1.0),\n",
              "  'Token[49]: \"point\"'/'O' (1.0),\n",
              "  'Token[50]: \"where\"'/'O' (1.0),\n",
              "  'Token[51]: \"i\"'/'O' (1.0),\n",
              "  'Token[52]: \"am\"'/'O' (1.0),\n",
              "  'Token[53]: \"in\"'/'O' (1.0),\n",
              "  'Token[54]: \"tears\"'/'O' (1.0),\n",
              "  'Token[55]: \"with\"'/'O' (1.0),\n",
              "  'Token[56]: \"the\"'/'O' (1.0),\n",
              "  'Token[57]: \"agony\"'/'B-Agony' (1.0),\n",
              "  'Token[58]: \",\"'/'O' (1.0),\n",
              "  'Token[59]: \"gp\"'/'O' (1.0),\n",
              "  'Token[60]: \"'\"'/'O' (1.0),\n",
              "  'Token[61]: \"s\"'/'O' (1.0),\n",
              "  'Token[62]: \"started\"'/'O' (1.0),\n",
              "  'Token[63]: \"me\"'/'O' (1.0),\n",
              "  'Token[64]: \"on\"'/'O' (1.0),\n",
              "  'Token[65]: \"75\"'/'O' (1.0),\n",
              "  'Token[66]: \"twice\"'/'O' (1.0),\n",
              "  'Token[67]: \"a\"'/'O' (1.0),\n",
              "  'Token[68]: \"day\"'/'O' (1.0),\n",
              "  'Token[69]: \"and\"'/'O' (1.0),\n",
              "  'Token[70]: \"i\"'/'O' (1.0),\n",
              "  'Token[71]: \"have\"'/'O' (1.0),\n",
              "  'Token[72]: \"to\"'/'O' (1.0),\n",
              "  'Token[73]: \"take\"'/'O' (1.0),\n",
              "  'Token[74]: \"it\"'/'O' (1.0),\n",
              "  'Token[75]: \".\"'/'O' (1.0),\n",
              "  'Token[76]: \"every\"'/'O' (1.0),\n",
              "  'Token[77]: \"day\"'/'O' (1.0),\n",
              "  'Token[78]: \"for\"'/'O' (1.0),\n",
              "  'Token[79]: \"the\"'/'O' (1.0),\n",
              "  'Token[80]: \"next\"'/'O' (1.0),\n",
              "  'Token[81]: \"month\"'/'O' (1.0),\n",
              "  'Token[82]: \"to\"'/'O' (1.0),\n",
              "  'Token[83]: \"see\"'/'O' (1.0),\n",
              "  'Token[84]: \"how\"'/'O' (1.0),\n",
              "  'Token[85]: \"i\"'/'O' (1.0),\n",
              "  'Token[86]: \"get\"'/'O' (1.0),\n",
              "  'Token[87]: \"on\"'/'O' (1.0),\n",
              "  'Token[88]: \",\"'/'O' (1.0),\n",
              "  'Token[89]: \"here\"'/'O' (1.0),\n",
              "  'Token[90]: \"goes\"'/'O' (1.0),\n",
              "  'Token[91]: \".\"'/'O' (1.0),\n",
              "  'Token[92]: \"so\"'/'O' (1.0),\n",
              "  'Token[93]: \"far\"'/'O' (1.0),\n",
              "  'Token[94]: \"its\"'/'O' (1.0),\n",
              "  'Token[95]: \"been\"'/'O' (1.0),\n",
              "  'Token[96]: \"very\"'/'O' (1.0),\n",
              "  'Token[97]: \"good\"'/'O' (1.0),\n",
              "  'Token[98]: \",\"'/'O' (1.0),\n",
              "  'Token[99]: \"pains\"'/'B-Pain' (1.0),\n",
              "  'Token[100]: \"almost\"'/'O' (1.0),\n",
              "  'Token[101]: \"gone\"'/'O' (1.0),\n",
              "  'Token[102]: \",\"'/'O' (1.0),\n",
              "  'Token[103]: \"but\"'/'O' (1.0),\n",
              "  'Token[104]: \"i\"'/'O' (1.0),\n",
              "  'Token[105]: \"feel\"'/'B-Malaise' (1.0),\n",
              "  'Token[106]: \"a\"'/'I-Malaise' (1.0),\n",
              "  'Token[107]: \"bit\"'/'I-Malaise' (1.0),\n",
              "  'Token[108]: \"weird\"'/'I-Malaise' (1.0),\n",
              "  'Token[109]: \",\"'/'O' (1.0),\n",
              "  'Token[110]: \"didnt\"'/'O' (1.0),\n",
              "  'Token[111]: \"have\"'/'O' (1.0),\n",
              "  'Token[112]: \"that\"'/'O' (1.0),\n",
              "  'Token[113]: \"when\"'/'O' (1.0),\n",
              "  'Token[114]: \"on\"'/'O' (1.0),\n",
              "  'Token[115]: \"50\"'/'O' (1.0),\n",
              "  'Token[116]: \".\"'/'O' (1.0)]}"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "temp.annotation_layers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "4CsGKKB6tGik"
      },
      "outputs": [],
      "source": [
        "\n",
        "# define your NER tags\n",
        "label_dict = {\n",
        "    'ADR': 'ADR',\n",
        "    'Drug': 'Drug',\n",
        "    'Finding': 'Finding',\n",
        "    'Symptom': 'Symptom',\n",
        "    'Disease': 'Disease'\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QmNsy8AltGik",
        "outputId": "196b793f-8a5c-4ad7-bfdd-fe89f883b52c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:47:49,281 Filtering empty sentences\n",
            "2023-04-21 16:47:49,302 Corpus: 948 train + 119 dev + 119 test sentences\n",
            "2023-04-21 16:47:49,304 Computing label dictionary. Progress:\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "948it [00:00, 6456.14it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:47:49,456 Dictionary created for label 'ner' with 12 values: O (seen 79057 times), I-ADR (seen 7953 times), B-ADR (seen 4712 times), B-Drug (seen 1396 times), I-Finding (seen 349 times), B-Finding (seen 309 times), B-Disease (seen 231 times), B-Symptom (seen 222 times), I-Symptom (seen 218 times), I-Disease (seen 169 times), I-Drug (seen 161 times)\n",
            "False\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "True True False 768\n",
            "{'is_token_embedding': True, 'is_document_embedding': False, 'allow_long_sentences': False, 'tokenizer': BertTokenizerFast(name_or_path='emilyalsentzer/Bio_ClinicalBERT', vocab_size=28996, model_max_length=512, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]', 'additional_special_tokens': ['[FLERT]']}, clean_up_tokenization_spaces=True), 'context_length': 64, 'context_dropout': 0.5, 'respect_document_boundaries': True, 'truncate': True, 'stride': 0, 'embedding_length': 768, 'name': 'transformer-emilyalsentzer/Bio_ClinicalBERT', 'fine_tune': True, 'use_lang_emb': False, 'force_max_length': True, 'feature_extractor': None, 'use_context_separator': True, 'needs_manual_ocr': False}\n",
            "BertTokenizerFast(name_or_path='emilyalsentzer/Bio_ClinicalBERT', vocab_size=28996, model_max_length=512, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]', 'additional_special_tokens': ['[FLERT]']}, clean_up_tokenization_spaces=True)\n",
            "2023-04-21 16:47:54,340 SequenceTagger predicts: Dictionary with 12 tags: <unk>, O, I-ADR, B-ADR, B-Drug, I-Finding, B-Finding, B-Disease, B-Symptom, I-Symptom, I-Disease, I-Drug\n",
            "True\n",
            "False\n"
          ]
        }
      ],
      "source": [
        "from flair.data import Corpus\n",
        "\n",
        "from flair.embeddings import WordEmbeddings, FlairEmbeddings, StackedEmbeddings, TransformerWordEmbeddings\n",
        "from flair.models import SequenceTagger\n",
        "from flair.trainers import ModelTrainer\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "from torch.optim import AdamW, SGD\n",
        "\n",
        "from flair.visual.training_curves import Plotter\n",
        "plotter = Plotter()\n",
        "\n",
        "\n",
        "\n",
        "train_docs, test_docs = train_test_split(docs, test_size=0.2, random_state=321)\n",
        "dev_docs, test_docs = train_test_split(test_docs, test_size=0.5, random_state=123)\n",
        "\n",
        "corpus = Corpus(train=train_docs, dev=dev_docs, test=test_docs) # Corpus: 948 train + 119 dev + 119 test sentences\n",
        "corpus.filter_empty_sentences()\n",
        "# # create a tag dictionary\n",
        "labels = ['ADR', 'Drug', 'Finding', 'Symptom', 'Disease']\n",
        "# tag_dictionary = Dictionary(add_unk=True)\n",
        "# for label in labels:\n",
        "#     tag_dictionary.add_item(label)\n",
        "tag_dictionary = corpus.make_label_dictionary(label_type='ner')\n",
        "print(tag_dictionary.span_labels)\n",
        "\n",
        "# create a stacked embedding using GloVe and Flair embeddings\n",
        "# stacked_embeddings = StackedEmbeddings([\n",
        "#     WordEmbeddings('glove',\n",
        "#                    force_cpu=False,\n",
        "#                 #    embedding_length=1024,\n",
        "#                   #  fine_tune=True,\n",
        "#                    ),\n",
        "#     # TransformerWordEmbeddings('emilyalsentzer/Bio_ClinicalBERT', layers=\"-1\",\n",
        "#     #                                    subtoken_pooling=\"first\",\n",
        "#     #                                    is_document_embedding=True,\n",
        "#     #                                    fine_tune=True,\n",
        "#     #                                    use_context=True,\n",
        "#     #                                    max_seq_length=1024,\n",
        "#     #                                    ),\n",
        "#     FlairEmbeddings('news-forward',\n",
        "#                     # fine_tune=True\n",
        "#                     ),\n",
        "#     FlairEmbeddings('news-backward',\n",
        "#                     # fine_tune=True\n",
        "#                     ),\n",
        "# ])\n",
        "# https://github.com/flairNLP/flair/blob/master/resources/docs/embeddings/FLAIR_EMBEDDINGS.md\n",
        "\n",
        "\n",
        "embeddings = TransformerWordEmbeddings(model='emilyalsentzer/Bio_ClinicalBERT',\n",
        "                                       layers=\"-1\",\n",
        "                                       subtoken_pooling=\"first\",\n",
        "                                       fine_tune=True,\n",
        "                                       use_context=True,\n",
        "                                       force_max_length=True,\n",
        "                                      #  truncate=True, # not accepted here\n",
        "                                      # #  is_document_embedding=True, # False is default\n",
        "                                      #  allow_long_sentences=True, # True is default; doesnt go through\n",
        "                                      #  embedding_length=512,\n",
        "                                       )\n",
        "# embeddings.allow_long_sentences = False\n",
        "embeddings.truncate = True\n",
        "# embeddings.embedding_length_internal = 512\n",
        "print(embeddings.force_max_length,embeddings.truncate,embeddings.allow_long_sentences,embeddings.embedding_length_internal)\n",
        "embeddings.tokenizer.model_max_length=512\n",
        "print(embeddings.to_args())\n",
        "print(embeddings.tokenizer)\n",
        "\n",
        "\n",
        "# create a SequenceTagger object and specify the tag dictionary and embeddings to use\n",
        "tagger = SequenceTagger(hidden_size=256,\n",
        "                        # embeddings=stacked_embeddings,\n",
        "                        embeddings=embeddings,\n",
        "                        tag_dictionary=tag_dictionary,\n",
        "                        tag_type='ner',\n",
        "                        use_crf=False,\n",
        "                        use_rnn=False,\n",
        "                        # reproject_embeddings=False\n",
        "                        )\n",
        "tagger.predict_spans = False\n",
        "\n",
        "print(tagger._determine_if_span_prediction_problem(tagger.label_dictionary))\n",
        "print(tagger.predict_spans)\n",
        "\n",
        "# https://github.com/flairNLP/flair/blob/master/flair/trainers/trainer.py\n",
        "# create a ModelTrainer object and train the tagger\n",
        "trainer = ModelTrainer(tagger, corpus)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "fh0zU2jHAA4J"
      },
      "outputs": [],
      "source": [
        "# !tensorboard --logdir=runs\n",
        "# http://localhost:6006/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UKTMzxV8zF9o",
        "outputId": "75274103-3302-435d-89d4-74ee26f8ae91"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:47:54,374 tensorboard logging path is runs/tb_logs_v3_Bio_ClinicalBERT\n",
            "2023-04-21 16:47:54,385 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:47:54,387 Model: \"SequenceTagger(\n",
            "  (embeddings): TransformerWordEmbeddings(\n",
            "    (model): BertModel(\n",
            "      (embeddings): BertEmbeddings(\n",
            "        (word_embeddings): Embedding(28997, 768)\n",
            "        (position_embeddings): Embedding(512, 768)\n",
            "        (token_type_embeddings): Embedding(2, 768)\n",
            "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
            "        (dropout): Dropout(p=0.1, inplace=False)\n",
            "      )\n",
            "      (encoder): BertEncoder(\n",
            "        (layer): ModuleList(\n",
            "          (0-11): 12 x BertLayer(\n",
            "            (attention): BertAttention(\n",
            "              (self): BertSelfAttention(\n",
            "                (query): Linear(in_features=768, out_features=768, bias=True)\n",
            "                (key): Linear(in_features=768, out_features=768, bias=True)\n",
            "                (value): Linear(in_features=768, out_features=768, bias=True)\n",
            "                (dropout): Dropout(p=0.1, inplace=False)\n",
            "              )\n",
            "              (output): BertSelfOutput(\n",
            "                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
            "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
            "                (dropout): Dropout(p=0.1, inplace=False)\n",
            "              )\n",
            "            )\n",
            "            (intermediate): BertIntermediate(\n",
            "              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
            "              (intermediate_act_fn): GELUActivation()\n",
            "            )\n",
            "            (output): BertOutput(\n",
            "              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
            "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
            "              (dropout): Dropout(p=0.1, inplace=False)\n",
            "            )\n",
            "          )\n",
            "        )\n",
            "      )\n",
            "      (pooler): BertPooler(\n",
            "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
            "        (activation): Tanh()\n",
            "      )\n",
            "    )\n",
            "  )\n",
            "  (locked_dropout): LockedDropout(p=0.5)\n",
            "  (embedding2nn): Linear(in_features=768, out_features=768, bias=True)\n",
            "  (linear): Linear(in_features=768, out_features=12, bias=True)\n",
            "  (loss_function): CrossEntropyLoss()\n",
            ")\"\n",
            "2023-04-21 16:47:54,389 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:47:54,391 Corpus: \"Corpus: 948 train + 119 dev + 119 test sentences\"\n",
            "2023-04-21 16:47:54,392 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:47:54,394 Parameters:\n",
            "2023-04-21 16:47:54,395  - learning_rate: \"0.000005\"\n",
            "2023-04-21 16:47:54,396  - mini_batch_size: \"40\"\n",
            "2023-04-21 16:47:54,398  - patience: \"1\"\n",
            "2023-04-21 16:47:54,400  - anneal_factor: \"0.8\"\n",
            "2023-04-21 16:47:54,401  - max_epochs: \"100\"\n",
            "2023-04-21 16:47:54,403  - shuffle: \"True\"\n",
            "2023-04-21 16:47:54,404  - train_with_dev: \"False\"\n",
            "2023-04-21 16:47:54,406  - batch_growth_annealing: \"False\"\n",
            "2023-04-21 16:47:54,408 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:47:54,409 Model training base path: \"flair_ner_model_v3_Bio_ClinicalBERT\"\n",
            "2023-04-21 16:47:54,411 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:47:54,413 Device: cuda:0\n",
            "2023-04-21 16:47:54,414 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:47:54,415 Embeddings storage mode: gpu\n",
            "2023-04-21 16:47:54,426 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:47:57,556 epoch 1 - iter 2/24 - loss 2.57818524 - time (sec): 3.13 - samples/sec: 2767.58 - lr: 0.000005\n",
            "2023-04-21 16:47:59,597 epoch 1 - iter 4/24 - loss 2.54933448 - time (sec): 5.17 - samples/sec: 3025.98 - lr: 0.000005\n",
            "2023-04-21 16:48:01,704 epoch 1 - iter 6/24 - loss 2.47916479 - time (sec): 7.28 - samples/sec: 3213.39 - lr: 0.000005\n",
            "2023-04-21 16:48:03,774 epoch 1 - iter 8/24 - loss 2.42659275 - time (sec): 9.35 - samples/sec: 3291.98 - lr: 0.000005\n",
            "2023-04-21 16:48:05,887 epoch 1 - iter 10/24 - loss 2.36948801 - time (sec): 11.46 - samples/sec: 3411.05 - lr: 0.000005\n",
            "2023-04-21 16:48:07,946 epoch 1 - iter 12/24 - loss 2.31905968 - time (sec): 13.52 - samples/sec: 3455.67 - lr: 0.000005\n",
            "2023-04-21 16:48:10,110 epoch 1 - iter 14/24 - loss 2.24942616 - time (sec): 15.68 - samples/sec: 3498.18 - lr: 0.000005\n",
            "2023-04-21 16:48:12,475 epoch 1 - iter 16/24 - loss 2.17450723 - time (sec): 18.05 - samples/sec: 3521.41 - lr: 0.000005\n",
            "2023-04-21 16:48:14,582 epoch 1 - iter 18/24 - loss 2.10068117 - time (sec): 20.15 - samples/sec: 3543.71 - lr: 0.000005\n",
            "2023-04-21 16:48:16,632 epoch 1 - iter 20/24 - loss 2.03880447 - time (sec): 22.20 - samples/sec: 3554.89 - lr: 0.000005\n",
            "2023-04-21 16:48:18,760 epoch 1 - iter 22/24 - loss 1.97326001 - time (sec): 24.33 - samples/sec: 3582.68 - lr: 0.000005\n",
            "2023-04-21 16:48:20,659 epoch 1 - iter 24/24 - loss 1.91325885 - time (sec): 26.23 - samples/sec: 3612.90 - lr: 0.000005\n",
            "2023-04-21 16:48:20,663 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:48:20,665 EPOCH 1 done: loss 1.9133 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.21it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:48:31,668 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:48:34,521 TRAIN : loss 1.0014984607696533 - f1-score (weighted avg)  0.0\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.23it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:48:36,012 Evaluating as a multi-label problem: False\n",
            "2023-04-21 16:48:36,073 DEV : loss 0.9971427321434021 - f1-score (weighted avg)  0.0\n",
            "2023-04-21 16:48:36,093 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 16:48:36,097 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:48:39,058 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:48:41,155 epoch 2 - iter 2/24 - loss 1.14608215 - time (sec): 2.09 - samples/sec: 3724.43 - lr: 0.000005\n",
            "2023-04-21 16:48:43,308 epoch 2 - iter 4/24 - loss 1.11580097 - time (sec): 4.25 - samples/sec: 3806.15 - lr: 0.000005\n",
            "2023-04-21 16:48:45,387 epoch 2 - iter 6/24 - loss 1.08650284 - time (sec): 6.33 - samples/sec: 3583.29 - lr: 0.000005\n",
            "2023-04-21 16:48:47,459 epoch 2 - iter 8/24 - loss 1.04818993 - time (sec): 8.40 - samples/sec: 3529.44 - lr: 0.000005\n",
            "2023-04-21 16:48:49,517 epoch 2 - iter 10/24 - loss 1.00690019 - time (sec): 10.46 - samples/sec: 3614.63 - lr: 0.000005\n",
            "2023-04-21 16:48:51,599 epoch 2 - iter 12/24 - loss 0.97730388 - time (sec): 12.54 - samples/sec: 3690.75 - lr: 0.000005\n",
            "2023-04-21 16:48:53,971 epoch 2 - iter 14/24 - loss 0.95805906 - time (sec): 14.91 - samples/sec: 3677.87 - lr: 0.000005\n",
            "2023-04-21 16:48:56,042 epoch 2 - iter 16/24 - loss 0.93705662 - time (sec): 16.98 - samples/sec: 3686.47 - lr: 0.000005\n",
            "2023-04-21 16:48:58,159 epoch 2 - iter 18/24 - loss 0.91788582 - time (sec): 19.10 - samples/sec: 3678.97 - lr: 0.000005\n",
            "2023-04-21 16:49:00,396 epoch 2 - iter 20/24 - loss 0.91124594 - time (sec): 21.34 - samples/sec: 3646.81 - lr: 0.000005\n",
            "2023-04-21 16:49:02,592 epoch 2 - iter 22/24 - loss 0.89326879 - time (sec): 23.53 - samples/sec: 3731.31 - lr: 0.000005\n",
            "2023-04-21 16:49:04,386 epoch 2 - iter 24/24 - loss 0.88258256 - time (sec): 25.33 - samples/sec: 3742.05 - lr: 0.000005\n",
            "2023-04-21 16:49:04,388 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:49:04,390 EPOCH 2 done: loss 0.8826 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:09<00:00,  2.42it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:49:14,431 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:49:16,181 TRAIN : loss 0.7047016620635986 - f1-score (weighted avg)  0.0\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.24it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:49:17,668 Evaluating as a multi-label problem: False\n",
            "2023-04-21 16:49:17,729 DEV : loss 0.702401340007782 - f1-score (weighted avg)  0.0\n",
            "2023-04-21 16:49:17,747 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 16:49:17,751 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:49:20,966 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:49:23,156 epoch 3 - iter 2/24 - loss 0.73595445 - time (sec): 2.19 - samples/sec: 3668.24 - lr: 0.000005\n",
            "2023-04-21 16:49:25,250 epoch 3 - iter 4/24 - loss 0.73313225 - time (sec): 4.28 - samples/sec: 3917.38 - lr: 0.000005\n",
            "2023-04-21 16:49:27,460 epoch 3 - iter 6/24 - loss 0.73548707 - time (sec): 6.49 - samples/sec: 3851.46 - lr: 0.000005\n",
            "2023-04-21 16:49:29,571 epoch 3 - iter 8/24 - loss 0.72182638 - time (sec): 8.60 - samples/sec: 3854.81 - lr: 0.000005\n",
            "2023-04-21 16:49:31,638 epoch 3 - iter 10/24 - loss 0.70858568 - time (sec): 10.67 - samples/sec: 3845.24 - lr: 0.000005\n",
            "2023-04-21 16:49:34,031 epoch 3 - iter 12/24 - loss 0.71174960 - time (sec): 13.06 - samples/sec: 3890.13 - lr: 0.000005\n",
            "2023-04-21 16:49:36,123 epoch 3 - iter 14/24 - loss 0.70494977 - time (sec): 15.16 - samples/sec: 3802.78 - lr: 0.000005\n",
            "2023-04-21 16:49:38,223 epoch 3 - iter 16/24 - loss 0.69859551 - time (sec): 17.26 - samples/sec: 3775.51 - lr: 0.000005\n",
            "2023-04-21 16:49:40,334 epoch 3 - iter 18/24 - loss 0.69215833 - time (sec): 19.37 - samples/sec: 3778.35 - lr: 0.000005\n",
            "2023-04-21 16:49:42,401 epoch 3 - iter 20/24 - loss 0.68871304 - time (sec): 21.43 - samples/sec: 3772.98 - lr: 0.000005\n",
            "2023-04-21 16:49:44,625 epoch 3 - iter 22/24 - loss 0.68586666 - time (sec): 23.66 - samples/sec: 3755.64 - lr: 0.000005\n",
            "2023-04-21 16:49:46,369 epoch 3 - iter 24/24 - loss 0.68012030 - time (sec): 25.40 - samples/sec: 3730.90 - lr: 0.000005\n",
            "2023-04-21 16:49:46,372 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:49:46,373 EPOCH 3 done: loss 0.6801 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.20it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:49:57,409 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:49:59,151 TRAIN : loss 0.5412646532058716 - f1-score (weighted avg)  0.0003\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:02<00:00,  1.30it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:50:01,599 Evaluating as a multi-label problem: False\n",
            "2023-04-21 16:50:01,661 DEV : loss 0.5448171496391296 - f1-score (weighted avg)  0.0\n",
            "2023-04-21 16:50:01,680 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 16:50:01,684 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:50:04,898 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:50:07,039 epoch 4 - iter 2/24 - loss 0.61987458 - time (sec): 2.14 - samples/sec: 3563.40 - lr: 0.000005\n",
            "2023-04-21 16:50:09,169 epoch 4 - iter 4/24 - loss 0.58387541 - time (sec): 4.27 - samples/sec: 3723.27 - lr: 0.000005\n",
            "2023-04-21 16:50:11,333 epoch 4 - iter 6/24 - loss 0.56536660 - time (sec): 6.43 - samples/sec: 3903.40 - lr: 0.000005\n",
            "2023-04-21 16:50:13,375 epoch 4 - iter 8/24 - loss 0.56575931 - time (sec): 8.48 - samples/sec: 3778.84 - lr: 0.000005\n",
            "2023-04-21 16:50:15,543 epoch 4 - iter 10/24 - loss 0.55712261 - time (sec): 10.64 - samples/sec: 3832.78 - lr: 0.000005\n",
            "2023-04-21 16:50:17,627 epoch 4 - iter 12/24 - loss 0.54909708 - time (sec): 12.73 - samples/sec: 3855.28 - lr: 0.000005\n",
            "2023-04-21 16:50:19,855 epoch 4 - iter 14/24 - loss 0.54591005 - time (sec): 14.95 - samples/sec: 3841.21 - lr: 0.000005\n",
            "2023-04-21 16:50:21,948 epoch 4 - iter 16/24 - loss 0.53441255 - time (sec): 17.05 - samples/sec: 3842.47 - lr: 0.000005\n",
            "2023-04-21 16:50:24,083 epoch 4 - iter 18/24 - loss 0.53128169 - time (sec): 19.18 - samples/sec: 3794.64 - lr: 0.000005\n",
            "2023-04-21 16:50:26,447 epoch 4 - iter 20/24 - loss 0.53537697 - time (sec): 21.55 - samples/sec: 3749.07 - lr: 0.000005\n",
            "2023-04-21 16:50:28,484 epoch 4 - iter 22/24 - loss 0.53266743 - time (sec): 23.58 - samples/sec: 3734.40 - lr: 0.000005\n",
            "2023-04-21 16:50:30,266 epoch 4 - iter 24/24 - loss 0.52858510 - time (sec): 25.37 - samples/sec: 3736.11 - lr: 0.000005\n",
            "2023-04-21 16:50:30,269 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:50:30,272 EPOCH 4 done: loss 0.5286 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:50:40,448 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:50:43,341 TRAIN : loss 0.4296320974826813 - f1-score (weighted avg)  0.3589\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.17it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:50:44,883 Evaluating as a multi-label problem: False\n",
            "2023-04-21 16:50:44,950 DEV : loss 0.4465876519680023 - f1-score (weighted avg)  0.3372\n",
            "2023-04-21 16:50:44,969 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 16:50:44,973 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:50:48,202 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:50:50,387 epoch 5 - iter 2/24 - loss 0.47176597 - time (sec): 2.18 - samples/sec: 3544.43 - lr: 0.000005\n",
            "2023-04-21 16:50:52,819 epoch 5 - iter 4/24 - loss 0.52517767 - time (sec): 4.62 - samples/sec: 3540.62 - lr: 0.000005\n",
            "2023-04-21 16:50:54,883 epoch 5 - iter 6/24 - loss 0.50853980 - time (sec): 6.68 - samples/sec: 3625.65 - lr: 0.000005\n",
            "2023-04-21 16:50:56,956 epoch 5 - iter 8/24 - loss 0.48751034 - time (sec): 8.75 - samples/sec: 3548.77 - lr: 0.000005\n",
            "2023-04-21 16:50:59,066 epoch 5 - iter 10/24 - loss 0.47599848 - time (sec): 10.86 - samples/sec: 3676.63 - lr: 0.000005\n",
            "2023-04-21 16:51:01,168 epoch 5 - iter 12/24 - loss 0.46643367 - time (sec): 12.96 - samples/sec: 3661.19 - lr: 0.000005\n",
            "2023-04-21 16:51:03,213 epoch 5 - iter 14/24 - loss 0.45743508 - time (sec): 15.01 - samples/sec: 3719.46 - lr: 0.000005\n",
            "2023-04-21 16:51:05,315 epoch 5 - iter 16/24 - loss 0.45334020 - time (sec): 17.11 - samples/sec: 3700.23 - lr: 0.000005\n",
            "2023-04-21 16:51:07,399 epoch 5 - iter 18/24 - loss 0.44927065 - time (sec): 19.20 - samples/sec: 3690.20 - lr: 0.000005\n",
            "2023-04-21 16:51:09,468 epoch 5 - iter 20/24 - loss 0.44367859 - time (sec): 21.26 - samples/sec: 3718.12 - lr: 0.000005\n",
            "2023-04-21 16:51:11,707 epoch 5 - iter 22/24 - loss 0.44282299 - time (sec): 23.50 - samples/sec: 3740.46 - lr: 0.000005\n",
            "2023-04-21 16:51:13,501 epoch 5 - iter 24/24 - loss 0.43793883 - time (sec): 25.30 - samples/sec: 3746.36 - lr: 0.000005\n",
            "2023-04-21 16:51:13,504 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:51:13,506 EPOCH 5 done: loss 0.4379 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:51:23,690 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:51:26,635 TRAIN : loss 0.3701850473880768 - f1-score (weighted avg)  0.5248\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.18it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:51:28,160 Evaluating as a multi-label problem: False\n",
            "2023-04-21 16:51:28,227 DEV : loss 0.3883552551269531 - f1-score (weighted avg)  0.4981\n",
            "2023-04-21 16:51:28,247 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 16:51:28,251 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:51:31,459 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:51:33,753 epoch 6 - iter 2/24 - loss 0.43700221 - time (sec): 2.29 - samples/sec: 3253.74 - lr: 0.000005\n",
            "2023-04-21 16:51:35,806 epoch 6 - iter 4/24 - loss 0.40009423 - time (sec): 4.35 - samples/sec: 3595.50 - lr: 0.000005\n",
            "2023-04-21 16:51:37,848 epoch 6 - iter 6/24 - loss 0.40144419 - time (sec): 6.39 - samples/sec: 3585.83 - lr: 0.000005\n",
            "2023-04-21 16:51:39,966 epoch 6 - iter 8/24 - loss 0.40243801 - time (sec): 8.51 - samples/sec: 3748.84 - lr: 0.000005\n",
            "2023-04-21 16:51:42,021 epoch 6 - iter 10/24 - loss 0.39557777 - time (sec): 10.56 - samples/sec: 3803.23 - lr: 0.000005\n",
            "2023-04-21 16:51:44,140 epoch 6 - iter 12/24 - loss 0.39024140 - time (sec): 12.68 - samples/sec: 3771.63 - lr: 0.000005\n",
            "2023-04-21 16:51:46,527 epoch 6 - iter 14/24 - loss 0.39948335 - time (sec): 15.07 - samples/sec: 3757.82 - lr: 0.000005\n",
            "2023-04-21 16:51:48,737 epoch 6 - iter 16/24 - loss 0.40051460 - time (sec): 17.28 - samples/sec: 3701.00 - lr: 0.000005\n",
            "2023-04-21 16:51:50,796 epoch 6 - iter 18/24 - loss 0.39537440 - time (sec): 19.34 - samples/sec: 3679.49 - lr: 0.000005\n",
            "2023-04-21 16:51:52,993 epoch 6 - iter 20/24 - loss 0.39331838 - time (sec): 21.53 - samples/sec: 3739.28 - lr: 0.000005\n",
            "2023-04-21 16:51:55,154 epoch 6 - iter 22/24 - loss 0.38981071 - time (sec): 23.69 - samples/sec: 3758.99 - lr: 0.000005\n",
            "2023-04-21 16:51:56,977 epoch 6 - iter 24/24 - loss 0.38935428 - time (sec): 25.52 - samples/sec: 3714.13 - lr: 0.000005\n",
            "2023-04-21 16:51:56,980 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:51:56,983 EPOCH 6 done: loss 0.3894 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:52:07,165 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:52:10,230 TRAIN : loss 0.33302438259124756 - f1-score (weighted avg)  0.5837\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.20it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:52:11,742 Evaluating as a multi-label problem: False\n",
            "2023-04-21 16:52:11,810 DEV : loss 0.3545033931732178 - f1-score (weighted avg)  0.5508\n",
            "2023-04-21 16:52:11,830 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 16:52:11,835 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:52:14,985 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:52:17,092 epoch 7 - iter 2/24 - loss 0.34886959 - time (sec): 2.10 - samples/sec: 4134.22 - lr: 0.000005\n",
            "2023-04-21 16:52:19,306 epoch 7 - iter 4/24 - loss 0.35639389 - time (sec): 4.32 - samples/sec: 3913.84 - lr: 0.000005\n",
            "2023-04-21 16:52:21,366 epoch 7 - iter 6/24 - loss 0.35670454 - time (sec): 6.38 - samples/sec: 3999.22 - lr: 0.000005\n",
            "2023-04-21 16:52:23,750 epoch 7 - iter 8/24 - loss 0.37631394 - time (sec): 8.76 - samples/sec: 3860.82 - lr: 0.000005\n",
            "2023-04-21 16:52:25,848 epoch 7 - iter 10/24 - loss 0.36935926 - time (sec): 10.86 - samples/sec: 3780.85 - lr: 0.000005\n",
            "2023-04-21 16:52:27,928 epoch 7 - iter 12/24 - loss 0.36753028 - time (sec): 12.94 - samples/sec: 3736.90 - lr: 0.000005\n",
            "2023-04-21 16:52:29,974 epoch 7 - iter 14/24 - loss 0.36710466 - time (sec): 14.99 - samples/sec: 3685.39 - lr: 0.000005\n",
            "2023-04-21 16:52:32,157 epoch 7 - iter 16/24 - loss 0.36304819 - time (sec): 17.17 - samples/sec: 3716.21 - lr: 0.000005\n",
            "2023-04-21 16:52:34,283 epoch 7 - iter 18/24 - loss 0.35975188 - time (sec): 19.30 - samples/sec: 3738.99 - lr: 0.000005\n",
            "2023-04-21 16:52:36,331 epoch 7 - iter 20/24 - loss 0.35427688 - time (sec): 21.34 - samples/sec: 3759.01 - lr: 0.000005\n",
            "2023-04-21 16:52:38,572 epoch 7 - iter 22/24 - loss 0.35448529 - time (sec): 23.58 - samples/sec: 3729.84 - lr: 0.000005\n",
            "2023-04-21 16:52:40,329 epoch 7 - iter 24/24 - loss 0.35374202 - time (sec): 25.34 - samples/sec: 3739.76 - lr: 0.000005\n",
            "2023-04-21 16:52:40,332 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:52:40,333 EPOCH 7 done: loss 0.3537 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:52:50,502 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:52:53,514 TRAIN : loss 0.30099642276763916 - f1-score (weighted avg)  0.635\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.21it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:52:55,019 Evaluating as a multi-label problem: False\n",
            "2023-04-21 16:52:55,087 DEV : loss 0.3265227973461151 - f1-score (weighted avg)  0.5989\n",
            "2023-04-21 16:52:55,106 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 16:52:55,111 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:52:58,277 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:53:00,451 epoch 8 - iter 2/24 - loss 0.30571522 - time (sec): 2.17 - samples/sec: 3992.62 - lr: 0.000005\n",
            "2023-04-21 16:53:02,552 epoch 8 - iter 4/24 - loss 0.31084503 - time (sec): 4.27 - samples/sec: 3758.45 - lr: 0.000005\n",
            "2023-04-21 16:53:04,605 epoch 8 - iter 6/24 - loss 0.30907091 - time (sec): 6.33 - samples/sec: 3881.51 - lr: 0.000005\n",
            "2023-04-21 16:53:06,784 epoch 8 - iter 8/24 - loss 0.30751920 - time (sec): 8.50 - samples/sec: 3881.35 - lr: 0.000005\n",
            "2023-04-21 16:53:08,985 epoch 8 - iter 10/24 - loss 0.31633903 - time (sec): 10.71 - samples/sec: 3850.49 - lr: 0.000005\n",
            "2023-04-21 16:53:11,060 epoch 8 - iter 12/24 - loss 0.32227983 - time (sec): 12.78 - samples/sec: 3827.42 - lr: 0.000005\n",
            "2023-04-21 16:53:13,118 epoch 8 - iter 14/24 - loss 0.32294660 - time (sec): 14.84 - samples/sec: 3821.92 - lr: 0.000005\n",
            "2023-04-21 16:53:15,171 epoch 8 - iter 16/24 - loss 0.31968512 - time (sec): 16.89 - samples/sec: 3808.97 - lr: 0.000005\n",
            "2023-04-21 16:53:17,388 epoch 8 - iter 18/24 - loss 0.32585108 - time (sec): 19.11 - samples/sec: 3789.16 - lr: 0.000005\n",
            "2023-04-21 16:53:19,511 epoch 8 - iter 20/24 - loss 0.32160884 - time (sec): 21.23 - samples/sec: 3772.37 - lr: 0.000005\n",
            "2023-04-21 16:53:21,660 epoch 8 - iter 22/24 - loss 0.31951112 - time (sec): 23.38 - samples/sec: 3739.83 - lr: 0.000005\n",
            "2023-04-21 16:53:23,687 epoch 8 - iter 24/24 - loss 0.32588489 - time (sec): 25.41 - samples/sec: 3730.06 - lr: 0.000005\n",
            "2023-04-21 16:53:23,689 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:53:23,691 EPOCH 8 done: loss 0.3259 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:53:33,893 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:53:36,961 TRAIN : loss 0.27741357684135437 - f1-score (weighted avg)  0.6664\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.16it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:53:38,504 Evaluating as a multi-label problem: False\n",
            "2023-04-21 16:53:38,572 DEV : loss 0.30621859431266785 - f1-score (weighted avg)  0.6297\n",
            "2023-04-21 16:53:38,593 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 16:53:38,597 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:53:41,732 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:53:43,950 epoch 9 - iter 2/24 - loss 0.31688988 - time (sec): 2.22 - samples/sec: 3909.87 - lr: 0.000005\n",
            "2023-04-21 16:53:45,967 epoch 9 - iter 4/24 - loss 0.32183026 - time (sec): 4.23 - samples/sec: 3615.54 - lr: 0.000005\n",
            "2023-04-21 16:53:48,072 epoch 9 - iter 6/24 - loss 0.30351351 - time (sec): 6.34 - samples/sec: 3683.06 - lr: 0.000005\n",
            "2023-04-21 16:53:50,229 epoch 9 - iter 8/24 - loss 0.30562408 - time (sec): 8.50 - samples/sec: 3698.73 - lr: 0.000005\n",
            "2023-04-21 16:53:52,754 epoch 9 - iter 10/24 - loss 0.32439006 - time (sec): 11.02 - samples/sec: 3734.72 - lr: 0.000005\n",
            "2023-04-21 16:53:54,884 epoch 9 - iter 12/24 - loss 0.31931659 - time (sec): 13.15 - samples/sec: 3779.33 - lr: 0.000005\n",
            "2023-04-21 16:53:56,994 epoch 9 - iter 14/24 - loss 0.31622237 - time (sec): 15.26 - samples/sec: 3749.16 - lr: 0.000005\n",
            "2023-04-21 16:53:59,043 epoch 9 - iter 16/24 - loss 0.31580581 - time (sec): 17.31 - samples/sec: 3750.92 - lr: 0.000005\n",
            "2023-04-21 16:54:01,105 epoch 9 - iter 18/24 - loss 0.31122566 - time (sec): 19.37 - samples/sec: 3749.19 - lr: 0.000005\n",
            "2023-04-21 16:54:03,179 epoch 9 - iter 20/24 - loss 0.30761386 - time (sec): 21.44 - samples/sec: 3750.11 - lr: 0.000005\n",
            "2023-04-21 16:54:05,286 epoch 9 - iter 22/24 - loss 0.30500577 - time (sec): 23.55 - samples/sec: 3750.62 - lr: 0.000005\n",
            "2023-04-21 16:54:07,047 epoch 9 - iter 24/24 - loss 0.30176319 - time (sec): 25.31 - samples/sec: 3743.89 - lr: 0.000005\n",
            "2023-04-21 16:54:07,050 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:54:07,051 EPOCH 9 done: loss 0.3018 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:54:17,271 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:54:20,326 TRAIN : loss 0.26389753818511963 - f1-score (weighted avg)  0.6839\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.20it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:54:21,841 Evaluating as a multi-label problem: False\n",
            "2023-04-21 16:54:21,908 DEV : loss 0.2985420525074005 - f1-score (weighted avg)  0.6408\n",
            "2023-04-21 16:54:21,927 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 16:54:21,931 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:54:25,067 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:54:27,154 epoch 10 - iter 2/24 - loss 0.27668418 - time (sec): 2.09 - samples/sec: 3516.14 - lr: 0.000005\n",
            "2023-04-21 16:54:29,359 epoch 10 - iter 4/24 - loss 0.26496635 - time (sec): 4.29 - samples/sec: 4057.68 - lr: 0.000005\n",
            "2023-04-21 16:54:31,479 epoch 10 - iter 6/24 - loss 0.25945701 - time (sec): 6.41 - samples/sec: 3889.13 - lr: 0.000005\n",
            "2023-04-21 16:54:33,519 epoch 10 - iter 8/24 - loss 0.26350975 - time (sec): 8.45 - samples/sec: 3790.29 - lr: 0.000005\n",
            "2023-04-21 16:54:35,646 epoch 10 - iter 10/24 - loss 0.26190825 - time (sec): 10.58 - samples/sec: 3812.90 - lr: 0.000005\n",
            "2023-04-21 16:54:37,759 epoch 10 - iter 12/24 - loss 0.26539202 - time (sec): 12.69 - samples/sec: 3808.66 - lr: 0.000005\n",
            "2023-04-21 16:54:39,850 epoch 10 - iter 14/24 - loss 0.26436383 - time (sec): 14.78 - samples/sec: 3832.60 - lr: 0.000005\n",
            "2023-04-21 16:54:41,978 epoch 10 - iter 16/24 - loss 0.26431832 - time (sec): 16.91 - samples/sec: 3814.44 - lr: 0.000005\n",
            "2023-04-21 16:54:44,155 epoch 10 - iter 18/24 - loss 0.26550005 - time (sec): 19.09 - samples/sec: 3779.91 - lr: 0.000005\n",
            "2023-04-21 16:54:46,212 epoch 10 - iter 20/24 - loss 0.26872040 - time (sec): 21.14 - samples/sec: 3785.05 - lr: 0.000005\n",
            "2023-04-21 16:54:48,569 epoch 10 - iter 22/24 - loss 0.28132496 - time (sec): 23.50 - samples/sec: 3764.30 - lr: 0.000005\n",
            "2023-04-21 16:54:50,438 epoch 10 - iter 24/24 - loss 0.28568337 - time (sec): 25.37 - samples/sec: 3735.62 - lr: 0.000005\n",
            "2023-04-21 16:54:50,441 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:54:50,442 EPOCH 10 done: loss 0.2857 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.33it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:55:00,893 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:55:02,720 TRAIN : loss 0.24874992668628693 - f1-score (weighted avg)  0.6971\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.18it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:55:04,242 Evaluating as a multi-label problem: False\n",
            "2023-04-21 16:55:04,309 DEV : loss 0.2881459891796112 - f1-score (weighted avg)  0.65\n",
            "2023-04-21 16:55:04,328 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 16:55:04,332 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:55:07,550 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:55:09,818 epoch 11 - iter 2/24 - loss 0.26197433 - time (sec): 2.27 - samples/sec: 3647.19 - lr: 0.000005\n",
            "2023-04-21 16:55:11,874 epoch 11 - iter 4/24 - loss 0.25881557 - time (sec): 4.32 - samples/sec: 3882.07 - lr: 0.000005\n",
            "2023-04-21 16:55:14,035 epoch 11 - iter 6/24 - loss 0.26355811 - time (sec): 6.48 - samples/sec: 3780.75 - lr: 0.000005\n",
            "2023-04-21 16:55:17,501 epoch 11 - iter 8/24 - loss 0.28763332 - time (sec): 9.95 - samples/sec: 3371.59 - lr: 0.000005\n",
            "2023-04-21 16:55:19,650 epoch 11 - iter 10/24 - loss 0.28038742 - time (sec): 12.10 - samples/sec: 3456.39 - lr: 0.000005\n",
            "2023-04-21 16:55:21,758 epoch 11 - iter 12/24 - loss 0.27213128 - time (sec): 14.21 - samples/sec: 3524.67 - lr: 0.000005\n",
            "2023-04-21 16:55:23,840 epoch 11 - iter 14/24 - loss 0.26884462 - time (sec): 16.29 - samples/sec: 3550.67 - lr: 0.000005\n",
            "2023-04-21 16:55:25,997 epoch 11 - iter 16/24 - loss 0.27056635 - time (sec): 18.44 - samples/sec: 3528.97 - lr: 0.000005\n",
            "2023-04-21 16:55:28,147 epoch 11 - iter 18/24 - loss 0.27155213 - time (sec): 20.59 - samples/sec: 3563.05 - lr: 0.000005\n",
            "2023-04-21 16:55:30,237 epoch 11 - iter 20/24 - loss 0.27142540 - time (sec): 22.68 - samples/sec: 3574.51 - lr: 0.000005\n",
            "2023-04-21 16:55:32,300 epoch 11 - iter 22/24 - loss 0.27157109 - time (sec): 24.75 - samples/sec: 3563.73 - lr: 0.000005\n",
            "2023-04-21 16:55:34,197 epoch 11 - iter 24/24 - loss 0.27165148 - time (sec): 26.64 - samples/sec: 3556.84 - lr: 0.000005\n",
            "2023-04-21 16:55:34,199 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:55:34,201 EPOCH 11 done: loss 0.2717 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:55:44,397 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:55:47,408 TRAIN : loss 0.2429502308368683 - f1-score (weighted avg)  0.7026\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.17it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:55:48,946 Evaluating as a multi-label problem: False\n",
            "2023-04-21 16:55:49,014 DEV : loss 0.2857285737991333 - f1-score (weighted avg)  0.6507\n",
            "2023-04-21 16:55:49,033 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 16:55:49,037 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:55:52,317 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:55:54,411 epoch 12 - iter 2/24 - loss 0.24659094 - time (sec): 2.09 - samples/sec: 3829.45 - lr: 0.000005\n",
            "2023-04-21 16:55:56,486 epoch 12 - iter 4/24 - loss 0.24500154 - time (sec): 4.17 - samples/sec: 3641.06 - lr: 0.000005\n",
            "2023-04-21 16:55:58,559 epoch 12 - iter 6/24 - loss 0.23915952 - time (sec): 6.24 - samples/sec: 3746.49 - lr: 0.000005\n",
            "2023-04-21 16:56:00,654 epoch 12 - iter 8/24 - loss 0.23975035 - time (sec): 8.33 - samples/sec: 3864.60 - lr: 0.000005\n",
            "2023-04-21 16:56:02,769 epoch 12 - iter 10/24 - loss 0.23601046 - time (sec): 10.45 - samples/sec: 3768.00 - lr: 0.000005\n",
            "2023-04-21 16:56:05,126 epoch 12 - iter 12/24 - loss 0.25453607 - time (sec): 12.81 - samples/sec: 3672.71 - lr: 0.000005\n",
            "2023-04-21 16:56:07,422 epoch 12 - iter 14/24 - loss 0.25725501 - time (sec): 15.10 - samples/sec: 3792.91 - lr: 0.000005\n",
            "2023-04-21 16:56:09,484 epoch 12 - iter 16/24 - loss 0.25966976 - time (sec): 17.16 - samples/sec: 3731.95 - lr: 0.000005\n",
            "2023-04-21 16:56:11,588 epoch 12 - iter 18/24 - loss 0.25845844 - time (sec): 19.27 - samples/sec: 3749.51 - lr: 0.000005\n",
            "2023-04-21 16:56:13,827 epoch 12 - iter 20/24 - loss 0.25758905 - time (sec): 21.51 - samples/sec: 3753.11 - lr: 0.000005\n",
            "2023-04-21 16:56:15,936 epoch 12 - iter 22/24 - loss 0.25992180 - time (sec): 23.62 - samples/sec: 3731.58 - lr: 0.000005\n",
            "2023-04-21 16:56:17,686 epoch 12 - iter 24/24 - loss 0.25798838 - time (sec): 25.37 - samples/sec: 3736.09 - lr: 0.000005\n",
            "2023-04-21 16:56:17,689 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:56:17,690 EPOCH 12 done: loss 0.2580 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:56:27,922 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:56:30,936 TRAIN : loss 0.2276505082845688 - f1-score (weighted avg)  0.7166\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.18it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:56:32,462 Evaluating as a multi-label problem: False\n",
            "2023-04-21 16:56:32,531 DEV : loss 0.2762390971183777 - f1-score (weighted avg)  0.6592\n",
            "2023-04-21 16:56:32,550 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 16:56:32,554 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:56:35,685 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:56:37,966 epoch 13 - iter 2/24 - loss 0.22082416 - time (sec): 2.28 - samples/sec: 3488.86 - lr: 0.000005\n",
            "2023-04-21 16:56:40,114 epoch 13 - iter 4/24 - loss 0.22538725 - time (sec): 4.43 - samples/sec: 3667.07 - lr: 0.000005\n",
            "2023-04-21 16:56:42,156 epoch 13 - iter 6/24 - loss 0.23058652 - time (sec): 6.47 - samples/sec: 3720.54 - lr: 0.000005\n",
            "2023-04-21 16:56:44,262 epoch 13 - iter 8/24 - loss 0.23500674 - time (sec): 8.58 - samples/sec: 3653.26 - lr: 0.000005\n",
            "2023-04-21 16:56:46,384 epoch 13 - iter 10/24 - loss 0.23826648 - time (sec): 10.70 - samples/sec: 3753.95 - lr: 0.000005\n",
            "2023-04-21 16:56:48,781 epoch 13 - iter 12/24 - loss 0.25986524 - time (sec): 13.09 - samples/sec: 3654.82 - lr: 0.000005\n",
            "2023-04-21 16:56:51,107 epoch 13 - iter 14/24 - loss 0.26116998 - time (sec): 15.42 - samples/sec: 3678.27 - lr: 0.000005\n",
            "2023-04-21 16:56:53,229 epoch 13 - iter 16/24 - loss 0.25954719 - time (sec): 17.54 - samples/sec: 3696.36 - lr: 0.000005\n",
            "2023-04-21 16:56:55,292 epoch 13 - iter 18/24 - loss 0.25669895 - time (sec): 19.61 - samples/sec: 3717.38 - lr: 0.000005\n",
            "2023-04-21 16:56:57,400 epoch 13 - iter 20/24 - loss 0.25390429 - time (sec): 21.71 - samples/sec: 3713.01 - lr: 0.000005\n",
            "2023-04-21 16:56:59,514 epoch 13 - iter 22/24 - loss 0.25382547 - time (sec): 23.83 - samples/sec: 3721.22 - lr: 0.000005\n",
            "2023-04-21 16:57:01,280 epoch 13 - iter 24/24 - loss 0.25130329 - time (sec): 25.59 - samples/sec: 3702.91 - lr: 0.000005\n",
            "2023-04-21 16:57:01,283 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:57:01,285 EPOCH 13 done: loss 0.2513 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:57:11,531 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:57:14,553 TRAIN : loss 0.2184978574514389 - f1-score (weighted avg)  0.725\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.19it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:57:16,071 Evaluating as a multi-label problem: False\n",
            "2023-04-21 16:57:16,143 DEV : loss 0.2717222571372986 - f1-score (weighted avg)  0.6626\n",
            "2023-04-21 16:57:16,162 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 16:57:16,166 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:57:19,451 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:57:21,557 epoch 14 - iter 2/24 - loss 0.25549115 - time (sec): 2.10 - samples/sec: 3642.06 - lr: 0.000005\n",
            "2023-04-21 16:57:23,989 epoch 14 - iter 4/24 - loss 0.28316661 - time (sec): 4.54 - samples/sec: 3788.29 - lr: 0.000005\n",
            "2023-04-21 16:57:26,148 epoch 14 - iter 6/24 - loss 0.26189950 - time (sec): 6.69 - samples/sec: 3792.55 - lr: 0.000005\n",
            "2023-04-21 16:57:28,197 epoch 14 - iter 8/24 - loss 0.25435277 - time (sec): 8.74 - samples/sec: 3706.53 - lr: 0.000005\n",
            "2023-04-21 16:57:30,441 epoch 14 - iter 10/24 - loss 0.25030294 - time (sec): 10.99 - samples/sec: 3688.72 - lr: 0.000005\n",
            "2023-04-21 16:57:32,558 epoch 14 - iter 12/24 - loss 0.24980942 - time (sec): 13.10 - samples/sec: 3730.22 - lr: 0.000005\n",
            "2023-04-21 16:57:34,726 epoch 14 - iter 14/24 - loss 0.24651126 - time (sec): 15.27 - samples/sec: 3736.89 - lr: 0.000005\n",
            "2023-04-21 16:57:36,796 epoch 14 - iter 16/24 - loss 0.24604182 - time (sec): 17.34 - samples/sec: 3657.40 - lr: 0.000005\n",
            "2023-04-21 16:57:38,923 epoch 14 - iter 18/24 - loss 0.24310332 - time (sec): 19.47 - samples/sec: 3650.55 - lr: 0.000005\n",
            "2023-04-21 16:57:41,013 epoch 14 - iter 20/24 - loss 0.23927297 - time (sec): 21.56 - samples/sec: 3629.12 - lr: 0.000005\n",
            "2023-04-21 16:57:43,293 epoch 14 - iter 22/24 - loss 0.24123678 - time (sec): 23.84 - samples/sec: 3656.24 - lr: 0.000005\n",
            "2023-04-21 16:57:45,062 epoch 14 - iter 24/24 - loss 0.24441584 - time (sec): 25.61 - samples/sec: 3700.66 - lr: 0.000005\n",
            "2023-04-21 16:57:45,065 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:57:45,067 EPOCH 14 done: loss 0.2444 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:57:55,256 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:57:57,118 TRAIN : loss 0.21191638708114624 - f1-score (weighted avg)  0.7293\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:02<00:00,  1.24it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:57:59,682 Evaluating as a multi-label problem: False\n",
            "2023-04-21 16:57:59,758 DEV : loss 0.26967212557792664 - f1-score (weighted avg)  0.6616\n",
            "2023-04-21 16:57:59,778 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 16:57:59,782 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:58:02,994 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:58:05,075 epoch 15 - iter 2/24 - loss 0.21921295 - time (sec): 2.08 - samples/sec: 3636.07 - lr: 0.000005\n",
            "2023-04-21 16:58:07,181 epoch 15 - iter 4/24 - loss 0.21955451 - time (sec): 4.18 - samples/sec: 3559.02 - lr: 0.000005\n",
            "2023-04-21 16:58:09,243 epoch 15 - iter 6/24 - loss 0.21491946 - time (sec): 6.25 - samples/sec: 3732.31 - lr: 0.000005\n",
            "2023-04-21 16:58:11,360 epoch 15 - iter 8/24 - loss 0.22113102 - time (sec): 8.36 - samples/sec: 3707.81 - lr: 0.000005\n",
            "2023-04-21 16:58:13,509 epoch 15 - iter 10/24 - loss 0.22049764 - time (sec): 10.51 - samples/sec: 3825.01 - lr: 0.000005\n",
            "2023-04-21 16:58:15,583 epoch 15 - iter 12/24 - loss 0.22537429 - time (sec): 12.59 - samples/sec: 3779.96 - lr: 0.000005\n",
            "2023-04-21 16:58:17,700 epoch 15 - iter 14/24 - loss 0.22474953 - time (sec): 14.70 - samples/sec: 3785.71 - lr: 0.000005\n",
            "2023-04-21 16:58:19,969 epoch 15 - iter 16/24 - loss 0.22866670 - time (sec): 16.97 - samples/sec: 3791.49 - lr: 0.000005\n",
            "2023-04-21 16:58:22,037 epoch 15 - iter 18/24 - loss 0.22632848 - time (sec): 19.04 - samples/sec: 3789.67 - lr: 0.000005\n",
            "2023-04-21 16:58:24,253 epoch 15 - iter 20/24 - loss 0.22550448 - time (sec): 21.26 - samples/sec: 3809.62 - lr: 0.000005\n",
            "2023-04-21 16:58:26,624 epoch 15 - iter 22/24 - loss 0.23488626 - time (sec): 23.63 - samples/sec: 3736.92 - lr: 0.000005\n",
            "2023-04-21 16:58:28,400 epoch 15 - iter 24/24 - loss 0.23438729 - time (sec): 25.40 - samples/sec: 3730.53 - lr: 0.000005\n",
            "2023-04-21 16:58:28,403 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:58:28,405 EPOCH 15 done: loss 0.2344 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:58:38,578 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:58:41,604 TRAIN : loss 0.20501959323883057 - f1-score (weighted avg)  0.7347\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.17it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:58:43,137 Evaluating as a multi-label problem: False\n",
            "2023-04-21 16:58:43,204 DEV : loss 0.26915591955184937 - f1-score (weighted avg)  0.6612\n",
            "2023-04-21 16:58:43,223 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 16:58:43,227 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:58:46,369 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:58:48,806 epoch 16 - iter 2/24 - loss 0.31755384 - time (sec): 2.43 - samples/sec: 3591.94 - lr: 0.000005\n",
            "2023-04-21 16:58:50,890 epoch 16 - iter 4/24 - loss 0.27102426 - time (sec): 4.52 - samples/sec: 3499.85 - lr: 0.000005\n",
            "2023-04-21 16:58:53,137 epoch 16 - iter 6/24 - loss 0.26015348 - time (sec): 6.77 - samples/sec: 3558.81 - lr: 0.000005\n",
            "2023-04-21 16:58:55,240 epoch 16 - iter 8/24 - loss 0.25096713 - time (sec): 8.87 - samples/sec: 3526.98 - lr: 0.000005\n",
            "2023-04-21 16:58:57,428 epoch 16 - iter 10/24 - loss 0.24823290 - time (sec): 11.06 - samples/sec: 3495.49 - lr: 0.000005\n",
            "2023-04-21 16:58:59,565 epoch 16 - iter 12/24 - loss 0.24244272 - time (sec): 13.19 - samples/sec: 3526.14 - lr: 0.000005\n",
            "2023-04-21 16:59:01,702 epoch 16 - iter 14/24 - loss 0.23338745 - time (sec): 15.33 - samples/sec: 3577.00 - lr: 0.000005\n",
            "2023-04-21 16:59:03,889 epoch 16 - iter 16/24 - loss 0.23394556 - time (sec): 17.52 - samples/sec: 3595.14 - lr: 0.000005\n",
            "2023-04-21 16:59:06,034 epoch 16 - iter 18/24 - loss 0.23138522 - time (sec): 19.66 - samples/sec: 3640.32 - lr: 0.000005\n",
            "2023-04-21 16:59:08,211 epoch 16 - iter 20/24 - loss 0.23042277 - time (sec): 21.84 - samples/sec: 3699.76 - lr: 0.000005\n",
            "2023-04-21 16:59:10,257 epoch 16 - iter 22/24 - loss 0.22974602 - time (sec): 23.89 - samples/sec: 3678.27 - lr: 0.000005\n",
            "2023-04-21 16:59:12,063 epoch 16 - iter 24/24 - loss 0.22754254 - time (sec): 25.69 - samples/sec: 3688.84 - lr: 0.000005\n",
            "2023-04-21 16:59:12,066 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:59:12,068 EPOCH 16 done: loss 0.2275 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:59:22,237 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:59:25,285 TRAIN : loss 0.20301224291324615 - f1-score (weighted avg)  0.7353\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.17it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:59:26,817 Evaluating as a multi-label problem: False\n",
            "2023-04-21 16:59:26,884 DEV : loss 0.2694757282733917 - f1-score (weighted avg)  0.6543\n",
            "2023-04-21 16:59:26,903 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 16:59:26,907 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 16:59:29,041 epoch 17 - iter 2/24 - loss 0.20602632 - time (sec): 2.13 - samples/sec: 3799.70 - lr: 0.000005\n",
            "2023-04-21 16:59:31,097 epoch 17 - iter 4/24 - loss 0.20014074 - time (sec): 4.19 - samples/sec: 3743.23 - lr: 0.000005\n",
            "2023-04-21 16:59:33,142 epoch 17 - iter 6/24 - loss 0.20618256 - time (sec): 6.23 - samples/sec: 3746.89 - lr: 0.000005\n",
            "2023-04-21 16:59:35,271 epoch 17 - iter 8/24 - loss 0.20640713 - time (sec): 8.36 - samples/sec: 3724.35 - lr: 0.000005\n",
            "2023-04-21 16:59:37,636 epoch 17 - iter 10/24 - loss 0.22280518 - time (sec): 10.73 - samples/sec: 3710.74 - lr: 0.000005\n",
            "2023-04-21 16:59:39,725 epoch 17 - iter 12/24 - loss 0.22248854 - time (sec): 12.82 - samples/sec: 3771.77 - lr: 0.000005\n",
            "2023-04-21 16:59:41,819 epoch 17 - iter 14/24 - loss 0.21798882 - time (sec): 14.91 - samples/sec: 3745.68 - lr: 0.000005\n",
            "2023-04-21 16:59:43,924 epoch 17 - iter 16/24 - loss 0.21785458 - time (sec): 17.02 - samples/sec: 3761.10 - lr: 0.000005\n",
            "2023-04-21 16:59:46,036 epoch 17 - iter 18/24 - loss 0.21303465 - time (sec): 19.13 - samples/sec: 3744.61 - lr: 0.000005\n",
            "2023-04-21 16:59:48,088 epoch 17 - iter 20/24 - loss 0.21248292 - time (sec): 21.18 - samples/sec: 3727.65 - lr: 0.000005\n",
            "2023-04-21 16:59:50,138 epoch 17 - iter 22/24 - loss 0.21025605 - time (sec): 23.23 - samples/sec: 3745.02 - lr: 0.000005\n",
            "2023-04-21 16:59:52,168 epoch 17 - iter 24/24 - loss 0.22080555 - time (sec): 25.26 - samples/sec: 3751.82 - lr: 0.000005\n",
            "2023-04-21 16:59:52,172 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 16:59:52,173 EPOCH 17 done: loss 0.2208 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:00:02,370 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:00:05,406 TRAIN : loss 0.19314999878406525 - f1-score (weighted avg)  0.7456\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.18it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:00:06,933 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:00:07,001 DEV : loss 0.26517966389656067 - f1-score (weighted avg)  0.6592\n",
            "2023-04-21 17:00:07,021 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:00:07,025 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:00:10,170 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:00:12,388 epoch 18 - iter 2/24 - loss 0.19680546 - time (sec): 2.22 - samples/sec: 3336.62 - lr: 0.000005\n",
            "2023-04-21 17:00:14,470 epoch 18 - iter 4/24 - loss 0.20684551 - time (sec): 4.30 - samples/sec: 3319.01 - lr: 0.000005\n",
            "2023-04-21 17:00:16,756 epoch 18 - iter 6/24 - loss 0.22623532 - time (sec): 6.58 - samples/sec: 3539.64 - lr: 0.000005\n",
            "2023-04-21 17:00:18,860 epoch 18 - iter 8/24 - loss 0.22210853 - time (sec): 8.69 - samples/sec: 3565.33 - lr: 0.000005\n",
            "2023-04-21 17:00:21,036 epoch 18 - iter 10/24 - loss 0.20876441 - time (sec): 10.86 - samples/sec: 3620.93 - lr: 0.000005\n",
            "2023-04-21 17:00:23,182 epoch 18 - iter 12/24 - loss 0.21026769 - time (sec): 13.01 - samples/sec: 3681.32 - lr: 0.000005\n",
            "2023-04-21 17:00:25,296 epoch 18 - iter 14/24 - loss 0.20821773 - time (sec): 15.12 - samples/sec: 3704.01 - lr: 0.000005\n",
            "2023-04-21 17:00:27,360 epoch 18 - iter 16/24 - loss 0.20615518 - time (sec): 17.19 - samples/sec: 3710.88 - lr: 0.000005\n",
            "2023-04-21 17:00:29,434 epoch 18 - iter 18/24 - loss 0.20612054 - time (sec): 19.26 - samples/sec: 3710.42 - lr: 0.000005\n",
            "2023-04-21 17:00:31,562 epoch 18 - iter 20/24 - loss 0.20590923 - time (sec): 21.39 - samples/sec: 3706.53 - lr: 0.000005\n",
            "2023-04-21 17:00:33,931 epoch 18 - iter 22/24 - loss 0.21446853 - time (sec): 23.76 - samples/sec: 3701.90 - lr: 0.000005\n",
            "2023-04-21 17:00:35,692 epoch 18 - iter 24/24 - loss 0.21359055 - time (sec): 25.52 - samples/sec: 3713.55 - lr: 0.000005\n",
            "2023-04-21 17:00:35,695 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:00:35,697 EPOCH 18 done: loss 0.2136 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:00:45,891 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:00:48,952 TRAIN : loss 0.18573273718357086 - f1-score (weighted avg)  0.7525\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.20it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:00:50,466 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:00:50,535 DEV : loss 0.2619098126888275 - f1-score (weighted avg)  0.6663\n",
            "2023-04-21 17:00:50,554 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:00:50,558 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:00:53,731 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:00:55,827 epoch 19 - iter 2/24 - loss 0.19745182 - time (sec): 2.09 - samples/sec: 3552.04 - lr: 0.000005\n",
            "2023-04-21 17:00:58,204 epoch 19 - iter 4/24 - loss 0.24051918 - time (sec): 4.47 - samples/sec: 3753.11 - lr: 0.000005\n",
            "2023-04-21 17:01:00,290 epoch 19 - iter 6/24 - loss 0.23079459 - time (sec): 6.56 - samples/sec: 3767.53 - lr: 0.000005\n",
            "2023-04-21 17:01:02,385 epoch 19 - iter 8/24 - loss 0.22228338 - time (sec): 8.65 - samples/sec: 3779.99 - lr: 0.000005\n",
            "2023-04-21 17:01:04,441 epoch 19 - iter 10/24 - loss 0.21261749 - time (sec): 10.71 - samples/sec: 3779.59 - lr: 0.000005\n",
            "2023-04-21 17:01:06,642 epoch 19 - iter 12/24 - loss 0.21054863 - time (sec): 12.91 - samples/sec: 3766.15 - lr: 0.000005\n",
            "2023-04-21 17:01:08,877 epoch 19 - iter 14/24 - loss 0.21147869 - time (sec): 15.14 - samples/sec: 3767.37 - lr: 0.000005\n",
            "2023-04-21 17:01:10,970 epoch 19 - iter 16/24 - loss 0.21265341 - time (sec): 17.24 - samples/sec: 3767.91 - lr: 0.000005\n",
            "2023-04-21 17:01:13,081 epoch 19 - iter 18/24 - loss 0.21127299 - time (sec): 19.35 - samples/sec: 3717.40 - lr: 0.000005\n",
            "2023-04-21 17:01:15,153 epoch 19 - iter 20/24 - loss 0.21049503 - time (sec): 21.42 - samples/sec: 3714.93 - lr: 0.000005\n",
            "2023-04-21 17:01:17,440 epoch 19 - iter 22/24 - loss 0.21216297 - time (sec): 23.71 - samples/sec: 3732.04 - lr: 0.000005\n",
            "2023-04-21 17:01:19,181 epoch 19 - iter 24/24 - loss 0.20957427 - time (sec): 25.45 - samples/sec: 3723.93 - lr: 0.000005\n",
            "2023-04-21 17:01:19,184 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:01:19,186 EPOCH 19 done: loss 0.2096 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:01:29,412 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:01:32,460 TRAIN : loss 0.1801718920469284 - f1-score (weighted avg)  0.7579\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.19it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:01:33,986 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:01:34,053 DEV : loss 0.26347246766090393 - f1-score (weighted avg)  0.6671\n",
            "2023-04-21 17:01:34,074 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:01:34,078 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:01:36,116 epoch 20 - iter 2/24 - loss 0.19047954 - time (sec): 2.04 - samples/sec: 3117.40 - lr: 0.000005\n",
            "2023-04-21 17:01:38,486 epoch 20 - iter 4/24 - loss 0.23370806 - time (sec): 4.41 - samples/sec: 3372.64 - lr: 0.000005\n",
            "2023-04-21 17:01:40,629 epoch 20 - iter 6/24 - loss 0.21365201 - time (sec): 6.55 - samples/sec: 3646.61 - lr: 0.000005\n",
            "2023-04-21 17:01:42,687 epoch 20 - iter 8/24 - loss 0.20623186 - time (sec): 8.61 - samples/sec: 3624.94 - lr: 0.000005\n",
            "2023-04-21 17:01:44,848 epoch 20 - iter 10/24 - loss 0.20762501 - time (sec): 10.77 - samples/sec: 3566.82 - lr: 0.000005\n",
            "2023-04-21 17:01:46,973 epoch 20 - iter 12/24 - loss 0.20141281 - time (sec): 12.89 - samples/sec: 3584.28 - lr: 0.000005\n",
            "2023-04-21 17:01:49,045 epoch 20 - iter 14/24 - loss 0.19859854 - time (sec): 14.97 - samples/sec: 3626.36 - lr: 0.000005\n",
            "2023-04-21 17:01:51,304 epoch 20 - iter 16/24 - loss 0.20103169 - time (sec): 17.22 - samples/sec: 3670.44 - lr: 0.000005\n",
            "2023-04-21 17:01:53,392 epoch 20 - iter 18/24 - loss 0.20137530 - time (sec): 19.31 - samples/sec: 3692.43 - lr: 0.000005\n",
            "2023-04-21 17:01:55,402 epoch 20 - iter 20/24 - loss 0.20229063 - time (sec): 21.32 - samples/sec: 3687.12 - lr: 0.000005\n",
            "2023-04-21 17:01:57,480 epoch 20 - iter 22/24 - loss 0.20146098 - time (sec): 23.40 - samples/sec: 3720.93 - lr: 0.000005\n",
            "2023-04-21 17:01:59,333 epoch 20 - iter 24/24 - loss 0.20240585 - time (sec): 25.25 - samples/sec: 3752.83 - lr: 0.000005\n",
            "2023-04-21 17:01:59,335 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:01:59,337 EPOCH 20 done: loss 0.2024 - lr 0.000005\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:09<00:00,  2.40it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:02:09,467 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:02:12,542 TRAIN : loss 0.17547939717769623 - f1-score (weighted avg)  0.7653\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.21it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:02:14,051 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:02:14,117 DEV : loss 0.26229968667030334 - f1-score (weighted avg)  0.6742\n",
            "2023-04-21 17:02:14,135 Epoch    20: reducing learning rate of group 0 to 4.0000e-06.\n",
            "2023-04-21 17:02:14,138 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:02:14,141 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:02:16,240 epoch 21 - iter 2/24 - loss 0.20427328 - time (sec): 2.10 - samples/sec: 3632.41 - lr: 0.000004\n",
            "2023-04-21 17:02:18,263 epoch 21 - iter 4/24 - loss 0.20645166 - time (sec): 4.12 - samples/sec: 3520.36 - lr: 0.000004\n",
            "2023-04-21 17:02:20,422 epoch 21 - iter 6/24 - loss 0.18495579 - time (sec): 6.28 - samples/sec: 3699.52 - lr: 0.000004\n",
            "2023-04-21 17:02:22,807 epoch 21 - iter 8/24 - loss 0.20726781 - time (sec): 8.67 - samples/sec: 3746.10 - lr: 0.000004\n",
            "2023-04-21 17:02:24,837 epoch 21 - iter 10/24 - loss 0.20193287 - time (sec): 10.69 - samples/sec: 3619.77 - lr: 0.000004\n",
            "2023-04-21 17:02:26,946 epoch 21 - iter 12/24 - loss 0.20092579 - time (sec): 12.80 - samples/sec: 3656.87 - lr: 0.000004\n",
            "2023-04-21 17:02:29,031 epoch 21 - iter 14/24 - loss 0.19984787 - time (sec): 14.89 - samples/sec: 3734.61 - lr: 0.000004\n",
            "2023-04-21 17:02:31,261 epoch 21 - iter 16/24 - loss 0.20199584 - time (sec): 17.12 - samples/sec: 3722.77 - lr: 0.000004\n",
            "2023-04-21 17:02:33,303 epoch 21 - iter 18/24 - loss 0.19948504 - time (sec): 19.16 - samples/sec: 3723.53 - lr: 0.000004\n",
            "2023-04-21 17:02:35,432 epoch 21 - iter 20/24 - loss 0.19762717 - time (sec): 21.29 - samples/sec: 3727.44 - lr: 0.000004\n",
            "2023-04-21 17:02:37,599 epoch 21 - iter 22/24 - loss 0.19716847 - time (sec): 23.46 - samples/sec: 3748.82 - lr: 0.000004\n",
            "2023-04-21 17:02:39,364 epoch 21 - iter 24/24 - loss 0.19729602 - time (sec): 25.22 - samples/sec: 3757.52 - lr: 0.000004\n",
            "2023-04-21 17:02:39,367 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:02:39,368 EPOCH 21 done: loss 0.1973 - lr 0.000004\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.40it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:02:49,512 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:02:51,343 TRAIN : loss 0.17473894357681274 - f1-score (weighted avg)  0.7659\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:02<00:00,  1.24it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:02:53,908 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:02:53,975 DEV : loss 0.2632267475128174 - f1-score (weighted avg)  0.6727\n",
            "2023-04-21 17:02:53,994 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:02:53,998 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:02:56,120 epoch 22 - iter 2/24 - loss 0.19543740 - time (sec): 2.12 - samples/sec: 4146.09 - lr: 0.000004\n",
            "2023-04-21 17:02:58,195 epoch 22 - iter 4/24 - loss 0.18890239 - time (sec): 4.20 - samples/sec: 3794.80 - lr: 0.000004\n",
            "2023-04-21 17:03:00,469 epoch 22 - iter 6/24 - loss 0.18941090 - time (sec): 6.47 - samples/sec: 3793.99 - lr: 0.000004\n",
            "2023-04-21 17:03:02,512 epoch 22 - iter 8/24 - loss 0.18707066 - time (sec): 8.51 - samples/sec: 3762.41 - lr: 0.000004\n",
            "2023-04-21 17:03:04,576 epoch 22 - iter 10/24 - loss 0.18495998 - time (sec): 10.58 - samples/sec: 3765.72 - lr: 0.000004\n",
            "2023-04-21 17:03:06,950 epoch 22 - iter 12/24 - loss 0.20386504 - time (sec): 12.95 - samples/sec: 3748.24 - lr: 0.000004\n",
            "2023-04-21 17:03:09,008 epoch 22 - iter 14/24 - loss 0.19944297 - time (sec): 15.01 - samples/sec: 3763.64 - lr: 0.000004\n",
            "2023-04-21 17:03:11,073 epoch 22 - iter 16/24 - loss 0.19761794 - time (sec): 17.07 - samples/sec: 3764.51 - lr: 0.000004\n",
            "2023-04-21 17:03:13,175 epoch 22 - iter 18/24 - loss 0.19817156 - time (sec): 19.18 - samples/sec: 3825.76 - lr: 0.000004\n",
            "2023-04-21 17:03:15,370 epoch 22 - iter 20/24 - loss 0.19752937 - time (sec): 21.37 - samples/sec: 3825.04 - lr: 0.000004\n",
            "2023-04-21 17:03:17,471 epoch 22 - iter 22/24 - loss 0.19397439 - time (sec): 23.47 - samples/sec: 3816.67 - lr: 0.000004\n",
            "2023-04-21 17:03:19,229 epoch 22 - iter 24/24 - loss 0.19301621 - time (sec): 25.23 - samples/sec: 3756.28 - lr: 0.000004\n",
            "2023-04-21 17:03:19,232 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:03:19,233 EPOCH 22 done: loss 0.1930 - lr 0.000004\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:03:29,441 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:03:32,462 TRAIN : loss 0.16334496438503265 - f1-score (weighted avg)  0.7812\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.21it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:03:33,975 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:03:34,051 DEV : loss 0.2634033262729645 - f1-score (weighted avg)  0.6796\n",
            "2023-04-21 17:03:34,071 Epoch    22: reducing learning rate of group 0 to 3.2000e-06.\n",
            "2023-04-21 17:03:34,073 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:03:34,076 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:03:36,138 epoch 23 - iter 2/24 - loss 0.20289997 - time (sec): 2.06 - samples/sec: 3531.58 - lr: 0.000003\n",
            "2023-04-21 17:03:38,496 epoch 23 - iter 4/24 - loss 0.23286946 - time (sec): 4.42 - samples/sec: 3663.11 - lr: 0.000003\n",
            "2023-04-21 17:03:40,580 epoch 23 - iter 6/24 - loss 0.21870707 - time (sec): 6.50 - samples/sec: 3707.81 - lr: 0.000003\n",
            "2023-04-21 17:03:42,632 epoch 23 - iter 8/24 - loss 0.21009698 - time (sec): 8.55 - samples/sec: 3680.42 - lr: 0.000003\n",
            "2023-04-21 17:03:44,850 epoch 23 - iter 10/24 - loss 0.20609717 - time (sec): 10.77 - samples/sec: 3681.00 - lr: 0.000003\n",
            "2023-04-21 17:03:46,978 epoch 23 - iter 12/24 - loss 0.19807973 - time (sec): 12.90 - samples/sec: 3668.42 - lr: 0.000003\n",
            "2023-04-21 17:03:49,091 epoch 23 - iter 14/24 - loss 0.19279137 - time (sec): 15.01 - samples/sec: 3725.80 - lr: 0.000003\n",
            "2023-04-21 17:03:51,142 epoch 23 - iter 16/24 - loss 0.18804625 - time (sec): 17.06 - samples/sec: 3720.09 - lr: 0.000003\n",
            "2023-04-21 17:03:53,387 epoch 23 - iter 18/24 - loss 0.19421988 - time (sec): 19.31 - samples/sec: 3742.79 - lr: 0.000003\n",
            "2023-04-21 17:03:55,551 epoch 23 - iter 20/24 - loss 0.19261010 - time (sec): 21.47 - samples/sec: 3726.56 - lr: 0.000003\n",
            "2023-04-21 17:03:57,641 epoch 23 - iter 22/24 - loss 0.18935297 - time (sec): 23.56 - samples/sec: 3775.80 - lr: 0.000003\n",
            "2023-04-21 17:03:59,377 epoch 23 - iter 24/24 - loss 0.18807098 - time (sec): 25.30 - samples/sec: 3745.92 - lr: 0.000003\n",
            "2023-04-21 17:03:59,380 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:03:59,382 EPOCH 23 done: loss 0.1881 - lr 0.000003\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:04:09,587 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:04:12,596 TRAIN : loss 0.16000136733055115 - f1-score (weighted avg)  0.7846\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.16it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:04:14,135 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:04:14,202 DEV : loss 0.2615758180618286 - f1-score (weighted avg)  0.6823\n",
            "2023-04-21 17:04:14,221 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:04:14,225 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:04:17,513 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:04:19,664 epoch 24 - iter 2/24 - loss 0.16829390 - time (sec): 2.15 - samples/sec: 3842.55 - lr: 0.000003\n",
            "2023-04-21 17:04:21,814 epoch 24 - iter 4/24 - loss 0.15000214 - time (sec): 4.30 - samples/sec: 3810.00 - lr: 0.000003\n",
            "2023-04-21 17:04:23,863 epoch 24 - iter 6/24 - loss 0.16643233 - time (sec): 6.35 - samples/sec: 3596.14 - lr: 0.000003\n",
            "2023-04-21 17:04:25,982 epoch 24 - iter 8/24 - loss 0.16737616 - time (sec): 8.47 - samples/sec: 3536.02 - lr: 0.000003\n",
            "2023-04-21 17:04:28,073 epoch 24 - iter 10/24 - loss 0.16969946 - time (sec): 10.56 - samples/sec: 3602.79 - lr: 0.000003\n",
            "2023-04-21 17:04:30,105 epoch 24 - iter 12/24 - loss 0.17151780 - time (sec): 12.59 - samples/sec: 3583.27 - lr: 0.000003\n",
            "2023-04-21 17:04:32,178 epoch 24 - iter 14/24 - loss 0.16952194 - time (sec): 14.66 - samples/sec: 3615.32 - lr: 0.000003\n",
            "2023-04-21 17:04:34,345 epoch 24 - iter 16/24 - loss 0.17274404 - time (sec): 16.83 - samples/sec: 3679.72 - lr: 0.000003\n",
            "2023-04-21 17:04:36,408 epoch 24 - iter 18/24 - loss 0.17099179 - time (sec): 18.89 - samples/sec: 3729.25 - lr: 0.000003\n",
            "2023-04-21 17:04:38,517 epoch 24 - iter 20/24 - loss 0.17268626 - time (sec): 21.00 - samples/sec: 3758.74 - lr: 0.000003\n",
            "2023-04-21 17:04:40,772 epoch 24 - iter 22/24 - loss 0.17793928 - time (sec): 23.26 - samples/sec: 3712.66 - lr: 0.000003\n",
            "2023-04-21 17:04:42,901 epoch 24 - iter 24/24 - loss 0.18580435 - time (sec): 25.39 - samples/sec: 3733.09 - lr: 0.000003\n",
            "2023-04-21 17:04:42,905 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:04:42,906 EPOCH 24 done: loss 0.1858 - lr 0.000003\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:04:53,067 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:04:56,107 TRAIN : loss 0.1576726734638214 - f1-score (weighted avg)  0.7894\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.19it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:04:57,633 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:04:57,702 DEV : loss 0.2632157504558563 - f1-score (weighted avg)  0.6867\n",
            "2023-04-21 17:04:57,722 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:04:57,727 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:04:59,904 epoch 25 - iter 2/24 - loss 0.16040134 - time (sec): 2.18 - samples/sec: 4228.20 - lr: 0.000003\n",
            "2023-04-21 17:05:02,021 epoch 25 - iter 4/24 - loss 0.16801733 - time (sec): 4.29 - samples/sec: 3884.22 - lr: 0.000003\n",
            "2023-04-21 17:05:04,248 epoch 25 - iter 6/24 - loss 0.18570929 - time (sec): 6.52 - samples/sec: 3753.69 - lr: 0.000003\n",
            "2023-04-21 17:05:06,311 epoch 25 - iter 8/24 - loss 0.18111643 - time (sec): 8.58 - samples/sec: 3657.09 - lr: 0.000003\n",
            "2023-04-21 17:05:08,419 epoch 25 - iter 10/24 - loss 0.17825838 - time (sec): 10.69 - samples/sec: 3812.17 - lr: 0.000003\n",
            "2023-04-21 17:05:10,479 epoch 25 - iter 12/24 - loss 0.17477048 - time (sec): 12.75 - samples/sec: 3732.21 - lr: 0.000003\n",
            "2023-04-21 17:05:12,515 epoch 25 - iter 14/24 - loss 0.17391512 - time (sec): 14.79 - samples/sec: 3745.01 - lr: 0.000003\n",
            "2023-04-21 17:05:14,626 epoch 25 - iter 16/24 - loss 0.17721180 - time (sec): 16.90 - samples/sec: 3801.13 - lr: 0.000003\n",
            "2023-04-21 17:05:16,717 epoch 25 - iter 18/24 - loss 0.17609355 - time (sec): 18.99 - samples/sec: 3806.53 - lr: 0.000003\n",
            "2023-04-21 17:05:19,128 epoch 25 - iter 20/24 - loss 0.18581066 - time (sec): 21.40 - samples/sec: 3770.50 - lr: 0.000003\n",
            "2023-04-21 17:05:21,180 epoch 25 - iter 22/24 - loss 0.18356571 - time (sec): 23.45 - samples/sec: 3743.64 - lr: 0.000003\n",
            "2023-04-21 17:05:22,951 epoch 25 - iter 24/24 - loss 0.18244653 - time (sec): 25.22 - samples/sec: 3757.34 - lr: 0.000003\n",
            "2023-04-21 17:05:22,954 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:05:22,955 EPOCH 25 done: loss 0.1824 - lr 0.000003\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:05:33,154 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:05:36,193 TRAIN : loss 0.1567198783159256 - f1-score (weighted avg)  0.7896\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.15it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:05:37,742 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:05:37,809 DEV : loss 0.2606967091560364 - f1-score (weighted avg)  0.6905\n",
            "2023-04-21 17:05:37,829 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:05:37,833 saving best model\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:05:41,104 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:05:43,276 epoch 26 - iter 2/24 - loss 0.17370874 - time (sec): 2.17 - samples/sec: 3603.49 - lr: 0.000003\n",
            "2023-04-21 17:05:45,316 epoch 26 - iter 4/24 - loss 0.17787060 - time (sec): 4.21 - samples/sec: 3709.70 - lr: 0.000003\n",
            "2023-04-21 17:05:47,431 epoch 26 - iter 6/24 - loss 0.17135708 - time (sec): 6.32 - samples/sec: 3747.00 - lr: 0.000003\n",
            "2023-04-21 17:05:49,528 epoch 26 - iter 8/24 - loss 0.18153526 - time (sec): 8.42 - samples/sec: 3601.58 - lr: 0.000003\n",
            "2023-04-21 17:05:51,610 epoch 26 - iter 10/24 - loss 0.18225668 - time (sec): 10.50 - samples/sec: 3751.44 - lr: 0.000003\n",
            "2023-04-21 17:05:53,745 epoch 26 - iter 12/24 - loss 0.18127888 - time (sec): 12.64 - samples/sec: 3732.72 - lr: 0.000003\n",
            "2023-04-21 17:05:56,137 epoch 26 - iter 14/24 - loss 0.19319640 - time (sec): 15.03 - samples/sec: 3691.72 - lr: 0.000003\n",
            "2023-04-21 17:05:58,260 epoch 26 - iter 16/24 - loss 0.18659234 - time (sec): 17.15 - samples/sec: 3696.65 - lr: 0.000003\n",
            "2023-04-21 17:06:00,349 epoch 26 - iter 18/24 - loss 0.18483262 - time (sec): 19.24 - samples/sec: 3675.05 - lr: 0.000003\n",
            "2023-04-21 17:06:02,430 epoch 26 - iter 20/24 - loss 0.18326674 - time (sec): 21.32 - samples/sec: 3661.02 - lr: 0.000003\n",
            "2023-04-21 17:06:04,571 epoch 26 - iter 22/24 - loss 0.17936439 - time (sec): 23.46 - samples/sec: 3709.59 - lr: 0.000003\n",
            "2023-04-21 17:06:06,553 epoch 26 - iter 24/24 - loss 0.18013396 - time (sec): 25.45 - samples/sec: 3724.25 - lr: 0.000003\n",
            "2023-04-21 17:06:06,557 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:06:06,558 EPOCH 26 done: loss 0.1801 - lr 0.000003\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.37it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:06:16,827 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:06:18,686 TRAIN : loss 0.1502903401851654 - f1-score (weighted avg)  0.7982\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:02<00:00,  1.24it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:06:21,261 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:06:21,330 DEV : loss 0.26139914989471436 - f1-score (weighted avg)  0.6925\n",
            "2023-04-21 17:06:21,350 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:06:21,354 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:06:23,436 epoch 27 - iter 2/24 - loss 0.18516318 - time (sec): 2.08 - samples/sec: 3886.37 - lr: 0.000003\n",
            "2023-04-21 17:06:25,519 epoch 27 - iter 4/24 - loss 0.18955293 - time (sec): 4.16 - samples/sec: 4154.61 - lr: 0.000003\n",
            "2023-04-21 17:06:27,671 epoch 27 - iter 6/24 - loss 0.17699956 - time (sec): 6.32 - samples/sec: 4008.48 - lr: 0.000003\n",
            "2023-04-21 17:06:30,039 epoch 27 - iter 8/24 - loss 0.19754688 - time (sec): 8.68 - samples/sec: 3886.99 - lr: 0.000003\n",
            "2023-04-21 17:06:32,075 epoch 27 - iter 10/24 - loss 0.18636318 - time (sec): 10.72 - samples/sec: 3796.62 - lr: 0.000003\n",
            "2023-04-21 17:06:34,105 epoch 27 - iter 12/24 - loss 0.18106380 - time (sec): 12.75 - samples/sec: 3787.66 - lr: 0.000003\n",
            "2023-04-21 17:06:36,393 epoch 27 - iter 14/24 - loss 0.18250396 - time (sec): 15.04 - samples/sec: 3738.16 - lr: 0.000003\n",
            "2023-04-21 17:06:38,445 epoch 27 - iter 16/24 - loss 0.18122761 - time (sec): 17.09 - samples/sec: 3765.10 - lr: 0.000003\n",
            "2023-04-21 17:06:40,581 epoch 27 - iter 18/24 - loss 0.18120969 - time (sec): 19.23 - samples/sec: 3774.25 - lr: 0.000003\n",
            "2023-04-21 17:06:42,707 epoch 27 - iter 20/24 - loss 0.17955955 - time (sec): 21.35 - samples/sec: 3762.99 - lr: 0.000003\n",
            "2023-04-21 17:06:44,823 epoch 27 - iter 22/24 - loss 0.17774185 - time (sec): 23.47 - samples/sec: 3744.19 - lr: 0.000003\n",
            "2023-04-21 17:06:46,639 epoch 27 - iter 24/24 - loss 0.17653239 - time (sec): 25.28 - samples/sec: 3748.29 - lr: 0.000003\n",
            "2023-04-21 17:06:46,643 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:06:46,644 EPOCH 27 done: loss 0.1765 - lr 0.000003\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:06:56,860 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:06:59,843 TRAIN : loss 0.15049158036708832 - f1-score (weighted avg)  0.7994\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.20it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:07:01,355 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:07:01,423 DEV : loss 0.2608695328235626 - f1-score (weighted avg)  0.6959\n",
            "2023-04-21 17:07:01,442 Epoch    27: reducing learning rate of group 0 to 2.5600e-06.\n",
            "2023-04-21 17:07:01,444 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:07:01,447 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:07:03,499 epoch 28 - iter 2/24 - loss 0.15636337 - time (sec): 2.05 - samples/sec: 3820.73 - lr: 0.000003\n",
            "2023-04-21 17:07:05,566 epoch 28 - iter 4/24 - loss 0.15187672 - time (sec): 4.12 - samples/sec: 3864.67 - lr: 0.000003\n",
            "2023-04-21 17:07:07,780 epoch 28 - iter 6/24 - loss 0.14953964 - time (sec): 6.33 - samples/sec: 3859.95 - lr: 0.000003\n",
            "2023-04-21 17:07:09,896 epoch 28 - iter 8/24 - loss 0.15084780 - time (sec): 8.45 - samples/sec: 3853.49 - lr: 0.000003\n",
            "2023-04-21 17:07:11,922 epoch 28 - iter 10/24 - loss 0.15570442 - time (sec): 10.47 - samples/sec: 3786.16 - lr: 0.000003\n",
            "2023-04-21 17:07:13,963 epoch 28 - iter 12/24 - loss 0.16006775 - time (sec): 12.51 - samples/sec: 3782.93 - lr: 0.000003\n",
            "2023-04-21 17:07:16,352 epoch 28 - iter 14/24 - loss 0.17750640 - time (sec): 14.90 - samples/sec: 3749.40 - lr: 0.000003\n",
            "2023-04-21 17:07:18,653 epoch 28 - iter 16/24 - loss 0.17868216 - time (sec): 17.20 - samples/sec: 3758.62 - lr: 0.000003\n",
            "2023-04-21 17:07:20,697 epoch 28 - iter 18/24 - loss 0.17437535 - time (sec): 19.25 - samples/sec: 3761.04 - lr: 0.000003\n",
            "2023-04-21 17:07:22,861 epoch 28 - iter 20/24 - loss 0.17312624 - time (sec): 21.41 - samples/sec: 3753.14 - lr: 0.000003\n",
            "2023-04-21 17:07:24,926 epoch 28 - iter 22/24 - loss 0.17160131 - time (sec): 23.48 - samples/sec: 3753.86 - lr: 0.000003\n",
            "2023-04-21 17:07:26,657 epoch 28 - iter 24/24 - loss 0.17316761 - time (sec): 25.21 - samples/sec: 3759.52 - lr: 0.000003\n",
            "2023-04-21 17:07:26,659 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:07:26,661 EPOCH 28 done: loss 0.1732 - lr 0.000003\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:07:36,879 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:07:39,883 TRAIN : loss 0.14514394104480743 - f1-score (weighted avg)  0.8064\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.19it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:07:41,400 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:07:41,466 DEV : loss 0.2633368968963623 - f1-score (weighted avg)  0.6985\n",
            "2023-04-21 17:07:41,485 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:07:41,488 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:07:43,570 epoch 29 - iter 2/24 - loss 0.16279533 - time (sec): 2.08 - samples/sec: 3792.98 - lr: 0.000003\n",
            "2023-04-21 17:07:45,647 epoch 29 - iter 4/24 - loss 0.15577320 - time (sec): 4.16 - samples/sec: 3814.13 - lr: 0.000003\n",
            "2023-04-21 17:07:47,741 epoch 29 - iter 6/24 - loss 0.16012240 - time (sec): 6.25 - samples/sec: 3710.27 - lr: 0.000003\n",
            "2023-04-21 17:07:50,113 epoch 29 - iter 8/24 - loss 0.18331202 - time (sec): 8.62 - samples/sec: 3716.48 - lr: 0.000003\n",
            "2023-04-21 17:07:52,284 epoch 29 - iter 10/24 - loss 0.17457058 - time (sec): 10.79 - samples/sec: 3798.98 - lr: 0.000003\n",
            "2023-04-21 17:07:54,366 epoch 29 - iter 12/24 - loss 0.16910636 - time (sec): 12.88 - samples/sec: 3812.36 - lr: 0.000003\n",
            "2023-04-21 17:07:56,632 epoch 29 - iter 14/24 - loss 0.17340046 - time (sec): 15.14 - samples/sec: 3787.21 - lr: 0.000003\n",
            "2023-04-21 17:07:58,752 epoch 29 - iter 16/24 - loss 0.16995225 - time (sec): 17.26 - samples/sec: 3780.04 - lr: 0.000003\n",
            "2023-04-21 17:08:00,842 epoch 29 - iter 18/24 - loss 0.17132495 - time (sec): 19.35 - samples/sec: 3770.90 - lr: 0.000003\n",
            "2023-04-21 17:08:02,943 epoch 29 - iter 20/24 - loss 0.17112629 - time (sec): 21.45 - samples/sec: 3752.03 - lr: 0.000003\n",
            "2023-04-21 17:08:04,999 epoch 29 - iter 22/24 - loss 0.17035135 - time (sec): 23.51 - samples/sec: 3791.59 - lr: 0.000003\n",
            "2023-04-21 17:08:06,795 epoch 29 - iter 24/24 - loss 0.16969555 - time (sec): 25.30 - samples/sec: 3745.13 - lr: 0.000003\n",
            "2023-04-21 17:08:06,798 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:08:06,799 EPOCH 29 done: loss 0.1697 - lr 0.000003\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:08:16,958 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:08:19,966 TRAIN : loss 0.142629474401474 - f1-score (weighted avg)  0.8091\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.19it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:08:21,483 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:08:21,549 DEV : loss 0.26206472516059875 - f1-score (weighted avg)  0.701\n",
            "2023-04-21 17:08:21,569 Epoch    29: reducing learning rate of group 0 to 2.0480e-06.\n",
            "2023-04-21 17:08:21,571 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:08:21,574 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:08:23,601 epoch 30 - iter 2/24 - loss 0.15789152 - time (sec): 2.02 - samples/sec: 3880.94 - lr: 0.000002\n",
            "2023-04-21 17:08:25,715 epoch 30 - iter 4/24 - loss 0.15219885 - time (sec): 4.14 - samples/sec: 3888.09 - lr: 0.000002\n",
            "2023-04-21 17:08:27,785 epoch 30 - iter 6/24 - loss 0.15034413 - time (sec): 6.21 - samples/sec: 3848.70 - lr: 0.000002\n",
            "2023-04-21 17:08:29,917 epoch 30 - iter 8/24 - loss 0.14469757 - time (sec): 8.34 - samples/sec: 3894.55 - lr: 0.000002\n",
            "2023-04-21 17:08:31,926 epoch 30 - iter 10/24 - loss 0.14941142 - time (sec): 10.35 - samples/sec: 3776.47 - lr: 0.000002\n",
            "2023-04-21 17:08:34,115 epoch 30 - iter 12/24 - loss 0.14544074 - time (sec): 12.54 - samples/sec: 3690.27 - lr: 0.000002\n",
            "2023-04-21 17:08:36,204 epoch 30 - iter 14/24 - loss 0.14615009 - time (sec): 14.63 - samples/sec: 3777.73 - lr: 0.000002\n",
            "2023-04-21 17:08:38,556 epoch 30 - iter 16/24 - loss 0.16844113 - time (sec): 16.98 - samples/sec: 3686.55 - lr: 0.000002\n",
            "2023-04-21 17:08:40,749 epoch 30 - iter 18/24 - loss 0.16899498 - time (sec): 19.17 - samples/sec: 3728.95 - lr: 0.000002\n",
            "2023-04-21 17:08:42,809 epoch 30 - iter 20/24 - loss 0.16648205 - time (sec): 21.23 - samples/sec: 3736.90 - lr: 0.000002\n",
            "2023-04-21 17:08:45,054 epoch 30 - iter 22/24 - loss 0.16930783 - time (sec): 23.48 - samples/sec: 3730.29 - lr: 0.000002\n",
            "2023-04-21 17:08:46,821 epoch 30 - iter 24/24 - loss 0.16764496 - time (sec): 25.24 - samples/sec: 3754.07 - lr: 0.000002\n",
            "2023-04-21 17:08:46,824 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:08:46,825 EPOCH 30 done: loss 0.1676 - lr 0.000002\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:08:56,995 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:09:00,029 TRAIN : loss 0.14165571331977844 - f1-score (weighted avg)  0.8105\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.19it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:09:01,551 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:09:01,618 DEV : loss 0.264420747756958 - f1-score (weighted avg)  0.702\n",
            "2023-04-21 17:09:01,637 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:09:01,641 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:09:03,725 epoch 31 - iter 2/24 - loss 0.14322001 - time (sec): 2.08 - samples/sec: 3641.51 - lr: 0.000002\n",
            "2023-04-21 17:09:05,981 epoch 31 - iter 4/24 - loss 0.16919476 - time (sec): 4.34 - samples/sec: 3723.76 - lr: 0.000002\n",
            "2023-04-21 17:09:08,054 epoch 31 - iter 6/24 - loss 0.16549815 - time (sec): 6.41 - samples/sec: 3700.34 - lr: 0.000002\n",
            "2023-04-21 17:09:10,210 epoch 31 - iter 8/24 - loss 0.15939214 - time (sec): 8.57 - samples/sec: 3825.84 - lr: 0.000002\n",
            "2023-04-21 17:09:12,339 epoch 31 - iter 10/24 - loss 0.15668489 - time (sec): 10.70 - samples/sec: 3815.10 - lr: 0.000002\n",
            "2023-04-21 17:09:14,434 epoch 31 - iter 12/24 - loss 0.15571150 - time (sec): 12.79 - samples/sec: 3798.93 - lr: 0.000002\n",
            "2023-04-21 17:09:16,505 epoch 31 - iter 14/24 - loss 0.15561607 - time (sec): 14.86 - samples/sec: 3834.09 - lr: 0.000002\n",
            "2023-04-21 17:09:18,568 epoch 31 - iter 16/24 - loss 0.15435752 - time (sec): 16.93 - samples/sec: 3781.24 - lr: 0.000002\n",
            "2023-04-21 17:09:20,663 epoch 31 - iter 18/24 - loss 0.15286676 - time (sec): 19.02 - samples/sec: 3810.64 - lr: 0.000002\n",
            "2023-04-21 17:09:22,725 epoch 31 - iter 20/24 - loss 0.15519087 - time (sec): 21.08 - samples/sec: 3807.37 - lr: 0.000002\n",
            "2023-04-21 17:09:24,785 epoch 31 - iter 22/24 - loss 0.15477058 - time (sec): 23.14 - samples/sec: 3804.48 - lr: 0.000002\n",
            "2023-04-21 17:09:26,930 epoch 31 - iter 24/24 - loss 0.16632845 - time (sec): 25.29 - samples/sec: 3747.74 - lr: 0.000002\n",
            "2023-04-21 17:09:26,933 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:09:26,934 EPOCH 31 done: loss 0.1663 - lr 0.000002\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:09:37,121 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:09:40,165 TRAIN : loss 0.14024987816810608 - f1-score (weighted avg)  0.8138\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.21it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:09:41,670 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:09:41,737 DEV : loss 0.2662420868873596 - f1-score (weighted avg)  0.7028\n",
            "2023-04-21 17:09:41,757 Epoch    31: reducing learning rate of group 0 to 1.6384e-06.\n",
            "2023-04-21 17:09:41,759 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:09:41,763 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:09:43,826 epoch 32 - iter 2/24 - loss 0.16335623 - time (sec): 2.06 - samples/sec: 4083.91 - lr: 0.000002\n",
            "2023-04-21 17:09:45,886 epoch 32 - iter 4/24 - loss 0.14919014 - time (sec): 4.12 - samples/sec: 3867.02 - lr: 0.000002\n",
            "2023-04-21 17:09:48,049 epoch 32 - iter 6/24 - loss 0.14471778 - time (sec): 6.28 - samples/sec: 3730.29 - lr: 0.000002\n",
            "2023-04-21 17:09:50,147 epoch 32 - iter 8/24 - loss 0.14342603 - time (sec): 8.38 - samples/sec: 3751.63 - lr: 0.000002\n",
            "2023-04-21 17:09:52,242 epoch 32 - iter 10/24 - loss 0.14594622 - time (sec): 10.48 - samples/sec: 3781.97 - lr: 0.000002\n",
            "2023-04-21 17:09:54,343 epoch 32 - iter 12/24 - loss 0.14644160 - time (sec): 12.58 - samples/sec: 3801.47 - lr: 0.000002\n",
            "2023-04-21 17:09:56,407 epoch 32 - iter 14/24 - loss 0.14619183 - time (sec): 14.64 - samples/sec: 3812.75 - lr: 0.000002\n",
            "2023-04-21 17:09:58,411 epoch 32 - iter 16/24 - loss 0.14615198 - time (sec): 16.65 - samples/sec: 3753.15 - lr: 0.000002\n",
            "2023-04-21 17:10:00,751 epoch 32 - iter 18/24 - loss 0.16213361 - time (sec): 18.99 - samples/sec: 3701.93 - lr: 0.000002\n",
            "2023-04-21 17:10:02,913 epoch 32 - iter 20/24 - loss 0.16118285 - time (sec): 21.15 - samples/sec: 3684.20 - lr: 0.000002\n",
            "2023-04-21 17:10:05,231 epoch 32 - iter 22/24 - loss 0.16427562 - time (sec): 23.47 - samples/sec: 3737.35 - lr: 0.000002\n",
            "2023-04-21 17:10:06,987 epoch 32 - iter 24/24 - loss 0.16476071 - time (sec): 25.22 - samples/sec: 3757.29 - lr: 0.000002\n",
            "2023-04-21 17:10:06,990 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:10:06,992 EPOCH 32 done: loss 0.1648 - lr 0.000002\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:10:17,166 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:10:19,018 TRAIN : loss 0.14122430980205536 - f1-score (weighted avg)  0.8131\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:02<00:00,  1.26it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:10:21,557 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:10:21,625 DEV : loss 0.2659316062927246 - f1-score (weighted avg)  0.7043\n",
            "2023-04-21 17:10:21,644 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:10:21,648 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:10:23,746 epoch 33 - iter 2/24 - loss 0.15192505 - time (sec): 2.10 - samples/sec: 3914.24 - lr: 0.000002\n",
            "2023-04-21 17:10:25,796 epoch 33 - iter 4/24 - loss 0.14517413 - time (sec): 4.15 - samples/sec: 3801.86 - lr: 0.000002\n",
            "2023-04-21 17:10:27,912 epoch 33 - iter 6/24 - loss 0.13685415 - time (sec): 6.26 - samples/sec: 3827.53 - lr: 0.000002\n",
            "2023-04-21 17:10:29,993 epoch 33 - iter 8/24 - loss 0.14708670 - time (sec): 8.34 - samples/sec: 3845.59 - lr: 0.000002\n",
            "2023-04-21 17:10:32,044 epoch 33 - iter 10/24 - loss 0.14414033 - time (sec): 10.39 - samples/sec: 3800.23 - lr: 0.000002\n",
            "2023-04-21 17:10:34,140 epoch 33 - iter 12/24 - loss 0.14365202 - time (sec): 12.49 - samples/sec: 3846.98 - lr: 0.000002\n",
            "2023-04-21 17:10:36,250 epoch 33 - iter 14/24 - loss 0.14358813 - time (sec): 14.60 - samples/sec: 3835.98 - lr: 0.000002\n",
            "2023-04-21 17:10:38,381 epoch 33 - iter 16/24 - loss 0.14169775 - time (sec): 16.73 - samples/sec: 3824.53 - lr: 0.000002\n",
            "2023-04-21 17:10:40,842 epoch 33 - iter 18/24 - loss 0.15569025 - time (sec): 19.19 - samples/sec: 3781.40 - lr: 0.000002\n",
            "2023-04-21 17:10:43,076 epoch 33 - iter 20/24 - loss 0.16217931 - time (sec): 21.43 - samples/sec: 3753.09 - lr: 0.000002\n",
            "2023-04-21 17:10:45,160 epoch 33 - iter 22/24 - loss 0.16202016 - time (sec): 23.51 - samples/sec: 3735.41 - lr: 0.000002\n",
            "2023-04-21 17:10:46,964 epoch 33 - iter 24/24 - loss 0.16131950 - time (sec): 25.31 - samples/sec: 3743.74 - lr: 0.000002\n",
            "2023-04-21 17:10:46,967 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:10:46,969 EPOCH 33 done: loss 0.1613 - lr 0.000002\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:10:57,149 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:11:00,174 TRAIN : loss 0.1365891396999359 - f1-score (weighted avg)  0.8191\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.18it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:11:01,704 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:11:01,772 DEV : loss 0.2656829059123993 - f1-score (weighted avg)  0.7088\n",
            "2023-04-21 17:11:01,791 Epoch    33: reducing learning rate of group 0 to 1.3107e-06.\n",
            "2023-04-21 17:11:01,793 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:11:01,796 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:11:03,923 epoch 34 - iter 2/24 - loss 0.13310845 - time (sec): 2.13 - samples/sec: 3613.70 - lr: 0.000001\n",
            "2023-04-21 17:11:05,941 epoch 34 - iter 4/24 - loss 0.14189541 - time (sec): 4.14 - samples/sec: 3551.38 - lr: 0.000001\n",
            "2023-04-21 17:11:08,046 epoch 34 - iter 6/24 - loss 0.13402643 - time (sec): 6.25 - samples/sec: 3583.52 - lr: 0.000001\n",
            "2023-04-21 17:11:10,289 epoch 34 - iter 8/24 - loss 0.14566046 - time (sec): 8.49 - samples/sec: 3544.17 - lr: 0.000001\n",
            "2023-04-21 17:11:12,423 epoch 34 - iter 10/24 - loss 0.14566345 - time (sec): 10.63 - samples/sec: 3695.11 - lr: 0.000001\n",
            "2023-04-21 17:11:14,537 epoch 34 - iter 12/24 - loss 0.14903632 - time (sec): 12.74 - samples/sec: 3685.18 - lr: 0.000001\n",
            "2023-04-21 17:11:16,616 epoch 34 - iter 14/24 - loss 0.15038724 - time (sec): 14.82 - samples/sec: 3697.08 - lr: 0.000001\n",
            "2023-04-21 17:11:18,996 epoch 34 - iter 16/24 - loss 0.16593879 - time (sec): 17.20 - samples/sec: 3717.58 - lr: 0.000001\n",
            "2023-04-21 17:11:21,091 epoch 34 - iter 18/24 - loss 0.16560481 - time (sec): 19.29 - samples/sec: 3707.09 - lr: 0.000001\n",
            "2023-04-21 17:11:23,177 epoch 34 - iter 20/24 - loss 0.16204607 - time (sec): 21.38 - samples/sec: 3724.61 - lr: 0.000001\n",
            "2023-04-21 17:11:25,253 epoch 34 - iter 22/24 - loss 0.15990259 - time (sec): 23.46 - samples/sec: 3722.66 - lr: 0.000001\n",
            "2023-04-21 17:11:27,067 epoch 34 - iter 24/24 - loss 0.15790929 - time (sec): 25.27 - samples/sec: 3750.45 - lr: 0.000001\n",
            "2023-04-21 17:11:27,070 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:11:27,072 EPOCH 34 done: loss 0.1579 - lr 0.000001\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:11:37,288 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:11:40,319 TRAIN : loss 0.13402067124843597 - f1-score (weighted avg)  0.8215\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.19it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:11:41,841 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:11:41,909 DEV : loss 0.26755988597869873 - f1-score (weighted avg)  0.7062\n",
            "2023-04-21 17:11:41,928 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:11:41,932 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:11:43,986 epoch 35 - iter 2/24 - loss 0.14255277 - time (sec): 2.05 - samples/sec: 3809.42 - lr: 0.000001\n",
            "2023-04-21 17:11:46,107 epoch 35 - iter 4/24 - loss 0.13925848 - time (sec): 4.17 - samples/sec: 3809.58 - lr: 0.000001\n",
            "2023-04-21 17:11:48,161 epoch 35 - iter 6/24 - loss 0.14100813 - time (sec): 6.23 - samples/sec: 3823.89 - lr: 0.000001\n",
            "2023-04-21 17:11:50,312 epoch 35 - iter 8/24 - loss 0.13941641 - time (sec): 8.38 - samples/sec: 3778.36 - lr: 0.000001\n",
            "2023-04-21 17:11:52,494 epoch 35 - iter 10/24 - loss 0.14434411 - time (sec): 10.56 - samples/sec: 3910.70 - lr: 0.000001\n",
            "2023-04-21 17:11:54,767 epoch 35 - iter 12/24 - loss 0.15028862 - time (sec): 12.83 - samples/sec: 3836.84 - lr: 0.000001\n",
            "2023-04-21 17:11:56,823 epoch 35 - iter 14/24 - loss 0.15132084 - time (sec): 14.89 - samples/sec: 3824.32 - lr: 0.000001\n",
            "2023-04-21 17:11:58,876 epoch 35 - iter 16/24 - loss 0.15264017 - time (sec): 16.94 - samples/sec: 3796.86 - lr: 0.000001\n",
            "2023-04-21 17:12:01,249 epoch 35 - iter 18/24 - loss 0.16367524 - time (sec): 19.32 - samples/sec: 3755.80 - lr: 0.000001\n",
            "2023-04-21 17:12:03,364 epoch 35 - iter 20/24 - loss 0.16161671 - time (sec): 21.43 - samples/sec: 3737.18 - lr: 0.000001\n",
            "2023-04-21 17:12:05,452 epoch 35 - iter 22/24 - loss 0.16047646 - time (sec): 23.52 - samples/sec: 3748.59 - lr: 0.000001\n",
            "2023-04-21 17:12:07,232 epoch 35 - iter 24/24 - loss 0.16044027 - time (sec): 25.30 - samples/sec: 3746.11 - lr: 0.000001\n",
            "2023-04-21 17:12:07,235 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:12:07,236 EPOCH 35 done: loss 0.1604 - lr 0.000001\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:12:17,407 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:12:20,455 TRAIN : loss 0.13475807011127472 - f1-score (weighted avg)  0.8209\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.19it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:12:21,982 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:12:22,053 DEV : loss 0.2680826187133789 - f1-score (weighted avg)  0.7082\n",
            "2023-04-21 17:12:22,073 Epoch    35: reducing learning rate of group 0 to 1.0486e-06.\n",
            "2023-04-21 17:12:22,075 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:12:22,079 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:12:24,138 epoch 36 - iter 2/24 - loss 0.16652450 - time (sec): 2.06 - samples/sec: 3975.35 - lr: 0.000001\n",
            "2023-04-21 17:12:26,318 epoch 36 - iter 4/24 - loss 0.16020084 - time (sec): 4.24 - samples/sec: 3877.94 - lr: 0.000001\n",
            "2023-04-21 17:12:28,360 epoch 36 - iter 6/24 - loss 0.16144842 - time (sec): 6.28 - samples/sec: 3752.03 - lr: 0.000001\n",
            "2023-04-21 17:12:30,758 epoch 36 - iter 8/24 - loss 0.18798280 - time (sec): 8.68 - samples/sec: 3670.85 - lr: 0.000001\n",
            "2023-04-21 17:12:32,892 epoch 36 - iter 10/24 - loss 0.17416800 - time (sec): 10.81 - samples/sec: 3774.25 - lr: 0.000001\n",
            "2023-04-21 17:12:34,978 epoch 36 - iter 12/24 - loss 0.16886573 - time (sec): 12.90 - samples/sec: 3692.21 - lr: 0.000001\n",
            "2023-04-21 17:12:37,126 epoch 36 - iter 14/24 - loss 0.16096759 - time (sec): 15.05 - samples/sec: 3818.70 - lr: 0.000001\n",
            "2023-04-21 17:12:39,189 epoch 36 - iter 16/24 - loss 0.16022584 - time (sec): 17.11 - samples/sec: 3793.91 - lr: 0.000001\n",
            "2023-04-21 17:12:41,259 epoch 36 - iter 18/24 - loss 0.15950809 - time (sec): 19.18 - samples/sec: 3762.74 - lr: 0.000001\n",
            "2023-04-21 17:12:43,395 epoch 36 - iter 20/24 - loss 0.15889480 - time (sec): 21.31 - samples/sec: 3753.99 - lr: 0.000001\n",
            "2023-04-21 17:12:45,645 epoch 36 - iter 22/24 - loss 0.16177921 - time (sec): 23.56 - samples/sec: 3731.76 - lr: 0.000001\n",
            "2023-04-21 17:12:47,426 epoch 36 - iter 24/24 - loss 0.16077901 - time (sec): 25.35 - samples/sec: 3739.17 - lr: 0.000001\n",
            "2023-04-21 17:12:47,428 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:12:47,430 EPOCH 36 done: loss 0.1608 - lr 0.000001\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:12:57,641 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:13:00,681 TRAIN : loss 0.13250567018985748 - f1-score (weighted avg)  0.8235\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.18it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:13:02,206 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:13:02,275 DEV : loss 0.2664056718349457 - f1-score (weighted avg)  0.7095\n",
            "2023-04-21 17:13:02,294 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:13:02,298 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:13:04,403 epoch 37 - iter 2/24 - loss 0.15032841 - time (sec): 2.10 - samples/sec: 3625.22 - lr: 0.000001\n",
            "2023-04-21 17:13:06,479 epoch 37 - iter 4/24 - loss 0.14677990 - time (sec): 4.18 - samples/sec: 3618.63 - lr: 0.000001\n",
            "2023-04-21 17:13:08,812 epoch 37 - iter 6/24 - loss 0.18192036 - time (sec): 6.51 - samples/sec: 3519.71 - lr: 0.000001\n",
            "2023-04-21 17:13:10,915 epoch 37 - iter 8/24 - loss 0.17213433 - time (sec): 8.62 - samples/sec: 3544.00 - lr: 0.000001\n",
            "2023-04-21 17:13:13,027 epoch 37 - iter 10/24 - loss 0.16763800 - time (sec): 10.73 - samples/sec: 3566.45 - lr: 0.000001\n",
            "2023-04-21 17:13:15,129 epoch 37 - iter 12/24 - loss 0.16279383 - time (sec): 12.83 - samples/sec: 3667.92 - lr: 0.000001\n",
            "2023-04-21 17:13:17,228 epoch 37 - iter 14/24 - loss 0.16038765 - time (sec): 14.93 - samples/sec: 3748.99 - lr: 0.000001\n",
            "2023-04-21 17:13:19,585 epoch 37 - iter 16/24 - loss 0.16594448 - time (sec): 17.29 - samples/sec: 3728.58 - lr: 0.000001\n",
            "2023-04-21 17:13:21,659 epoch 37 - iter 18/24 - loss 0.16233187 - time (sec): 19.36 - samples/sec: 3716.94 - lr: 0.000001\n",
            "2023-04-21 17:13:23,742 epoch 37 - iter 20/24 - loss 0.15869799 - time (sec): 21.44 - samples/sec: 3738.70 - lr: 0.000001\n",
            "2023-04-21 17:13:25,862 epoch 37 - iter 22/24 - loss 0.15688516 - time (sec): 23.56 - samples/sec: 3785.22 - lr: 0.000001\n",
            "2023-04-21 17:13:27,670 epoch 37 - iter 24/24 - loss 0.15722596 - time (sec): 25.37 - samples/sec: 3735.48 - lr: 0.000001\n",
            "2023-04-21 17:13:27,673 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:13:27,675 EPOCH 37 done: loss 0.1572 - lr 0.000001\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:13:37,899 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:13:40,943 TRAIN : loss 0.13095971941947937 - f1-score (weighted avg)  0.8268\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.19it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:13:42,467 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:13:42,536 DEV : loss 0.2681761384010315 - f1-score (weighted avg)  0.709\n",
            "2023-04-21 17:13:42,555 Epoch    37: reducing learning rate of group 0 to 8.3886e-07.\n",
            "2023-04-21 17:13:42,557 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:13:42,560 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:13:44,707 epoch 38 - iter 2/24 - loss 0.14153139 - time (sec): 2.15 - samples/sec: 3843.31 - lr: 0.000001\n",
            "2023-04-21 17:13:46,806 epoch 38 - iter 4/24 - loss 0.14130543 - time (sec): 4.24 - samples/sec: 3711.56 - lr: 0.000001\n",
            "2023-04-21 17:13:48,888 epoch 38 - iter 6/24 - loss 0.15002923 - time (sec): 6.33 - samples/sec: 3714.54 - lr: 0.000001\n",
            "2023-04-21 17:13:51,056 epoch 38 - iter 8/24 - loss 0.15021319 - time (sec): 8.49 - samples/sec: 3731.16 - lr: 0.000001\n",
            "2023-04-21 17:13:53,163 epoch 38 - iter 10/24 - loss 0.14915480 - time (sec): 10.60 - samples/sec: 3697.80 - lr: 0.000001\n",
            "2023-04-21 17:13:55,184 epoch 38 - iter 12/24 - loss 0.14643649 - time (sec): 12.62 - samples/sec: 3690.08 - lr: 0.000001\n",
            "2023-04-21 17:13:57,267 epoch 38 - iter 14/24 - loss 0.14497792 - time (sec): 14.71 - samples/sec: 3719.80 - lr: 0.000001\n",
            "2023-04-21 17:13:59,562 epoch 38 - iter 16/24 - loss 0.14840324 - time (sec): 17.00 - samples/sec: 3729.03 - lr: 0.000001\n",
            "2023-04-21 17:14:01,980 epoch 38 - iter 18/24 - loss 0.16001086 - time (sec): 19.42 - samples/sec: 3729.00 - lr: 0.000001\n",
            "2023-04-21 17:14:04,010 epoch 38 - iter 20/24 - loss 0.15846635 - time (sec): 21.45 - samples/sec: 3719.42 - lr: 0.000001\n",
            "2023-04-21 17:14:06,131 epoch 38 - iter 22/24 - loss 0.15651346 - time (sec): 23.57 - samples/sec: 3737.86 - lr: 0.000001\n",
            "2023-04-21 17:14:07,924 epoch 38 - iter 24/24 - loss 0.15545783 - time (sec): 25.36 - samples/sec: 3736.58 - lr: 0.000001\n",
            "2023-04-21 17:14:07,927 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:14:07,929 EPOCH 38 done: loss 0.1555 - lr 0.000001\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:14:18,120 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:14:19,950 TRAIN : loss 0.1307717114686966 - f1-score (weighted avg)  0.827\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:02<00:00,  1.24it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:14:22,511 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:14:22,578 DEV : loss 0.26739153265953064 - f1-score (weighted avg)  0.7097\n",
            "2023-04-21 17:14:22,597 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:14:22,601 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:14:24,637 epoch 39 - iter 2/24 - loss 0.15611818 - time (sec): 2.03 - samples/sec: 3263.84 - lr: 0.000001\n",
            "2023-04-21 17:14:26,721 epoch 39 - iter 4/24 - loss 0.14211039 - time (sec): 4.12 - samples/sec: 3737.89 - lr: 0.000001\n",
            "2023-04-21 17:14:28,854 epoch 39 - iter 6/24 - loss 0.13685074 - time (sec): 6.25 - samples/sec: 3805.87 - lr: 0.000001\n",
            "2023-04-21 17:14:30,986 epoch 39 - iter 8/24 - loss 0.13619391 - time (sec): 8.38 - samples/sec: 3820.56 - lr: 0.000001\n",
            "2023-04-21 17:14:33,340 epoch 39 - iter 10/24 - loss 0.16206788 - time (sec): 10.74 - samples/sec: 3762.87 - lr: 0.000001\n",
            "2023-04-21 17:14:35,408 epoch 39 - iter 12/24 - loss 0.15814314 - time (sec): 12.81 - samples/sec: 3823.20 - lr: 0.000001\n",
            "2023-04-21 17:14:37,580 epoch 39 - iter 14/24 - loss 0.15726054 - time (sec): 14.98 - samples/sec: 3862.33 - lr: 0.000001\n",
            "2023-04-21 17:14:39,649 epoch 39 - iter 16/24 - loss 0.15694084 - time (sec): 17.05 - samples/sec: 3823.15 - lr: 0.000001\n",
            "2023-04-21 17:14:41,921 epoch 39 - iter 18/24 - loss 0.15858195 - time (sec): 19.32 - samples/sec: 3768.78 - lr: 0.000001\n",
            "2023-04-21 17:14:44,028 epoch 39 - iter 20/24 - loss 0.15871666 - time (sec): 21.43 - samples/sec: 3765.94 - lr: 0.000001\n",
            "2023-04-21 17:14:46,108 epoch 39 - iter 22/24 - loss 0.15797850 - time (sec): 23.51 - samples/sec: 3763.83 - lr: 0.000001\n",
            "2023-04-21 17:14:47,924 epoch 39 - iter 24/24 - loss 0.15774896 - time (sec): 25.32 - samples/sec: 3742.70 - lr: 0.000001\n",
            "2023-04-21 17:14:47,927 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:14:47,929 EPOCH 39 done: loss 0.1577 - lr 0.000001\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:14:58,151 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:15:01,194 TRAIN : loss 0.13168403506278992 - f1-score (weighted avg)  0.8247\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.15it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:15:02,743 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:15:02,813 DEV : loss 0.26865077018737793 - f1-score (weighted avg)  0.7078\n",
            "2023-04-21 17:15:02,833 Epoch    39: reducing learning rate of group 0 to 6.7109e-07.\n",
            "2023-04-21 17:15:02,835 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:15:02,838 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:15:04,969 epoch 40 - iter 2/24 - loss 0.13989402 - time (sec): 2.13 - samples/sec: 3940.80 - lr: 0.000001\n",
            "2023-04-21 17:15:07,066 epoch 40 - iter 4/24 - loss 0.15640581 - time (sec): 4.23 - samples/sec: 4072.43 - lr: 0.000001\n",
            "2023-04-21 17:15:09,121 epoch 40 - iter 6/24 - loss 0.15498478 - time (sec): 6.28 - samples/sec: 3801.46 - lr: 0.000001\n",
            "2023-04-21 17:15:11,204 epoch 40 - iter 8/24 - loss 0.15288106 - time (sec): 8.36 - samples/sec: 3725.73 - lr: 0.000001\n",
            "2023-04-21 17:15:13,389 epoch 40 - iter 10/24 - loss 0.15169255 - time (sec): 10.55 - samples/sec: 3800.11 - lr: 0.000001\n",
            "2023-04-21 17:15:15,430 epoch 40 - iter 12/24 - loss 0.14503892 - time (sec): 12.59 - samples/sec: 3782.13 - lr: 0.000001\n",
            "2023-04-21 17:15:17,437 epoch 40 - iter 14/24 - loss 0.14838765 - time (sec): 14.60 - samples/sec: 3723.25 - lr: 0.000001\n",
            "2023-04-21 17:15:19,509 epoch 40 - iter 16/24 - loss 0.14773516 - time (sec): 16.67 - samples/sec: 3728.76 - lr: 0.000001\n",
            "2023-04-21 17:15:21,667 epoch 40 - iter 18/24 - loss 0.14697731 - time (sec): 18.83 - samples/sec: 3731.98 - lr: 0.000001\n",
            "2023-04-21 17:15:24,062 epoch 40 - iter 20/24 - loss 0.15808631 - time (sec): 21.22 - samples/sec: 3723.40 - lr: 0.000001\n",
            "2023-04-21 17:15:26,374 epoch 40 - iter 22/24 - loss 0.15897146 - time (sec): 23.53 - samples/sec: 3793.76 - lr: 0.000001\n",
            "2023-04-21 17:15:28,086 epoch 40 - iter 24/24 - loss 0.15779885 - time (sec): 25.25 - samples/sec: 3753.92 - lr: 0.000001\n",
            "2023-04-21 17:15:28,088 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:15:28,090 EPOCH 40 done: loss 0.1578 - lr 0.000001\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:15:38,321 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:15:41,359 TRAIN : loss 0.12874804437160492 - f1-score (weighted avg)  0.8294\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.19it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:15:42,876 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:15:42,943 DEV : loss 0.26827526092529297 - f1-score (weighted avg)  0.7089\n",
            "2023-04-21 17:15:42,962 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:15:42,966 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:15:45,011 epoch 41 - iter 2/24 - loss 0.16410351 - time (sec): 2.04 - samples/sec: 3837.33 - lr: 0.000001\n",
            "2023-04-21 17:15:47,065 epoch 41 - iter 4/24 - loss 0.16047931 - time (sec): 4.10 - samples/sec: 3785.48 - lr: 0.000001\n",
            "2023-04-21 17:15:49,135 epoch 41 - iter 6/24 - loss 0.14701093 - time (sec): 6.17 - samples/sec: 3763.55 - lr: 0.000001\n",
            "2023-04-21 17:15:51,329 epoch 41 - iter 8/24 - loss 0.14961696 - time (sec): 8.36 - samples/sec: 3724.29 - lr: 0.000001\n",
            "2023-04-21 17:15:53,451 epoch 41 - iter 10/24 - loss 0.14881661 - time (sec): 10.48 - samples/sec: 3712.94 - lr: 0.000001\n",
            "2023-04-21 17:15:55,459 epoch 41 - iter 12/24 - loss 0.15055906 - time (sec): 12.49 - samples/sec: 3670.15 - lr: 0.000001\n",
            "2023-04-21 17:15:57,524 epoch 41 - iter 14/24 - loss 0.14780822 - time (sec): 14.56 - samples/sec: 3668.75 - lr: 0.000001\n",
            "2023-04-21 17:15:59,915 epoch 41 - iter 16/24 - loss 0.15905122 - time (sec): 16.95 - samples/sec: 3684.79 - lr: 0.000001\n",
            "2023-04-21 17:16:01,965 epoch 41 - iter 18/24 - loss 0.15915730 - time (sec): 19.00 - samples/sec: 3670.94 - lr: 0.000001\n",
            "2023-04-21 17:16:04,053 epoch 41 - iter 20/24 - loss 0.15709534 - time (sec): 21.08 - samples/sec: 3711.86 - lr: 0.000001\n",
            "2023-04-21 17:16:06,244 epoch 41 - iter 22/24 - loss 0.15575388 - time (sec): 23.28 - samples/sec: 3756.31 - lr: 0.000001\n",
            "2023-04-21 17:16:08,196 epoch 41 - iter 24/24 - loss 0.15646062 - time (sec): 25.23 - samples/sec: 3756.49 - lr: 0.000001\n",
            "2023-04-21 17:16:08,199 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:16:08,201 EPOCH 41 done: loss 0.1565 - lr 0.000001\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.36it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:16:18,510 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:16:21,627 TRAIN : loss 0.1291791945695877 - f1-score (weighted avg)  0.828\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.18it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:16:23,155 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:16:23,225 DEV : loss 0.2683766186237335 - f1-score (weighted avg)  0.7065\n",
            "2023-04-21 17:16:23,245 Epoch    41: reducing learning rate of group 0 to 5.3687e-07.\n",
            "2023-04-21 17:16:23,247 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:16:23,251 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:16:25,328 epoch 42 - iter 2/24 - loss 0.11896010 - time (sec): 2.08 - samples/sec: 3745.47 - lr: 0.000001\n",
            "2023-04-21 17:16:27,396 epoch 42 - iter 4/24 - loss 0.14390888 - time (sec): 4.14 - samples/sec: 3693.60 - lr: 0.000001\n",
            "2023-04-21 17:16:29,454 epoch 42 - iter 6/24 - loss 0.14144907 - time (sec): 6.20 - samples/sec: 3621.02 - lr: 0.000001\n",
            "2023-04-21 17:16:31,648 epoch 42 - iter 8/24 - loss 0.13908232 - time (sec): 8.40 - samples/sec: 3685.35 - lr: 0.000001\n",
            "2023-04-21 17:16:33,782 epoch 42 - iter 10/24 - loss 0.14023778 - time (sec): 10.53 - samples/sec: 3595.88 - lr: 0.000001\n",
            "2023-04-21 17:16:36,036 epoch 42 - iter 12/24 - loss 0.14680172 - time (sec): 12.78 - samples/sec: 3705.54 - lr: 0.000001\n",
            "2023-04-21 17:16:38,152 epoch 42 - iter 14/24 - loss 0.14387933 - time (sec): 14.90 - samples/sec: 3697.56 - lr: 0.000001\n",
            "2023-04-21 17:16:40,558 epoch 42 - iter 16/24 - loss 0.15753343 - time (sec): 17.31 - samples/sec: 3743.49 - lr: 0.000001\n",
            "2023-04-21 17:16:42,634 epoch 42 - iter 18/24 - loss 0.15431085 - time (sec): 19.38 - samples/sec: 3662.85 - lr: 0.000001\n",
            "2023-04-21 17:16:44,716 epoch 42 - iter 20/24 - loss 0.15444837 - time (sec): 21.46 - samples/sec: 3719.34 - lr: 0.000001\n",
            "2023-04-21 17:16:46,793 epoch 42 - iter 22/24 - loss 0.15270969 - time (sec): 23.54 - samples/sec: 3733.65 - lr: 0.000001\n",
            "2023-04-21 17:16:48,593 epoch 42 - iter 24/24 - loss 0.15349076 - time (sec): 25.34 - samples/sec: 3739.78 - lr: 0.000001\n",
            "2023-04-21 17:16:48,596 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:16:48,598 EPOCH 42 done: loss 0.1535 - lr 0.000001\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.37it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:16:58,859 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:17:01,997 TRAIN : loss 0.13100774586200714 - f1-score (weighted avg)  0.8264\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.20it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:17:03,519 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:17:03,589 DEV : loss 0.26898422837257385 - f1-score (weighted avg)  0.7075\n",
            "2023-04-21 17:17:03,608 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:17:03,613 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:17:05,913 epoch 43 - iter 2/24 - loss 0.17143309 - time (sec): 2.30 - samples/sec: 3524.44 - lr: 0.000001\n",
            "2023-04-21 17:17:08,028 epoch 43 - iter 4/24 - loss 0.17210281 - time (sec): 4.41 - samples/sec: 3730.94 - lr: 0.000001\n",
            "2023-04-21 17:17:10,146 epoch 43 - iter 6/24 - loss 0.16166796 - time (sec): 6.53 - samples/sec: 3622.96 - lr: 0.000001\n",
            "2023-04-21 17:17:12,202 epoch 43 - iter 8/24 - loss 0.15280909 - time (sec): 8.59 - samples/sec: 3730.08 - lr: 0.000001\n",
            "2023-04-21 17:17:14,277 epoch 43 - iter 10/24 - loss 0.15299668 - time (sec): 10.66 - samples/sec: 3735.77 - lr: 0.000001\n",
            "2023-04-21 17:17:16,373 epoch 43 - iter 12/24 - loss 0.15022210 - time (sec): 12.76 - samples/sec: 3731.69 - lr: 0.000001\n",
            "2023-04-21 17:17:18,473 epoch 43 - iter 14/24 - loss 0.15018354 - time (sec): 14.86 - samples/sec: 3698.36 - lr: 0.000001\n",
            "2023-04-21 17:17:20,592 epoch 43 - iter 16/24 - loss 0.14803352 - time (sec): 16.98 - samples/sec: 3739.99 - lr: 0.000001\n",
            "2023-04-21 17:17:22,652 epoch 43 - iter 18/24 - loss 0.14796478 - time (sec): 19.04 - samples/sec: 3745.63 - lr: 0.000001\n",
            "2023-04-21 17:17:25,016 epoch 43 - iter 20/24 - loss 0.15952611 - time (sec): 21.40 - samples/sec: 3713.07 - lr: 0.000001\n",
            "2023-04-21 17:17:27,175 epoch 43 - iter 22/24 - loss 0.15621254 - time (sec): 23.56 - samples/sec: 3732.05 - lr: 0.000001\n",
            "2023-04-21 17:17:28,957 epoch 43 - iter 24/24 - loss 0.15595605 - time (sec): 25.34 - samples/sec: 3739.57 - lr: 0.000001\n",
            "2023-04-21 17:17:28,960 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:17:28,962 EPOCH 43 done: loss 0.1560 - lr 0.000001\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.37it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:17:39,221 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:17:41,084 TRAIN : loss 0.1276307851076126 - f1-score (weighted avg)  0.8303\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:02<00:00,  1.23it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:17:43,679 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:17:43,748 DEV : loss 0.2681693136692047 - f1-score (weighted avg)  0.7079\n",
            "2023-04-21 17:17:43,768 Epoch    43: reducing learning rate of group 0 to 4.2950e-07.\n",
            "2023-04-21 17:17:43,770 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:17:43,774 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:17:45,860 epoch 44 - iter 2/24 - loss 0.12268441 - time (sec): 2.08 - samples/sec: 4047.38 - lr: 0.000000\n",
            "2023-04-21 17:17:47,989 epoch 44 - iter 4/24 - loss 0.12348619 - time (sec): 4.21 - samples/sec: 3942.22 - lr: 0.000000\n",
            "2023-04-21 17:17:50,059 epoch 44 - iter 6/24 - loss 0.12969885 - time (sec): 6.28 - samples/sec: 3777.12 - lr: 0.000000\n",
            "2023-04-21 17:17:52,185 epoch 44 - iter 8/24 - loss 0.12643401 - time (sec): 8.41 - samples/sec: 3624.85 - lr: 0.000000\n",
            "2023-04-21 17:17:54,326 epoch 44 - iter 10/24 - loss 0.12969142 - time (sec): 10.55 - samples/sec: 3804.45 - lr: 0.000000\n",
            "2023-04-21 17:17:56,377 epoch 44 - iter 12/24 - loss 0.13450471 - time (sec): 12.60 - samples/sec: 3779.57 - lr: 0.000000\n",
            "2023-04-21 17:17:58,496 epoch 44 - iter 14/24 - loss 0.13371003 - time (sec): 14.72 - samples/sec: 3754.76 - lr: 0.000000\n",
            "2023-04-21 17:18:00,849 epoch 44 - iter 16/24 - loss 0.14909118 - time (sec): 17.07 - samples/sec: 3743.39 - lr: 0.000000\n",
            "2023-04-21 17:18:02,934 epoch 44 - iter 18/24 - loss 0.14776394 - time (sec): 19.16 - samples/sec: 3713.19 - lr: 0.000000\n",
            "2023-04-21 17:18:05,186 epoch 44 - iter 20/24 - loss 0.15146502 - time (sec): 21.41 - samples/sec: 3677.27 - lr: 0.000000\n",
            "2023-04-21 17:18:07,357 epoch 44 - iter 22/24 - loss 0.15457487 - time (sec): 23.58 - samples/sec: 3676.13 - lr: 0.000000\n",
            "2023-04-21 17:18:09,128 epoch 44 - iter 24/24 - loss 0.15251509 - time (sec): 25.35 - samples/sec: 3738.09 - lr: 0.000000\n",
            "2023-04-21 17:18:09,132 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:18:09,133 EPOCH 44 done: loss 0.1525 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.36it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:18:19,436 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:18:22,513 TRAIN : loss 0.1274845004081726 - f1-score (weighted avg)  0.8304\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.16it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:18:24,057 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:18:24,126 DEV : loss 0.2686128616333008 - f1-score (weighted avg)  0.7073\n",
            "2023-04-21 17:18:24,146 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:18:24,151 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:18:26,417 epoch 45 - iter 2/24 - loss 0.18541977 - time (sec): 2.26 - samples/sec: 3859.59 - lr: 0.000000\n",
            "2023-04-21 17:18:28,487 epoch 45 - iter 4/24 - loss 0.17137995 - time (sec): 4.33 - samples/sec: 4021.35 - lr: 0.000000\n",
            "2023-04-21 17:18:30,551 epoch 45 - iter 6/24 - loss 0.16340714 - time (sec): 6.40 - samples/sec: 3799.87 - lr: 0.000000\n",
            "2023-04-21 17:18:32,756 epoch 45 - iter 8/24 - loss 0.16203884 - time (sec): 8.60 - samples/sec: 3830.56 - lr: 0.000000\n",
            "2023-04-21 17:18:34,805 epoch 45 - iter 10/24 - loss 0.15526898 - time (sec): 10.65 - samples/sec: 3744.12 - lr: 0.000000\n",
            "2023-04-21 17:18:37,226 epoch 45 - iter 12/24 - loss 0.17430993 - time (sec): 13.07 - samples/sec: 3674.69 - lr: 0.000000\n",
            "2023-04-21 17:18:39,275 epoch 45 - iter 14/24 - loss 0.16871559 - time (sec): 15.12 - samples/sec: 3658.77 - lr: 0.000000\n",
            "2023-04-21 17:18:41,387 epoch 45 - iter 16/24 - loss 0.16361474 - time (sec): 17.23 - samples/sec: 3704.75 - lr: 0.000000\n",
            "2023-04-21 17:18:43,568 epoch 45 - iter 18/24 - loss 0.15940927 - time (sec): 19.42 - samples/sec: 3718.20 - lr: 0.000000\n",
            "2023-04-21 17:18:45,609 epoch 45 - iter 20/24 - loss 0.15669464 - time (sec): 21.46 - samples/sec: 3721.30 - lr: 0.000000\n",
            "2023-04-21 17:18:47,721 epoch 45 - iter 22/24 - loss 0.15324206 - time (sec): 23.57 - samples/sec: 3744.27 - lr: 0.000000\n",
            "2023-04-21 17:18:49,475 epoch 45 - iter 24/24 - loss 0.15199291 - time (sec): 25.32 - samples/sec: 3742.43 - lr: 0.000000\n",
            "2023-04-21 17:18:49,479 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:18:49,481 EPOCH 45 done: loss 0.1520 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.36it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:18:59,777 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:19:02,846 TRAIN : loss 0.12759065628051758 - f1-score (weighted avg)  0.8304\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.15it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:19:04,392 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:19:04,462 DEV : loss 0.26854217052459717 - f1-score (weighted avg)  0.7084\n",
            "2023-04-21 17:19:04,482 Epoch    45: reducing learning rate of group 0 to 3.4360e-07.\n",
            "2023-04-21 17:19:04,484 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:19:04,493 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:19:06,672 epoch 46 - iter 2/24 - loss 0.15265320 - time (sec): 2.18 - samples/sec: 3427.55 - lr: 0.000000\n",
            "2023-04-21 17:19:08,736 epoch 46 - iter 4/24 - loss 0.15599707 - time (sec): 4.24 - samples/sec: 3615.12 - lr: 0.000000\n",
            "2023-04-21 17:19:11,105 epoch 46 - iter 6/24 - loss 0.18656661 - time (sec): 6.61 - samples/sec: 3777.36 - lr: 0.000000\n",
            "2023-04-21 17:19:13,353 epoch 46 - iter 8/24 - loss 0.18661229 - time (sec): 8.86 - samples/sec: 3611.70 - lr: 0.000000\n",
            "2023-04-21 17:19:15,408 epoch 46 - iter 10/24 - loss 0.18032756 - time (sec): 10.91 - samples/sec: 3651.39 - lr: 0.000000\n",
            "2023-04-21 17:19:17,527 epoch 46 - iter 12/24 - loss 0.16888941 - time (sec): 13.03 - samples/sec: 3618.02 - lr: 0.000000\n",
            "2023-04-21 17:19:19,659 epoch 46 - iter 14/24 - loss 0.16160846 - time (sec): 15.16 - samples/sec: 3677.40 - lr: 0.000000\n",
            "2023-04-21 17:19:21,775 epoch 46 - iter 16/24 - loss 0.15718728 - time (sec): 17.28 - samples/sec: 3685.62 - lr: 0.000000\n",
            "2023-04-21 17:19:23,940 epoch 46 - iter 18/24 - loss 0.15398674 - time (sec): 19.45 - samples/sec: 3705.01 - lr: 0.000000\n",
            "2023-04-21 17:19:26,061 epoch 46 - iter 20/24 - loss 0.15218400 - time (sec): 21.57 - samples/sec: 3720.35 - lr: 0.000000\n",
            "2023-04-21 17:19:28,144 epoch 46 - iter 22/24 - loss 0.15071611 - time (sec): 23.65 - samples/sec: 3753.36 - lr: 0.000000\n",
            "2023-04-21 17:19:29,862 epoch 46 - iter 24/24 - loss 0.15038057 - time (sec): 25.37 - samples/sec: 3735.81 - lr: 0.000000\n",
            "2023-04-21 17:19:29,865 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:19:29,867 EPOCH 46 done: loss 0.1504 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.40it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:19:40,019 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:19:43,024 TRAIN : loss 0.12628552317619324 - f1-score (weighted avg)  0.8321\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.20it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:19:44,539 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:19:44,608 DEV : loss 0.2689170837402344 - f1-score (weighted avg)  0.708\n",
            "2023-04-21 17:19:44,626 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:19:44,630 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:19:46,811 epoch 47 - iter 2/24 - loss 0.13438064 - time (sec): 2.18 - samples/sec: 3861.58 - lr: 0.000000\n",
            "2023-04-21 17:19:48,862 epoch 47 - iter 4/24 - loss 0.14613103 - time (sec): 4.23 - samples/sec: 3769.39 - lr: 0.000000\n",
            "2023-04-21 17:19:50,905 epoch 47 - iter 6/24 - loss 0.14548266 - time (sec): 6.27 - samples/sec: 3617.63 - lr: 0.000000\n",
            "2023-04-21 17:19:53,014 epoch 47 - iter 8/24 - loss 0.14946683 - time (sec): 8.38 - samples/sec: 3680.50 - lr: 0.000000\n",
            "2023-04-21 17:19:55,116 epoch 47 - iter 10/24 - loss 0.14730538 - time (sec): 10.48 - samples/sec: 3632.41 - lr: 0.000000\n",
            "2023-04-21 17:19:57,151 epoch 47 - iter 12/24 - loss 0.14684788 - time (sec): 12.52 - samples/sec: 3608.43 - lr: 0.000000\n",
            "2023-04-21 17:19:59,185 epoch 47 - iter 14/24 - loss 0.14581293 - time (sec): 14.55 - samples/sec: 3636.38 - lr: 0.000000\n",
            "2023-04-21 17:20:01,324 epoch 47 - iter 16/24 - loss 0.14695897 - time (sec): 16.69 - samples/sec: 3709.81 - lr: 0.000000\n",
            "2023-04-21 17:20:03,631 epoch 47 - iter 18/24 - loss 0.14891212 - time (sec): 19.00 - samples/sec: 3675.90 - lr: 0.000000\n",
            "2023-04-21 17:20:05,709 epoch 47 - iter 20/24 - loss 0.14517175 - time (sec): 21.08 - samples/sec: 3736.87 - lr: 0.000000\n",
            "2023-04-21 17:20:08,173 epoch 47 - iter 22/24 - loss 0.15548726 - time (sec): 23.54 - samples/sec: 3727.90 - lr: 0.000000\n",
            "2023-04-21 17:20:09,960 epoch 47 - iter 24/24 - loss 0.15094246 - time (sec): 25.33 - samples/sec: 3741.60 - lr: 0.000000\n",
            "2023-04-21 17:20:09,963 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:20:09,965 EPOCH 47 done: loss 0.1509 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:20:20,181 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:20:23,195 TRAIN : loss 0.12696237862110138 - f1-score (weighted avg)  0.8314\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.17it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:20:24,729 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:20:24,796 DEV : loss 0.2692607343196869 - f1-score (weighted avg)  0.7104\n",
            "2023-04-21 17:20:24,814 Epoch    47: reducing learning rate of group 0 to 2.7488e-07.\n",
            "2023-04-21 17:20:24,816 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:20:24,820 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:20:26,867 epoch 48 - iter 2/24 - loss 0.13271927 - time (sec): 2.05 - samples/sec: 3411.20 - lr: 0.000000\n",
            "2023-04-21 17:20:28,982 epoch 48 - iter 4/24 - loss 0.13850390 - time (sec): 4.16 - samples/sec: 3669.37 - lr: 0.000000\n",
            "2023-04-21 17:20:31,065 epoch 48 - iter 6/24 - loss 0.13442970 - time (sec): 6.24 - samples/sec: 3708.48 - lr: 0.000000\n",
            "2023-04-21 17:20:33,364 epoch 48 - iter 8/24 - loss 0.15374772 - time (sec): 8.54 - samples/sec: 3629.45 - lr: 0.000000\n",
            "2023-04-21 17:20:35,726 epoch 48 - iter 10/24 - loss 0.17096268 - time (sec): 10.91 - samples/sec: 3653.17 - lr: 0.000000\n",
            "2023-04-21 17:20:37,835 epoch 48 - iter 12/24 - loss 0.16540822 - time (sec): 13.01 - samples/sec: 3669.25 - lr: 0.000000\n",
            "2023-04-21 17:20:39,911 epoch 48 - iter 14/24 - loss 0.16090222 - time (sec): 15.09 - samples/sec: 3671.05 - lr: 0.000000\n",
            "2023-04-21 17:20:42,059 epoch 48 - iter 16/24 - loss 0.15607435 - time (sec): 17.24 - samples/sec: 3672.15 - lr: 0.000000\n",
            "2023-04-21 17:20:44,254 epoch 48 - iter 18/24 - loss 0.15407481 - time (sec): 19.43 - samples/sec: 3690.97 - lr: 0.000000\n",
            "2023-04-21 17:20:46,332 epoch 48 - iter 20/24 - loss 0.15330777 - time (sec): 21.51 - samples/sec: 3670.74 - lr: 0.000000\n",
            "2023-04-21 17:20:48,430 epoch 48 - iter 22/24 - loss 0.15273652 - time (sec): 23.61 - samples/sec: 3701.32 - lr: 0.000000\n",
            "2023-04-21 17:20:50,243 epoch 48 - iter 24/24 - loss 0.15175203 - time (sec): 25.42 - samples/sec: 3727.91 - lr: 0.000000\n",
            "2023-04-21 17:20:50,246 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:20:50,247 EPOCH 48 done: loss 0.1518 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.37it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:21:00,493 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:21:03,550 TRAIN : loss 0.1265224814414978 - f1-score (weighted avg)  0.8315\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.19it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:21:05,070 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:21:05,145 DEV : loss 0.269550621509552 - f1-score (weighted avg)  0.7066\n",
            "2023-04-21 17:21:05,164 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:21:05,168 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:21:07,574 epoch 49 - iter 2/24 - loss 0.25562378 - time (sec): 2.40 - samples/sec: 3318.64 - lr: 0.000000\n",
            "2023-04-21 17:21:09,658 epoch 49 - iter 4/24 - loss 0.20706254 - time (sec): 4.49 - samples/sec: 3553.85 - lr: 0.000000\n",
            "2023-04-21 17:21:11,728 epoch 49 - iter 6/24 - loss 0.19282621 - time (sec): 6.56 - samples/sec: 3407.29 - lr: 0.000000\n",
            "2023-04-21 17:21:13,897 epoch 49 - iter 8/24 - loss 0.17522181 - time (sec): 8.73 - samples/sec: 3585.05 - lr: 0.000000\n",
            "2023-04-21 17:21:15,971 epoch 49 - iter 10/24 - loss 0.16547703 - time (sec): 10.80 - samples/sec: 3618.33 - lr: 0.000000\n",
            "2023-04-21 17:21:18,124 epoch 49 - iter 12/24 - loss 0.16028784 - time (sec): 12.95 - samples/sec: 3670.82 - lr: 0.000000\n",
            "2023-04-21 17:21:20,371 epoch 49 - iter 14/24 - loss 0.16058404 - time (sec): 15.20 - samples/sec: 3728.81 - lr: 0.000000\n",
            "2023-04-21 17:21:22,450 epoch 49 - iter 16/24 - loss 0.15696201 - time (sec): 17.28 - samples/sec: 3750.30 - lr: 0.000000\n",
            "2023-04-21 17:21:24,558 epoch 49 - iter 18/24 - loss 0.15464504 - time (sec): 19.39 - samples/sec: 3695.90 - lr: 0.000000\n",
            "2023-04-21 17:21:26,664 epoch 49 - iter 20/24 - loss 0.15324577 - time (sec): 21.49 - samples/sec: 3715.14 - lr: 0.000000\n",
            "2023-04-21 17:21:28,697 epoch 49 - iter 22/24 - loss 0.15338557 - time (sec): 23.53 - samples/sec: 3715.53 - lr: 0.000000\n",
            "2023-04-21 17:21:30,478 epoch 49 - iter 24/24 - loss 0.15144310 - time (sec): 25.31 - samples/sec: 3744.52 - lr: 0.000000\n",
            "2023-04-21 17:21:30,481 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:21:30,483 EPOCH 49 done: loss 0.1514 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:21:40,709 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:21:43,756 TRAIN : loss 0.12582992017269135 - f1-score (weighted avg)  0.8325\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.18it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:21:45,283 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:21:45,350 DEV : loss 0.2704709768295288 - f1-score (weighted avg)  0.7066\n",
            "2023-04-21 17:21:45,369 Epoch    49: reducing learning rate of group 0 to 2.1990e-07.\n",
            "2023-04-21 17:21:45,371 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:21:45,375 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:21:47,443 epoch 50 - iter 2/24 - loss 0.13870169 - time (sec): 2.07 - samples/sec: 3900.28 - lr: 0.000000\n",
            "2023-04-21 17:21:49,597 epoch 50 - iter 4/24 - loss 0.14219207 - time (sec): 4.22 - samples/sec: 4101.56 - lr: 0.000000\n",
            "2023-04-21 17:21:51,685 epoch 50 - iter 6/24 - loss 0.13929456 - time (sec): 6.31 - samples/sec: 4001.13 - lr: 0.000000\n",
            "2023-04-21 17:21:53,791 epoch 50 - iter 8/24 - loss 0.13752643 - time (sec): 8.41 - samples/sec: 3893.05 - lr: 0.000000\n",
            "2023-04-21 17:21:56,018 epoch 50 - iter 10/24 - loss 0.14538286 - time (sec): 10.64 - samples/sec: 3805.95 - lr: 0.000000\n",
            "2023-04-21 17:21:58,107 epoch 50 - iter 12/24 - loss 0.14365411 - time (sec): 12.73 - samples/sec: 3745.97 - lr: 0.000000\n",
            "2023-04-21 17:22:00,189 epoch 50 - iter 14/24 - loss 0.14096018 - time (sec): 14.81 - samples/sec: 3733.87 - lr: 0.000000\n",
            "2023-04-21 17:22:02,252 epoch 50 - iter 16/24 - loss 0.14070946 - time (sec): 16.88 - samples/sec: 3745.83 - lr: 0.000000\n",
            "2023-04-21 17:22:04,305 epoch 50 - iter 18/24 - loss 0.14215245 - time (sec): 18.93 - samples/sec: 3789.65 - lr: 0.000000\n",
            "2023-04-21 17:22:06,734 epoch 50 - iter 20/24 - loss 0.15215979 - time (sec): 21.36 - samples/sec: 3761.53 - lr: 0.000000\n",
            "2023-04-21 17:22:08,901 epoch 50 - iter 22/24 - loss 0.15179562 - time (sec): 23.52 - samples/sec: 3750.24 - lr: 0.000000\n",
            "2023-04-21 17:22:10,668 epoch 50 - iter 24/24 - loss 0.15095932 - time (sec): 25.29 - samples/sec: 3747.18 - lr: 0.000000\n",
            "2023-04-21 17:22:10,671 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:22:10,673 EPOCH 50 done: loss 0.1510 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.40it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:22:20,835 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:22:22,678 TRAIN : loss 0.12536340951919556 - f1-score (weighted avg)  0.8333\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:02<00:00,  1.26it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:22:25,217 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:22:25,284 DEV : loss 0.27035364508628845 - f1-score (weighted avg)  0.7074\n",
            "2023-04-21 17:22:25,304 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:22:25,308 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:22:27,433 epoch 51 - iter 2/24 - loss 0.13888277 - time (sec): 2.12 - samples/sec: 3727.69 - lr: 0.000000\n",
            "2023-04-21 17:22:29,524 epoch 51 - iter 4/24 - loss 0.13954141 - time (sec): 4.21 - samples/sec: 4046.12 - lr: 0.000000\n",
            "2023-04-21 17:22:31,593 epoch 51 - iter 6/24 - loss 0.14192146 - time (sec): 6.28 - samples/sec: 3964.80 - lr: 0.000000\n",
            "2023-04-21 17:22:33,712 epoch 51 - iter 8/24 - loss 0.14010361 - time (sec): 8.40 - samples/sec: 3858.60 - lr: 0.000000\n",
            "2023-04-21 17:22:35,772 epoch 51 - iter 10/24 - loss 0.13834067 - time (sec): 10.46 - samples/sec: 3821.84 - lr: 0.000000\n",
            "2023-04-21 17:22:37,851 epoch 51 - iter 12/24 - loss 0.13417801 - time (sec): 12.54 - samples/sec: 3772.79 - lr: 0.000000\n",
            "2023-04-21 17:22:40,015 epoch 51 - iter 14/24 - loss 0.13696545 - time (sec): 14.71 - samples/sec: 3818.46 - lr: 0.000000\n",
            "2023-04-21 17:22:42,078 epoch 51 - iter 16/24 - loss 0.13599499 - time (sec): 16.77 - samples/sec: 3776.69 - lr: 0.000000\n",
            "2023-04-21 17:22:44,554 epoch 51 - iter 18/24 - loss 0.15228844 - time (sec): 19.25 - samples/sec: 3745.93 - lr: 0.000000\n",
            "2023-04-21 17:22:46,612 epoch 51 - iter 20/24 - loss 0.15149646 - time (sec): 21.30 - samples/sec: 3728.23 - lr: 0.000000\n",
            "2023-04-21 17:22:48,860 epoch 51 - iter 22/24 - loss 0.15429422 - time (sec): 23.55 - samples/sec: 3707.24 - lr: 0.000000\n",
            "2023-04-21 17:22:50,638 epoch 51 - iter 24/24 - loss 0.15374436 - time (sec): 25.33 - samples/sec: 3741.66 - lr: 0.000000\n",
            "2023-04-21 17:22:50,640 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:22:50,642 EPOCH 51 done: loss 0.1537 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:23:00,807 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:23:03,850 TRAIN : loss 0.12637798488140106 - f1-score (weighted avg)  0.8317\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.17it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:23:05,391 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:23:05,459 DEV : loss 0.2702518403530121 - f1-score (weighted avg)  0.7081\n",
            "2023-04-21 17:23:05,478 Epoch    51: reducing learning rate of group 0 to 1.7592e-07.\n",
            "2023-04-21 17:23:05,480 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:23:05,483 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:23:07,533 epoch 52 - iter 2/24 - loss 0.11635596 - time (sec): 2.05 - samples/sec: 3442.73 - lr: 0.000000\n",
            "2023-04-21 17:23:09,689 epoch 52 - iter 4/24 - loss 0.11204622 - time (sec): 4.20 - samples/sec: 3746.10 - lr: 0.000000\n",
            "2023-04-21 17:23:11,706 epoch 52 - iter 6/24 - loss 0.12299628 - time (sec): 6.22 - samples/sec: 3721.20 - lr: 0.000000\n",
            "2023-04-21 17:23:14,102 epoch 52 - iter 8/24 - loss 0.15296259 - time (sec): 8.62 - samples/sec: 3703.76 - lr: 0.000000\n",
            "2023-04-21 17:23:16,463 epoch 52 - iter 10/24 - loss 0.16030778 - time (sec): 10.98 - samples/sec: 3695.23 - lr: 0.000000\n",
            "2023-04-21 17:23:18,581 epoch 52 - iter 12/24 - loss 0.15930061 - time (sec): 13.10 - samples/sec: 3606.77 - lr: 0.000000\n",
            "2023-04-21 17:23:20,676 epoch 52 - iter 14/24 - loss 0.15458218 - time (sec): 15.19 - samples/sec: 3605.96 - lr: 0.000000\n",
            "2023-04-21 17:23:22,786 epoch 52 - iter 16/24 - loss 0.14966074 - time (sec): 17.30 - samples/sec: 3705.85 - lr: 0.000000\n",
            "2023-04-21 17:23:24,880 epoch 52 - iter 18/24 - loss 0.14783551 - time (sec): 19.39 - samples/sec: 3737.40 - lr: 0.000000\n",
            "2023-04-21 17:23:26,951 epoch 52 - iter 20/24 - loss 0.14786396 - time (sec): 21.47 - samples/sec: 3756.05 - lr: 0.000000\n",
            "2023-04-21 17:23:28,989 epoch 52 - iter 22/24 - loss 0.14795457 - time (sec): 23.50 - samples/sec: 3777.49 - lr: 0.000000\n",
            "2023-04-21 17:23:30,736 epoch 52 - iter 24/24 - loss 0.15033049 - time (sec): 25.25 - samples/sec: 3753.11 - lr: 0.000000\n",
            "2023-04-21 17:23:30,740 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:23:30,741 EPOCH 52 done: loss 0.1503 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:23:40,947 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:23:44,032 TRAIN : loss 0.1265721470117569 - f1-score (weighted avg)  0.8315\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.20it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:23:45,550 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:23:45,617 DEV : loss 0.2704090476036072 - f1-score (weighted avg)  0.7093\n",
            "2023-04-21 17:23:45,637 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:23:45,641 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:23:47,747 epoch 53 - iter 2/24 - loss 0.12593138 - time (sec): 2.10 - samples/sec: 3720.61 - lr: 0.000000\n",
            "2023-04-21 17:23:49,840 epoch 53 - iter 4/24 - loss 0.14658089 - time (sec): 4.20 - samples/sec: 3849.97 - lr: 0.000000\n",
            "2023-04-21 17:23:51,909 epoch 53 - iter 6/24 - loss 0.14241342 - time (sec): 6.27 - samples/sec: 3900.98 - lr: 0.000000\n",
            "2023-04-21 17:23:54,278 epoch 53 - iter 8/24 - loss 0.17679577 - time (sec): 8.64 - samples/sec: 3845.72 - lr: 0.000000\n",
            "2023-04-21 17:23:56,294 epoch 53 - iter 10/24 - loss 0.17036356 - time (sec): 10.65 - samples/sec: 3764.02 - lr: 0.000000\n",
            "2023-04-21 17:23:58,488 epoch 53 - iter 12/24 - loss 0.16539461 - time (sec): 12.85 - samples/sec: 3781.49 - lr: 0.000000\n",
            "2023-04-21 17:24:00,616 epoch 53 - iter 14/24 - loss 0.16156854 - time (sec): 14.97 - samples/sec: 3834.61 - lr: 0.000000\n",
            "2023-04-21 17:24:02,885 epoch 53 - iter 16/24 - loss 0.16214576 - time (sec): 17.24 - samples/sec: 3843.84 - lr: 0.000000\n",
            "2023-04-21 17:24:04,950 epoch 53 - iter 18/24 - loss 0.15701906 - time (sec): 19.31 - samples/sec: 3802.61 - lr: 0.000000\n",
            "2023-04-21 17:24:06,985 epoch 53 - iter 20/24 - loss 0.15658595 - time (sec): 21.34 - samples/sec: 3749.32 - lr: 0.000000\n",
            "2023-04-21 17:24:09,105 epoch 53 - iter 22/24 - loss 0.15356089 - time (sec): 23.46 - samples/sec: 3755.71 - lr: 0.000000\n",
            "2023-04-21 17:24:10,949 epoch 53 - iter 24/24 - loss 0.15207344 - time (sec): 25.31 - samples/sec: 3744.99 - lr: 0.000000\n",
            "2023-04-21 17:24:10,951 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:24:10,953 EPOCH 53 done: loss 0.1521 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:24:21,141 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:24:24,188 TRAIN : loss 0.12522505223751068 - f1-score (weighted avg)  0.8335\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.16it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:24:25,740 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:24:25,808 DEV : loss 0.27019989490509033 - f1-score (weighted avg)  0.7078\n",
            "2023-04-21 17:24:25,827 Epoch    53: reducing learning rate of group 0 to 1.4074e-07.\n",
            "2023-04-21 17:24:25,829 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:24:25,832 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:24:27,955 epoch 54 - iter 2/24 - loss 0.15031607 - time (sec): 2.12 - samples/sec: 3622.85 - lr: 0.000000\n",
            "2023-04-21 17:24:30,054 epoch 54 - iter 4/24 - loss 0.14411525 - time (sec): 4.22 - samples/sec: 3696.05 - lr: 0.000000\n",
            "2023-04-21 17:24:32,148 epoch 54 - iter 6/24 - loss 0.13967864 - time (sec): 6.31 - samples/sec: 3607.03 - lr: 0.000000\n",
            "2023-04-21 17:24:34,224 epoch 54 - iter 8/24 - loss 0.14423433 - time (sec): 8.39 - samples/sec: 3631.48 - lr: 0.000000\n",
            "2023-04-21 17:24:36,394 epoch 54 - iter 10/24 - loss 0.14281616 - time (sec): 10.56 - samples/sec: 3662.18 - lr: 0.000000\n",
            "2023-04-21 17:24:38,646 epoch 54 - iter 12/24 - loss 0.15133104 - time (sec): 12.81 - samples/sec: 3656.98 - lr: 0.000000\n",
            "2023-04-21 17:24:40,743 epoch 54 - iter 14/24 - loss 0.14626229 - time (sec): 14.91 - samples/sec: 3658.35 - lr: 0.000000\n",
            "2023-04-21 17:24:42,895 epoch 54 - iter 16/24 - loss 0.14438975 - time (sec): 17.06 - samples/sec: 3724.56 - lr: 0.000000\n",
            "2023-04-21 17:24:44,935 epoch 54 - iter 18/24 - loss 0.14414099 - time (sec): 19.10 - samples/sec: 3715.77 - lr: 0.000000\n",
            "2023-04-21 17:24:47,022 epoch 54 - iter 20/24 - loss 0.14303649 - time (sec): 21.19 - samples/sec: 3717.56 - lr: 0.000000\n",
            "2023-04-21 17:24:49,094 epoch 54 - iter 22/24 - loss 0.14418363 - time (sec): 23.26 - samples/sec: 3707.00 - lr: 0.000000\n",
            "2023-04-21 17:24:51,180 epoch 54 - iter 24/24 - loss 0.15183075 - time (sec): 25.35 - samples/sec: 3739.08 - lr: 0.000000\n",
            "2023-04-21 17:24:51,183 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:24:51,185 EPOCH 54 done: loss 0.1518 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:25:01,400 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:25:04,468 TRAIN : loss 0.12545932829380035 - f1-score (weighted avg)  0.8331\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.17it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:25:06,009 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:25:06,078 DEV : loss 0.27055057883262634 - f1-score (weighted avg)  0.7094\n",
            "2023-04-21 17:25:06,097 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:25:06,101 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:25:08,185 epoch 55 - iter 2/24 - loss 0.13520731 - time (sec): 2.08 - samples/sec: 3784.85 - lr: 0.000000\n",
            "2023-04-21 17:25:10,268 epoch 55 - iter 4/24 - loss 0.13442247 - time (sec): 4.16 - samples/sec: 3637.80 - lr: 0.000000\n",
            "2023-04-21 17:25:12,366 epoch 55 - iter 6/24 - loss 0.14047342 - time (sec): 6.26 - samples/sec: 3644.01 - lr: 0.000000\n",
            "2023-04-21 17:25:14,448 epoch 55 - iter 8/24 - loss 0.13747660 - time (sec): 8.35 - samples/sec: 3732.71 - lr: 0.000000\n",
            "2023-04-21 17:25:16,804 epoch 55 - iter 10/24 - loss 0.15806364 - time (sec): 10.70 - samples/sec: 3728.25 - lr: 0.000000\n",
            "2023-04-21 17:25:19,073 epoch 55 - iter 12/24 - loss 0.16178419 - time (sec): 12.97 - samples/sec: 3825.52 - lr: 0.000000\n",
            "2023-04-21 17:25:21,179 epoch 55 - iter 14/24 - loss 0.15855406 - time (sec): 15.08 - samples/sec: 3761.09 - lr: 0.000000\n",
            "2023-04-21 17:25:23,219 epoch 55 - iter 16/24 - loss 0.15603773 - time (sec): 17.12 - samples/sec: 3730.98 - lr: 0.000000\n",
            "2023-04-21 17:25:25,372 epoch 55 - iter 18/24 - loss 0.15728374 - time (sec): 19.27 - samples/sec: 3716.06 - lr: 0.000000\n",
            "2023-04-21 17:25:27,426 epoch 55 - iter 20/24 - loss 0.15395684 - time (sec): 21.32 - samples/sec: 3711.82 - lr: 0.000000\n",
            "2023-04-21 17:25:29,542 epoch 55 - iter 22/24 - loss 0.15174925 - time (sec): 23.44 - samples/sec: 3738.50 - lr: 0.000000\n",
            "2023-04-21 17:25:31,305 epoch 55 - iter 24/24 - loss 0.15088304 - time (sec): 25.20 - samples/sec: 3760.40 - lr: 0.000000\n",
            "2023-04-21 17:25:31,308 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:25:31,309 EPOCH 55 done: loss 0.1509 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:25:41,494 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:25:43,318 TRAIN : loss 0.1248856633901596 - f1-score (weighted avg)  0.834\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.18it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:25:44,847 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:25:44,913 DEV : loss 0.2707728445529938 - f1-score (weighted avg)  0.7096\n",
            "2023-04-21 17:25:44,932 Epoch    55: reducing learning rate of group 0 to 1.1259e-07.\n",
            "2023-04-21 17:25:44,934 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:25:44,937 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:25:48,035 epoch 56 - iter 2/24 - loss 0.12242870 - time (sec): 3.10 - samples/sec: 2335.02 - lr: 0.000000\n",
            "2023-04-21 17:25:50,090 epoch 56 - iter 4/24 - loss 0.13072213 - time (sec): 5.15 - samples/sec: 3023.09 - lr: 0.000000\n",
            "2023-04-21 17:25:52,183 epoch 56 - iter 6/24 - loss 0.12553852 - time (sec): 7.24 - samples/sec: 3248.26 - lr: 0.000000\n",
            "2023-04-21 17:25:54,223 epoch 56 - iter 8/24 - loss 0.12727240 - time (sec): 9.28 - samples/sec: 3319.12 - lr: 0.000000\n",
            "2023-04-21 17:25:56,650 epoch 56 - iter 10/24 - loss 0.16052459 - time (sec): 11.71 - samples/sec: 3400.81 - lr: 0.000000\n",
            "2023-04-21 17:25:58,714 epoch 56 - iter 12/24 - loss 0.15787583 - time (sec): 13.78 - samples/sec: 3432.51 - lr: 0.000000\n",
            "2023-04-21 17:26:00,813 epoch 56 - iter 14/24 - loss 0.15284817 - time (sec): 15.87 - samples/sec: 3457.96 - lr: 0.000000\n",
            "2023-04-21 17:26:02,850 epoch 56 - iter 16/24 - loss 0.14973984 - time (sec): 17.91 - samples/sec: 3456.92 - lr: 0.000000\n",
            "2023-04-21 17:26:04,986 epoch 56 - iter 18/24 - loss 0.14980207 - time (sec): 20.05 - samples/sec: 3575.74 - lr: 0.000000\n",
            "2023-04-21 17:26:07,174 epoch 56 - iter 20/24 - loss 0.14986822 - time (sec): 22.23 - samples/sec: 3596.73 - lr: 0.000000\n",
            "2023-04-21 17:26:09,303 epoch 56 - iter 22/24 - loss 0.14862506 - time (sec): 24.36 - samples/sec: 3609.38 - lr: 0.000000\n",
            "2023-04-21 17:26:11,104 epoch 56 - iter 24/24 - loss 0.14932285 - time (sec): 26.16 - samples/sec: 3622.08 - lr: 0.000000\n",
            "2023-04-21 17:26:11,107 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:26:11,108 EPOCH 56 done: loss 0.1493 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:26:21,299 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:26:24,322 TRAIN : loss 0.12492360919713974 - f1-score (weighted avg)  0.834\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.20it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:26:25,842 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:26:25,909 DEV : loss 0.27095913887023926 - f1-score (weighted avg)  0.7094\n",
            "2023-04-21 17:26:25,930 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:26:25,934 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:26:28,088 epoch 57 - iter 2/24 - loss 0.14551478 - time (sec): 2.15 - samples/sec: 3635.24 - lr: 0.000000\n",
            "2023-04-21 17:26:30,276 epoch 57 - iter 4/24 - loss 0.13976386 - time (sec): 4.34 - samples/sec: 3545.12 - lr: 0.000000\n",
            "2023-04-21 17:26:32,531 epoch 57 - iter 6/24 - loss 0.14756035 - time (sec): 6.60 - samples/sec: 3547.08 - lr: 0.000000\n",
            "2023-04-21 17:26:34,617 epoch 57 - iter 8/24 - loss 0.14162785 - time (sec): 8.68 - samples/sec: 3704.94 - lr: 0.000000\n",
            "2023-04-21 17:26:36,666 epoch 57 - iter 10/24 - loss 0.14240678 - time (sec): 10.73 - samples/sec: 3671.89 - lr: 0.000000\n",
            "2023-04-21 17:26:38,730 epoch 57 - iter 12/24 - loss 0.14021609 - time (sec): 12.79 - samples/sec: 3717.96 - lr: 0.000000\n",
            "2023-04-21 17:26:41,130 epoch 57 - iter 14/24 - loss 0.15517245 - time (sec): 15.19 - samples/sec: 3730.99 - lr: 0.000000\n",
            "2023-04-21 17:26:43,265 epoch 57 - iter 16/24 - loss 0.15607887 - time (sec): 17.33 - samples/sec: 3698.12 - lr: 0.000000\n",
            "2023-04-21 17:26:45,375 epoch 57 - iter 18/24 - loss 0.15485253 - time (sec): 19.44 - samples/sec: 3683.51 - lr: 0.000000\n",
            "2023-04-21 17:26:47,493 epoch 57 - iter 20/24 - loss 0.15408642 - time (sec): 21.56 - samples/sec: 3711.38 - lr: 0.000000\n",
            "2023-04-21 17:26:49,534 epoch 57 - iter 22/24 - loss 0.15330935 - time (sec): 23.60 - samples/sec: 3716.92 - lr: 0.000000\n",
            "2023-04-21 17:26:51,292 epoch 57 - iter 24/24 - loss 0.15154504 - time (sec): 25.36 - samples/sec: 3737.44 - lr: 0.000000\n",
            "2023-04-21 17:26:51,295 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:26:51,297 EPOCH 57 done: loss 0.1515 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:27:01,509 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:27:04,539 TRAIN : loss 0.12523958086967468 - f1-score (weighted avg)  0.8336\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.19it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:27:06,061 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:27:06,130 DEV : loss 0.27104467153549194 - f1-score (weighted avg)  0.7091\n",
            "2023-04-21 17:27:06,149 Epoch    57: reducing learning rate of group 0 to 9.0072e-08.\n",
            "2023-04-21 17:27:06,151 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:27:06,154 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:27:08,258 epoch 58 - iter 2/24 - loss 0.14915834 - time (sec): 2.10 - samples/sec: 3653.32 - lr: 0.000000\n",
            "2023-04-21 17:27:10,369 epoch 58 - iter 4/24 - loss 0.13587778 - time (sec): 4.21 - samples/sec: 3611.75 - lr: 0.000000\n",
            "2023-04-21 17:27:12,517 epoch 58 - iter 6/24 - loss 0.13627196 - time (sec): 6.36 - samples/sec: 3910.84 - lr: 0.000000\n",
            "2023-04-21 17:27:14,560 epoch 58 - iter 8/24 - loss 0.13846063 - time (sec): 8.40 - samples/sec: 3873.52 - lr: 0.000000\n",
            "2023-04-21 17:27:16,615 epoch 58 - iter 10/24 - loss 0.13800713 - time (sec): 10.46 - samples/sec: 3832.40 - lr: 0.000000\n",
            "2023-04-21 17:27:18,744 epoch 58 - iter 12/24 - loss 0.13940030 - time (sec): 12.59 - samples/sec: 3857.95 - lr: 0.000000\n",
            "2023-04-21 17:27:20,814 epoch 58 - iter 14/24 - loss 0.13803719 - time (sec): 14.66 - samples/sec: 3825.12 - lr: 0.000000\n",
            "2023-04-21 17:27:23,077 epoch 58 - iter 16/24 - loss 0.14461916 - time (sec): 16.92 - samples/sec: 3821.30 - lr: 0.000000\n",
            "2023-04-21 17:27:25,139 epoch 58 - iter 18/24 - loss 0.14234658 - time (sec): 18.98 - samples/sec: 3792.03 - lr: 0.000000\n",
            "2023-04-21 17:27:27,175 epoch 58 - iter 20/24 - loss 0.14057490 - time (sec): 21.02 - samples/sec: 3776.25 - lr: 0.000000\n",
            "2023-04-21 17:27:29,582 epoch 58 - iter 22/24 - loss 0.14940845 - time (sec): 23.43 - samples/sec: 3738.96 - lr: 0.000000\n",
            "2023-04-21 17:27:31,414 epoch 58 - iter 24/24 - loss 0.14952841 - time (sec): 25.26 - samples/sec: 3752.06 - lr: 0.000000\n",
            "2023-04-21 17:27:31,417 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:27:31,418 EPOCH 58 done: loss 0.1495 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.37it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:27:41,667 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:27:44,763 TRAIN : loss 0.12519559264183044 - f1-score (weighted avg)  0.8339\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.14it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:27:46,320 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:27:46,389 DEV : loss 0.27106931805610657 - f1-score (weighted avg)  0.7096\n",
            "2023-04-21 17:27:46,409 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:27:46,413 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:27:48,496 epoch 59 - iter 2/24 - loss 0.14704487 - time (sec): 2.08 - samples/sec: 4038.44 - lr: 0.000000\n",
            "2023-04-21 17:27:50,868 epoch 59 - iter 4/24 - loss 0.18421822 - time (sec): 4.45 - samples/sec: 3780.82 - lr: 0.000000\n",
            "2023-04-21 17:27:52,903 epoch 59 - iter 6/24 - loss 0.17373608 - time (sec): 6.49 - samples/sec: 3733.46 - lr: 0.000000\n",
            "2023-04-21 17:27:54,981 epoch 59 - iter 8/24 - loss 0.17003673 - time (sec): 8.57 - samples/sec: 3729.87 - lr: 0.000000\n",
            "2023-04-21 17:27:57,270 epoch 59 - iter 10/24 - loss 0.16663637 - time (sec): 10.86 - samples/sec: 3718.05 - lr: 0.000000\n",
            "2023-04-21 17:27:59,341 epoch 59 - iter 12/24 - loss 0.16321005 - time (sec): 12.93 - samples/sec: 3709.03 - lr: 0.000000\n",
            "2023-04-21 17:28:01,431 epoch 59 - iter 14/24 - loss 0.16050897 - time (sec): 15.02 - samples/sec: 3688.03 - lr: 0.000000\n",
            "2023-04-21 17:28:03,504 epoch 59 - iter 16/24 - loss 0.15813563 - time (sec): 17.09 - samples/sec: 3700.86 - lr: 0.000000\n",
            "2023-04-21 17:28:05,646 epoch 59 - iter 18/24 - loss 0.15690710 - time (sec): 19.23 - samples/sec: 3747.04 - lr: 0.000000\n",
            "2023-04-21 17:28:07,706 epoch 59 - iter 20/24 - loss 0.15434290 - time (sec): 21.29 - samples/sec: 3716.59 - lr: 0.000000\n",
            "2023-04-21 17:28:09,923 epoch 59 - iter 22/24 - loss 0.15382667 - time (sec): 23.51 - samples/sec: 3725.16 - lr: 0.000000\n",
            "2023-04-21 17:28:11,762 epoch 59 - iter 24/24 - loss 0.15102209 - time (sec): 25.35 - samples/sec: 3738.89 - lr: 0.000000\n",
            "2023-04-21 17:28:11,764 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:28:11,766 EPOCH 59 done: loss 0.1510 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:28:21,992 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:28:25,053 TRAIN : loss 0.12478934973478317 - f1-score (weighted avg)  0.8341\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.17it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:28:26,589 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:28:26,658 DEV : loss 0.27102187275886536 - f1-score (weighted avg)  0.7094\n",
            "2023-04-21 17:28:26,677 Epoch    59: reducing learning rate of group 0 to 7.2058e-08.\n",
            "2023-04-21 17:28:26,679 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:28:26,683 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:28:28,929 epoch 60 - iter 2/24 - loss 0.18108234 - time (sec): 2.24 - samples/sec: 3559.09 - lr: 0.000000\n",
            "2023-04-21 17:28:31,031 epoch 60 - iter 4/24 - loss 0.16329052 - time (sec): 4.35 - samples/sec: 3533.68 - lr: 0.000000\n",
            "2023-04-21 17:28:33,189 epoch 60 - iter 6/24 - loss 0.16027510 - time (sec): 6.51 - samples/sec: 3427.94 - lr: 0.000000\n",
            "2023-04-21 17:28:35,353 epoch 60 - iter 8/24 - loss 0.15184601 - time (sec): 8.67 - samples/sec: 3559.43 - lr: 0.000000\n",
            "2023-04-21 17:28:37,431 epoch 60 - iter 10/24 - loss 0.14692804 - time (sec): 10.75 - samples/sec: 3611.24 - lr: 0.000000\n",
            "2023-04-21 17:28:39,518 epoch 60 - iter 12/24 - loss 0.14521487 - time (sec): 12.83 - samples/sec: 3619.40 - lr: 0.000000\n",
            "2023-04-21 17:28:41,633 epoch 60 - iter 14/24 - loss 0.14329011 - time (sec): 14.95 - samples/sec: 3658.09 - lr: 0.000000\n",
            "2023-04-21 17:28:43,708 epoch 60 - iter 16/24 - loss 0.14136115 - time (sec): 17.02 - samples/sec: 3650.85 - lr: 0.000000\n",
            "2023-04-21 17:28:45,794 epoch 60 - iter 18/24 - loss 0.14034085 - time (sec): 19.11 - samples/sec: 3665.70 - lr: 0.000000\n",
            "2023-04-21 17:28:47,823 epoch 60 - iter 20/24 - loss 0.14143298 - time (sec): 21.14 - samples/sec: 3673.40 - lr: 0.000000\n",
            "2023-04-21 17:28:50,221 epoch 60 - iter 22/24 - loss 0.14847592 - time (sec): 23.54 - samples/sec: 3745.49 - lr: 0.000000\n",
            "2023-04-21 17:28:51,978 epoch 60 - iter 24/24 - loss 0.14920689 - time (sec): 25.29 - samples/sec: 3746.80 - lr: 0.000000\n",
            "2023-04-21 17:28:51,980 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:28:51,983 EPOCH 60 done: loss 0.1492 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:29:02,151 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:29:03,997 TRAIN : loss 0.12480632960796356 - f1-score (weighted avg)  0.8344\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:02<00:00,  1.25it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:29:06,545 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:29:06,612 DEV : loss 0.27113455533981323 - f1-score (weighted avg)  0.7094\n",
            "2023-04-21 17:29:06,631 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:29:06,634 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:29:08,770 epoch 61 - iter 2/24 - loss 0.15398627 - time (sec): 2.13 - samples/sec: 3961.20 - lr: 0.000000\n",
            "2023-04-21 17:29:10,865 epoch 61 - iter 4/24 - loss 0.13854431 - time (sec): 4.23 - samples/sec: 3874.48 - lr: 0.000000\n",
            "2023-04-21 17:29:12,930 epoch 61 - iter 6/24 - loss 0.14107560 - time (sec): 6.29 - samples/sec: 3720.41 - lr: 0.000000\n",
            "2023-04-21 17:29:14,989 epoch 61 - iter 8/24 - loss 0.14568789 - time (sec): 8.35 - samples/sec: 3779.52 - lr: 0.000000\n",
            "2023-04-21 17:29:17,074 epoch 61 - iter 10/24 - loss 0.14639566 - time (sec): 10.44 - samples/sec: 3740.29 - lr: 0.000000\n",
            "2023-04-21 17:29:19,164 epoch 61 - iter 12/24 - loss 0.14461584 - time (sec): 12.53 - samples/sec: 3735.54 - lr: 0.000000\n",
            "2023-04-21 17:29:21,251 epoch 61 - iter 14/24 - loss 0.13981809 - time (sec): 14.61 - samples/sec: 3777.87 - lr: 0.000000\n",
            "2023-04-21 17:29:23,333 epoch 61 - iter 16/24 - loss 0.14050636 - time (sec): 16.70 - samples/sec: 3818.60 - lr: 0.000000\n",
            "2023-04-21 17:29:25,614 epoch 61 - iter 18/24 - loss 0.14347047 - time (sec): 18.98 - samples/sec: 3759.01 - lr: 0.000000\n",
            "2023-04-21 17:29:27,723 epoch 61 - iter 20/24 - loss 0.14133314 - time (sec): 21.09 - samples/sec: 3766.84 - lr: 0.000000\n",
            "2023-04-21 17:29:30,103 epoch 61 - iter 22/24 - loss 0.15166629 - time (sec): 23.47 - samples/sec: 3751.50 - lr: 0.000000\n",
            "2023-04-21 17:29:31,939 epoch 61 - iter 24/24 - loss 0.15124594 - time (sec): 25.30 - samples/sec: 3745.39 - lr: 0.000000\n",
            "2023-04-21 17:29:31,942 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:29:31,944 EPOCH 61 done: loss 0.1512 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:29:42,132 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:29:45,170 TRAIN : loss 0.12515075504779816 - f1-score (weighted avg)  0.8338\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.22it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:29:46,674 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:29:46,742 DEV : loss 0.2711751163005829 - f1-score (weighted avg)  0.7096\n",
            "2023-04-21 17:29:46,761 Epoch    61: reducing learning rate of group 0 to 5.7646e-08.\n",
            "2023-04-21 17:29:46,762 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:29:46,766 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:29:48,840 epoch 62 - iter 2/24 - loss 0.13217571 - time (sec): 2.07 - samples/sec: 3244.24 - lr: 0.000000\n",
            "2023-04-21 17:29:50,954 epoch 62 - iter 4/24 - loss 0.13546045 - time (sec): 4.19 - samples/sec: 3788.87 - lr: 0.000000\n",
            "2023-04-21 17:29:53,103 epoch 62 - iter 6/24 - loss 0.14243554 - time (sec): 6.34 - samples/sec: 3769.45 - lr: 0.000000\n",
            "2023-04-21 17:29:55,177 epoch 62 - iter 8/24 - loss 0.13913369 - time (sec): 8.41 - samples/sec: 3847.61 - lr: 0.000000\n",
            "2023-04-21 17:29:57,270 epoch 62 - iter 10/24 - loss 0.13811823 - time (sec): 10.50 - samples/sec: 3782.03 - lr: 0.000000\n",
            "2023-04-21 17:29:59,637 epoch 62 - iter 12/24 - loss 0.15765858 - time (sec): 12.87 - samples/sec: 3730.26 - lr: 0.000000\n",
            "2023-04-21 17:30:01,775 epoch 62 - iter 14/24 - loss 0.15377184 - time (sec): 15.01 - samples/sec: 3707.30 - lr: 0.000000\n",
            "2023-04-21 17:30:03,856 epoch 62 - iter 16/24 - loss 0.15283468 - time (sec): 17.09 - samples/sec: 3730.41 - lr: 0.000000\n",
            "2023-04-21 17:30:05,944 epoch 62 - iter 18/24 - loss 0.15031111 - time (sec): 19.18 - samples/sec: 3760.19 - lr: 0.000000\n",
            "2023-04-21 17:30:08,011 epoch 62 - iter 20/24 - loss 0.14951823 - time (sec): 21.24 - samples/sec: 3763.24 - lr: 0.000000\n",
            "2023-04-21 17:30:10,259 epoch 62 - iter 22/24 - loss 0.15230821 - time (sec): 23.49 - samples/sec: 3746.11 - lr: 0.000000\n",
            "2023-04-21 17:30:12,030 epoch 62 - iter 24/24 - loss 0.15012655 - time (sec): 25.26 - samples/sec: 3751.40 - lr: 0.000000\n",
            "2023-04-21 17:30:12,033 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:30:12,034 EPOCH 62 done: loss 0.1501 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.40it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:30:22,188 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:30:25,207 TRAIN : loss 0.12481807172298431 - f1-score (weighted avg)  0.8344\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.21it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:30:26,718 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:30:26,785 DEV : loss 0.2711625099182129 - f1-score (weighted avg)  0.7096\n",
            "2023-04-21 17:30:26,804 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:30:26,808 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:30:28,950 epoch 63 - iter 2/24 - loss 0.18042099 - time (sec): 2.14 - samples/sec: 4307.53 - lr: 0.000000\n",
            "2023-04-21 17:30:31,299 epoch 63 - iter 4/24 - loss 0.20920481 - time (sec): 4.49 - samples/sec: 3915.76 - lr: 0.000000\n",
            "2023-04-21 17:30:33,371 epoch 63 - iter 6/24 - loss 0.18706588 - time (sec): 6.56 - samples/sec: 3843.78 - lr: 0.000000\n",
            "2023-04-21 17:30:35,421 epoch 63 - iter 8/24 - loss 0.17089071 - time (sec): 8.61 - samples/sec: 3835.79 - lr: 0.000000\n",
            "2023-04-21 17:30:37,484 epoch 63 - iter 10/24 - loss 0.16870674 - time (sec): 10.67 - samples/sec: 3718.61 - lr: 0.000000\n",
            "2023-04-21 17:30:39,606 epoch 63 - iter 12/24 - loss 0.16342470 - time (sec): 12.80 - samples/sec: 3677.06 - lr: 0.000000\n",
            "2023-04-21 17:30:41,640 epoch 63 - iter 14/24 - loss 0.15578989 - time (sec): 14.83 - samples/sec: 3689.31 - lr: 0.000000\n",
            "2023-04-21 17:30:43,815 epoch 63 - iter 16/24 - loss 0.15445890 - time (sec): 17.01 - samples/sec: 3758.09 - lr: 0.000000\n",
            "2023-04-21 17:30:45,853 epoch 63 - iter 18/24 - loss 0.14998970 - time (sec): 19.04 - samples/sec: 3734.48 - lr: 0.000000\n",
            "2023-04-21 17:30:47,962 epoch 63 - iter 20/24 - loss 0.14954221 - time (sec): 21.15 - samples/sec: 3779.34 - lr: 0.000000\n",
            "2023-04-21 17:30:50,069 epoch 63 - iter 22/24 - loss 0.14765232 - time (sec): 23.26 - samples/sec: 3766.75 - lr: 0.000000\n",
            "2023-04-21 17:30:51,956 epoch 63 - iter 24/24 - loss 0.14957943 - time (sec): 25.15 - samples/sec: 3768.66 - lr: 0.000000\n",
            "2023-04-21 17:30:51,960 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:30:51,961 EPOCH 63 done: loss 0.1496 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:31:02,152 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:31:05,160 TRAIN : loss 0.12446706742048264 - f1-score (weighted avg)  0.835\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.16it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:31:06,706 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:31:06,774 DEV : loss 0.2711469829082489 - f1-score (weighted avg)  0.7094\n",
            "2023-04-21 17:31:06,794 Epoch    63: reducing learning rate of group 0 to 4.6117e-08.\n",
            "2023-04-21 17:31:06,796 BAD EPOCHS (no improvement): 2\n",
            "2023-04-21 17:31:06,799 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:31:08,902 epoch 64 - iter 2/24 - loss 0.15193391 - time (sec): 2.10 - samples/sec: 4218.83 - lr: 0.000000\n",
            "2023-04-21 17:31:11,021 epoch 64 - iter 4/24 - loss 0.13056189 - time (sec): 4.22 - samples/sec: 3932.46 - lr: 0.000000\n",
            "2023-04-21 17:31:13,119 epoch 64 - iter 6/24 - loss 0.14014756 - time (sec): 6.32 - samples/sec: 3964.10 - lr: 0.000000\n",
            "2023-04-21 17:31:15,225 epoch 64 - iter 8/24 - loss 0.14080066 - time (sec): 8.42 - samples/sec: 3949.45 - lr: 0.000000\n",
            "2023-04-21 17:31:17,292 epoch 64 - iter 10/24 - loss 0.13920549 - time (sec): 10.49 - samples/sec: 3962.69 - lr: 0.000000\n",
            "2023-04-21 17:31:19,338 epoch 64 - iter 12/24 - loss 0.13897816 - time (sec): 12.54 - samples/sec: 3874.35 - lr: 0.000000\n",
            "2023-04-21 17:31:21,398 epoch 64 - iter 14/24 - loss 0.14127544 - time (sec): 14.60 - samples/sec: 3874.25 - lr: 0.000000\n",
            "2023-04-21 17:31:23,457 epoch 64 - iter 16/24 - loss 0.13956265 - time (sec): 16.66 - samples/sec: 3878.12 - lr: 0.000000\n",
            "2023-04-21 17:31:25,514 epoch 64 - iter 18/24 - loss 0.13763029 - time (sec): 18.71 - samples/sec: 3847.69 - lr: 0.000000\n",
            "2023-04-21 17:31:27,567 epoch 64 - iter 20/24 - loss 0.13942362 - time (sec): 20.77 - samples/sec: 3808.43 - lr: 0.000000\n",
            "2023-04-21 17:31:29,789 epoch 64 - iter 22/24 - loss 0.14245581 - time (sec): 22.99 - samples/sec: 3774.42 - lr: 0.000000\n",
            "2023-04-21 17:31:31,928 epoch 64 - iter 24/24 - loss 0.15167155 - time (sec): 25.13 - samples/sec: 3771.61 - lr: 0.000000\n",
            "2023-04-21 17:31:31,931 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:31:31,932 EPOCH 64 done: loss 0.1517 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:31:42,106 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:31:45,139 TRAIN : loss 0.12474220991134644 - f1-score (weighted avg)  0.8345\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.21it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:31:46,649 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:31:46,717 DEV : loss 0.2711397111415863 - f1-score (weighted avg)  0.7094\n",
            "2023-04-21 17:31:46,737 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:31:46,741 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:31:48,869 epoch 65 - iter 2/24 - loss 0.14129040 - time (sec): 2.13 - samples/sec: 3654.96 - lr: 0.000000\n",
            "2023-04-21 17:31:50,961 epoch 65 - iter 4/24 - loss 0.13072520 - time (sec): 4.22 - samples/sec: 3886.71 - lr: 0.000000\n",
            "2023-04-21 17:31:53,006 epoch 65 - iter 6/24 - loss 0.12503115 - time (sec): 6.26 - samples/sec: 3773.34 - lr: 0.000000\n",
            "2023-04-21 17:31:55,069 epoch 65 - iter 8/24 - loss 0.12866093 - time (sec): 8.33 - samples/sec: 3766.90 - lr: 0.000000\n",
            "2023-04-21 17:31:57,101 epoch 65 - iter 10/24 - loss 0.12799262 - time (sec): 10.36 - samples/sec: 3739.99 - lr: 0.000000\n",
            "2023-04-21 17:31:59,651 epoch 65 - iter 12/24 - loss 0.15764544 - time (sec): 12.91 - samples/sec: 3653.75 - lr: 0.000000\n",
            "2023-04-21 17:32:01,768 epoch 65 - iter 14/24 - loss 0.15494864 - time (sec): 15.03 - samples/sec: 3704.94 - lr: 0.000000\n",
            "2023-04-21 17:32:03,883 epoch 65 - iter 16/24 - loss 0.15074236 - time (sec): 17.14 - samples/sec: 3738.86 - lr: 0.000000\n",
            "2023-04-21 17:32:06,034 epoch 65 - iter 18/24 - loss 0.14736271 - time (sec): 19.29 - samples/sec: 3722.34 - lr: 0.000000\n",
            "2023-04-21 17:32:08,211 epoch 65 - iter 20/24 - loss 0.14768298 - time (sec): 21.47 - samples/sec: 3760.87 - lr: 0.000000\n",
            "2023-04-21 17:32:10,306 epoch 65 - iter 22/24 - loss 0.14805862 - time (sec): 23.56 - samples/sec: 3762.97 - lr: 0.000000\n",
            "2023-04-21 17:32:12,035 epoch 65 - iter 24/24 - loss 0.14712818 - time (sec): 25.29 - samples/sec: 3746.87 - lr: 0.000000\n",
            "2023-04-21 17:32:12,038 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:32:12,040 EPOCH 65 done: loss 0.1471 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:32:22,226 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:32:25,283 TRAIN : loss 0.12491733580827713 - f1-score (weighted avg)  0.8342\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.18it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:32:26,812 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:32:26,882 DEV : loss 0.27128058671951294 - f1-score (weighted avg)  0.7096\n",
            "2023-04-21 17:32:26,901 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:32:26,905 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:32:29,149 epoch 66 - iter 2/24 - loss 0.18706668 - time (sec): 2.24 - samples/sec: 3912.38 - lr: 0.000000\n",
            "2023-04-21 17:32:31,509 epoch 66 - iter 4/24 - loss 0.21303839 - time (sec): 4.60 - samples/sec: 3905.63 - lr: 0.000000\n",
            "2023-04-21 17:32:33,569 epoch 66 - iter 6/24 - loss 0.19317773 - time (sec): 6.66 - samples/sec: 3881.51 - lr: 0.000000\n",
            "2023-04-21 17:32:35,623 epoch 66 - iter 8/24 - loss 0.18106014 - time (sec): 8.72 - samples/sec: 3748.16 - lr: 0.000000\n",
            "2023-04-21 17:32:37,702 epoch 66 - iter 10/24 - loss 0.17194626 - time (sec): 10.80 - samples/sec: 3726.45 - lr: 0.000000\n",
            "2023-04-21 17:32:39,922 epoch 66 - iter 12/24 - loss 0.16553575 - time (sec): 13.01 - samples/sec: 3798.11 - lr: 0.000000\n",
            "2023-04-21 17:32:41,998 epoch 66 - iter 14/24 - loss 0.16173715 - time (sec): 15.09 - samples/sec: 3853.39 - lr: 0.000000\n",
            "2023-04-21 17:32:44,090 epoch 66 - iter 16/24 - loss 0.15717797 - time (sec): 17.18 - samples/sec: 3868.23 - lr: 0.000000\n",
            "2023-04-21 17:32:46,143 epoch 66 - iter 18/24 - loss 0.15583641 - time (sec): 19.24 - samples/sec: 3855.97 - lr: 0.000000\n",
            "2023-04-21 17:32:48,242 epoch 66 - iter 20/24 - loss 0.15345302 - time (sec): 21.34 - samples/sec: 3814.02 - lr: 0.000000\n",
            "2023-04-21 17:32:50,320 epoch 66 - iter 22/24 - loss 0.15039340 - time (sec): 23.41 - samples/sec: 3792.12 - lr: 0.000000\n",
            "2023-04-21 17:32:52,105 epoch 66 - iter 24/24 - loss 0.15020566 - time (sec): 25.20 - samples/sec: 3760.92 - lr: 0.000000\n",
            "2023-04-21 17:32:52,108 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:32:52,109 EPOCH 66 done: loss 0.1502 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:33:02,315 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:33:04,146 TRAIN : loss 0.1249786987900734 - f1-score (weighted avg)  0.834\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:02<00:00,  1.26it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:33:06,683 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:33:06,751 DEV : loss 0.2713092863559723 - f1-score (weighted avg)  0.7092\n",
            "2023-04-21 17:33:06,770 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:33:06,773 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:33:08,821 epoch 67 - iter 2/24 - loss 0.13565348 - time (sec): 2.05 - samples/sec: 3587.95 - lr: 0.000000\n",
            "2023-04-21 17:33:10,882 epoch 67 - iter 4/24 - loss 0.12648309 - time (sec): 4.11 - samples/sec: 3786.23 - lr: 0.000000\n",
            "2023-04-21 17:33:13,250 epoch 67 - iter 6/24 - loss 0.16508737 - time (sec): 6.48 - samples/sec: 3705.79 - lr: 0.000000\n",
            "2023-04-21 17:33:15,398 epoch 67 - iter 8/24 - loss 0.15874194 - time (sec): 8.62 - samples/sec: 3714.52 - lr: 0.000000\n",
            "2023-04-21 17:33:17,520 epoch 67 - iter 10/24 - loss 0.14901662 - time (sec): 10.75 - samples/sec: 3821.52 - lr: 0.000000\n",
            "2023-04-21 17:33:19,596 epoch 67 - iter 12/24 - loss 0.14854165 - time (sec): 12.82 - samples/sec: 3837.27 - lr: 0.000000\n",
            "2023-04-21 17:33:21,681 epoch 67 - iter 14/24 - loss 0.14625232 - time (sec): 14.91 - samples/sec: 3849.31 - lr: 0.000000\n",
            "2023-04-21 17:33:23,763 epoch 67 - iter 16/24 - loss 0.14712670 - time (sec): 16.99 - samples/sec: 3809.96 - lr: 0.000000\n",
            "2023-04-21 17:33:25,855 epoch 67 - iter 18/24 - loss 0.14701284 - time (sec): 19.08 - samples/sec: 3761.66 - lr: 0.000000\n",
            "2023-04-21 17:33:28,101 epoch 67 - iter 20/24 - loss 0.14497365 - time (sec): 21.33 - samples/sec: 3772.63 - lr: 0.000000\n",
            "2023-04-21 17:33:30,180 epoch 67 - iter 22/24 - loss 0.14577036 - time (sec): 23.41 - samples/sec: 3751.26 - lr: 0.000000\n",
            "2023-04-21 17:33:32,129 epoch 67 - iter 24/24 - loss 0.14890476 - time (sec): 25.35 - samples/sec: 3737.87 - lr: 0.000000\n",
            "2023-04-21 17:33:32,132 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:33:32,133 EPOCH 67 done: loss 0.1489 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:33:42,357 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:33:45,363 TRAIN : loss 0.12468621879816055 - f1-score (weighted avg)  0.8348\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.17it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:33:46,901 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:33:46,967 DEV : loss 0.2713741362094879 - f1-score (weighted avg)  0.7094\n",
            "2023-04-21 17:33:46,986 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:33:46,990 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:33:49,365 epoch 68 - iter 2/24 - loss 0.25116090 - time (sec): 2.37 - samples/sec: 3577.57 - lr: 0.000000\n",
            "2023-04-21 17:33:51,651 epoch 68 - iter 4/24 - loss 0.21085132 - time (sec): 4.66 - samples/sec: 3706.92 - lr: 0.000000\n",
            "2023-04-21 17:33:53,679 epoch 68 - iter 6/24 - loss 0.19370565 - time (sec): 6.69 - samples/sec: 3707.31 - lr: 0.000000\n",
            "2023-04-21 17:33:55,752 epoch 68 - iter 8/24 - loss 0.17984430 - time (sec): 8.76 - samples/sec: 3668.03 - lr: 0.000000\n",
            "2023-04-21 17:33:57,888 epoch 68 - iter 10/24 - loss 0.17018515 - time (sec): 10.90 - samples/sec: 3732.20 - lr: 0.000000\n",
            "2023-04-21 17:33:59,920 epoch 68 - iter 12/24 - loss 0.16683159 - time (sec): 12.93 - samples/sec: 3651.22 - lr: 0.000000\n",
            "2023-04-21 17:34:01,983 epoch 68 - iter 14/24 - loss 0.16231678 - time (sec): 14.99 - samples/sec: 3686.75 - lr: 0.000000\n",
            "2023-04-21 17:34:04,083 epoch 68 - iter 16/24 - loss 0.16039981 - time (sec): 17.09 - samples/sec: 3724.67 - lr: 0.000000\n",
            "2023-04-21 17:34:06,249 epoch 68 - iter 18/24 - loss 0.15806714 - time (sec): 19.26 - samples/sec: 3769.37 - lr: 0.000000\n",
            "2023-04-21 17:34:08,349 epoch 68 - iter 20/24 - loss 0.15420227 - time (sec): 21.36 - samples/sec: 3739.74 - lr: 0.000000\n",
            "2023-04-21 17:34:10,447 epoch 68 - iter 22/24 - loss 0.15203116 - time (sec): 23.46 - samples/sec: 3762.85 - lr: 0.000000\n",
            "2023-04-21 17:34:12,208 epoch 68 - iter 24/24 - loss 0.15046876 - time (sec): 25.22 - samples/sec: 3758.28 - lr: 0.000000\n",
            "2023-04-21 17:34:12,211 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:34:12,212 EPOCH 68 done: loss 0.1505 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.40it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:34:22,364 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:34:25,414 TRAIN : loss 0.12473449856042862 - f1-score (weighted avg)  0.8346\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.21it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:34:26,927 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:34:26,995 DEV : loss 0.2713915705680847 - f1-score (weighted avg)  0.7096\n",
            "2023-04-21 17:34:27,013 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:34:27,025 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:34:29,573 epoch 69 - iter 2/24 - loss 0.28203122 - time (sec): 2.55 - samples/sec: 3506.65 - lr: 0.000000\n",
            "2023-04-21 17:34:31,699 epoch 69 - iter 4/24 - loss 0.20353789 - time (sec): 4.67 - samples/sec: 3718.76 - lr: 0.000000\n",
            "2023-04-21 17:34:33,784 epoch 69 - iter 6/24 - loss 0.18227364 - time (sec): 6.76 - samples/sec: 3770.87 - lr: 0.000000\n",
            "2023-04-21 17:34:35,899 epoch 69 - iter 8/24 - loss 0.16550405 - time (sec): 8.87 - samples/sec: 3833.87 - lr: 0.000000\n",
            "2023-04-21 17:34:38,017 epoch 69 - iter 10/24 - loss 0.16595664 - time (sec): 10.99 - samples/sec: 3757.60 - lr: 0.000000\n",
            "2023-04-21 17:34:40,170 epoch 69 - iter 12/24 - loss 0.16112162 - time (sec): 13.14 - samples/sec: 3704.19 - lr: 0.000000\n",
            "2023-04-21 17:34:42,253 epoch 69 - iter 14/24 - loss 0.15789289 - time (sec): 15.23 - samples/sec: 3655.35 - lr: 0.000000\n",
            "2023-04-21 17:34:44,386 epoch 69 - iter 16/24 - loss 0.15579204 - time (sec): 17.36 - samples/sec: 3664.46 - lr: 0.000000\n",
            "2023-04-21 17:34:46,421 epoch 69 - iter 18/24 - loss 0.15252985 - time (sec): 19.39 - samples/sec: 3667.19 - lr: 0.000000\n",
            "2023-04-21 17:34:48,526 epoch 69 - iter 20/24 - loss 0.14990412 - time (sec): 21.50 - samples/sec: 3695.96 - lr: 0.000000\n",
            "2023-04-21 17:34:50,584 epoch 69 - iter 22/24 - loss 0.15025153 - time (sec): 23.56 - samples/sec: 3726.12 - lr: 0.000000\n",
            "2023-04-21 17:34:52,346 epoch 69 - iter 24/24 - loss 0.14999657 - time (sec): 25.32 - samples/sec: 3743.08 - lr: 0.000000\n",
            "2023-04-21 17:34:52,349 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:34:52,350 EPOCH 69 done: loss 0.1500 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:35:02,547 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:35:05,600 TRAIN : loss 0.12464357167482376 - f1-score (weighted avg)  0.8347\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.20it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:35:07,121 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:35:07,189 DEV : loss 0.27138805389404297 - f1-score (weighted avg)  0.7096\n",
            "2023-04-21 17:35:07,208 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:35:07,212 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:35:09,304 epoch 70 - iter 2/24 - loss 0.11991895 - time (sec): 2.09 - samples/sec: 3630.68 - lr: 0.000000\n",
            "2023-04-21 17:35:11,411 epoch 70 - iter 4/24 - loss 0.13680889 - time (sec): 4.20 - samples/sec: 3926.73 - lr: 0.000000\n",
            "2023-04-21 17:35:13,511 epoch 70 - iter 6/24 - loss 0.13347359 - time (sec): 6.30 - samples/sec: 3867.32 - lr: 0.000000\n",
            "2023-04-21 17:35:15,674 epoch 70 - iter 8/24 - loss 0.13451576 - time (sec): 8.46 - samples/sec: 3880.77 - lr: 0.000000\n",
            "2023-04-21 17:35:17,779 epoch 70 - iter 10/24 - loss 0.13802149 - time (sec): 10.57 - samples/sec: 3820.69 - lr: 0.000000\n",
            "2023-04-21 17:35:19,843 epoch 70 - iter 12/24 - loss 0.13864608 - time (sec): 12.63 - samples/sec: 3804.07 - lr: 0.000000\n",
            "2023-04-21 17:35:21,911 epoch 70 - iter 14/24 - loss 0.14133315 - time (sec): 14.70 - samples/sec: 3816.11 - lr: 0.000000\n",
            "2023-04-21 17:35:24,290 epoch 70 - iter 16/24 - loss 0.15544940 - time (sec): 17.08 - samples/sec: 3763.86 - lr: 0.000000\n",
            "2023-04-21 17:35:26,324 epoch 70 - iter 18/24 - loss 0.14981471 - time (sec): 19.11 - samples/sec: 3748.48 - lr: 0.000000\n",
            "2023-04-21 17:35:28,450 epoch 70 - iter 20/24 - loss 0.14860916 - time (sec): 21.24 - samples/sec: 3738.82 - lr: 0.000000\n",
            "2023-04-21 17:35:30,527 epoch 70 - iter 22/24 - loss 0.14689124 - time (sec): 23.31 - samples/sec: 3759.67 - lr: 0.000000\n",
            "2023-04-21 17:35:32,527 epoch 70 - iter 24/24 - loss 0.15000433 - time (sec): 25.31 - samples/sec: 3743.82 - lr: 0.000000\n",
            "2023-04-21 17:35:32,530 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:35:32,531 EPOCH 70 done: loss 0.1500 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:35:42,734 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:35:45,806 TRAIN : loss 0.12457387894392014 - f1-score (weighted avg)  0.8349\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.16it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:35:47,352 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:35:47,421 DEV : loss 0.2713777422904968 - f1-score (weighted avg)  0.7096\n",
            "2023-04-21 17:35:47,441 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:35:47,446 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:35:49,919 epoch 71 - iter 2/24 - loss 0.23221548 - time (sec): 2.47 - samples/sec: 3536.34 - lr: 0.000000\n",
            "2023-04-21 17:35:51,978 epoch 71 - iter 4/24 - loss 0.17904432 - time (sec): 4.53 - samples/sec: 3586.99 - lr: 0.000000\n",
            "2023-04-21 17:35:54,118 epoch 71 - iter 6/24 - loss 0.16668795 - time (sec): 6.67 - samples/sec: 3716.83 - lr: 0.000000\n",
            "2023-04-21 17:35:56,222 epoch 71 - iter 8/24 - loss 0.15934786 - time (sec): 8.77 - samples/sec: 3752.62 - lr: 0.000000\n",
            "2023-04-21 17:35:58,320 epoch 71 - iter 10/24 - loss 0.15444999 - time (sec): 10.87 - samples/sec: 3653.01 - lr: 0.000000\n",
            "2023-04-21 17:36:00,407 epoch 71 - iter 12/24 - loss 0.15228587 - time (sec): 12.96 - samples/sec: 3627.40 - lr: 0.000000\n",
            "2023-04-21 17:36:02,450 epoch 71 - iter 14/24 - loss 0.14947328 - time (sec): 15.00 - samples/sec: 3661.03 - lr: 0.000000\n",
            "2023-04-21 17:36:04,516 epoch 71 - iter 16/24 - loss 0.14863868 - time (sec): 17.07 - samples/sec: 3662.09 - lr: 0.000000\n",
            "2023-04-21 17:36:06,583 epoch 71 - iter 18/24 - loss 0.14639917 - time (sec): 19.14 - samples/sec: 3713.93 - lr: 0.000000\n",
            "2023-04-21 17:36:08,833 epoch 71 - iter 20/24 - loss 0.15144555 - time (sec): 21.39 - samples/sec: 3725.66 - lr: 0.000000\n",
            "2023-04-21 17:36:10,939 epoch 71 - iter 22/24 - loss 0.15096509 - time (sec): 23.49 - samples/sec: 3736.82 - lr: 0.000000\n",
            "2023-04-21 17:36:12,706 epoch 71 - iter 24/24 - loss 0.14802903 - time (sec): 25.26 - samples/sec: 3752.00 - lr: 0.000000\n",
            "2023-04-21 17:36:12,709 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:36:12,711 EPOCH 71 done: loss 0.1480 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:36:22,939 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:36:25,991 TRAIN : loss 0.12442554533481598 - f1-score (weighted avg)  0.8352\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.18it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:36:27,519 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:36:27,590 DEV : loss 0.2714381217956543 - f1-score (weighted avg)  0.7092\n",
            "2023-04-21 17:36:27,610 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:36:27,614 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:36:29,970 epoch 72 - iter 2/24 - loss 0.23229100 - time (sec): 2.35 - samples/sec: 3519.88 - lr: 0.000000\n",
            "2023-04-21 17:36:32,141 epoch 72 - iter 4/24 - loss 0.18457235 - time (sec): 4.53 - samples/sec: 3929.28 - lr: 0.000000\n",
            "2023-04-21 17:36:34,237 epoch 72 - iter 6/24 - loss 0.16046830 - time (sec): 6.62 - samples/sec: 3842.30 - lr: 0.000000\n",
            "2023-04-21 17:36:36,362 epoch 72 - iter 8/24 - loss 0.15831190 - time (sec): 8.75 - samples/sec: 3818.39 - lr: 0.000000\n",
            "2023-04-21 17:36:38,459 epoch 72 - iter 10/24 - loss 0.15345369 - time (sec): 10.84 - samples/sec: 3911.81 - lr: 0.000000\n",
            "2023-04-21 17:36:40,577 epoch 72 - iter 12/24 - loss 0.15037556 - time (sec): 12.96 - samples/sec: 3870.66 - lr: 0.000000\n",
            "2023-04-21 17:36:42,822 epoch 72 - iter 14/24 - loss 0.15410474 - time (sec): 15.21 - samples/sec: 3865.54 - lr: 0.000000\n",
            "2023-04-21 17:36:44,928 epoch 72 - iter 16/24 - loss 0.15249279 - time (sec): 17.31 - samples/sec: 3767.11 - lr: 0.000000\n",
            "2023-04-21 17:36:46,995 epoch 72 - iter 18/24 - loss 0.15044470 - time (sec): 19.38 - samples/sec: 3723.90 - lr: 0.000000\n",
            "2023-04-21 17:36:49,047 epoch 72 - iter 20/24 - loss 0.15062748 - time (sec): 21.43 - samples/sec: 3747.64 - lr: 0.000000\n",
            "2023-04-21 17:36:51,078 epoch 72 - iter 22/24 - loss 0.14989908 - time (sec): 23.46 - samples/sec: 3737.48 - lr: 0.000000\n",
            "2023-04-21 17:36:52,876 epoch 72 - iter 24/24 - loss 0.14893241 - time (sec): 25.26 - samples/sec: 3751.81 - lr: 0.000000\n",
            "2023-04-21 17:36:52,879 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:36:52,880 EPOCH 72 done: loss 0.1489 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:37:03,096 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:37:04,924 TRAIN : loss 0.12442406266927719 - f1-score (weighted avg)  0.8352\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:02<00:00,  1.26it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:37:07,462 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:37:07,531 DEV : loss 0.27148303389549255 - f1-score (weighted avg)  0.7095\n",
            "2023-04-21 17:37:07,551 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:37:07,555 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:37:09,876 epoch 73 - iter 2/24 - loss 0.16557088 - time (sec): 2.32 - samples/sec: 3728.94 - lr: 0.000000\n",
            "2023-04-21 17:37:12,045 epoch 73 - iter 4/24 - loss 0.17018465 - time (sec): 4.49 - samples/sec: 3823.63 - lr: 0.000000\n",
            "2023-04-21 17:37:14,151 epoch 73 - iter 6/24 - loss 0.16010167 - time (sec): 6.59 - samples/sec: 3950.70 - lr: 0.000000\n",
            "2023-04-21 17:37:16,190 epoch 73 - iter 8/24 - loss 0.15445899 - time (sec): 8.63 - samples/sec: 3873.32 - lr: 0.000000\n",
            "2023-04-21 17:37:18,604 epoch 73 - iter 10/24 - loss 0.16611365 - time (sec): 11.05 - samples/sec: 3965.44 - lr: 0.000000\n",
            "2023-04-21 17:37:20,660 epoch 73 - iter 12/24 - loss 0.16114041 - time (sec): 13.10 - samples/sec: 3842.28 - lr: 0.000000\n",
            "2023-04-21 17:37:22,699 epoch 73 - iter 14/24 - loss 0.15636495 - time (sec): 15.14 - samples/sec: 3771.74 - lr: 0.000000\n",
            "2023-04-21 17:37:24,761 epoch 73 - iter 16/24 - loss 0.15558245 - time (sec): 17.20 - samples/sec: 3781.31 - lr: 0.000000\n",
            "2023-04-21 17:37:26,863 epoch 73 - iter 18/24 - loss 0.15273638 - time (sec): 19.31 - samples/sec: 3807.94 - lr: 0.000000\n",
            "2023-04-21 17:37:28,954 epoch 73 - iter 20/24 - loss 0.15211588 - time (sec): 21.40 - samples/sec: 3782.27 - lr: 0.000000\n",
            "2023-04-21 17:37:31,055 epoch 73 - iter 22/24 - loss 0.14848661 - time (sec): 23.50 - samples/sec: 3754.22 - lr: 0.000000\n",
            "2023-04-21 17:37:32,799 epoch 73 - iter 24/24 - loss 0.14837642 - time (sec): 25.24 - samples/sec: 3754.46 - lr: 0.000000\n",
            "2023-04-21 17:37:32,802 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:37:32,804 EPOCH 73 done: loss 0.1484 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:37:43,001 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:37:46,011 TRAIN : loss 0.12451758980751038 - f1-score (weighted avg)  0.8353\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.21it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:37:47,523 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:37:47,591 DEV : loss 0.27155181765556335 - f1-score (weighted avg)  0.7099\n",
            "2023-04-21 17:37:47,609 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:37:47,613 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:37:49,656 epoch 74 - iter 2/24 - loss 0.13865146 - time (sec): 2.04 - samples/sec: 3787.25 - lr: 0.000000\n",
            "2023-04-21 17:37:51,881 epoch 74 - iter 4/24 - loss 0.16045122 - time (sec): 4.27 - samples/sec: 3324.79 - lr: 0.000000\n",
            "2023-04-21 17:37:53,912 epoch 74 - iter 6/24 - loss 0.15515764 - time (sec): 6.30 - samples/sec: 3392.36 - lr: 0.000000\n",
            "2023-04-21 17:37:56,049 epoch 74 - iter 8/24 - loss 0.14439856 - time (sec): 8.43 - samples/sec: 3601.27 - lr: 0.000000\n",
            "2023-04-21 17:37:58,084 epoch 74 - iter 10/24 - loss 0.13845369 - time (sec): 10.47 - samples/sec: 3637.29 - lr: 0.000000\n",
            "2023-04-21 17:38:00,137 epoch 74 - iter 12/24 - loss 0.13830877 - time (sec): 12.52 - samples/sec: 3632.15 - lr: 0.000000\n",
            "2023-04-21 17:38:02,294 epoch 74 - iter 14/24 - loss 0.13617619 - time (sec): 14.68 - samples/sec: 3697.69 - lr: 0.000000\n",
            "2023-04-21 17:38:04,405 epoch 74 - iter 16/24 - loss 0.13678572 - time (sec): 16.79 - samples/sec: 3757.87 - lr: 0.000000\n",
            "2023-04-21 17:38:06,590 epoch 74 - iter 18/24 - loss 0.13871342 - time (sec): 18.97 - samples/sec: 3770.75 - lr: 0.000000\n",
            "2023-04-21 17:38:08,971 epoch 74 - iter 20/24 - loss 0.15014791 - time (sec): 21.36 - samples/sec: 3739.81 - lr: 0.000000\n",
            "2023-04-21 17:38:11,048 epoch 74 - iter 22/24 - loss 0.14901071 - time (sec): 23.43 - samples/sec: 3770.61 - lr: 0.000000\n",
            "2023-04-21 17:38:12,851 epoch 74 - iter 24/24 - loss 0.14808806 - time (sec): 25.24 - samples/sec: 3755.39 - lr: 0.000000\n",
            "2023-04-21 17:38:12,854 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:38:12,855 EPOCH 74 done: loss 0.1481 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:38:23,070 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:38:26,110 TRAIN : loss 0.12436036020517349 - f1-score (weighted avg)  0.8354\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.16it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:38:27,654 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:38:27,722 DEV : loss 0.27158552408218384 - f1-score (weighted avg)  0.7098\n",
            "2023-04-21 17:38:27,741 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:38:27,745 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:38:29,903 epoch 75 - iter 2/24 - loss 0.14327748 - time (sec): 2.16 - samples/sec: 3941.90 - lr: 0.000000\n",
            "2023-04-21 17:38:31,959 epoch 75 - iter 4/24 - loss 0.14239713 - time (sec): 4.21 - samples/sec: 3910.39 - lr: 0.000000\n",
            "2023-04-21 17:38:34,069 epoch 75 - iter 6/24 - loss 0.14107976 - time (sec): 6.32 - samples/sec: 3862.42 - lr: 0.000000\n",
            "2023-04-21 17:38:36,075 epoch 75 - iter 8/24 - loss 0.13837214 - time (sec): 8.33 - samples/sec: 3767.20 - lr: 0.000000\n",
            "2023-04-21 17:38:38,130 epoch 75 - iter 10/24 - loss 0.13552904 - time (sec): 10.38 - samples/sec: 3753.01 - lr: 0.000000\n",
            "2023-04-21 17:38:40,192 epoch 75 - iter 12/24 - loss 0.13447859 - time (sec): 12.45 - samples/sec: 3780.57 - lr: 0.000000\n",
            "2023-04-21 17:38:42,550 epoch 75 - iter 14/24 - loss 0.14307641 - time (sec): 14.80 - samples/sec: 3749.64 - lr: 0.000000\n",
            "2023-04-21 17:38:44,950 epoch 75 - iter 16/24 - loss 0.15914182 - time (sec): 17.20 - samples/sec: 3726.72 - lr: 0.000000\n",
            "2023-04-21 17:38:47,007 epoch 75 - iter 18/24 - loss 0.15640597 - time (sec): 19.26 - samples/sec: 3685.68 - lr: 0.000000\n",
            "2023-04-21 17:38:49,123 epoch 75 - iter 20/24 - loss 0.15260402 - time (sec): 21.38 - samples/sec: 3753.01 - lr: 0.000000\n",
            "2023-04-21 17:38:51,204 epoch 75 - iter 22/24 - loss 0.15050137 - time (sec): 23.46 - samples/sec: 3770.41 - lr: 0.000000\n",
            "2023-04-21 17:38:52,947 epoch 75 - iter 24/24 - loss 0.14915069 - time (sec): 25.20 - samples/sec: 3760.51 - lr: 0.000000\n",
            "2023-04-21 17:38:52,950 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:38:52,952 EPOCH 75 done: loss 0.1492 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:39:03,145 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:39:06,198 TRAIN : loss 0.12461801618337631 - f1-score (weighted avg)  0.8349\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.17it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:39:07,733 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:39:07,801 DEV : loss 0.2716026306152344 - f1-score (weighted avg)  0.7094\n",
            "2023-04-21 17:39:07,820 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:39:07,824 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:39:09,863 epoch 76 - iter 2/24 - loss 0.13443294 - time (sec): 2.04 - samples/sec: 3133.29 - lr: 0.000000\n",
            "2023-04-21 17:39:11,930 epoch 76 - iter 4/24 - loss 0.14184089 - time (sec): 4.10 - samples/sec: 3437.59 - lr: 0.000000\n",
            "2023-04-21 17:39:13,998 epoch 76 - iter 6/24 - loss 0.13364167 - time (sec): 6.17 - samples/sec: 3534.16 - lr: 0.000000\n",
            "2023-04-21 17:39:16,079 epoch 76 - iter 8/24 - loss 0.13469627 - time (sec): 8.25 - samples/sec: 3608.47 - lr: 0.000000\n",
            "2023-04-21 17:39:18,431 epoch 76 - iter 10/24 - loss 0.15543443 - time (sec): 10.61 - samples/sec: 3602.29 - lr: 0.000000\n",
            "2023-04-21 17:39:20,780 epoch 76 - iter 12/24 - loss 0.16220869 - time (sec): 12.95 - samples/sec: 3652.82 - lr: 0.000000\n",
            "2023-04-21 17:39:22,878 epoch 76 - iter 14/24 - loss 0.15737373 - time (sec): 15.05 - samples/sec: 3701.63 - lr: 0.000000\n",
            "2023-04-21 17:39:25,024 epoch 76 - iter 16/24 - loss 0.15302649 - time (sec): 17.20 - samples/sec: 3740.18 - lr: 0.000000\n",
            "2023-04-21 17:39:27,095 epoch 76 - iter 18/24 - loss 0.15246184 - time (sec): 19.27 - samples/sec: 3765.34 - lr: 0.000000\n",
            "2023-04-21 17:39:29,208 epoch 76 - iter 20/24 - loss 0.15006973 - time (sec): 21.38 - samples/sec: 3761.52 - lr: 0.000000\n",
            "2023-04-21 17:39:31,324 epoch 76 - iter 22/24 - loss 0.15058057 - time (sec): 23.50 - samples/sec: 3750.52 - lr: 0.000000\n",
            "2023-04-21 17:39:33,151 epoch 76 - iter 24/24 - loss 0.15118622 - time (sec): 25.33 - samples/sec: 3742.10 - lr: 0.000000\n",
            "2023-04-21 17:39:33,154 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:39:33,156 EPOCH 76 done: loss 0.1512 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:39:43,356 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:39:45,194 TRAIN : loss 0.12453605234622955 - f1-score (weighted avg)  0.835\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:02<00:00,  1.26it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:39:47,730 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:39:47,802 DEV : loss 0.27156636118888855 - f1-score (weighted avg)  0.7096\n",
            "2023-04-21 17:39:47,822 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:39:47,826 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:39:49,882 epoch 77 - iter 2/24 - loss 0.12447316 - time (sec): 2.05 - samples/sec: 3924.87 - lr: 0.000000\n",
            "2023-04-21 17:39:51,950 epoch 77 - iter 4/24 - loss 0.11969615 - time (sec): 4.12 - samples/sec: 3906.78 - lr: 0.000000\n",
            "2023-04-21 17:39:54,062 epoch 77 - iter 6/24 - loss 0.12004830 - time (sec): 6.23 - samples/sec: 3727.61 - lr: 0.000000\n",
            "2023-04-21 17:39:56,282 epoch 77 - iter 8/24 - loss 0.12260884 - time (sec): 8.45 - samples/sec: 3734.58 - lr: 0.000000\n",
            "2023-04-21 17:39:58,369 epoch 77 - iter 10/24 - loss 0.12688360 - time (sec): 10.54 - samples/sec: 3781.94 - lr: 0.000000\n",
            "2023-04-21 17:40:00,534 epoch 77 - iter 12/24 - loss 0.13059179 - time (sec): 12.71 - samples/sec: 3802.34 - lr: 0.000000\n",
            "2023-04-21 17:40:02,578 epoch 77 - iter 14/24 - loss 0.13111019 - time (sec): 14.75 - samples/sec: 3735.59 - lr: 0.000000\n",
            "2023-04-21 17:40:04,818 epoch 77 - iter 16/24 - loss 0.13855024 - time (sec): 16.99 - samples/sec: 3791.41 - lr: 0.000000\n",
            "2023-04-21 17:40:06,950 epoch 77 - iter 18/24 - loss 0.13721857 - time (sec): 19.12 - samples/sec: 3782.91 - lr: 0.000000\n",
            "2023-04-21 17:40:09,357 epoch 77 - iter 20/24 - loss 0.15082122 - time (sec): 21.53 - samples/sec: 3722.76 - lr: 0.000000\n",
            "2023-04-21 17:40:11,402 epoch 77 - iter 22/24 - loss 0.14992706 - time (sec): 23.57 - samples/sec: 3721.37 - lr: 0.000000\n",
            "2023-04-21 17:40:13,186 epoch 77 - iter 24/24 - loss 0.14802959 - time (sec): 25.36 - samples/sec: 3737.19 - lr: 0.000000\n",
            "2023-04-21 17:40:13,189 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:40:13,191 EPOCH 77 done: loss 0.1480 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:40:23,389 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:40:26,402 TRAIN : loss 0.12452424317598343 - f1-score (weighted avg)  0.8349\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.18it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:40:27,931 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:40:28,000 DEV : loss 0.2715824544429779 - f1-score (weighted avg)  0.7094\n",
            "2023-04-21 17:40:28,020 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:40:28,024 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:40:30,168 epoch 78 - iter 2/24 - loss 0.13925458 - time (sec): 2.14 - samples/sec: 4219.49 - lr: 0.000000\n",
            "2023-04-21 17:40:32,411 epoch 78 - iter 4/24 - loss 0.14931514 - time (sec): 4.39 - samples/sec: 3879.19 - lr: 0.000000\n",
            "2023-04-21 17:40:34,480 epoch 78 - iter 6/24 - loss 0.15122716 - time (sec): 6.46 - samples/sec: 3812.37 - lr: 0.000000\n",
            "2023-04-21 17:40:36,577 epoch 78 - iter 8/24 - loss 0.14441139 - time (sec): 8.55 - samples/sec: 3873.10 - lr: 0.000000\n",
            "2023-04-21 17:40:38,608 epoch 78 - iter 10/24 - loss 0.14913105 - time (sec): 10.58 - samples/sec: 3791.81 - lr: 0.000000\n",
            "2023-04-21 17:40:40,698 epoch 78 - iter 12/24 - loss 0.14864579 - time (sec): 12.67 - samples/sec: 3786.66 - lr: 0.000000\n",
            "2023-04-21 17:40:42,754 epoch 78 - iter 14/24 - loss 0.14492601 - time (sec): 14.73 - samples/sec: 3767.65 - lr: 0.000000\n",
            "2023-04-21 17:40:45,106 epoch 78 - iter 16/24 - loss 0.15528752 - time (sec): 17.08 - samples/sec: 3771.61 - lr: 0.000000\n",
            "2023-04-21 17:40:47,242 epoch 78 - iter 18/24 - loss 0.15323400 - time (sec): 19.22 - samples/sec: 3742.60 - lr: 0.000000\n",
            "2023-04-21 17:40:49,330 epoch 78 - iter 20/24 - loss 0.15030373 - time (sec): 21.31 - samples/sec: 3754.18 - lr: 0.000000\n",
            "2023-04-21 17:40:51,507 epoch 78 - iter 22/24 - loss 0.14864623 - time (sec): 23.48 - samples/sec: 3801.12 - lr: 0.000000\n",
            "2023-04-21 17:40:53,305 epoch 78 - iter 24/24 - loss 0.14834902 - time (sec): 25.28 - samples/sec: 3748.79 - lr: 0.000000\n",
            "2023-04-21 17:40:53,308 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:40:53,309 EPOCH 78 done: loss 0.1483 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:41:03,488 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:41:06,529 TRAIN : loss 0.12429899722337723 - f1-score (weighted avg)  0.8354\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.19it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:41:08,051 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:41:08,120 DEV : loss 0.2716139554977417 - f1-score (weighted avg)  0.7095\n",
            "2023-04-21 17:41:08,140 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:41:08,144 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:41:10,198 epoch 79 - iter 2/24 - loss 0.14224000 - time (sec): 2.05 - samples/sec: 4026.44 - lr: 0.000000\n",
            "2023-04-21 17:41:12,265 epoch 79 - iter 4/24 - loss 0.15026033 - time (sec): 4.12 - samples/sec: 3708.13 - lr: 0.000000\n",
            "2023-04-21 17:41:14,315 epoch 79 - iter 6/24 - loss 0.14099831 - time (sec): 6.17 - samples/sec: 3606.09 - lr: 0.000000\n",
            "2023-04-21 17:41:16,443 epoch 79 - iter 8/24 - loss 0.14034769 - time (sec): 8.30 - samples/sec: 3805.07 - lr: 0.000000\n",
            "2023-04-21 17:41:18,732 epoch 79 - iter 10/24 - loss 0.15130525 - time (sec): 10.59 - samples/sec: 3703.77 - lr: 0.000000\n",
            "2023-04-21 17:41:20,902 epoch 79 - iter 12/24 - loss 0.14685222 - time (sec): 12.76 - samples/sec: 3740.85 - lr: 0.000000\n",
            "2023-04-21 17:41:23,007 epoch 79 - iter 14/24 - loss 0.14694329 - time (sec): 14.86 - samples/sec: 3741.41 - lr: 0.000000\n",
            "2023-04-21 17:41:25,126 epoch 79 - iter 16/24 - loss 0.14239040 - time (sec): 16.98 - samples/sec: 3768.63 - lr: 0.000000\n",
            "2023-04-21 17:41:27,511 epoch 79 - iter 18/24 - loss 0.15599903 - time (sec): 19.37 - samples/sec: 3791.66 - lr: 0.000000\n",
            "2023-04-21 17:41:29,588 epoch 79 - iter 20/24 - loss 0.15515896 - time (sec): 21.44 - samples/sec: 3773.67 - lr: 0.000000\n",
            "2023-04-21 17:41:31,653 epoch 79 - iter 22/24 - loss 0.15230070 - time (sec): 23.51 - samples/sec: 3775.45 - lr: 0.000000\n",
            "2023-04-21 17:41:33,405 epoch 79 - iter 24/24 - loss 0.15069413 - time (sec): 25.26 - samples/sec: 3751.97 - lr: 0.000000\n",
            "2023-04-21 17:41:33,407 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:41:33,409 EPOCH 79 done: loss 0.1507 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:41:43,610 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:41:46,634 TRAIN : loss 0.12428856641054153 - f1-score (weighted avg)  0.8356\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.18it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:41:48,162 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:41:48,231 DEV : loss 0.2717282772064209 - f1-score (weighted avg)  0.7097\n",
            "2023-04-21 17:41:48,253 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:41:48,257 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:41:50,301 epoch 80 - iter 2/24 - loss 0.11720552 - time (sec): 2.04 - samples/sec: 3660.21 - lr: 0.000000\n",
            "2023-04-21 17:41:52,383 epoch 80 - iter 4/24 - loss 0.13971159 - time (sec): 4.12 - samples/sec: 3863.15 - lr: 0.000000\n",
            "2023-04-21 17:41:54,619 epoch 80 - iter 6/24 - loss 0.14137516 - time (sec): 6.36 - samples/sec: 3837.15 - lr: 0.000000\n",
            "2023-04-21 17:41:56,693 epoch 80 - iter 8/24 - loss 0.14034920 - time (sec): 8.43 - samples/sec: 3723.20 - lr: 0.000000\n",
            "2023-04-21 17:41:59,084 epoch 80 - iter 10/24 - loss 0.16218318 - time (sec): 10.83 - samples/sec: 3799.87 - lr: 0.000000\n",
            "2023-04-21 17:42:01,227 epoch 80 - iter 12/24 - loss 0.15897846 - time (sec): 12.97 - samples/sec: 3818.04 - lr: 0.000000\n",
            "2023-04-21 17:42:03,288 epoch 80 - iter 14/24 - loss 0.15574347 - time (sec): 15.03 - samples/sec: 3780.88 - lr: 0.000000\n",
            "2023-04-21 17:42:05,469 epoch 80 - iter 16/24 - loss 0.15275591 - time (sec): 17.21 - samples/sec: 3776.58 - lr: 0.000000\n",
            "2023-04-21 17:42:07,523 epoch 80 - iter 18/24 - loss 0.15052576 - time (sec): 19.26 - samples/sec: 3790.78 - lr: 0.000000\n",
            "2023-04-21 17:42:09,550 epoch 80 - iter 20/24 - loss 0.14861061 - time (sec): 21.29 - samples/sec: 3757.00 - lr: 0.000000\n",
            "2023-04-21 17:42:11,733 epoch 80 - iter 22/24 - loss 0.14801782 - time (sec): 23.47 - samples/sec: 3767.71 - lr: 0.000000\n",
            "2023-04-21 17:42:13,530 epoch 80 - iter 24/24 - loss 0.14709187 - time (sec): 25.27 - samples/sec: 3750.13 - lr: 0.000000\n",
            "2023-04-21 17:42:13,532 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:42:13,534 EPOCH 80 done: loss 0.1471 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.37it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:42:23,799 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:42:26,823 TRAIN : loss 0.12399660050868988 - f1-score (weighted avg)  0.8359\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.17it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:42:28,360 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:42:28,427 DEV : loss 0.2717648148536682 - f1-score (weighted avg)  0.7097\n",
            "2023-04-21 17:42:28,447 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:42:28,450 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:42:30,850 epoch 81 - iter 2/24 - loss 0.23409136 - time (sec): 2.40 - samples/sec: 3527.89 - lr: 0.000000\n",
            "2023-04-21 17:42:32,904 epoch 81 - iter 4/24 - loss 0.18164643 - time (sec): 4.45 - samples/sec: 3640.10 - lr: 0.000000\n",
            "2023-04-21 17:42:35,049 epoch 81 - iter 6/24 - loss 0.16306248 - time (sec): 6.60 - samples/sec: 3746.83 - lr: 0.000000\n",
            "2023-04-21 17:42:37,128 epoch 81 - iter 8/24 - loss 0.15610870 - time (sec): 8.68 - samples/sec: 3645.00 - lr: 0.000000\n",
            "2023-04-21 17:42:39,247 epoch 81 - iter 10/24 - loss 0.15169763 - time (sec): 10.80 - samples/sec: 3718.70 - lr: 0.000000\n",
            "2023-04-21 17:42:41,354 epoch 81 - iter 12/24 - loss 0.15198936 - time (sec): 12.90 - samples/sec: 3834.01 - lr: 0.000000\n",
            "2023-04-21 17:42:43,392 epoch 81 - iter 14/24 - loss 0.14961663 - time (sec): 14.94 - samples/sec: 3801.86 - lr: 0.000000\n",
            "2023-04-21 17:42:45,484 epoch 81 - iter 16/24 - loss 0.14647369 - time (sec): 17.03 - samples/sec: 3813.34 - lr: 0.000000\n",
            "2023-04-21 17:42:47,541 epoch 81 - iter 18/24 - loss 0.14641610 - time (sec): 19.09 - samples/sec: 3790.81 - lr: 0.000000\n",
            "2023-04-21 17:42:49,689 epoch 81 - iter 20/24 - loss 0.14853073 - time (sec): 21.24 - samples/sec: 3738.36 - lr: 0.000000\n",
            "2023-04-21 17:42:51,955 epoch 81 - iter 22/24 - loss 0.15109104 - time (sec): 23.50 - samples/sec: 3778.31 - lr: 0.000000\n",
            "2023-04-21 17:42:53,688 epoch 81 - iter 24/24 - loss 0.15120272 - time (sec): 25.24 - samples/sec: 3755.27 - lr: 0.000000\n",
            "2023-04-21 17:42:53,692 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:42:53,693 EPOCH 81 done: loss 0.1512 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:43:03,930 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:43:06,969 TRAIN : loss 0.1239466443657875 - f1-score (weighted avg)  0.8359\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.20it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:43:08,486 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:43:08,554 DEV : loss 0.27177661657333374 - f1-score (weighted avg)  0.7101\n",
            "2023-04-21 17:43:08,573 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:43:08,577 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:43:10,659 epoch 82 - iter 2/24 - loss 0.14130119 - time (sec): 2.08 - samples/sec: 3863.14 - lr: 0.000000\n",
            "2023-04-21 17:43:12,747 epoch 82 - iter 4/24 - loss 0.12006580 - time (sec): 4.17 - samples/sec: 3651.22 - lr: 0.000000\n",
            "2023-04-21 17:43:14,797 epoch 82 - iter 6/24 - loss 0.13165182 - time (sec): 6.22 - samples/sec: 3719.02 - lr: 0.000000\n",
            "2023-04-21 17:43:17,053 epoch 82 - iter 8/24 - loss 0.14896194 - time (sec): 8.47 - samples/sec: 3609.04 - lr: 0.000000\n",
            "2023-04-21 17:43:19,092 epoch 82 - iter 10/24 - loss 0.14550634 - time (sec): 10.51 - samples/sec: 3620.55 - lr: 0.000000\n",
            "2023-04-21 17:43:21,118 epoch 82 - iter 12/24 - loss 0.14461503 - time (sec): 12.54 - samples/sec: 3558.14 - lr: 0.000000\n",
            "2023-04-21 17:43:23,263 epoch 82 - iter 14/24 - loss 0.14133567 - time (sec): 14.68 - samples/sec: 3665.87 - lr: 0.000000\n",
            "2023-04-21 17:43:25,359 epoch 82 - iter 16/24 - loss 0.14264248 - time (sec): 16.78 - samples/sec: 3691.86 - lr: 0.000000\n",
            "2023-04-21 17:43:27,604 epoch 82 - iter 18/24 - loss 0.14185067 - time (sec): 19.02 - samples/sec: 3756.46 - lr: 0.000000\n",
            "2023-04-21 17:43:29,970 epoch 82 - iter 20/24 - loss 0.15276085 - time (sec): 21.39 - samples/sec: 3715.75 - lr: 0.000000\n",
            "2023-04-21 17:43:32,089 epoch 82 - iter 22/24 - loss 0.14993754 - time (sec): 23.51 - samples/sec: 3712.25 - lr: 0.000000\n",
            "2023-04-21 17:43:33,851 epoch 82 - iter 24/24 - loss 0.14944809 - time (sec): 25.27 - samples/sec: 3749.99 - lr: 0.000000\n",
            "2023-04-21 17:43:33,854 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:43:33,855 EPOCH 82 done: loss 0.1494 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:43:44,097 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:43:47,143 TRAIN : loss 0.12389671802520752 - f1-score (weighted avg)  0.8362\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.16it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:43:48,682 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:43:48,749 DEV : loss 0.27182257175445557 - f1-score (weighted avg)  0.7098\n",
            "2023-04-21 17:43:48,769 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:43:48,773 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:43:50,935 epoch 83 - iter 2/24 - loss 0.12978936 - time (sec): 2.16 - samples/sec: 3701.74 - lr: 0.000000\n",
            "2023-04-21 17:43:52,986 epoch 83 - iter 4/24 - loss 0.13064639 - time (sec): 4.21 - samples/sec: 3755.49 - lr: 0.000000\n",
            "2023-04-21 17:43:55,064 epoch 83 - iter 6/24 - loss 0.13399516 - time (sec): 6.29 - samples/sec: 3798.05 - lr: 0.000000\n",
            "2023-04-21 17:43:57,298 epoch 83 - iter 8/24 - loss 0.15251365 - time (sec): 8.52 - samples/sec: 3627.08 - lr: 0.000000\n",
            "2023-04-21 17:43:59,465 epoch 83 - iter 10/24 - loss 0.14749979 - time (sec): 10.69 - samples/sec: 3661.02 - lr: 0.000000\n",
            "2023-04-21 17:44:01,510 epoch 83 - iter 12/24 - loss 0.14702029 - time (sec): 12.74 - samples/sec: 3611.73 - lr: 0.000000\n",
            "2023-04-21 17:44:03,647 epoch 83 - iter 14/24 - loss 0.14618972 - time (sec): 14.87 - samples/sec: 3702.60 - lr: 0.000000\n",
            "2023-04-21 17:44:05,771 epoch 83 - iter 16/24 - loss 0.14374823 - time (sec): 17.00 - samples/sec: 3763.46 - lr: 0.000000\n",
            "2023-04-21 17:44:07,857 epoch 83 - iter 18/24 - loss 0.14235547 - time (sec): 19.08 - samples/sec: 3734.26 - lr: 0.000000\n",
            "2023-04-21 17:44:09,921 epoch 83 - iter 20/24 - loss 0.14242138 - time (sec): 21.15 - samples/sec: 3735.50 - lr: 0.000000\n",
            "2023-04-21 17:44:12,288 epoch 83 - iter 22/24 - loss 0.15198779 - time (sec): 23.51 - samples/sec: 3737.27 - lr: 0.000000\n",
            "2023-04-21 17:44:14,072 epoch 83 - iter 24/24 - loss 0.15154025 - time (sec): 25.30 - samples/sec: 3746.20 - lr: 0.000000\n",
            "2023-04-21 17:44:14,075 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:44:14,076 EPOCH 83 done: loss 0.1515 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:44:24,303 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:44:27,348 TRAIN : loss 0.12382270395755768 - f1-score (weighted avg)  0.8361\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.20it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:44:28,867 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:44:28,938 DEV : loss 0.27174699306488037 - f1-score (weighted avg)  0.7096\n",
            "2023-04-21 17:44:28,958 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:44:28,962 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:44:31,073 epoch 84 - iter 2/24 - loss 0.11430579 - time (sec): 2.11 - samples/sec: 3746.27 - lr: 0.000000\n",
            "2023-04-21 17:44:33,326 epoch 84 - iter 4/24 - loss 0.13625453 - time (sec): 4.36 - samples/sec: 3712.11 - lr: 0.000000\n",
            "2023-04-21 17:44:35,455 epoch 84 - iter 6/24 - loss 0.13222055 - time (sec): 6.49 - samples/sec: 3776.14 - lr: 0.000000\n",
            "2023-04-21 17:44:37,498 epoch 84 - iter 8/24 - loss 0.13610576 - time (sec): 8.53 - samples/sec: 3781.40 - lr: 0.000000\n",
            "2023-04-21 17:44:39,690 epoch 84 - iter 10/24 - loss 0.14155338 - time (sec): 10.73 - samples/sec: 3848.40 - lr: 0.000000\n",
            "2023-04-21 17:44:41,752 epoch 84 - iter 12/24 - loss 0.14388726 - time (sec): 12.79 - samples/sec: 3805.13 - lr: 0.000000\n",
            "2023-04-21 17:44:43,790 epoch 84 - iter 14/24 - loss 0.14207421 - time (sec): 14.83 - samples/sec: 3818.04 - lr: 0.000000\n",
            "2023-04-21 17:44:45,917 epoch 84 - iter 16/24 - loss 0.14077276 - time (sec): 16.95 - samples/sec: 3725.66 - lr: 0.000000\n",
            "2023-04-21 17:44:48,023 epoch 84 - iter 18/24 - loss 0.14029586 - time (sec): 19.06 - samples/sec: 3724.70 - lr: 0.000000\n",
            "2023-04-21 17:44:50,389 epoch 84 - iter 20/24 - loss 0.15245023 - time (sec): 21.43 - samples/sec: 3687.93 - lr: 0.000000\n",
            "2023-04-21 17:44:52,528 epoch 84 - iter 22/24 - loss 0.15113381 - time (sec): 23.56 - samples/sec: 3735.98 - lr: 0.000000\n",
            "2023-04-21 17:44:54,313 epoch 84 - iter 24/24 - loss 0.14905608 - time (sec): 25.35 - samples/sec: 3738.58 - lr: 0.000000\n",
            "2023-04-21 17:44:54,316 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:44:54,317 EPOCH 84 done: loss 0.1491 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:45:04,527 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:45:06,361 TRAIN : loss 0.12391597777605057 - f1-score (weighted avg)  0.836\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:02<00:00,  1.26it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:45:08,906 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:45:08,984 DEV : loss 0.27178847789764404 - f1-score (weighted avg)  0.7096\n",
            "2023-04-21 17:45:09,004 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:45:09,009 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:45:11,099 epoch 85 - iter 2/24 - loss 0.13331651 - time (sec): 2.09 - samples/sec: 3321.98 - lr: 0.000000\n",
            "2023-04-21 17:45:13,437 epoch 85 - iter 4/24 - loss 0.19910150 - time (sec): 4.43 - samples/sec: 3218.45 - lr: 0.000000\n",
            "2023-04-21 17:45:15,570 epoch 85 - iter 6/24 - loss 0.17418907 - time (sec): 6.56 - samples/sec: 3480.83 - lr: 0.000000\n",
            "2023-04-21 17:45:17,620 epoch 85 - iter 8/24 - loss 0.17089246 - time (sec): 8.61 - samples/sec: 3478.58 - lr: 0.000000\n",
            "2023-04-21 17:45:19,713 epoch 85 - iter 10/24 - loss 0.16195408 - time (sec): 10.70 - samples/sec: 3518.79 - lr: 0.000000\n",
            "2023-04-21 17:45:21,806 epoch 85 - iter 12/24 - loss 0.15233886 - time (sec): 12.80 - samples/sec: 3628.39 - lr: 0.000000\n",
            "2023-04-21 17:45:24,066 epoch 85 - iter 14/24 - loss 0.15486844 - time (sec): 15.06 - samples/sec: 3663.78 - lr: 0.000000\n",
            "2023-04-21 17:45:26,113 epoch 85 - iter 16/24 - loss 0.15125503 - time (sec): 17.10 - samples/sec: 3667.23 - lr: 0.000000\n",
            "2023-04-21 17:45:28,303 epoch 85 - iter 18/24 - loss 0.15152204 - time (sec): 19.29 - samples/sec: 3699.53 - lr: 0.000000\n",
            "2023-04-21 17:45:30,417 epoch 85 - iter 20/24 - loss 0.14860317 - time (sec): 21.41 - samples/sec: 3699.30 - lr: 0.000000\n",
            "2023-04-21 17:45:32,490 epoch 85 - iter 22/24 - loss 0.15048544 - time (sec): 23.48 - samples/sec: 3715.69 - lr: 0.000000\n",
            "2023-04-21 17:45:34,390 epoch 85 - iter 24/24 - loss 0.14859384 - time (sec): 25.38 - samples/sec: 3734.01 - lr: 0.000000\n",
            "2023-04-21 17:45:34,393 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:45:34,394 EPOCH 85 done: loss 0.1486 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:45:44,602 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:45:47,622 TRAIN : loss 0.1240968108177185 - f1-score (weighted avg)  0.8361\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.22it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:45:49,127 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:45:49,194 DEV : loss 0.27189287543296814 - f1-score (weighted avg)  0.7099\n",
            "2023-04-21 17:45:49,214 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:45:49,218 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:45:51,351 epoch 86 - iter 2/24 - loss 0.12983061 - time (sec): 2.13 - samples/sec: 3978.27 - lr: 0.000000\n",
            "2023-04-21 17:45:53,511 epoch 86 - iter 4/24 - loss 0.14603168 - time (sec): 4.29 - samples/sec: 3783.10 - lr: 0.000000\n",
            "2023-04-21 17:45:55,625 epoch 86 - iter 6/24 - loss 0.14816771 - time (sec): 6.41 - samples/sec: 3708.67 - lr: 0.000000\n",
            "2023-04-21 17:45:57,708 epoch 86 - iter 8/24 - loss 0.14010520 - time (sec): 8.49 - samples/sec: 3738.44 - lr: 0.000000\n",
            "2023-04-21 17:45:59,830 epoch 86 - iter 10/24 - loss 0.13317339 - time (sec): 10.61 - samples/sec: 3735.98 - lr: 0.000000\n",
            "2023-04-21 17:46:01,889 epoch 86 - iter 12/24 - loss 0.12999621 - time (sec): 12.67 - samples/sec: 3771.34 - lr: 0.000000\n",
            "2023-04-21 17:46:04,010 epoch 86 - iter 14/24 - loss 0.13261123 - time (sec): 14.79 - samples/sec: 3824.02 - lr: 0.000000\n",
            "2023-04-21 17:46:06,079 epoch 86 - iter 16/24 - loss 0.13096448 - time (sec): 16.86 - samples/sec: 3788.64 - lr: 0.000000\n",
            "2023-04-21 17:46:08,169 epoch 86 - iter 18/24 - loss 0.13267611 - time (sec): 18.95 - samples/sec: 3809.22 - lr: 0.000000\n",
            "2023-04-21 17:46:10,211 epoch 86 - iter 20/24 - loss 0.13432408 - time (sec): 20.99 - samples/sec: 3770.42 - lr: 0.000000\n",
            "2023-04-21 17:46:12,450 epoch 86 - iter 22/24 - loss 0.13955788 - time (sec): 23.23 - samples/sec: 3744.14 - lr: 0.000000\n",
            "2023-04-21 17:46:14,534 epoch 86 - iter 24/24 - loss 0.14967844 - time (sec): 25.31 - samples/sec: 3743.68 - lr: 0.000000\n",
            "2023-04-21 17:46:14,537 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:46:14,538 EPOCH 86 done: loss 0.1497 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:46:24,701 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:46:27,835 TRAIN : loss 0.12367137521505356 - f1-score (weighted avg)  0.8363\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.20it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:46:29,353 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:46:29,420 DEV : loss 0.27187711000442505 - f1-score (weighted avg)  0.7094\n",
            "2023-04-21 17:46:29,442 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:46:29,446 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:46:31,611 epoch 87 - iter 2/24 - loss 0.14478421 - time (sec): 2.16 - samples/sec: 3728.61 - lr: 0.000000\n",
            "2023-04-21 17:46:33,718 epoch 87 - iter 4/24 - loss 0.13350175 - time (sec): 4.27 - samples/sec: 3607.10 - lr: 0.000000\n",
            "2023-04-21 17:46:35,742 epoch 87 - iter 6/24 - loss 0.12941010 - time (sec): 6.29 - samples/sec: 3665.18 - lr: 0.000000\n",
            "2023-04-21 17:46:38,143 epoch 87 - iter 8/24 - loss 0.15536371 - time (sec): 8.70 - samples/sec: 3779.88 - lr: 0.000000\n",
            "2023-04-21 17:46:40,253 epoch 87 - iter 10/24 - loss 0.15358712 - time (sec): 10.81 - samples/sec: 3774.11 - lr: 0.000000\n",
            "2023-04-21 17:46:42,366 epoch 87 - iter 12/24 - loss 0.15004937 - time (sec): 12.92 - samples/sec: 3736.71 - lr: 0.000000\n",
            "2023-04-21 17:46:44,422 epoch 87 - iter 14/24 - loss 0.15290317 - time (sec): 14.97 - samples/sec: 3710.57 - lr: 0.000000\n",
            "2023-04-21 17:46:46,525 epoch 87 - iter 16/24 - loss 0.15099215 - time (sec): 17.08 - samples/sec: 3709.20 - lr: 0.000000\n",
            "2023-04-21 17:46:48,563 epoch 87 - iter 18/24 - loss 0.14842746 - time (sec): 19.12 - samples/sec: 3746.81 - lr: 0.000000\n",
            "2023-04-21 17:46:50,793 epoch 87 - iter 20/24 - loss 0.15071070 - time (sec): 21.35 - samples/sec: 3738.46 - lr: 0.000000\n",
            "2023-04-21 17:46:52,863 epoch 87 - iter 22/24 - loss 0.14907677 - time (sec): 23.42 - samples/sec: 3722.85 - lr: 0.000000\n",
            "2023-04-21 17:46:54,684 epoch 87 - iter 24/24 - loss 0.14790450 - time (sec): 25.24 - samples/sec: 3755.20 - lr: 0.000000\n",
            "2023-04-21 17:46:54,687 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:46:54,689 EPOCH 87 done: loss 0.1479 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:47:04,853 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:47:07,902 TRAIN : loss 0.12349922209978104 - f1-score (weighted avg)  0.8367\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.19it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:47:09,429 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:47:09,496 DEV : loss 0.2718559503555298 - f1-score (weighted avg)  0.7088\n",
            "2023-04-21 17:47:09,515 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:47:09,519 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:47:11,566 epoch 88 - iter 2/24 - loss 0.15862834 - time (sec): 2.05 - samples/sec: 4018.31 - lr: 0.000000\n",
            "2023-04-21 17:47:13,624 epoch 88 - iter 4/24 - loss 0.14684239 - time (sec): 4.10 - samples/sec: 3943.98 - lr: 0.000000\n",
            "2023-04-21 17:47:15,721 epoch 88 - iter 6/24 - loss 0.14152495 - time (sec): 6.20 - samples/sec: 3902.64 - lr: 0.000000\n",
            "2023-04-21 17:47:17,804 epoch 88 - iter 8/24 - loss 0.13829934 - time (sec): 8.28 - samples/sec: 3816.79 - lr: 0.000000\n",
            "2023-04-21 17:47:19,926 epoch 88 - iter 10/24 - loss 0.13284658 - time (sec): 10.41 - samples/sec: 3786.92 - lr: 0.000000\n",
            "2023-04-21 17:47:21,966 epoch 88 - iter 12/24 - loss 0.12991134 - time (sec): 12.45 - samples/sec: 3745.17 - lr: 0.000000\n",
            "2023-04-21 17:47:24,182 epoch 88 - iter 14/24 - loss 0.13591992 - time (sec): 14.66 - samples/sec: 3730.37 - lr: 0.000000\n",
            "2023-04-21 17:47:26,336 epoch 88 - iter 16/24 - loss 0.13986194 - time (sec): 16.81 - samples/sec: 3716.90 - lr: 0.000000\n",
            "2023-04-21 17:47:28,474 epoch 88 - iter 18/24 - loss 0.13838526 - time (sec): 18.95 - samples/sec: 3759.92 - lr: 0.000000\n",
            "2023-04-21 17:47:30,869 epoch 88 - iter 20/24 - loss 0.14937990 - time (sec): 21.35 - samples/sec: 3789.76 - lr: 0.000000\n",
            "2023-04-21 17:47:32,967 epoch 88 - iter 22/24 - loss 0.14705405 - time (sec): 23.45 - samples/sec: 3800.13 - lr: 0.000000\n",
            "2023-04-21 17:47:34,693 epoch 88 - iter 24/24 - loss 0.14811136 - time (sec): 25.17 - samples/sec: 3764.81 - lr: 0.000000\n",
            "2023-04-21 17:47:34,696 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:47:34,698 EPOCH 88 done: loss 0.1481 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:47:44,886 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:47:47,957 TRAIN : loss 0.12371096014976501 - f1-score (weighted avg)  0.8364\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.16it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:47:49,499 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:47:49,567 DEV : loss 0.2718922793865204 - f1-score (weighted avg)  0.71\n",
            "2023-04-21 17:47:49,587 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:47:49,591 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:47:51,605 epoch 89 - iter 2/24 - loss 0.11089772 - time (sec): 2.01 - samples/sec: 3356.66 - lr: 0.000000\n",
            "2023-04-21 17:47:53,675 epoch 89 - iter 4/24 - loss 0.12431217 - time (sec): 4.08 - samples/sec: 3618.15 - lr: 0.000000\n",
            "2023-04-21 17:47:56,146 epoch 89 - iter 6/24 - loss 0.17151088 - time (sec): 6.55 - samples/sec: 3635.87 - lr: 0.000000\n",
            "2023-04-21 17:47:58,169 epoch 89 - iter 8/24 - loss 0.16545152 - time (sec): 8.58 - samples/sec: 3663.05 - lr: 0.000000\n",
            "2023-04-21 17:48:00,282 epoch 89 - iter 10/24 - loss 0.15649926 - time (sec): 10.69 - samples/sec: 3738.30 - lr: 0.000000\n",
            "2023-04-21 17:48:02,344 epoch 89 - iter 12/24 - loss 0.15616772 - time (sec): 12.75 - samples/sec: 3690.62 - lr: 0.000000\n",
            "2023-04-21 17:48:04,497 epoch 89 - iter 14/24 - loss 0.15144162 - time (sec): 14.90 - samples/sec: 3715.01 - lr: 0.000000\n",
            "2023-04-21 17:48:06,596 epoch 89 - iter 16/24 - loss 0.14931617 - time (sec): 17.00 - samples/sec: 3752.55 - lr: 0.000000\n",
            "2023-04-21 17:48:08,814 epoch 89 - iter 18/24 - loss 0.15163392 - time (sec): 19.22 - samples/sec: 3744.09 - lr: 0.000000\n",
            "2023-04-21 17:48:10,893 epoch 89 - iter 20/24 - loss 0.15107352 - time (sec): 21.30 - samples/sec: 3747.92 - lr: 0.000000\n",
            "2023-04-21 17:48:12,979 epoch 89 - iter 22/24 - loss 0.15102117 - time (sec): 23.39 - samples/sec: 3726.38 - lr: 0.000000\n",
            "2023-04-21 17:48:14,820 epoch 89 - iter 24/24 - loss 0.14911456 - time (sec): 25.23 - samples/sec: 3756.59 - lr: 0.000000\n",
            "2023-04-21 17:48:14,824 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:48:14,825 EPOCH 89 done: loss 0.1491 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:48:25,054 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:48:28,121 TRAIN : loss 0.12384466826915741 - f1-score (weighted avg)  0.8364\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.21it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:48:29,641 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:48:29,711 DEV : loss 0.2719272971153259 - f1-score (weighted avg)  0.709\n",
            "2023-04-21 17:48:29,730 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:48:29,734 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:48:31,917 epoch 90 - iter 2/24 - loss 0.12306519 - time (sec): 2.18 - samples/sec: 4128.62 - lr: 0.000000\n",
            "2023-04-21 17:48:34,032 epoch 90 - iter 4/24 - loss 0.12909060 - time (sec): 4.30 - samples/sec: 3815.34 - lr: 0.000000\n",
            "2023-04-21 17:48:36,289 epoch 90 - iter 6/24 - loss 0.14260589 - time (sec): 6.55 - samples/sec: 3865.92 - lr: 0.000000\n",
            "2023-04-21 17:48:38,402 epoch 90 - iter 8/24 - loss 0.14257830 - time (sec): 8.67 - samples/sec: 3841.40 - lr: 0.000000\n",
            "2023-04-21 17:48:40,417 epoch 90 - iter 10/24 - loss 0.14410955 - time (sec): 10.68 - samples/sec: 3818.07 - lr: 0.000000\n",
            "2023-04-21 17:48:42,486 epoch 90 - iter 12/24 - loss 0.14814029 - time (sec): 12.75 - samples/sec: 3777.39 - lr: 0.000000\n",
            "2023-04-21 17:48:44,581 epoch 90 - iter 14/24 - loss 0.14573021 - time (sec): 14.84 - samples/sec: 3767.36 - lr: 0.000000\n",
            "2023-04-21 17:48:46,956 epoch 90 - iter 16/24 - loss 0.15862256 - time (sec): 17.22 - samples/sec: 3686.72 - lr: 0.000000\n",
            "2023-04-21 17:48:49,106 epoch 90 - iter 18/24 - loss 0.15634706 - time (sec): 19.37 - samples/sec: 3669.86 - lr: 0.000000\n",
            "2023-04-21 17:48:51,231 epoch 90 - iter 20/24 - loss 0.15211535 - time (sec): 21.49 - samples/sec: 3711.12 - lr: 0.000000\n",
            "2023-04-21 17:48:53,267 epoch 90 - iter 22/24 - loss 0.15114280 - time (sec): 23.53 - samples/sec: 3733.38 - lr: 0.000000\n",
            "2023-04-21 17:48:55,049 epoch 90 - iter 24/24 - loss 0.14806031 - time (sec): 25.31 - samples/sec: 3743.93 - lr: 0.000000\n",
            "2023-04-21 17:48:55,052 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:48:55,053 EPOCH 90 done: loss 0.1481 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:49:05,246 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:49:07,080 TRAIN : loss 0.12404933571815491 - f1-score (weighted avg)  0.836\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.18it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:49:08,611 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:49:08,678 DEV : loss 0.2720516324043274 - f1-score (weighted avg)  0.709\n",
            "2023-04-21 17:49:08,696 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:49:08,700 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:49:10,734 epoch 91 - iter 2/24 - loss 0.14703168 - time (sec): 2.03 - samples/sec: 3111.57 - lr: 0.000000\n",
            "2023-04-21 17:49:12,852 epoch 91 - iter 4/24 - loss 0.14241487 - time (sec): 4.15 - samples/sec: 3447.64 - lr: 0.000000\n",
            "2023-04-21 17:49:14,923 epoch 91 - iter 6/24 - loss 0.14105516 - time (sec): 6.22 - samples/sec: 3503.08 - lr: 0.000000\n",
            "2023-04-21 17:49:18,369 epoch 91 - iter 8/24 - loss 0.16845210 - time (sec): 9.67 - samples/sec: 3193.12 - lr: 0.000000\n",
            "2023-04-21 17:49:20,522 epoch 91 - iter 10/24 - loss 0.16224096 - time (sec): 11.82 - samples/sec: 3211.49 - lr: 0.000000\n",
            "2023-04-21 17:49:22,606 epoch 91 - iter 12/24 - loss 0.15376659 - time (sec): 13.90 - samples/sec: 3311.41 - lr: 0.000000\n",
            "2023-04-21 17:49:24,620 epoch 91 - iter 14/24 - loss 0.15126915 - time (sec): 15.92 - samples/sec: 3371.01 - lr: 0.000000\n",
            "2023-04-21 17:49:26,777 epoch 91 - iter 16/24 - loss 0.14877360 - time (sec): 18.07 - samples/sec: 3484.63 - lr: 0.000000\n",
            "2023-04-21 17:49:29,025 epoch 91 - iter 18/24 - loss 0.15160994 - time (sec): 20.32 - samples/sec: 3523.57 - lr: 0.000000\n",
            "2023-04-21 17:49:31,109 epoch 91 - iter 20/24 - loss 0.15024907 - time (sec): 22.41 - samples/sec: 3590.40 - lr: 0.000000\n",
            "2023-04-21 17:49:33,256 epoch 91 - iter 22/24 - loss 0.15016464 - time (sec): 24.55 - samples/sec: 3575.33 - lr: 0.000000\n",
            "2023-04-21 17:49:35,018 epoch 91 - iter 24/24 - loss 0.14918384 - time (sec): 26.32 - samples/sec: 3601.25 - lr: 0.000000\n",
            "2023-04-21 17:49:35,021 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:49:35,022 EPOCH 91 done: loss 0.1492 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:49:45,239 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:49:48,254 TRAIN : loss 0.1239415630698204 - f1-score (weighted avg)  0.8362\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.18it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:49:49,792 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:49:49,861 DEV : loss 0.27205708622932434 - f1-score (weighted avg)  0.709\n",
            "2023-04-21 17:49:49,881 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:49:49,885 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:49:51,947 epoch 92 - iter 2/24 - loss 0.11874656 - time (sec): 2.06 - samples/sec: 3809.61 - lr: 0.000000\n",
            "2023-04-21 17:49:54,244 epoch 92 - iter 4/24 - loss 0.15375139 - time (sec): 4.36 - samples/sec: 3520.16 - lr: 0.000000\n",
            "2023-04-21 17:49:56,355 epoch 92 - iter 6/24 - loss 0.14333738 - time (sec): 6.47 - samples/sec: 3637.08 - lr: 0.000000\n",
            "2023-04-21 17:49:58,373 epoch 92 - iter 8/24 - loss 0.15500999 - time (sec): 8.49 - samples/sec: 3631.11 - lr: 0.000000\n",
            "2023-04-21 17:50:00,730 epoch 92 - iter 10/24 - loss 0.17410072 - time (sec): 10.84 - samples/sec: 3572.75 - lr: 0.000000\n",
            "2023-04-21 17:50:02,844 epoch 92 - iter 12/24 - loss 0.16786094 - time (sec): 12.96 - samples/sec: 3644.12 - lr: 0.000000\n",
            "2023-04-21 17:50:04,963 epoch 92 - iter 14/24 - loss 0.16058197 - time (sec): 15.08 - samples/sec: 3695.13 - lr: 0.000000\n",
            "2023-04-21 17:50:07,048 epoch 92 - iter 16/24 - loss 0.15580508 - time (sec): 17.16 - samples/sec: 3722.46 - lr: 0.000000\n",
            "2023-04-21 17:50:09,170 epoch 92 - iter 18/24 - loss 0.15194352 - time (sec): 19.28 - samples/sec: 3741.30 - lr: 0.000000\n",
            "2023-04-21 17:50:11,337 epoch 92 - iter 20/24 - loss 0.15358553 - time (sec): 21.45 - samples/sec: 3778.46 - lr: 0.000000\n",
            "2023-04-21 17:50:13,427 epoch 92 - iter 22/24 - loss 0.14994706 - time (sec): 23.54 - samples/sec: 3763.10 - lr: 0.000000\n",
            "2023-04-21 17:50:15,188 epoch 92 - iter 24/24 - loss 0.14936330 - time (sec): 25.30 - samples/sec: 3745.63 - lr: 0.000000\n",
            "2023-04-21 17:50:15,191 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:50:15,193 EPOCH 92 done: loss 0.1494 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:50:25,401 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:50:28,418 TRAIN : loss 0.12391345202922821 - f1-score (weighted avg)  0.8363\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.23it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:50:29,918 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:50:29,985 DEV : loss 0.27208825945854187 - f1-score (weighted avg)  0.709\n",
            "2023-04-21 17:50:30,004 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:50:30,010 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:50:32,146 epoch 93 - iter 2/24 - loss 0.14054750 - time (sec): 2.13 - samples/sec: 4127.13 - lr: 0.000000\n",
            "2023-04-21 17:50:34,237 epoch 93 - iter 4/24 - loss 0.13249211 - time (sec): 4.23 - samples/sec: 4021.60 - lr: 0.000000\n",
            "2023-04-21 17:50:36,475 epoch 93 - iter 6/24 - loss 0.15063876 - time (sec): 6.46 - samples/sec: 3894.77 - lr: 0.000000\n",
            "2023-04-21 17:50:38,567 epoch 93 - iter 8/24 - loss 0.14715331 - time (sec): 8.55 - samples/sec: 3802.13 - lr: 0.000000\n",
            "2023-04-21 17:50:40,597 epoch 93 - iter 10/24 - loss 0.14238761 - time (sec): 10.58 - samples/sec: 3748.35 - lr: 0.000000\n",
            "2023-04-21 17:50:42,697 epoch 93 - iter 12/24 - loss 0.14074015 - time (sec): 12.68 - samples/sec: 3813.29 - lr: 0.000000\n",
            "2023-04-21 17:50:44,751 epoch 93 - iter 14/24 - loss 0.14023683 - time (sec): 14.74 - samples/sec: 3821.62 - lr: 0.000000\n",
            "2023-04-21 17:50:46,987 epoch 93 - iter 16/24 - loss 0.14200071 - time (sec): 16.97 - samples/sec: 3801.07 - lr: 0.000000\n",
            "2023-04-21 17:50:49,042 epoch 93 - iter 18/24 - loss 0.14255821 - time (sec): 19.03 - samples/sec: 3797.52 - lr: 0.000000\n",
            "2023-04-21 17:50:51,090 epoch 93 - iter 20/24 - loss 0.14139784 - time (sec): 21.08 - samples/sec: 3796.03 - lr: 0.000000\n",
            "2023-04-21 17:50:53,163 epoch 93 - iter 22/24 - loss 0.14090507 - time (sec): 23.15 - samples/sec: 3764.10 - lr: 0.000000\n",
            "2023-04-21 17:50:55,259 epoch 93 - iter 24/24 - loss 0.14888224 - time (sec): 25.25 - samples/sec: 3753.68 - lr: 0.000000\n",
            "2023-04-21 17:50:55,262 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:50:55,263 EPOCH 93 done: loss 0.1489 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.37it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:51:05,550 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:51:08,569 TRAIN : loss 0.12386798113584518 - f1-score (weighted avg)  0.8364\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.17it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:51:10,109 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:51:10,177 DEV : loss 0.2721557021141052 - f1-score (weighted avg)  0.709\n",
            "2023-04-21 17:51:10,197 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:51:10,201 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:51:12,317 epoch 94 - iter 2/24 - loss 0.14549342 - time (sec): 2.11 - samples/sec: 3634.70 - lr: 0.000000\n",
            "2023-04-21 17:51:14,677 epoch 94 - iter 4/24 - loss 0.18541583 - time (sec): 4.47 - samples/sec: 3746.97 - lr: 0.000000\n",
            "2023-04-21 17:51:16,801 epoch 94 - iter 6/24 - loss 0.16787420 - time (sec): 6.60 - samples/sec: 3820.06 - lr: 0.000000\n",
            "2023-04-21 17:51:18,902 epoch 94 - iter 8/24 - loss 0.15437725 - time (sec): 8.70 - samples/sec: 3808.95 - lr: 0.000000\n",
            "2023-04-21 17:51:20,977 epoch 94 - iter 10/24 - loss 0.15460296 - time (sec): 10.77 - samples/sec: 3873.28 - lr: 0.000000\n",
            "2023-04-21 17:51:23,045 epoch 94 - iter 12/24 - loss 0.14814015 - time (sec): 12.84 - samples/sec: 3859.10 - lr: 0.000000\n",
            "2023-04-21 17:51:25,287 epoch 94 - iter 14/24 - loss 0.14846284 - time (sec): 15.08 - samples/sec: 3825.74 - lr: 0.000000\n",
            "2023-04-21 17:51:27,323 epoch 94 - iter 16/24 - loss 0.14912268 - time (sec): 17.12 - samples/sec: 3790.79 - lr: 0.000000\n",
            "2023-04-21 17:51:29,557 epoch 94 - iter 18/24 - loss 0.15090422 - time (sec): 19.35 - samples/sec: 3755.46 - lr: 0.000000\n",
            "2023-04-21 17:51:31,660 epoch 94 - iter 20/24 - loss 0.14819326 - time (sec): 21.46 - samples/sec: 3780.16 - lr: 0.000000\n",
            "2023-04-21 17:51:33,731 epoch 94 - iter 22/24 - loss 0.14713371 - time (sec): 23.53 - samples/sec: 3767.04 - lr: 0.000000\n",
            "2023-04-21 17:51:35,492 epoch 94 - iter 24/24 - loss 0.14670264 - time (sec): 25.29 - samples/sec: 3747.47 - lr: 0.000000\n",
            "2023-04-21 17:51:35,494 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:51:35,497 EPOCH 94 done: loss 0.1467 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:51:45,696 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:51:48,771 TRAIN : loss 0.12380514293909073 - f1-score (weighted avg)  0.8365\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.21it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:51:50,281 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:51:50,350 DEV : loss 0.2721424102783203 - f1-score (weighted avg)  0.709\n",
            "2023-04-21 17:51:50,369 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:51:50,375 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:51:52,430 epoch 95 - iter 2/24 - loss 0.14443556 - time (sec): 2.05 - samples/sec: 3902.59 - lr: 0.000000\n",
            "2023-04-21 17:51:54,460 epoch 95 - iter 4/24 - loss 0.14320138 - time (sec): 4.08 - samples/sec: 3736.00 - lr: 0.000000\n",
            "2023-04-21 17:51:56,561 epoch 95 - iter 6/24 - loss 0.14585061 - time (sec): 6.18 - samples/sec: 3689.72 - lr: 0.000000\n",
            "2023-04-21 17:51:58,592 epoch 95 - iter 8/24 - loss 0.15015136 - time (sec): 8.22 - samples/sec: 3637.39 - lr: 0.000000\n",
            "2023-04-21 17:52:00,816 epoch 95 - iter 10/24 - loss 0.14340778 - time (sec): 10.44 - samples/sec: 3680.67 - lr: 0.000000\n",
            "2023-04-21 17:52:02,934 epoch 95 - iter 12/24 - loss 0.14196658 - time (sec): 12.56 - samples/sec: 3690.93 - lr: 0.000000\n",
            "2023-04-21 17:52:05,353 epoch 95 - iter 14/24 - loss 0.15634525 - time (sec): 14.98 - samples/sec: 3684.28 - lr: 0.000000\n",
            "2023-04-21 17:52:07,379 epoch 95 - iter 16/24 - loss 0.15173679 - time (sec): 17.00 - samples/sec: 3700.21 - lr: 0.000000\n",
            "2023-04-21 17:52:09,439 epoch 95 - iter 18/24 - loss 0.14636690 - time (sec): 19.06 - samples/sec: 3768.76 - lr: 0.000000\n",
            "2023-04-21 17:52:11,470 epoch 95 - iter 20/24 - loss 0.14384435 - time (sec): 21.09 - samples/sec: 3755.02 - lr: 0.000000\n",
            "2023-04-21 17:52:13,720 epoch 95 - iter 22/24 - loss 0.14915711 - time (sec): 23.34 - samples/sec: 3742.79 - lr: 0.000000\n",
            "2023-04-21 17:52:15,485 epoch 95 - iter 24/24 - loss 0.14851666 - time (sec): 25.11 - samples/sec: 3774.38 - lr: 0.000000\n",
            "2023-04-21 17:52:15,488 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:52:15,489 EPOCH 95 done: loss 0.1485 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:52:25,707 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:52:27,558 TRAIN : loss 0.12350770831108093 - f1-score (weighted avg)  0.8369\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:02<00:00,  1.25it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:52:30,118 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:52:30,187 DEV : loss 0.272136926651001 - f1-score (weighted avg)  0.7085\n",
            "2023-04-21 17:52:30,206 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:52:30,210 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:52:32,314 epoch 96 - iter 2/24 - loss 0.12548519 - time (sec): 2.10 - samples/sec: 3722.67 - lr: 0.000000\n",
            "2023-04-21 17:52:34,531 epoch 96 - iter 4/24 - loss 0.14852404 - time (sec): 4.32 - samples/sec: 3707.46 - lr: 0.000000\n",
            "2023-04-21 17:52:36,608 epoch 96 - iter 6/24 - loss 0.13907149 - time (sec): 6.40 - samples/sec: 3700.89 - lr: 0.000000\n",
            "2023-04-21 17:52:38,984 epoch 96 - iter 8/24 - loss 0.16261768 - time (sec): 8.77 - samples/sec: 3715.71 - lr: 0.000000\n",
            "2023-04-21 17:52:41,050 epoch 96 - iter 10/24 - loss 0.16410311 - time (sec): 10.84 - samples/sec: 3656.54 - lr: 0.000000\n",
            "2023-04-21 17:52:43,152 epoch 96 - iter 12/24 - loss 0.15840292 - time (sec): 12.94 - samples/sec: 3721.94 - lr: 0.000000\n",
            "2023-04-21 17:52:45,317 epoch 96 - iter 14/24 - loss 0.15229447 - time (sec): 15.11 - samples/sec: 3816.02 - lr: 0.000000\n",
            "2023-04-21 17:52:47,355 epoch 96 - iter 16/24 - loss 0.15266282 - time (sec): 17.14 - samples/sec: 3759.45 - lr: 0.000000\n",
            "2023-04-21 17:52:49,421 epoch 96 - iter 18/24 - loss 0.15068425 - time (sec): 19.21 - samples/sec: 3745.23 - lr: 0.000000\n",
            "2023-04-21 17:52:51,533 epoch 96 - iter 20/24 - loss 0.14906551 - time (sec): 21.32 - samples/sec: 3733.08 - lr: 0.000000\n",
            "2023-04-21 17:52:53,649 epoch 96 - iter 22/24 - loss 0.14902899 - time (sec): 23.44 - samples/sec: 3740.57 - lr: 0.000000\n",
            "2023-04-21 17:52:55,440 epoch 96 - iter 24/24 - loss 0.14774451 - time (sec): 25.23 - samples/sec: 3756.37 - lr: 0.000000\n",
            "2023-04-21 17:52:55,443 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:52:55,445 EPOCH 96 done: loss 0.1477 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.39it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:53:05,633 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:53:08,672 TRAIN : loss 0.12346755713224411 - f1-score (weighted avg)  0.837\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.19it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:53:10,203 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:53:10,271 DEV : loss 0.2720809578895569 - f1-score (weighted avg)  0.7088\n",
            "2023-04-21 17:53:10,291 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:53:10,295 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:53:12,394 epoch 97 - iter 2/24 - loss 0.13088939 - time (sec): 2.10 - samples/sec: 3475.78 - lr: 0.000000\n",
            "2023-04-21 17:53:14,607 epoch 97 - iter 4/24 - loss 0.13271809 - time (sec): 4.31 - samples/sec: 3738.07 - lr: 0.000000\n",
            "2023-04-21 17:53:16,707 epoch 97 - iter 6/24 - loss 0.13139126 - time (sec): 6.41 - samples/sec: 3732.14 - lr: 0.000000\n",
            "2023-04-21 17:53:18,797 epoch 97 - iter 8/24 - loss 0.12705655 - time (sec): 8.50 - samples/sec: 3847.21 - lr: 0.000000\n",
            "2023-04-21 17:53:20,839 epoch 97 - iter 10/24 - loss 0.12888720 - time (sec): 10.54 - samples/sec: 3765.94 - lr: 0.000000\n",
            "2023-04-21 17:53:22,966 epoch 97 - iter 12/24 - loss 0.13351243 - time (sec): 12.67 - samples/sec: 3844.39 - lr: 0.000000\n",
            "2023-04-21 17:53:25,037 epoch 97 - iter 14/24 - loss 0.13444153 - time (sec): 14.74 - samples/sec: 3827.07 - lr: 0.000000\n",
            "2023-04-21 17:53:27,326 epoch 97 - iter 16/24 - loss 0.13825096 - time (sec): 17.03 - samples/sec: 3765.31 - lr: 0.000000\n",
            "2023-04-21 17:53:29,455 epoch 97 - iter 18/24 - loss 0.13716732 - time (sec): 19.16 - samples/sec: 3808.95 - lr: 0.000000\n",
            "2023-04-21 17:53:31,462 epoch 97 - iter 20/24 - loss 0.13574664 - time (sec): 21.17 - samples/sec: 3757.69 - lr: 0.000000\n",
            "2023-04-21 17:53:33,559 epoch 97 - iter 22/24 - loss 0.13736163 - time (sec): 23.26 - samples/sec: 3747.06 - lr: 0.000000\n",
            "2023-04-21 17:53:35,649 epoch 97 - iter 24/24 - loss 0.14785693 - time (sec): 25.35 - samples/sec: 3738.17 - lr: 0.000000\n",
            "2023-04-21 17:53:35,652 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:53:35,653 EPOCH 97 done: loss 0.1479 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:53:45,885 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:53:48,942 TRAIN : loss 0.12341625988483429 - f1-score (weighted avg)  0.8368\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.18it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:53:50,478 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:53:50,556 DEV : loss 0.27212226390838623 - f1-score (weighted avg)  0.7082\n",
            "2023-04-21 17:53:50,577 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:53:50,582 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:53:52,954 epoch 98 - iter 2/24 - loss 0.24578086 - time (sec): 2.37 - samples/sec: 3583.80 - lr: 0.000000\n",
            "2023-04-21 17:53:55,074 epoch 98 - iter 4/24 - loss 0.18658507 - time (sec): 4.49 - samples/sec: 3572.21 - lr: 0.000000\n",
            "2023-04-21 17:53:57,145 epoch 98 - iter 6/24 - loss 0.17448493 - time (sec): 6.56 - samples/sec: 3634.93 - lr: 0.000000\n",
            "2023-04-21 17:53:59,217 epoch 98 - iter 8/24 - loss 0.16327990 - time (sec): 8.63 - samples/sec: 3645.05 - lr: 0.000000\n",
            "2023-04-21 17:54:01,265 epoch 98 - iter 10/24 - loss 0.15530116 - time (sec): 10.68 - samples/sec: 3658.65 - lr: 0.000000\n",
            "2023-04-21 17:54:03,383 epoch 98 - iter 12/24 - loss 0.15321068 - time (sec): 12.80 - samples/sec: 3746.23 - lr: 0.000000\n",
            "2023-04-21 17:54:05,678 epoch 98 - iter 14/24 - loss 0.15675551 - time (sec): 15.09 - samples/sec: 3786.92 - lr: 0.000000\n",
            "2023-04-21 17:54:07,732 epoch 98 - iter 16/24 - loss 0.15381997 - time (sec): 17.15 - samples/sec: 3798.88 - lr: 0.000000\n",
            "2023-04-21 17:54:09,872 epoch 98 - iter 18/24 - loss 0.15231299 - time (sec): 19.29 - samples/sec: 3763.60 - lr: 0.000000\n",
            "2023-04-21 17:54:11,889 epoch 98 - iter 20/24 - loss 0.15131481 - time (sec): 21.31 - samples/sec: 3742.73 - lr: 0.000000\n",
            "2023-04-21 17:54:14,081 epoch 98 - iter 22/24 - loss 0.15029825 - time (sec): 23.50 - samples/sec: 3798.63 - lr: 0.000000\n",
            "2023-04-21 17:54:15,818 epoch 98 - iter 24/24 - loss 0.15040014 - time (sec): 25.23 - samples/sec: 3755.56 - lr: 0.000000\n",
            "2023-04-21 17:54:15,821 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:54:15,823 EPOCH 98 done: loss 0.1504 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:54:26,032 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:54:29,057 TRAIN : loss 0.12365560978651047 - f1-score (weighted avg)  0.8366\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.19it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:54:30,580 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:54:30,647 DEV : loss 0.2720682621002197 - f1-score (weighted avg)  0.709\n",
            "2023-04-21 17:54:30,666 BAD EPOCHS (no improvement): 1\n",
            "2023-04-21 17:54:30,670 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:54:32,784 epoch 99 - iter 2/24 - loss 0.11531549 - time (sec): 2.11 - samples/sec: 4016.38 - lr: 0.000000\n",
            "2023-04-21 17:54:34,834 epoch 99 - iter 4/24 - loss 0.11996914 - time (sec): 4.16 - samples/sec: 3914.99 - lr: 0.000000\n",
            "2023-04-21 17:54:37,273 epoch 99 - iter 6/24 - loss 0.15889910 - time (sec): 6.60 - samples/sec: 4034.12 - lr: 0.000000\n",
            "2023-04-21 17:54:39,559 epoch 99 - iter 8/24 - loss 0.15331721 - time (sec): 8.89 - samples/sec: 3967.82 - lr: 0.000000\n",
            "2023-04-21 17:54:41,586 epoch 99 - iter 10/24 - loss 0.15364412 - time (sec): 10.91 - samples/sec: 3856.13 - lr: 0.000000\n",
            "2023-04-21 17:54:43,615 epoch 99 - iter 12/24 - loss 0.15278305 - time (sec): 12.94 - samples/sec: 3889.64 - lr: 0.000000\n",
            "2023-04-21 17:54:45,689 epoch 99 - iter 14/24 - loss 0.15543236 - time (sec): 15.02 - samples/sec: 3847.30 - lr: 0.000000\n",
            "2023-04-21 17:54:47,735 epoch 99 - iter 16/24 - loss 0.15566232 - time (sec): 17.06 - samples/sec: 3802.44 - lr: 0.000000\n",
            "2023-04-21 17:54:49,863 epoch 99 - iter 18/24 - loss 0.15077516 - time (sec): 19.19 - samples/sec: 3818.41 - lr: 0.000000\n",
            "2023-04-21 17:54:51,944 epoch 99 - iter 20/24 - loss 0.14832680 - time (sec): 21.27 - samples/sec: 3799.05 - lr: 0.000000\n",
            "2023-04-21 17:54:54,074 epoch 99 - iter 22/24 - loss 0.14778959 - time (sec): 23.40 - samples/sec: 3778.78 - lr: 0.000000\n",
            "2023-04-21 17:54:55,785 epoch 99 - iter 24/24 - loss 0.14738049 - time (sec): 25.11 - samples/sec: 3773.70 - lr: 0.000000\n",
            "2023-04-21 17:54:55,789 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:54:55,790 EPOCH 99 done: loss 0.1474 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:55:05,995 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:55:09,035 TRAIN : loss 0.12380121648311615 - f1-score (weighted avg)  0.8362\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.16it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:55:10,578 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:55:10,647 DEV : loss 0.272084504365921 - f1-score (weighted avg)  0.7091\n",
            "2023-04-21 17:55:10,667 BAD EPOCHS (no improvement): 0\n",
            "2023-04-21 17:55:10,671 ----------------------------------------------------------------------------------------------------\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:55:12,851 epoch 100 - iter 2/24 - loss 0.16236589 - time (sec): 2.18 - samples/sec: 3506.93 - lr: 0.000000\n",
            "2023-04-21 17:55:14,930 epoch 100 - iter 4/24 - loss 0.15524924 - time (sec): 4.26 - samples/sec: 3442.50 - lr: 0.000000\n",
            "2023-04-21 17:55:17,073 epoch 100 - iter 6/24 - loss 0.15019155 - time (sec): 6.40 - samples/sec: 3654.08 - lr: 0.000000\n",
            "2023-04-21 17:55:19,136 epoch 100 - iter 8/24 - loss 0.14965521 - time (sec): 8.46 - samples/sec: 3745.46 - lr: 0.000000\n",
            "2023-04-21 17:55:21,262 epoch 100 - iter 10/24 - loss 0.14849888 - time (sec): 10.59 - samples/sec: 3757.23 - lr: 0.000000\n",
            "2023-04-21 17:55:23,304 epoch 100 - iter 12/24 - loss 0.14584253 - time (sec): 12.63 - samples/sec: 3678.49 - lr: 0.000000\n",
            "2023-04-21 17:55:25,565 epoch 100 - iter 14/24 - loss 0.15092624 - time (sec): 14.89 - samples/sec: 3686.33 - lr: 0.000000\n",
            "2023-04-21 17:55:27,646 epoch 100 - iter 16/24 - loss 0.14726150 - time (sec): 16.97 - samples/sec: 3708.10 - lr: 0.000000\n",
            "2023-04-21 17:55:29,764 epoch 100 - iter 18/24 - loss 0.14359263 - time (sec): 19.09 - samples/sec: 3699.10 - lr: 0.000000\n",
            "2023-04-21 17:55:31,844 epoch 100 - iter 20/24 - loss 0.14273459 - time (sec): 21.17 - samples/sec: 3744.71 - lr: 0.000000\n",
            "2023-04-21 17:55:33,961 epoch 100 - iter 22/24 - loss 0.14031023 - time (sec): 23.29 - samples/sec: 3733.12 - lr: 0.000000\n",
            "2023-04-21 17:55:36,026 epoch 100 - iter 24/24 - loss 0.14878001 - time (sec): 25.35 - samples/sec: 3738.06 - lr: 0.000000\n",
            "2023-04-21 17:55:36,029 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:55:36,030 EPOCH 100 done: loss 0.1488 - lr 0.000000\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.38it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:55:46,239 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:55:48,078 TRAIN : loss 0.12391065806150436 - f1-score (weighted avg)  0.8361\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:02<00:00,  1.25it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:55:50,639 Evaluating as a multi-label problem: False\n",
            "2023-04-21 17:55:50,708 DEV : loss 0.2720869481563568 - f1-score (weighted avg)  0.7086\n",
            "2023-04-21 17:55:50,727 BAD EPOCHS (no improvement): 1\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:55:53,778 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:55:57,845 SequenceTagger predicts: Dictionary with 12 tags: <unk>, O, I-ADR, B-ADR, B-Drug, I-Finding, B-Finding, B-Disease, B-Symptom, I-Symptom, I-Disease, I-Drug\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3/3 [00:01<00:00,  2.29it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:55:59,327 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:55:59,551 0.7402\t0.7536\t0.7469\t0.9315\n",
            "2023-04-21 17:55:59,553 \n",
            "Results:\n",
            "- F-score (micro) 0.7469\n",
            "- F-score (macro) 0.3345\n",
            "- Accuracy 0.9315\n",
            "\n",
            "By class:\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "       I-ADR     0.7135    0.8035    0.7558       967\n",
            "       B-ADR     0.7477    0.8385    0.7905       576\n",
            "      B-Drug     0.8866    0.9635    0.9234       219\n",
            "   B-Finding     0.3182    0.1346    0.1892        52\n",
            "      I-Drug     0.9000    0.3462    0.5000        52\n",
            "   I-Finding     0.3333    0.0556    0.0952        36\n",
            "   I-Symptom     0.0000    0.0000    0.0000        33\n",
            "   B-Symptom     0.0000    0.0000    0.0000        22\n",
            "   B-Disease     0.5000    0.0500    0.0909        20\n",
            "   I-Disease     0.0000    0.0000    0.0000        12\n",
            "\n",
            "   micro avg     0.7402    0.7536    0.7469      1989\n",
            "   macro avg     0.4399    0.3192    0.3345      1989\n",
            "weighted avg     0.7039    0.7536    0.7187      1989\n",
            " samples avg     0.1267    0.1266    0.1266      1989\n",
            "\n",
            "2023-04-21 17:55:59,554 ----------------------------------------------------------------------------------------------------\n"
          ]
        }
      ],
      "source": [
        "hist = trainer.train('flair_ner_model'+suf,\n",
        "              embeddings_storage_mode='gpu',\n",
        "              gold_label_dictionary_for_eval=tag_dictionary,\n",
        "              mini_batch_size = 40,\n",
        "              eval_batch_size = 40,\n",
        "              patience=1,\n",
        "              anneal_factor = 0.8,\n",
        "              anneal_against_dev_loss=True,\n",
        "              main_evaluation_metric = (\"weighted avg\", \"f1-score\"),\n",
        "              metrics_for_tensorboard=[(\"weighted avg\", 'precision'), (\"weighted avg\", 'recall'), (\"macro avg\", 'f1-score'), (\"macro avg\", 'precision'), (\"macro avg\", 'recall')],\n",
        "              use_tensorboard=True,\n",
        "              tensorboard_comment='Flair NER',\n",
        "              tensorboard_log_dir='runs/tb_logs'+suf,\n",
        "              min_learning_rate=3e-8,\n",
        "              optimizer=AdamW,\n",
        "              learning_rate=5.0e-6,\n",
        "              weight_decay=0.001,\n",
        "              max_epochs=100,\n",
        "              monitor_train=True,\n",
        "              save_optimizer_state=True,)\n",
        "# hist"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "ExiLRreqXsDJ"
      },
      "outputs": [],
      "source": [
        "# print(hist.keys())\n",
        "# hist"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "P5Uu5qkN0pia",
        "outputId": "79b49e4a-3940-4570-ba6f-8e33e88e7193"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "_v3_Bio_ClinicalBERT\n",
            "2023-04-21 17:55:59,581 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:55:59,582 WARNING: No LOSS found for test split in this data.\n",
            "2023-04-21 17:55:59,583 Are you sure you want to plot LOSS and not another value?\n",
            "2023-04-21 17:55:59,585 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:55:59,610 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:55:59,611 WARNING: No F1 found for test split in this data.\n",
            "2023-04-21 17:55:59,612 Are you sure you want to plot F1 and not another value?\n",
            "2023-04-21 17:55:59,614 ----------------------------------------------------------------------------------------------------\n",
            "2023-04-21 17:56:00,695 Loss and F1 plots are saved in flair_ner_model_v3_Bio_ClinicalBERT/training.png\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAABdYAAAPgCAYAAAA/fy5BAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAD03UlEQVR4nOzdeZhT9d3+8fskmSSTWZkBZpFtEERAWQRFwI3HUUDlKXW3tCxuv1o3nOKCVdTailq1qFCpPira1rpUxSqIAgoqsonSiiIIDovCsDPDbMkkOb8/sswEZt8yy/t1XbmSc3KSfE8mJ+J9Pvl8DdM0TQEAAAAAAAAAgFqxRHsAAAAAAAAAAAC0JgTrAAAAAAAAAADUAcE6AAAAAAAAAAB1QLAOAAAAAAAAAEAdEKwDAAAAAAAAAFAHBOsAAAAAAAAAANQBwToAAAAAAAAAAHVAsA4AAAAAAAAAQB0QrAMAAAAAAAAAUAcE6wAAAAAAAAAA1EFUg/WZM2fq1FNPVUJCgjp37qzx48dr06ZNNT7ujTfe0Iknniin06mTTz5ZCxcujLjfNE3NmDFDGRkZio2NVXZ2tr7//vum2g0AAAAAAAAAQDsS1WB9+fLluvHGG7Vq1SotXrxYZWVlOv/881VUVFTlYz7//HNdddVVuuaaa/TVV19p/PjxGj9+vDZs2BDe5tFHH9VTTz2luXPnavXq1YqLi9Po0aNVWlraHLsFAAAAAAAAAGjDDNM0zWgPImTfvn3q3Lmzli9frrPOOqvSba644goVFRXpvffeC687/fTTNWjQIM2dO1emaSozM1O//e1vNW3aNElSfn6+0tLSNG/ePF155ZXNsi8AAAAAAAAAgLbJFu0BVJSfny9JSklJqXKblStXKicnJ2Ld6NGjNX/+fElSbm6u8vLylJ2dHb4/KSlJw4YN08qVKysN1t1ut9xud3jZ7/fr4MGDSk1NlWEYDdklAAAAAAAAoMUxTVNHjhxRZmamLBamYQTqqsUE636/X1OnTtXIkSN10kknVbldXl6e0tLSItalpaUpLy8vfH9oXVXbHG3mzJl64IEHGjJ8AAAAAAAAoNXZuXOnunTpEu1hAK1OiwnWb7zxRm3YsEGfffZZs7/29OnTI6rg8/Pz1a1bN+3cuVOJiYnNPh4AAAAAAACgKRUUFKhr165KSEiI9lCAVqlFBOs33XST3nvvPX3yySc1niFLT0/Xnj17Itbt2bNH6enp4ftD6zIyMiK2GTRoUKXP6XA45HA4jlmfmJhIsA4AAAAAAIA2izbIQP1EtYGSaZq66aab9Pbbb+ujjz5SVlZWjY8ZPny4li5dGrFu8eLFGj58uCQpKytL6enpEdsUFBRo9erV4W0AAAAAAAAAAKivqFas33jjjXrllVf0zjvvKCEhIdwDPSkpSbGxsZKkiRMn6rjjjtPMmTMlSbfeeqvOPvtsPf7447rwwgv16quv6osvvtCzzz4rKXCWberUqfrDH/6g3r17KysrS/fee68yMzM1fvz4qOwnAAAAAAAAAKDtiGqw/swzz0iSzjnnnIj1L774oiZPnixJ2rFjR8TMxCNGjNArr7yie+65R3fffbd69+6t+fPnR0x4escdd6ioqEjXX3+9Dh8+rDPOOEOLFi2S0+ls8n0CAAAAAAAAALRthmmaZrQH0dIUFBQoKSlJ+fn59FgHAAAAAABAm0P+BTRMVHusAwAAAAAAAADQ2hCsAwAAAAAAAABQBwTrAAAAAAAAAADUAcE6AAAAAAAAAAB1QLAOAAAAAAAAAEAdEKwDAAAAAAAAAFAHBOsAAAAAAAAAANQBwToAAAAAAAAAAHVAsA4AAAAAAAAAQB0QrAMAAAAAAAAAUAcE6wAAAAAAAAAA1IEt2gNA8/p8y37tyi/V6T1T1KWDK9rDAQAAAAAAAIBWh4r1dmbWku817Y3/6D8786M9FAAAAAAAAABolQjW25k4h1WSVOTxRnkkAAAAAAAAANA6Eay3My5HoPtPkZtgHQAAAAAAAADqg2C9nYm3B4L1Yo8vyiMBAAAAAAAAgNaJYL2dcQVbwRRSsQ4AAAAAAAAA9UKw3s7EhSrWCdYBAAAAAAAAoF4I1tuZuGCP9UI3rWAAAAAAAAAAoD4I1tuZuGArmGIPFesAAAAAAAAAUB8E6+1MqBVMEZOXAgAAAAAAAEC9EKy3M6GK9SJ6rAMAAAAAAABAvRCstzOhHusE6wAAAAAAAABQPwTr7Ywr3AqGYB0AAAAAAAAA6oNgvZ2JD1asF7vpsQ4AAAAAAAAA9UGw3s647IEe64W0ggEAAAAAAACAeiFYb2dCPdbdXr+8Pn+URwMAAAAAAAAArQ/BejsT57CGbxd5aAcDAAAAAAAAAHVFsN7O2K0W2SyGJKmYCUwBAAAAAAAAoM4I1tsZwzDC7WCKmMAUAAAAAAAAAOqMYL0digtOYFrEBKYAAAAAAAAAUGcE6+1QuGKdVjAAAAAAAAAAUGcE6+2Qi1YwAAAAAAAAAFBvBOvtUKgVDJOXAgAAAAAAAEDdEay3Q6FWMIX0WAcAAAAAAACAOiNYb4fCFeu0ggEAAAAAAACAOiNYb4eYvBQAAAAAAAAA6o9gvR0KB+u0ggEAAAAAAACAOiNYb4fi7KGKdVrBAAAAAAAAAEBdEay3Q3GOQI91KtYBAAAAAAAAoO4I1tuh8lYwVKwDAAAAAAAAQF0RrLdDLjsV6wAAAAAAAABQXwTr7VCox3qxh2AdAAAAAAAAAOqKYL0dCrWCKaRiHQAAAAAAAADqjGC9HQpNXlrsocc6AAAAAAAAANRVVIP1Tz75ROPGjVNmZqYMw9D8+fOr3X7y5MkyDOOYS//+/cPb3H///cfcf+KJJzbxnrQu5ZOXUrEOAAAAAAAAAHUV1WC9qKhIAwcO1Jw5c2q1/ZNPPqndu3eHLzt37lRKSoouu+yyiO369+8fsd1nn33WFMNvtUI91os8PpmmGeXRAAAAAAAAAEDrYovmi48dO1Zjx46t9fZJSUlKSkoKL8+fP1+HDh3SlClTIraz2WxKT09vtHG2NaFWMD6/KbfXL2eMNcojAgAAAAAAAIDWo1X3WH/++eeVnZ2t7t27R6z//vvvlZmZqZ49e2rChAnasWNHtc/jdrtVUFAQcWnLXPby8ym0gwEAAAAAAACAumm1wfquXbv0/vvv69prr41YP2zYMM2bN0+LFi3SM888o9zcXJ155pk6cuRIlc81c+bMcDV8UlKSunbt2tTDjyqrxZAzJvCnZwJTAAAAAAAAAKibVhusv/TSS0pOTtb48eMj1o8dO1aXXXaZBgwYoNGjR2vhwoU6fPiwXn/99Sqfa/r06crPzw9fdu7c2cSjj7744ASmhVSsAwAAAAAAAECdRLXHen2ZpqkXXnhBv/rVr2S326vdNjk5WSeccIK2bNlS5TYOh0MOh6Oxh9miBdrBeFTsIVgHAAAAAAAAgLpolRXry5cv15YtW3TNNdfUuG1hYaG2bt2qjIyMZhhZ6xEXrFgvctMKBgAAAAAAAADqIqrBemFhodavX6/169dLknJzc7V+/frwZKPTp0/XxIkTj3nc888/r2HDhumkk0465r5p06Zp+fLl2rZtmz7//HP9/Oc/l9Vq1VVXXdWk+9LaxNmtkpi8FAAAAAAAAADqKqqtYL744guNGjUqvJyTkyNJmjRpkubNm6fdu3eHQ/aQ/Px8vfnmm3ryyScrfc4ff/xRV111lQ4cOKBOnTrpjDPO0KpVq9SpU6em25FWKFyxzuSlAAAAAAAAAFAnUQ3WzznnHJmmWeX98+bNO2ZdUlKSiouLq3zMq6++2hhDa/PiHFSsAwAAAAAAAEB9tMoe62i4wOSlUhGTlwIAAAAAAABAnRCst1Px4clLCdYBAAAAAAAAoC4I1tspV3jyUnqsAwAAAAAAAEBdEKy3U3FUrAMAAAAAAABAvRCst1NxwYr1Yg8V6wAAAAAAAABQFwTr7VS4Yp3JSwEAAAAAAACgTgjW2ylawQAAAAAAAABA/RCst1PlwTqtYAAAAAAAAACgLgjW26lQj3VawQAAAAAAAABA3RCst1MuOxXrAAAAAAAAAFAfBOvtVDw91gEAAAAAAACgXgjW2ymXI9AKpqTMJ5/fjPJoAAAAAAAAAKD1IFhvp0IV61IgXAcAAAAAAAAA1A7BejvlsFlkMQK3aQcDAAAAAAAAALVHsN5OGYahOPqsAwAAAAAAAECdEay3Y3H2ULBOKxgAAAAAAAAAqC2C9XYsNIFpkYeKdQAAAAAAAACoLYL1diyeVjAAAAAAAAAAUGcE6+2Yyx6qWKcVDAAAAAAAAADUFsF6O0bFOgAAAAAAAADUHcF6O+ayE6wDAAAAAAAAQF0RrLdjccGK9WJawQAAAAAAAABArRGst2NxoR7rVKwDAAAAAAAAQK0RrLdjoYr1Ig/BOgAAAAAAAADUFsF6OxbnCFWs0woGAAAAAAAAAGqLYL0dY/JSAAAAAAAAAKg7gvV2LJ5WMAAAAAAAAABQZwTr7ZjLTisYAAAAAAAAAKgrgvV2LFSxXkzFOgAAAAAAAADUGsF6O+YKtYKhYh0AAAAAAAAAao1gvR2LdwRbwVCxDgAAAAAAAAC1RrDejrnsoYp1gnUAAAAAAAAAqC2C9XYsLhisl/lMebz+KI8GAAAAAAAAAFoHgvV2zBVsBSNRtQ4AAAAAAAAAtUWw3o7FWC2y2wIfAfqsAwAAAAAAAEDtEKy3c/GOUJ91X5RHAgAAAAAAAACtA8F6O+eyB9rBULEOAAAAAAAAALVDsN7OhSrWi6lYBwAAAAAAAIBaIVhv50IV64VMXgoAAAAAAAAAtUKw3s7FhSrWaQUDAAAAAAAAALVCsN7OxdlDk5cSrAMAAAAAAABAbRCst3MuR2jyUnqsAwAAAAAAAEBtEKy3c6HJS6lYBwAAAAAAAIDaIVhv51zhVjBUrAMAAAAAAABAbUQ1WP/kk080btw4ZWZmyjAMzZ8/v9rtly1bJsMwjrnk5eVFbDdnzhz16NFDTqdTw4YN05o1a5pwL1q3+GArGCYvBQAAAAAAAIDaiWqwXlRUpIEDB2rOnDl1etymTZu0e/fu8KVz587h+1577TXl5OTovvvu05dffqmBAwdq9OjR2rt3b2MPv00IVawX0goGAAAAAAAAAGrFFs0XHzt2rMaOHVvnx3Xu3FnJycmV3vfEE0/ouuuu05QpUyRJc+fO1YIFC/TCCy/orrvuashw26RQj/ViJi8FAAAAAAAAgFpplT3WBw0apIyMDJ133nlasWJFeL3H49G6deuUnZ0dXmexWJSdna2VK1dGY6gtnivYCoaKdQAAAAAAAAConVYVrGdkZGju3Ll688039eabb6pr164655xz9OWXX0qS9u/fL5/Pp7S0tIjHpaWlHdOHvSK3262CgoKIS3sRZw9VrBOsAwAAAAAAAEBtRLUVTF316dNHffr0CS+PGDFCW7du1Z///Gf97W9/q/fzzpw5Uw888EBjDLHViQu2gily0woGAAAAAAAAAGqjVVWsV+a0007Tli1bJEkdO3aU1WrVnj17IrbZs2eP0tPTq3yO6dOnKz8/P3zZuXNnk465JXHZA61gimgFAwAAAAAAAAC10uqD9fXr1ysjI0OSZLfbNWTIEC1dujR8v9/v19KlSzV8+PAqn8PhcCgxMTHi0l7EhyvWCdYBAAAAAAAAoDai2gqmsLAwXG0uSbm5uVq/fr1SUlLUrVs3TZ8+XT/99JNefvllSdKsWbOUlZWl/v37q7S0VP/3f/+njz76SB9++GH4OXJycjRp0iQNHTpUp512mmbNmqWioiJNmTKl2fevNQhNXlpc5pPfb8piMaI8IgAAAAAAAABo2aIarH/xxRcaNWpUeDknJ0eSNGnSJM2bN0+7d+/Wjh07wvd7PB799re/1U8//SSXy6UBAwZoyZIlEc9xxRVXaN++fZoxY4by8vI0aNAgLVq06JgJTREQqlg3TanU65PL3qra7gMAAAAAAABAszNM0zSjPYiWpqCgQElJScrPz2/zbWH8flPH/26hTFNa87tz1TnBGe0hAQAAAAAAoIm1p/wLaAqtvsc6GsZiMeSKCbaDcfuiPBoAAAAAAAAAaPkI1qG4YDuYQiYwBQAAAAAAAIAaEawjHKwXe6hYBwAAAAAAAICaEKxDcY5AK5giKtYBAAAAAAAAoEYE65DLHqhYL/IQrAMAAAAAAABATQjWofhQKxgmLwUAAAAAAACAGhGsQy57oBUMk5cCAAAAAAAAQM0I1lFesU4rGAAAAAAAAACoEcE6wj3WC2kFAwAAAAAAAAA1IliH4hyBVjBUrAMAAAAAAABAzQjWoThHqGKdYB0AAAAAAAAAakKwDsUFJy8tphUMAAAAAAAAANSIYB3hivUiWsEAAAAAAAAAQI0I1hGevLSIVjAAAAAAAAAAUCOCdSg+WLFe7KEVDAAAAAAAAADUhGAdcjkCPdaZvBQAAAAAAAAAakawDsXZqVgHAAAAAAAAgNoiWIfiqFgHAAAAAAAAgFojWEe4Yt3j9avM54/yaAAAAAAAAACgZSNYh+KCk5dKUrGbdjAAAAAAAAAAUB2CdchusyjGakiSijy0gwEAAAAAAACA6hCsQ1J51XoxwToAAAAAAAAAVItgHZLK+6wX0goGAAAAAAAAAKpFsA5JUpzDKkkqdlOxDgAAAAAAAADVIViHJMkVrlgnWAcAAAAAAACA6hCsQ1KFinUPrWAAAAAAAAAAoDoE65BUscc6FesAAAAAAAAAUB2CdUiS4hyBYL3YQ7AOAAAAAAAAANUhWIek8lYwhW5awQAAAAAAAABAdQjWIam8FUwxrWAAAAAAAAAAoFoE65BU3gqmiMlLAQAAAAAAAKBaBOuQJLnsgVYwRVSsAwAAAAAAAEC1CNYhiclLAQAAAAAAAKC2CNYhqTxYL6RiHQAAAAAAAACqRbAOSVJcsBVMMT3WAQAAAAAAAKBaBOuQRMU6AAAAAAAAANQWwTokSXH2YI91NxXrAAAAAAAAAFAdgnVIkuIcgVYwRUxeCgAAAAAAAADVIliHpPJWMEVur0zTjPJoAAAAAAAAAKDlIliHpPJg3W9Kbq8/yqMBAAAAAAAAgJaLYB2SpNgYa/g2E5gCAAAAAAAAQNUI1iFJslqMcLjOBKYAAAAAAAAAUDWCdYSF2sFQsQ4AAAAAAAAAVSNYR1icI1ix7iFYBwAAAAAAAICqEKwjLM5OxToAAAAAAAAA1CSqwfonn3yicePGKTMzU4ZhaP78+dVu/9Zbb+m8885Tp06dlJiYqOHDh+uDDz6I2Ob++++XYRgRlxNPPLEJ96LtKK9Yp8c6AAAAAAAAAFQlqsF6UVGRBg4cqDlz5tRq+08++UTnnXeeFi5cqHXr1mnUqFEaN26cvvrqq4jt+vfvr927d4cvn332WVMMv80J9VgvomIdAAAAAAAAAKpki+aLjx07VmPHjq319rNmzYpYfuihh/TOO+/o3Xff1eDBg8PrbTab0tPTG2uY7UaoFQzBOgAAAAAAAABUrVX3WPf7/Tpy5IhSUlIi1n///ffKzMxUz549NWHCBO3YsSNKI2xdXPZAK5giWsEAAAAAAAAAQJWiWrHeUI899pgKCwt1+eWXh9cNGzZM8+bNU58+fbR792498MADOvPMM7VhwwYlJCRU+jxut1tutzu8XFBQ0ORjb4loBQMAAAAAAAAANWu1wforr7yiBx54QO+88446d+4cXl+xtcyAAQM0bNgwde/eXa+//rquueaaSp9r5syZeuCBB5p8zC0dk5cCAAAAAAAAQM1aZSuYV199Vddee61ef/11ZWdnV7ttcnKyTjjhBG3ZsqXKbaZPn678/PzwZefOnY095FYhVLFeSMU6AAAAAAAAAFSp1QXr//znPzVlyhT985//1IUXXljj9oWFhdq6dasyMjKq3MbhcCgxMTHi0h6FJi8t9hCsAwAAAAAAAEBVotoKprCwMKKSPDc3V+vXr1dKSoq6deum6dOn66efftLLL78sKdD+ZdKkSXryySc1bNgw5eXlSZJiY2OVlJQkSZo2bZrGjRun7t27a9euXbrvvvtktVp11VVXNf8OtjLlPdZpBQMAAAAAAAAAVYlqxfoXX3yhwYMHa/DgwZKknJwcDR48WDNmzJAk7d69Wzt27Ahv/+yzz8rr9erGG29URkZG+HLrrbeGt/nxxx911VVXqU+fPrr88suVmpqqVatWqVOnTs27c61QnD3QY53JSwEAAAAAAACgalGtWD/nnHNkmmaV98+bNy9iedmyZTU+56uvvtrAUbVfrlDFOpOXAgAAAAAAAECVWl2PdTSdeAcV6wAAAAAAAABQE4J1hLmYvBQAAAAAAAAAakSwjrD4YCuYQirWAQAAAAAAAKBKBOsIcwUnLy0t88vnr7r3PQAAAAAAAAC0ZwTrCItzlM9lW0Q7GAAAAAAAAACoFME6whw2i6wWQ5JU7PZFeTQAAAAAAAAA0DIRrCPMMAzFBdvBULEOAAAAAAAAAJUjWEeEUDuYIiYwBQAAAAAAAIBKEawjQmgC0yJawQAAAAAAAABApQjWESGeinUAAAAAAAAAqBbBOiK47MFgnR7rAAAAAAAAAFApgnVEKO+xTisYAAAAAAAAAKgMwToixDkCPdaLqVgHAAAAAAAAgEoRrCMCFesAAAAAAAAAUD2CdUSIswcq1umxDgAAAAAAAACVI1hHhPDkpW6CdQAAAAAAAACoDME6IsQ7CNYBAAAAAAAAoDoE64jgcoRawdBjHQAAAAAAAAAqQ7COCFSsAwAAAAAAAED1CNYRIdxjnYp1AAAAAAAAAKgUwToixIVawVCxDgAAAAAAAACVIlhHhLhgxXoxwToAAAAAAAAAVIpgHRHiHLSCAQAAAAAAAIDqEKwjQsVWMKZpRnk0AAAAAAAAANDyEKwjQmjyUq/flMfnj/JoAAAAAAAAAKDlIVhHhDi7NXy7yE07GAAAAAAAAAA4GsE6ItisFjlsgY9FEROYAgAAAAAAAMAxCNZxjPjwBKYE6wAAAAAAAABwNIJ1HMMVnsCUVjAAAAAAAAAAcDSCdRwjLjiBaTEV6wAAAAAAAABwDIJ1HCMu1AqGHusAAAAAAAAAcAyCdRzDZacVDAAAAAAAAABUhWAdx2DyUgAAAAAAAACoWr2C9ZdeekkLFiwIL99xxx1KTk7WiBEjtH379kYbHKLDZQ+1gqFiHQAAAAAAAACOVq9g/aGHHlJsbKwkaeXKlZozZ44effRRdezYUbfddlujDhDNL94RagVDxToAAAAAAAAAHM1Wnwft3LlTvXr1kiTNnz9fl1xyia6//nqNHDlS55xzTmOOD1HgohUMAAAAAAAAAFSpXhXr8fHxOnDggCTpww8/1HnnnSdJcjqdKikpabzRISrCPdapWAcAAAAAAACAY9SrYv28887Ttddeq8GDB2vz5s264IILJEnffPONevTo0ZjjQxS47MFWMB56rAMAAAAAAADA0epVsT5nzhwNHz5c+/bt05tvvqnU1FRJ0rp163TVVVc16gDR/OKCFevFVKwDAAAAAAAAwDHqVbGenJys2bNnH7P+gQceaPCAEH1x9lArGCrWAQAAAAAAAOBo9apYX7RokT777LPw8pw5czRo0CD94he/0KFDhxptcIgOlyPUCoaKdQAAAAAAAAA4Wr2C9dtvv10FBQWSpK+//lq//e1vdcEFFyg3N1c5OTmNOkA0PyYvBQAAAAAAAICq1asVTG5urvr16ydJevPNN3XRRRfpoYce0pdffhmeyBStF5OXAgAAAAAAAEDV6lWxbrfbVVxcLElasmSJzj//fElSSkpKuJIdrRcV6wAAAAAAAABQtXpVrJ9xxhnKycnRyJEjtWbNGr322muSpM2bN6tLly6NOkA0P1dw8tJij09+vymLxYjyiAAAAAAAAACg5ahXxfrs2bNls9n0r3/9S88884yOO+44SdL777+vMWPG1Pp5PvnkE40bN06ZmZkyDEPz58+v8THLli3TKaecIofDoV69emnevHnHbDNnzhz16NFDTqdTw4YN05o1a2o9JpRXrEtSSRntYAAAAAAAAACgonpVrHfr1k3vvffeMev//Oc/1+l5ioqKNHDgQF199dW6+OKLa9w+NzdXF154oX7961/rH//4h5YuXaprr71WGRkZGj16tCTptddeU05OjubOnathw4Zp1qxZGj16tDZt2qTOnTvXaXztlTPGIsOQTDPQDibOUa+PCQAAAAAAAAC0SYZpmmZ9Hujz+TR//nxt3LhRktS/f3/97//+r6xWa/0GYhh6++23NX78+Cq3ufPOO7VgwQJt2LAhvO7KK6/U4cOHtWjRIknSsGHDdOqpp2r27NmSJL/fr65du+rmm2/WXXfdVauxFBQUKCkpSfn5+UpMTKzX/rR2J933gQrdXn087RxldYyL9nAAAAAAAADQiMi/gIapVyuYLVu2qG/fvpo4caLeeustvfXWW/rlL3+p/v37a+vWrY09xrCVK1cqOzs7Yt3o0aO1cuVKSZLH49G6desitrFYLMrOzg5vUxm3262CgoKIS3sX5wicIGECUwAAAAAAAACIVK9g/ZZbbtHxxx+vnTt36ssvv9SXX36pHTt2KCsrS7fccktjjzEsLy9PaWlpEevS0tJUUFCgkpIS7d+/Xz6fr9Jt8vLyqnzemTNnKikpKXzp2rVrk4y/NYkLTmBKsA4AAAAAAAAAkeoVrC9fvlyPPvqoUlJSwutSU1P18MMPa/ny5Y02uOYyffp05efnhy87d+6M9pCiLtRXvchDsA4AAAAAAAAAFdVrVkqHw6EjR44cs76wsFB2u73Bg6pKenq69uzZE7Fuz549SkxMVGxsrKxWq6xWa6XbpKenV/m8DodDDoejScbcWrnsoVYwviiPBAAAAAAAAABalnpVrF900UW6/vrrtXr1apmmKdM0tWrVKv3617/W//7v/zb2GMOGDx+upUuXRqxbvHixhg8fLkmy2+0aMmRIxDZ+v19Lly4Nb4PaiXfQCgYAAAAAAAAAKlOvYP2pp57S8ccfr+HDh8vpdMrpdGrEiBHq1auXZs2aVevnKSws1Pr167V+/XpJUm5urtavX68dO3ZICrRomThxYnj7X//61/rhhx90xx136LvvvtNf/vIXvf7667rtttvC2+Tk5Oi5557TSy+9pI0bN+qGG25QUVGRpkyZUp9dbbdc4VYwVKwDAAAAAAAAQEX1agWTnJysd955R1u2bNHGjRslSX379lWvXr3q9DxffPGFRo0aFV7OycmRJE2aNEnz5s3T7t27wyG7JGVlZWnBggW67bbb9OSTT6pLly76v//7P40ePTq8zRVXXKF9+/ZpxowZysvL06BBg7Ro0aJjJjRF9eKCrWCKqVgHAAAAAAAAgAiGaZpmbTYMhd618cQTT9R7QC1BQUGBkpKSlJ+fr8TExGgPJyoefO9bPf9Zrv7f2T01fWzfaA8HAAAAAAAAjYj8C2iYWlesf/XVV7XazjCMeg8GLUd5xTqtYAAAABAFphm4qJLrMEMyjGOvW9r/kxxdy9QY46tdfVQVjzFrt9ziBN+30N86fPvo+xpZU75vFR/bVM9b5eu0YlXuX1XrgiI+H03wGaryuZr781nNe1Df563ytZpRte9nY/39qvjbWWyS3VX/5wbQJtU6WP/444+bchxoYeKYvBQAGpdpSj6PVFYslZUELp6iwLW/TPKVSX5f4LbfW/WyFAyNLFVfLNbANqYZfA5v4GL6gsu+CsveyHX+ssjHhC++yNsVx1FVsCWjfDwWW4XL0cuhddbAe+T1SN7S4O1SyeuucKmwXqpk/6t6b6yVj8Mac+y4Qn8rr1vyVXjt8DpP+Tp/Pf47WWUIeNS13xf8G/nL/3bhv9lR61TVfldYb7EG/16qW0hk+o8KOP0V1vkjt4kIQHXUcsXrCn87S4W/UcVxGtby+2oMKCoEqaa/wmfVf9Rn3R95PJhmNZ/dSq4j9rmKwLchIUNl+1PVvteolgFgY4232mWp8qCnqsCvMYOaCn/DOj+0lvtX7WehtvtSYXzHvG+qw/MAAJpEjzOlye9FexQAWph69VhH21c+eSnBOoCjeN2S+4hUmh+4dh+R3AWSp7g8xLNYI4PMo9cZlmC4XBwIlz1FwdvFUllwOXy7OBiC+XVMoBe++I5almoOvio8h/+o5/H7KiwHw2mZFcLXSsLYisuGIZWVRoboZcXBABRAi9KS8sqWNJbaaBXjbcDJjmbdv8oqQgHUX1Un+KRW8uXVSjRhJf7RGv3EKwA0HME6KhXvCLSCKaIVDND0ykqlI7sDl4Jdwdt5gXDZXyb5vBUqmoOVyz5PhSrmYIXxMWHz0ZdQpaUCVaAWWzDoDgXCR6+zBh7jLogM0H2e6L5fbYFhlexxUkysZHNKVntk5bQ1JhjeWyNvW2zBqtmqTjBUPCngD2x79N80/DxHrTMqVm/XUGEeUfVcwwmM0AmKo6vgfVVUxFtjAu+JzRG8BG9bj1q2OSQZ1X/Wjz5RckzlfWXV+cH/7lntwde1VxiDPfLa5gi8H3WqhK3sfapifbhiu8LfKHx91HqpmvfhqPei1lXGoeWK1e8VfolQsSL+mAr8iuukY6q/KxtvqPo+fNss/+yE3rvw5+7o5Qq3Iz7jNXzmQ1XoR1cXV1eFXtsK9zqrrIK8in2utdr8VL4hYUg144tYV5uWCFV8VkKfs8ruq+xvdPT3T4N+RVCbKn9Ttfo8RHwuzGqes8Jype9XI/z9jg7AGvzriOZSxTEfsUkThm5N+b416q9Vqnjeqta1tLZJdXLUPtRnX8xqvsfq8zy1+l5sAk3WIqUR3uOm1Bh/v9r+squl7TuAFoFgHZVy2alYRxthmoGWEWUlwUDaXR5M+zzlt70V17uraUVx1DpfWSAItMYEw1F7hduOyPUWi1S0vzw8L9gVuJQcjPa7VD/2BMkRvDgTpRiXJLNCiwpvNa0sfIFQOcYVCJjtccHbLskeX347Ji5wbbUfG+RVeakkxKipTcnRrTKOaUkRCpL9qrZFSqhdi+mXYpyB/Yhxle9r6Npmj8ZfDAAAABW11MAYtcPfD0CUEayjUvHBVjBMXoqo8XoC1dGl+cGWI6HbBeW33YWSJ3QJtQ8pDK6vsNwa2m9YHVJiZuCSkCElpAcC61C1cqiS+JjlUKWztYawuUIgHariq67PdmidDMmRGBmgOxIC4bfFGu13DQAAAAAAICoI1lEplz0QmBUyeWnb4vdL7nyp5LBUckgqPVx+232kkjYJFZfLypdDld9ed6ASPFTJfcx18LZhqNoWBhXXl5UEQvPQxISNyihv4xCuJD+60txReSuKiu0gbM5AxbHNGQi5/d4K1e/uyEr40ESModYtrtTy8LzidWwHKi0AAAAAAABaCYJ1VCouVLFOK5iWzTQDoXjhXqkwL3B9JE8q3BO4FO0PhueHAgF6ab7q3TewQeOUJK9Un8Jxe7zkTApUTTsTI287EgLtSEKtRBzxge1DyxXvi3FJVr7yAAAAAAAA0HCkTKhUKFhn8tJGYpqBgLtov1S0r8IluOzzVDJJio5aDk7U5D4SDM/3BsJzn7vu44lxBSqknclSbHLgtiPhqIkTq5vAMKZC5XaFKu6Yo5ZDEw7KjOytXXGSunAfbn953+2KATrtRgAAAAAAANDCEKyjUnHBVjAen18er192myXKI2ohQq1USvPLK8BL8wOheWnF9Yel4gOR4bm/Cav/ncmBntzxnaX44HVCuhTX6dgA3ZnMxIkAAAAAAABAAxCso1Iue/lHo9jjlb29BLF+v3Rkt3Twh/LLodzA9eEdgYkzG9JKxZEkxXUMBN4Vr23OwP3hHttHz25eYdkeFwzP06SENCmuc6BSHAAAAAAAAECzIFhHpew2i+xWizw+v4o8PiW7oj2iRlZWIu3+r7Tna+lgboUQfVvtJs20xQYqwJ1JwUuF26H1rsoCdEfT7hcAAAAAAACAJkewjirFOazyFPtV5G7lE5iapnRgq/TjWumnL6Qfv5D2bKi6NYvFJiV3k1J6ll86ZEkdekiu1ED/bwJyAAAAAAAAoN0iWEeVXHabDhWXtb5gvfhgIDwPheg/rQv0PD9aXCcpc7CU2ltKySoP0ZO6SlYODQAAAAAAAACVIz1EleIcgQlMiz2+KI+kGqX50u7/SLu+Cl7WB3qiH83mlDIGSscNlboMCVwnd6vQwxwAAAAAAAAAaodgHVWKcwQ+HoUtpWLdfSQYoq8vD9IPbq1829RewRA9eEk7SbLGNOtwAQAAAAAAALRNBOuoUpw98PEo9kQxWD+0Xfr8aSl3ubT/e0nmsdskdwu0dMkYFLweKLlSmnukAAAAAAAAANoJgnVUKdQKptAdhVYw+T9KnzwmffV3yV9Wvj6payA4zxxcHqbHpTb/+AAAAAAAAAC0WwTrqFK4Yr05W8EU7JI+fUL68iXJ5wms63mONOzXgdYu8Z2abywAAAAAAAAAUAmCdVQpMTbQk3zfEXfTv9iRPdJnf5a+eEHyBV+vx5nSOdOlHiOb/vUBAAAAAAAAoJYI1lGlAV2SJElrtx9quhcp3CetmCWtfV7ylgTWdRsujbpbyjqr6V4XAAAAAAAAAOqJYB1VGtYz0Lt8w0/5KnR7Fe9oxI9L0QHp86ekNc9KZcWBdV1ODQTqPUdJhtF4rwUAAAAAAAAAjYhgHVU6LjlWXTrE6sdDJVq3/ZDOPqGR+pt/M1/6982SuyCwnHmKNOp3Uq9zCdQBAAAAAAAAtHiWaA8ALdvpwar1VT8caPiTeT3S+3dJb0wKhOrpJ0tXvSZd95HUO5tQHQAAAAAAAECrQLCOag3LSpEkrW5osH54pzTvAmn1M4HlkVOl65ZJfcYQqAMAAAAAAABoVWgFg2qFKtb/+2O+ij1euez1+MhsWSK9eZ1UclByJknj50onXtDIIwUAAAAAAACA5kHFOqrVpUOsjkuOlddvat32Q3V7sN8nffyQ9PdLA6F6xkDp/31CqA4AAAAAAACgVSNYR7UMw6jQDuZg7R9YuE/6+8XS8kckmdLQq6WrP5Q69GiScQIAAAAAAABAcyFYR42G9QwG67m17LO+Y5X01zOlH5ZJMS7p4ueki/4sxTibbpAAAAAAAAAA0EzosY4ahfqsr995WCUen2Lt1so3NE1p5Wxp8X2S6ZM6niBd/jep84nNOFoAAAAAAAAAaFoE6+3NV3+XDm2v00O6ydS9rm0q9Hi1/90V6trBVfmGu9dL338YuH3SJdK4pyRHfMPGCwAAAAAAAAAtDMF6e/OfV6Vtn9bpIYaka6TAp+XrGja2xEhjZkqnXisZRv3GCAAAAAAAAAAtGMF6e3PiRVKnurdm2bz3iFZuPaC0RKfG9E+vfCOrXRpwuZQ5qGFjBAAAAAAAAIAWjGC9vTn91/V6mG1foe57fLnsBRb99/zz5Yypos86AAAAAAAAALRxlmgPAK1DVsc4dUpwyOP1a/3Ow9EeDgAAAAAAAABEDcE6asUwDJ3eM1WStOqHA1EeDQAAAAAAAABED8E6am1YVookafUPB6M8EgAAAAAAAACIHoJ11NrpPQPB+pc7Dsnt9UV5NAAAAAAAAAAQHQTrqLXjO8WrY7xdbq9f/9mZH+3hAAAAAAAAAEBUEKyj1gzD0LCsQJ/11fRZBwAAAAAAANBOEayjToYF28GszqXPOgAAAAAAAID2iWAddRKqWP9i+0F5vP4ojwYAAAAAAAAAmh/BOuqkd+d4pcTZVVrm19c/HY72cAAAAAAAAACg2RGso04sFkOn9Qi0g1n1A+1gAAAAAAAAALQ/BOuos1Cf9VVMYAoAAAAAAACgHWoRwfqcOXPUo0cPOZ1ODRs2TGvWrKly23POOUeGYRxzufDCC8PbTJ48+Zj7x4wZ0xy70i6c3jPQZ33d9kMq89FnHQAAAAAAAED7EvVg/bXXXlNOTo7uu+8+ffnllxo4cKBGjx6tvXv3Vrr9W2+9pd27d4cvGzZskNVq1WWXXRax3ZgxYyK2++c//9kcu9Mu9ElLULIrRsUenzb8lB/t4QAAAAAAAABAs4p6sP7EE0/ouuuu05QpU9SvXz/NnTtXLpdLL7zwQqXbp6SkKD09PXxZvHixXC7XMcG6w+GI2K5Dhw7NsTvtgsVi6FT6rAMAAAAAAABop6IarHs8Hq1bt07Z2dnhdRaLRdnZ2Vq5cmWtnuP555/XlVdeqbi4uIj1y5YtU+fOndWnTx/dcMMNOnCg6n7gbrdbBQUFERdUL9QOZnUufdYBAAAAAAAAtC9RDdb3798vn8+ntLS0iPVpaWnKy8ur8fFr1qzRhg0bdO2110asHzNmjF5++WUtXbpUjzzyiJYvX66xY8fK5/NV+jwzZ85UUlJS+NK1a9f671Q7MSwrULH+xbZD8tJnHQAAAAAAAEA7Yov2ABri+eef18knn6zTTjstYv2VV14Zvn3yySdrwIABOv7447Vs2TKde+65xzzP9OnTlZOTE14uKCggXK9B34xEJThtOlLq1Te7CjSwa3K0hwQAAAAAAAAAzSKqFesdO3aU1WrVnj17Itbv2bNH6enp1T62qKhIr776qq655poaX6dnz57q2LGjtmzZUun9DodDiYmJERdUz2oxwlXrtIMBAAAAAAAA0J5ENVi32+0aMmSIli5dGl7n9/u1dOlSDR8+vNrHvvHGG3K73frlL39Z4+v8+OOPOnDggDIyMho8ZpQblhXss84EpgAAAAAAAADakagG65KUk5Oj5557Ti+99JI2btyoG264QUVFRZoyZYokaeLEiZo+ffoxj3v++ec1fvx4paamRqwvLCzU7bffrlWrVmnbtm1aunSpfvazn6lXr14aPXp0s+xTezGsZ6BifU3uQfn8ZpRHAwAAAAAAAADNI+o91q+44grt27dPM2bMUF5engYNGqRFixaFJzTdsWOHLJbI/H/Tpk367LPP9OGHHx7zfFarVf/973/10ksv6fDhw8rMzNT555+vBx98UA6Ho1n2qb3ol5GoBIdNR9xebdxdoJOOS4r2kAAAAAAAAACgyRmmaVJqfJSCggIlJSUpPz+ffus1mPLiGn28aZ/uubCvrj2zZ7SHAwAAAAAAgFog/wIaJuqtYNC6DesZaMWzij7rAAAAAAAAANoJgnU0yOnBYH3ttoPy02cdAAAAAAAAQDtAsI4GOSkzUXF2q/JLyvRd3pFoDwcAAAAAAAAAmhzBOhrEZrVoSI8USdKqHw5EeTQAAAAAAAAA0PQI1tFgp/cMBOurcwnWAQAAAAAAALR9BOtosGFZgT7ra3Lpsw4AAAAAAACg7SNYR4MN6JKk2BirDhWXafNe+qwDAAAAAAAAaNsI1tFgMVaLhvboIEla/cPBKI8GAAAAAAAAAJoWwToaxbCsQJ/1z7bsj/JIAAAAAAAAAKBpEayjUYw6sbMk6ZPN+1Tk9kZ5NAAAAAAAAADQdAjW0Sj6ZSSqR6pLbq9fS7/bG+3hAAAAAAAAAECTIVhHozAMQxecnCFJev/r3VEeDQAAAAAAAAA0HYJ1NJpQsP7xpr0q9tAOBgAAAAAAAEDbRLCORtM/M1HdUlwqLfPr4+/2RXs4AAAAAAAAANAkCNbRaAzD0NiT0yVJC2kHAwAAAAAAAKCNIlhHo7ow2A7mo+/2qsTji/JoAAAAAAAAAKDxEayjUZ18XJK6dIhVSZlPyzbtjfZwAAAAAAAAAKDREayjURmGEZ7EdAHtYAAAAAAAAAC0QQTraHQXVGgHU1pGOxgAAAAAAAAAbQvBOhrdwC5JOi45VsUen5Zt2hft4QAAAAAAAABAoyJYR6MzDENjT0qXJL2/gXYwAAAAAAAAANoWgnU0ibHBdjBLN9IOBgAAAAAAAEDbQrCOJjG4a7IykpwqdHv16ff7oz0cAAAAAAAAAGg0BOtoEhaLoTHBdjALv6YdDAAAAAAAAIC2g2AdTebCYDuYJd/ukdtLOxgAAAAAAAAAbQPBOprMKd06KC3RoSNurz6jHQwAAAAAAACANoJgHU3GYjE09qRA1foC2sEAAAAAAAAAaCMI1tGkLgi2g1n87R55vP4ojwYAAAAAAAAAGo5gHU1qSPcO6pTg0JFSr1ZsoR0MAAAAAAAAgNaPYB1NymoxNPakdEm0gwEAAAAAAADQNhCso8mF2sF8+E0e7WAAAAAAAAAAtHoE62hyp/ZIUcd4hwpKvfp8K+1gAAAAAAAAALRuBOtoclaLoTEnpUmS3v86L8qjAQAAAAAAAICGIVhHs7jgpEA7mA++zVOZj3YwAAAAAAAAAFovgnU0i9OyUpQaZ9fh4jKt+uFAtIcDAAAAAAAAAPVGsI5mYbNadH7/dEnSwq93R3k0AAAAAAAAAFB/BOtoNheeHGwH880eeWkHAwAAAAAAAKCVIlhHszm9Z4o6uGJ0sMij1bkHoz0cAAAAAAAAAKgXgnU0G5vVotHBdjALaAcDAAAAAAAAoJUiWEezuiDUDmZDnnx+M8qjAQAAAAAAAIC6I1hHsxp+fKqSXTE6UOTR6twD0R4OAAAAAAAAANQZwTqaVYzVovP7pUmSFtIOBgAAAAAAAEArRLCOZhdqBzP/q106UOiO8mgAAAAAAAAAoG4I1tHszurdSScdl6hCt1dPLf0+2sMBAAAAAAAAgDppEcH6nDlz1KNHDzmdTg0bNkxr1qypctt58+bJMIyIi9PpjNjGNE3NmDFDGRkZio2NVXZ2tr7/ngC3pbBYDN09tq8k6R+rd+iHfYVRHhEAAAAAAAAA1F7Ug/XXXntNOTk5uu+++/Tll19q4MCBGj16tPbu3VvlYxITE7V79+7wZfv27RH3P/roo3rqqac0d+5crV69WnFxcRo9erRKS0ubendQSyN6ddT/nNhZXr+pRxZ9F+3hAAAAAAAAAECtRT1Yf+KJJ3TddddpypQp6tevn+bOnSuXy6UXXnihyscYhqH09PTwJS0tLXyfaZqaNWuW7rnnHv3sZz/TgAED9PLLL2vXrl2aP39+M+wRamv62BNlMaQPvtmjtdsORns4AAAAAAAAAFArUQ3WPR6P1q1bp+zs7PA6i8Wi7OxsrVy5ssrHFRYWqnv37uratat+9rOf6Ztvvgnfl5ubq7y8vIjnTEpK0rBhw6p9TjS/3mkJuuLUbpKkPy7YKNM0ozwiAAAAAAAAAKhZVIP1/fv3y+fzRVScS1JaWpry8vIqfUyfPn30wgsv6J133tHf//53+f1+jRgxQj/++KMkhR9Xl+d0u90qKCiIuKB53HZeb7nsVq3feVgLvt4d7eEAAAAAAAAAQI2i3gqmroYPH66JEydq0KBBOvvss/XWW2+pU6dO+utf/1rv55w5c6aSkpLCl65duzbiiFGdzglO/b+zjpckPbLoO7m9viiPCAAAAAAAAACqF9VgvWPHjrJardqzZ0/E+j179ig9Pb1WzxETE6PBgwdry5YtkhR+XF2ec/r06crPzw9fdu7cWdddQQNcd1aWOic4tPNgif62cnvNDwAAAAAAAACAKIpqsG632zVkyBAtXbo0vM7v92vp0qUaPnx4rZ7D5/Pp66+/VkZGhiQpKytL6enpEc9ZUFCg1atXV/mcDodDiYmJERc0H5fdpt+ef4Ik6emPtii/uCzKIwIAAAAAAACAqkW9FUxOTo6ee+45vfTSS9q4caNuuOEGFRUVacqUKZKkiRMnavr06eHtf//73+vDDz/UDz/8oC+//FK//OUvtX37dl177bWSJMMwNHXqVP3hD3/Qv//9b3399deaOHGiMjMzNX78+GjsImrh0iFd1SctQfklZZr98ffRHg4AAAAAAAAAVMkW7QFcccUV2rdvn2bMmKG8vDwNGjRIixYtCk8+umPHDlks5fn/oUOHdN111ykvL08dOnTQkCFD9Pnnn6tfv37hbe644w4VFRXp+uuv1+HDh3XGGWdo0aJFcjqdzb5/qB2rxdD0C07U5BfX6qXPt2vi8B7qmuKK9rAAAAAAAAAA4BiGaZpmtAfR0hQUFCgpKUn5+fm0hWlGpmnqV8+v0Wdb9mvcwEw9fdXgaA8JAAAAAACgTSL/Ahom6q1ggBDDCFStG4b07n92af3Ow9EeEgAAAAAAAAAcg2AdLUr/zCRdPLiLJOmhBRvFDyoAAAAAAAAAtDQE62hxpo0+QQ6bRWu2HdTib/dEezgAAAAAAAAAEIFgHS1ORlKsrj0zS5L08KLvVObzR3lEAAAAAAAAAFCOYB0t0q/PPl6pcXb9sK9Ir67ZEe3hAAAAAAAAAEAYwTpapARnjKZm95YkzVryvY6UlkV5RAAAAAAAAAAQQLCOFuvK07qpZ6c4HSjyaO7yrdEeDgAAAAAAAABIIlhHCxZjteiuMSdKkv66/Act+O/uKI8IAAAAAAAAAAjW0cKd1y9NFw8+Tl6/qZv/+aXmf/VTtIcEAAAAAAAAoJ0jWEeLZhiG/nTZQF02pIv8pnTb6+v1+tqd0R4WAAAAAAAAgHaMYB0tntVi6JFLBmjCsG4yTemON/+rv6/aHu1hAQAAAAAAAGinCNbRKlgshv4w/iRNGdlDknTP/A16/rPc6A4KAAAAAAAAQLtEsI5WwzAMzbion3599vGSpAff+1bPLNsa5VEBAAAAAAAAaG8I1tGqGIahO8f00a3n9pYkPbLoO81aslmmaUZ5ZAAAAAAAAADaC4J1tDqGYei2807Q7aP7SJJmLflef/pgE+E6AAAAAAAAgGZBsI5W68ZRvXTPhX0lSX9ZtlV/XLCRcB0AAAAAAABAkyNYR6t27Zk99eDP+kuS/u+zXM145xv5/YTrAAAAAAAAAJoOwTpavV8N76FHLjlZhiH9bdV23fXWf+Xx+qM9LAAAAAAAAABtFME62oQrTu2mJy4fKIshvf7Fj/r5X1Zoy97CaA8LAAAAAAAAQBtEsI424+eDu+i5iUPVwRWjb3YV6KKnP9XfV22n7zoAAAAAAACARkWwjjbl3L5pWjT1LJ3Zu6NKy/y6Z/4GXffyFzpQ6I720AAAAAAAAAC0EQTraHPSEp16acppuufCvrJbLVqyca/GPPmplm/eF+2hAQAAAAAAAGgDCNbRJlkshq49s6fm3zhSvTvHa98Rtya9sEa/f/dblZb5oj08AAAAAAAAAK0YwTratH6ZiXr35jM0aXh3SdILK3I1fs4Kbco7EuWRAQAAAAAAAGitCNbR5jljrHrgZyfphclD1THeru/yjmjc7M80b0UuE5sCAAAAAAAAqDOCdbQb/3Nimt6/9Syd06eTPF6/7n/3W01+ca22HyiK9tAAAAAAAAAAtCIE62hXOiU49OLkU/XA//aX3WbR8s37lP3Ecv3hvW+VX1wW7eEBAAAAAAAAaAUI1tHuGIahSSN6aMHNZ+jM3h1V5jP1f5/l6qw/faznP8uVx+uP9hABAAAAAAAAtGCGSZPpYxQUFCgpKUn5+flKTEyM9nDQxJZv3qeHFmzUpj2BCU27p7p015gTNeakdBmGEeXRAQAAAAAAND7yL6BhCNYrwRdL++Pzm3rji516fPFm7TviliQN7d5Bv7uwrwZ36xDl0QEAAAAAADQu8i+gYQjWK8EXS/tV5Pbqr5/8oGc/2arSskBLmIsGZOjOMSeqa4oryqMDAAAAAABoHORfQMMQrFeCLxbk5Zfq8Q836V9f/ijTlOxWiyaP7KFfn328UuLs0R4eAAAAAABAg5B/AQ1DsF4JvlgQ8u2uAj20cKM+27JfkuSyWzVhWDddd2ZPdU50Rnl0AAAAAAAA9UP+BTQMwXol+GJBRaZpatmmfXp88SZt+KlAkmS3WXT50C76f2cdT4sYAAAAAADQ6pB/AQ1DsF4JvlhQGdM0tWzzPs3+aIvWbT8kSbJZDI0ffJx+c87x6tkpPsojBAAAAAAAqB3yL6BhCNYrwRcLqmOaplb9cFBzPt4SbhFjGNKFJ2foxlG91DeDzwwAAAAAAGjZyL+AhiFYrwRfLKitr3Yc0pyPt2jJxr3hddl9O+vGUb00uFuHKI4MAAAAAACgauRfQMMQrFeCLxbU1be7CjRn2RYt/Hq3QkdUv4xEjTkpXWNOSlfvzvEyDCO6gwQAAAAAAAgi/wIahmC9EnyxoL627ivUM8u2av5XP8nrLz+0enaK05j+gZD95OOSCNkBAAAAAEBUkX8BDUOwXgm+WNBQB4s8WrJxjz7YkKdPv98vj88fvu+45Fid3z9NY0/K0JDuHWS1ELIDAAAAAIDmRf4FNAzBeiX4YkFjOlJapo837dMHG/L08aa9Kvb4wvd1jLfrvH7pOq9fZ53eM1Uuuy2KIwUAAAAAAO0F+RfQMATrleCLBU2ltMynTzbv06Jv8rTk2z0qKPWG77NbLTo1q4POPqGTzj6hs05Ioy87AAAAAABoGuRfQMMQrFeCLxY0hzKfX6t+OKBFG/K0bNM+/XS4JOL+9ESnzjqho84+obPO6NVRSa6YKI0UAAAAAAC0NeRfQMMQrFeCLxY0N9M09cP+Ii3ftE+ffL9PK7cekNtb3pfdYkiDuibr7BM666wTOmpAl2R6swMAAAAAgHoj/wIahmC9EnyxINpKy3xak3tQn2zep+Wb9+n7vYUR9yfFxmhkr1Sd2buTzujVUV1TXFEaKQAAAAAAaI3Iv4CGaRHB+pw5c/SnP/1JeXl5GjhwoJ5++mmddtpplW773HPP6eWXX9aGDRskSUOGDNFDDz0Usf3kyZP10ksvRTxu9OjRWrRoUa3GwxcLWppdh0v0yeZ9WrZpn1Zs3a8jFXqzS1JWxzid2bujzuzdSaf3TFGCk7YxAAAAAACgam0p//L7/fJ4PNEeBtoAu90ui8VSq22jHqy/9tprmjhxoubOnathw4Zp1qxZeuONN7Rp0yZ17tz5mO0nTJigkSNHasSIEXI6nXrkkUf09ttv65tvvtFxxx0nKRCs79mzRy+++GL4cQ6HQx06dKjVmNrSFwvaHq/Pr//+lK9PN+/Xp9/v01c7D8vnLz+MbRZDp3TroDN7d9SpWSnqkRqnzgkOWWgdAwAAAAAAgtpK/uXxeJSbmyu/31/zxkANLBaLsrKyZLfba9w26sH6sGHDdOqpp2r27NmSAmeYunbtqptvvll33XVXjY/3+Xzq0KGDZs+erYkTJ0oKBOuHDx/W/Pnz6zWmtvLFgvahoLRMq7Ye0KffB4L2bQeKj9nGYbOoa4pL3YKX0O3uqS517eBSrN0ahZEDAAAAAIBoaQv5l2ma2rFjh8rKypSZmVnrSmOgMn6/X7t27VJMTIy6desmw6i+SNXWTOOqlMfj0bp16zR9+vTwOovFouzsbK1cubJWz1FcXKyysjKlpKRErF+2bJk6d+6sDh066H/+53/0hz/8QampqY06fqAlSHTG6Pz+6Tq/f7okaceBYn26ZZ8+3bxf3+4u0E+HS+T2+rVlb6G2HNWrPaRTgiMQtKe41C3VpR6pceqWGlhOibPX+EUCAAAAAADQ3Lxer4qLi5WZmSmXi/nn0HCdOnXSrl275PV6FRNTfavlqAbr+/fvl8/nU1paWsT6tLQ0fffdd7V6jjvvvFOZmZnKzs4OrxszZowuvvhiZWVlaevWrbr77rs1duxYrVy5UlbrsZW5brdbbrc7vFxQUFDPPQKir1uqSxNSu2vCsO6SpDKfX7sPl2rHwWLtOFis7QeLtDN0+0CxjpR6te+IW/uOuLVu+6Fjni/eYVP31EB1e7eUOHUPBu/9MhKV5KKXOwAAAAAAiA6fzydJtWrbAdRG6LPk8/ladrDeUA8//LBeffVVLVu2TE6nM7z+yiuvDN8++eSTNWDAAB1//PFatmyZzj333GOeZ+bMmXrggQeaZcxAc4uxWtQtNVCJXpn84jJtP1ik7QdCYXv57d35pSp0e/XNrgJ9s+vYE07dUlw6uUuSTj4ucDkpM4mwHQAAAAAANCt+aY/GUpfPUlSD9Y4dO8pqtWrPnj0R6/fs2aP09PRqH/vYY4/p4Ycf1pIlSzRgwIBqt+3Zs6c6duyoLVu2VBqsT58+XTk5OeHlgoICde3atQ57ArReSa4YDXAla0CX5GPuKy3zaWewsn37wWLtOFCkbQeK9cP+Qu08WBKugl/w393hx3RPdemk4wjbAQAAAAAA0HZFNVi32+0aMmSIli5dqvHjx0sKNIlfunSpbrrppiof9+ijj+qPf/yjPvjgAw0dOrTG1/nxxx914MABZWRkVHq/w+GQw+Go1z4AbZkzxqreaQnqnZZwzH2Hiz3a8FOBvv4pXxt+ytd/fzqsnQdLAiH8gciwvWO8I9BOJtjDvWJbmVR6uAMAAAAAANRbjx49NHXqVE2dOrVW2y9btkyjRo3SoUOHlJyc3GTjmjdvnqZOnarDhw832WtEU9RbweTk5GjSpEkaOnSoTjvtNM2aNUtFRUWaMmWKJGnixIk67rjjNHPmTEnSI488ohkzZuiVV15Rjx49lJeXJ0mKj49XfHy8CgsL9cADD+iSSy5Renq6tm7dqjvuuEO9evXS6NGjo7afQFuT7LLrjN4ddUbvjuF1VYXt+wvd2l9YdQ/3rsGJU7sHW9Z0T4lTtxSXMpOdslmZ0RsAAAAAALQd55xzjgYNGqRZs2Y1yvOtXbtWcXFxtd5+xIgR2r17t5KSkhrl9durqAfrV1xxhfbt26cZM2YoLy9PgwYN0qJFi8ITmu7YsUMWS3mw9swzz8jj8ejSSy+NeJ777rtP999/v6xWq/773//qpZde0uHDh5WZmanzzz9fDz74IFXpQBOrLGwvKC3T9v3F5X3cDwRu7zhQrN0FgR7uG3cXaOPuY3u4Wy2GjkuOVbcUVyB8T3WpW0rwkupSopMWMwAAAAAAoO0xTVM+n082W83xbadOner03Ha7vcY23KiZYZqmGe1BtDQFBQVKSkpSfn6+EhMToz0coM0qLfPpx0Ml2hEM3bcfKA70dA/2bvd4/dU+voMrRj06xikrNU49OsaFb3fvSOgOAAAAAEB12kL+VVpaqtzcXGVlZcnpdEZ7OLUyefJkvfTSSxHrcnNztW3bNo0aNUoLFy7UPffco6+//loffvihunbtqpycHK1atUpFRUXq27evZs6cqezs7PDjj24FYxiGnnvuOS1YsEAffPCBjjvuOD3++OP63//9X0nHtoIJtWx57bXXNHXqVO3cuVNnnHGGXnzxxXBrba/Xq5ycHL388suyWq269tprlZeXp/z8fM2fP7/Sfa2sFcwzzzyjxx57TDt37lRWVpbuuece/epXv5IUOJnwwAMP6IUXXtCePXuUmpqqSy+9VE899ZQk6S9/+Yv+/Oc/a+fOnUpKStKZZ56pf/3rX43xZwmry2cq6hXrANovZ4xVvTrHq1fn+GPu8/tN7T3i1o6Dxdp+oCgicN9xoFgHijw6VFymQzsO66sdh495fGqcPRC2p8Ypq6NLXTq4lBJnj7g4Y6zNsJcAAAAAAKA5mKapkjJfVF47NsZaqznknnzySW3evFknnXSSfv/730sKVJxv27ZNknTXXXfpscceU8+ePdWhQwft3LlTF1xwgf74xz/K4XDo5Zdf1rhx47Rp0yZ169atytd54IEH9Oijj+pPf/qTnn76aU2YMEHbt29XSkpKpdsXFxfrscce09/+9jdZLBb98pe/1LRp0/SPf/xDUqA99z/+8Q+9+OKL6tu3r5588knNnz9fo0aNqvV79Pbbb+vWW2/VrFmzlJ2drffee09TpkxRly5dNGrUKL355pv685//rFdffVX9+/dXXl6e/vOf/0iSvvjiC91yyy3629/+phEjRujgwYP69NNPa/3aTYFgHUCLZLEYSk9yKj3JqdOyjv3SL3R7tW1/oNJ924Ei5e4v0rb9Rdp2oEj7Cz06UBS4VNbXPcRlt0aG7a7AdccEh3qkutQ9NRDMx9oJ4AEAAAAAaOlKynzqN+ODqLz2t78fLZe95qg1KSlJdrtdLper0nYsv//973XeeeeFl1NSUjRw4MDw8oMPPqi3335b//73v3XTTTdV+TqTJ0/WVVddJUl66KGH9NRTT2nNmjUaM2ZMpduXlZVp7ty5Ov744yVJN910Uzj4l6Snn35a06dP189//nNJ0uzZs7Vw4cIa97eixx57TJMnT9ZvfvMbSQpX4j/22GMaNWqUduzYofT0dGVnZysmJkbdunXTaaedJinQLjwuLk4XXXSREhIS1L17dw0ePLhOr9/YCNYBtErxDptOOi5JJx137EQbR0rLtP1AcThszz1QpLz8Uh0s8oQvXr+pYo9PxZ4S/XiopNrXSk90qnuqS1nBdjM9UuPUo2NgklVCdwAAAAAA0FiGDh0asVxYWKj7779fCxYs0O7du+X1elVSUqIdO3ZU+zwDBgwI346Li1NiYqL27t1b5fYulyscqktSRkZGePv8/Hzt2bMnHHJLktVq1ZAhQ+T3V9/Gt6KNGzfq+uuvj1g3cuRIPfnkk5Kkyy67TLNmzVLPnj01ZswYXXDBBRo3bpxsNpvOO+88de/ePXzfmDFj9POf/1wul6vWr9/YCNYBtDkJzpgqQ3cp8NOwglKvDgWr2g8Fw/ZAexmP8vJLtT1YBV9Q6lVeQanyCkq1OvfgMc+Vnhioqk9LdCgt0am0RKc6J5TfTk90KjHWVqufgwEAAAAAgPqLjbHq29+PjtprN4a4uLiI5WnTpmnx4sV67LHH1KtXL8XGxurSSy+Vx+Op9nliYiLnnjMMo9oQvLLtm3tqzq5du2rTpk1asmSJFi9erN/85jf605/+pOXLlyshIUFffvmlli1bpg8//FAzZszQ/fffr7Vr1yo5OblZxxlCsA6g3TEMQ0mxMUqKDUx+Wp1DRR5tO1AUbDcT6Pe+bf+xoXt1HDZLMGh3qGO8Q6nxdqXEOdQx3q7UuMBy6HZSbIwsFkJ4AAAAAADqyjCMWrVjiTa73S6fr3a94FesWKHJkyeHW7AUFhaG+7E3l6SkJKWlpWnt2rU666yzJEk+n09ffvmlBg0aVOvn6du3r1asWKFJkyaF161YsUL9+vULL8fGxmrcuHEaN26cbrzxRp144on6+uuvdcopp8hmsyk7O1vZ2dm67777lJycrI8++kgXX3xxo+1rXbT8TxoARFGHOLs6xNk1uFuHY+47VOTRjoPF2lNQGry4A9dH3NobDNwPF5fJ7fUHJl09WFzj61kthjq4AkF7pwSHOgWr3zsnONQ5IRDOd05wqnOig8lXAQAAAABohXr06KHVq1dr27Ztio+Pr3JCUUnq3bu33nrrLY0bN06GYejee++tU/uVxnLzzTdr5syZ6tWrl0488UQ9/fTTOnToUJ1+oX/77bfr8ssv1+DBg5Wdna13331Xb731lpYsWSJJmjdvnnw+n4YNGyaXy6W///3vio2NVffu3fXee+/phx9+0FlnnaUOHTpo4cKF8vv96tOnT1Ptco0I1gGgnkKhe3VKy3zad8QdDt4PFLkDk6sWunWg0KMDRe7ARKuFHuWXlMnnN7W/0K39hW59l3ek2udOcNrCoXsHl12xdqvi7Fa5HDa5YoLXdmvwYgvfF++wKjFYse+wEc4DAAAAANCcpk2bpkmTJqlfv34qKSlRbm5ulds+8cQTuvrqqzVixAh17NhRd955pwoKCppxtAF33nmn8vLyNHHiRFmtVl1//fUaPXq0rNba5wrjx4/Xk08+qccee0y33nqrsrKy9OKLL+qcc86RJCUnJ+vhhx9WTk6OfD6fTj75ZL377rtKTU1VcnKy3nrrLd1///0qLS1V79699c9//lP9+/dvoj2umWE2d7OcVqCgoEBJSUnKz89XYmJitIcDoJ3weP06VOwJBuse7Tvi1t4jpdpb4A6H83uD60rLGufstDPGouRYe7g1TihwT3YFl502uRw2xdltcjmsgetgWB8XDu5tstK+BgAAAABalbaQf5WWlio3N1dZWVlyOp3RHk674vf71bdvX11++eV68MEHoz2cRlOXzxQV6wDQQtjDvdir/+IOTb66Lxi67zlSqoISr4o8XpV4fCpy+1Ts8arYU35d5PGp2B24faS0TEfcXpmmVFrmV15ZzX3ia+KMsSjOblOSK0bJsTHq4LIHb9vVwRUI6pNddiW7gvfFxijJFaMEBxO7AgAAAADQ0m3fvl0ffvihzj77bLndbs2ePVu5ubn6xS9+Ee2hRQ3BOgC0MhUnX+3VOaFez+H3mzpS6lV+Sdkxl8MlgbY0BSVlKijxqtjjVZHHFwjtPV4Vu4PXHp98/sCPnkrL/Cot8+hAUfWzkh/NYii8L5VVzJdf7MFwPnh/rF3OGAuhPAAAAAAAzcBisWjevHmaNm2aTNPUSSedpCVLlqhv377RHlrUEKwDQDtksRhKcgWqxuvLNE25vf5w4F7o9iq/uEyHisuUX+LRoeIyHS4u0+Fijw4Xl+lQcSCwP1QcuM/j9ctvSoeCj6kru82i5IggPhC+xztsirVbFRsTaFnjDF7HxljltFvlirEqNtjOxmGzym6zyG61yG6zKMZqUYzVILAHAAAAAKCCrl27asWKFdEeRotCsA4AqBfDMOSMCQTXNU3iWpnSMl9ktXxxqGK+LFwxn18SCOZD6wtKAmG912/K4/UHe867G3m/pBirRQ6rRTEVQne7zSJH+GKVI6bCbZsluGyVM8Yipy0wQWyyK1CFnxwbaIUT6ltvs1oadcwAAAAAAKB5EawDAKIiFMrX1FP+aKZpqsjjKw/di8vD+EPFHhW7fSr2+FRS5lOJx6uSssByaZmv/LbHp+KywLpQ5Xz58wcmkvV4/VLjZvZhCY5gP/oKrW2OboOTXKE9TmhdvMMm05SKPF4VuX0qdHtVFLwUur3BXw74wus8Pr9kSn7TlN8M7JvfNGWGlhVabyrGGvwFgKu8L364BU9wLJwQAAAAAAAggGAdANCqGIaheIdN8Q6bjkuObZTn9AUr4D1ev9y+QNhe5itf5/H55C7zy+3zB669Prm9/sClzCdPeH3gvhKPTwWloVY45VX5hW6vJOmI26sjbq9+PFRSp3FaLUa4r300JDhtgSp8Z0zwxEigOt8ZE6jgd8ZYg8uWQOudGKtirIZKyvyBXv1HTawbuRy4HXmSI3Jfj95zm8UI/5og8MsCq+zWo9cF1rtirEpw2pQYG6MEp00JztC1TYnOwD6Fli2GoeKywIS/hcFJf8MnMTzlJy6K3D55/f6I+QBCv0wInQxxxlib/g/ThpmmqTKfSYsmAAAAAC0OwToAoN2zWoxAX3a7VVL9+87XpMznD7SzqdD+5nCJJ9gGxxueOPbotjj5xWXy+PwRobrVYijOblW8wyaXw6Y4h03xDqvi7IGTDnEOm+w2iyyGZDEMKXgdWjYUOElhMQwZRqBK/3CJJ3wi4HBwbIeLy3SkNHhCoNQbvF23EwLtmcNmCYfsic4YWS2GrJbA+26xGLKG/i4WQ1bDkMUSWLZajGNaDTnDLYgC65wx5fdJks8fOEnkN83wdeB2YMJiX4X1Xp8pr98vr9+Uz2cGrv2myvz+iGW/aYY/I5ZgsB36HBlVfLZCF6sl8BkL7K8qrA88T6E78HkqdJepMPjZOuL2qrDUG7wvcDIqFKwnxdqVFGuLOIGRVMmEx6YplXp9wUmVA79McXvLb4fWu71++UyzwrGg8O3wdXhd4JizWSyyWQ3FWC2yWgzFWAzZrMF1wftslsD9Ff9W1bWPslkNlXp8KgqeXCoJ3i4JnnQK3S5y+1Tq9clhtch51DwSgds2xdot4eVYu1V+U3IH99UTPBno8ZafHKx42xJu71V+wizWHmpxFXnCzDAU+GVQ8NdA5b8SKr8uLgv8Osjj88tuLX/fKt6OsVpktwXe19AcF6FjwWoxwu+7teIxUuE4Ka3wi6TQCbry296IsSU4bOqY4FDHeIc6Ba87xtsV77DV6aRNaZlPR0q9KigNtCcr85kVTuQZslutwTk7Ik/yhV7DNM3AyVjvUSdqy/zBk7TlfxtL6Bg6ar8jjq/gcl3YbcG/pT1wHRtjbTO/SPL5zfLPYujzWHbU56HCr9icMdbAydXYwEnW0C/FAieQG2eSdNM0VVrmP2oC+PKTy16/qeQKk7Qnu+yKs1tr/dqmaeqI26uDhYFJ5A8WeXSwyK38kjJZLZaIY9oZOgkeE3lsO2xWef3+8PtTEn6vvBVulx9PVsMI/7vDZQ/9G8Qml8Ma/jdInN2qOIdNMQ38bIVOsHr9fpV5A/+dKgv+myjcgi/G2uDXaUlMM/DfYTP468JQnUFoufx2eRGC5ajvCWvov2Ft9KS0aQaO9cJgoUPo15tur1/xDpsSKxRTxMbU/ngCgLoiWAcAoJnEWC1KjXcoNd5Rp8eF/qc8v6RMNmugYt9ha5z/4a8Nr88fDvoPF5epoLRM7rLI4LI0HFyGQsvAbY/PL1dM4H+uY+1WxdkD/xPuslsDJwQqLtutslgq36ej15oq/6VBKIQq84V+YeCv8GuDwHWxx1sehoVC3NKy8HVBSSA8qMhiKBgO2BTnsFa4Xb5sNQwVlJafjMivMDeA35TcXr/2FLi1p6CJ+gq1E2U+U/sL3dpfyPuIxuWwWSoE7Q51SrBLChzXR0q9KigpC4boge8Pj9dfr9eJsQZCL3c9H9/U7NZAABua/NsZPDljKPBdW1bhZJzXZ8rrC972m+GQ02+a5SdBjjkhomPWh05aVTxpF1oOBYKh736v3y+vL3BSIvT6nnDY6leZP7CuMX/UFWM1gqF74BJjMWQq1FJNkW3V/IH1Ct5f5jNV5A6dHPPKrOO4QicTK7Zm6+CKkctu1eGSMh0Ih+huHSoKnHxvqSzBE2TlJ4LKTygffZ+pwOerzBf45WBZ8HNWG1aLIafNEm416AifULCEfz1WMag2JemotnhmcBufv/y1vUeNpSz0GfQH1lsMyWa1KMZiKMZmCZ4oDJw8tFmN8MlEm8UiwwgUWVQ8QVDmDe1v3fe5NgxDESfnrMGTqC6HVa6YwL/NQv8Gi7Xb5Aoe+3GOwHeB129G/nsv+G+9kgr//nMHb5sKnYQ2ZLVYgtdG+NpaYdliGOFfIoZODpQvK2LZ6/OXtz0MtTw86leO1bFZjMhfLToCJ9PiHLaIX62GTnqW/9vSF/FvyqE9UvTcxKGN9JcB0FYQrAMA0MIZRsWK+uZnq+cJgdamzOdXYWmgejDeYWtQtaJpmip0eyPC9iOlZeFK8ND/uPuCPe99flW4Hbh4fH6VhipaK7QaOrrKtdQbOCEQqgYPVb6HK+ON8ip5q0WyWSwR/6Nrsxrl1dgWQ1Zr+f8UGwqFD5F9+f3BVCIUMPnNyP79FffFbwYq5v2mKV+Fx8Y5rEpwBuYOCLXhiXfEKN5pC1ebxTttcsXYVOTxHnXywhOxHJrcOL+kTIYkRzBYKQ9ZysMWp80Svt9qKBykhCr/KgYv/oq3/WY4vAuHi8FgxVshcAn/T3oVraPCt70+lflClYYKVpwHTj7FVjjZ5LIHfokSG6xOL/P5IyrCq5pHoqTMJ0v4lw/llfKhiZjtR63zm6ZKQifGQifJvOXVvaWhuScqcNgsgTAmGMTEhm/bFBtjkctuk81ihN+bisFR6HYoIA2EVYFfEfj95cdH+LMT/CxVPG6ctsjXdYVvl79/ob97obtM+494tC94gmb/EbeKPIG/x4+HSurUmsswFPyMxshus4T/3mUVApijg7HA3/rYFCj89wn9AiX4d4mxWSTTDL8foV+gRB5P5ffV9pvKVOAXSqFK7lCA5fEF9qEg+Aul1s4wFK7Gd0Z8NqzhINFhs8rt9amgJPD9EvoVQkGpN3wy4UBRIMBuLC57qMI7eG23ymIY4XliDgd/oVafk4kuu1UpcXalxtmVEmdXsssun9+MOPld8dg++hc8MVajwq9fjj6mym87Y6zy+81wa7TCCu3RAnPABG6Hwn6/Kfmr+PzXV2UnqnyhMXl81Tyy8flDc/NIUjO/dm2YpuQ1g+9/cHhFHp8OFEV1WI3GMKT4CkUPdptVRW5v+OSoL3gC8FBxmQ4VlzXotfJLGvZ4AG0TwToAAIACvyjoEGdvlOcyDCPYxz1GXRvlGdu3JFeMMhtpToWWJFQV2Zy/QGkIv99UqTcQxsbGVP0Lk9ai2OONDNsL3dp3xC2LYSgxOBdDoC1IeaVjYmyM4u22GvfdHzw55vEFKqpD7bxCLZyObhETDaZphtskVWzlU7FliqlAiGm1BCpyraH2R8GTcuVtkQLVuMecMKyiPZXPbwZaWlQ4gRWuGq5QBR464RVqHVTeRihwMjDcSsgWrBi2BqruG3JMmaap4uBcKeHQvaRMXr+/vIWaFGjVo8gq+9B9McGTTnH2QHuUOLutVsdMqL1FaI6Ww8We8K/FDhV7VOLxKdkVo5RgeJ4a51BKfCBMb8icHqZpNvpnMfRrMbfXf8xJodBJMrPC7VDFf+jvG/qbx1gDn7nyym8jorVSxZPMFU8WlFSornYHT0CXt/oq/9uFllVhOVRhXvH1Q22+Klaj2ywWmTLDJwnDFee+YEV68CRb6JcWpmkGW0VZKt3PiH0OfMAqjDFy7AqPu1z4OAuecPMFjznTjDwWj27vU3G+m5KI2z7ZrJFtwmLt1sDJaVvgWKvYZsgwjGCQ7Q8H2hXby3n9/nA7usAJQSP8k8TQroT+tuXLgUKBeEd5y6FQiB7vqL7NS+hYDv86MfirxYKSQOhe5PaGW++Vt+6yRi4Hv6sdNovincRnaHl69OihqVOnaurUqZICx9Dbb7+t8ePHV7r9tm3blJWVpa+++kqDBg2q9+s21vPUZPLkyTp8+LDmz5/fZK/RUHwzAAAAAFEQ+Fl865ng1mIx5LK3nf99cNlt6pZqU7dUV6M/t8ViyGmxtugJjI1wX32rkqM9mBbECPYPj3PYlJHU/K8daI9ma9aTiU1xgicQSjbOyeqqVPwMJzXhHDlonSoey+lJzmgPB2gWu3fvVocOHRr1OSsLt7t27ardu3erY8eOjfparVHb+ZcxAAAAAAAAALRD6enpzfI6Vqu12V6rpWs7U2cDAAAAAAAAQAv27LPPKjMzU35/5Pw1P/vZz3T11VdLkrZu3aqf/exnSktLU3x8vE499VQtWbKk2uc1DCOisnzNmjUaPHiwnE6nhg4dqq+++ipie5/Pp2uuuUZZWVmKjY1Vnz599OSTT4bvv//++/XSSy/pnXfeCbSkMgwtW7ZM27Ztk2EYWr9+fXjb5cuX67TTTpPD4VBGRobuuusueb3lc7ecc845uuWWW3THHXcoJSVF6enpuv/+++v0vrndbt1yyy3q3LmznE6nzjjjDK1duzZ8/6FDhzRhwgR16tRJsbGx6t27t1588UVJksfj0U033aSMjAw5nU51795dM2fOrNPrV4aKdQAAAAAAAACtn2lKZcXRee0Y17ETIFTisssu080336yPP/5Y5557riTp4MGDWrRokRYuXChJKiws1AUXXKA//vGPcjgcevnllzVu3Dht2rRJ3bp1q/E1CgsLddFFF+m8887T3//+d+Xm5urWW2+N2Mbv96tLly564403lJqaqs8//1zXX3+9MjIydPnll2vatGnauHGjCgoKwgF1SkqKdu3aFfE8P/30ky644AJNnjxZL7/8sr777jtdd911cjqdEeH5Sy+9pJycHK1evVorV67U5MmTNXLkSJ133nk17o8k3XHHHXrzzTf10ksvqXv37nr00Uc1evRobdmyRSkpKbr33nv17bff6v3331fHjh21ZcsWlZQEJqh/6qmn9O9//1uvv/66unXrpp07d2rnzp21et3qEKwDAAAAAAAAaP3KiqWHMqPz2nfvkuxxNW7WoUMHjR07Vq+88ko4WP/Xv/6ljh07atSoUZKkgQMHauDAgeHHPPjgg3r77bf173//WzfddFONr/HKK6/I7/fr+eefl9PpVP/+/fXjjz/qhhtuCG8TExOjBx54ILyclZWllStX6vXXX9fll1+u+Ph4xcbGyu12V9v65S9/+Yu6du2q2bNnyzAMnXjiidq1a5fuvPNOzZgxQxZLoGHKgAEDdN9990mSevfurdmzZ2vp0qW1CtaLior0zDPPaN68eRo7dqwk6bnnntPixYv1/PPP6/bbb9eOHTs0ePBgDR06VFJgcteQHTt2qHfv3jrjjDNkGIa6d+9e42vWBq1gAAAAAAAAAKCZTJgwQW+++abcbrck6R//+IeuvPLKcAhdWFioadOmqW/fvkpOTlZ8fLw2btyoHTt21Or5N27cqAEDBsjpLJ+8d/jw4cdsN2fOHA0ZMkSdOnVSfHy8nn322Vq/RsXXGj58eMRk2CNHjlRhYaF+/PHH8LoBAwZEPC4jI0N79+6t1Wts3bpVZWVlGjlyZHhdTEyMTjvtNG3cuFGSdMMNN+jVV1/VoEGDdMcdd+jzzz8Pbzt58mStX79effr00S233KIPP/ywTvtYFSrWAQAAAAAAALR+Ma5A5Xi0XruWxo0bJ9M0tWDBAp166qn69NNP9ec//zl8/7Rp07R48WI99thj6tWrl2JjY3XppZfK4/E02nBfffVVTZs2TY8//riGDx+uhIQE/elPf9Lq1asb7TUqiomJiVg2DOOYPvMNMXbsWG3fvl0LFy7U4sWLde655+rGG2/UY489plNOOUW5ubl6//33tWTJEl1++eXKzs7Wv/71rwa9JsE6AAAAAAAAgNbPMGrVjiXanE6nLr74Yv3jH//Qli1b1KdPH51yyinh+1esWKHJkyfr5z//uaRABfu2bdtq/fx9+/bV3/72N5WWloar1letWhWxzYoVKzRixAj95je/Ca/bunVrxDZ2u10+n6/G13rzzTdlmma4an3FihVKSEhQly5daj3m6hx//PGy2+1asWJFuI1LWVmZ1q5dq6lTp4a369SpkyZNmqRJkybpzDPP1O23367HHntMkpSYmKgrrrhCV1xxhS699FKNGTNGBw8eVEpKSr3HRSsYAAAAAAAAAGhGEyZM0IIFC/TCCy9owoQJEff17t1bb731ltavX6///Oc/+sUvflGn6u5f/OIXMgxD1113nb799lstXLgwHDBXfI0vvvhCH3zwgTZv3qx7771Xa9eujdimR48e+u9//6tNmzZp//79KisrO+a1fvOb32jnzp26+eab9d133+mdd97Rfffdp5ycnHBrm4aKi4vTDTfcoNtvv12LFi3St99+q+uuu07FxcW65pprJEkzZszQO++8oy1btuibb77Re++9p759+0qSnnjiCf3zn//Ud999p82bN+uNN95Qenq6kpOTGzQugnUAAAAAAAAAaEb/8z//o5SUFG3atEm/+MUvIu574okn1KFDB40YMULjxo3T6NGjIyraaxIfH693331XX3/9tQYPHqzf/e53euSRRyK2+X//7//p4osv1hVXXKFhw4bpwIEDEdXrknTdddepT58+Gjp0qDp16qQVK1Yc81rHHXecFi5cqDVr1mjgwIH69a9/rWuuuUb33HNPHd6Nmj388MO65JJL9Ktf/UqnnHKKtmzZog8++EAdOnSQFKiunz59ugYMGKCzzjpLVqtVr776qiQpISFBjz76qIYOHapTTz1V27Zt08KFCxsc/BumaZoN3rM2pqCgQElJScrPz1diYmK0hwMAAAAAAAA0qraQf5WWlio3N1dZWVkRE3UC9VWXzxQV6wAAAAAAAAAA1AHBOgAAAAAAAAAAdUCwDgAAAAAAAABAHRCsAwAAAAAAAABQBwTrAAAAAAAAAFot0zSjPQS0EXX5LBGsAwAAAAAAAGh1rFarJMnj8UR5JGgrQp+l0GerOramHkxrFDozUVBQEOWRAAAAAAAAAI0vlHu15mpvm80ml8ulffv2KSYmRhYLNcSoP7/fr3379snlcslmqzk2J1ivxJEjRyRJXbt2jfJIAAAAAAAAgKZz5MgRJSUlRXsY9WIYhjIyMpSbm6vt27dHezhoAywWi7p16ybDMGrc1jBb82mpJuL3+7Vr1y4lJCTU6k1sTQoKCtS1a1ft3LlTiYmJ0R4O0K5w/AHRwbEHRAfHHhA9HH9AdLS2Y880TR05ckSZmZmtvtLb7/fTDgaNwm631/p4oGK9EhaLRV26dIn2MJpUYmJiq/iSB9oijj8gOjj2gOjg2AOih+MPiI7WdOy11kr1o1ksFjmdzmgPA+1M6z4dBQAAAAAAAABAMyNYBwAAAAAAAACgDgjW2xmHw6H77rtPDocj2kMB2h2OPyA6OPaA6ODYA6KH4w+IDo49oH1h8lIAAAAAAAAAAOqAinUAAAAAAAAAAOqAYB0AAAAAAAAAgDogWAcAAAAAAAAAoA4I1gEAAAAAAAAAqAOC9XZmzpw56tGjh5xOp4YNG6Y1a9ZEe0hAmzJz5kydeuqpSkhIUOfOnTV+/Hht2rQpYpvS0lLdeOONSk1NVXx8vC655BLt2bMnSiMG2qaHH35YhmFo6tSp4XUce0DT+Omnn/TLX/5Sqampio2N1cknn6wvvvgifL9pmpoxY4YyMjIUGxur7Oxsff/991EcMdA2+Hw+3XvvvcrKylJsbKyOP/54PfjggzJNM7wNxx/QcJ988onGjRunzMxMGYah+fPnR9xfm+Ps4MGDmjBhghITE5WcnKxrrrlGhYWFzbgXAJoCwXo78tprryknJ0f33XefvvzySw0cOFCjR4/W3r17oz00oM1Yvny5brzxRq1atUqLFy9WWVmZzj//fBUVFYW3ue222/Tuu+/qjTfe0PLly7Vr1y5dfPHFURw10LasXbtWf/3rXzVgwICI9Rx7QOM7dOiQRo4cqZiYGL3//vv69ttv9fjjj6tDhw7hbR599FE99dRTmjt3rlavXq24uDiNHj1apaWlURw50Po98sgjeuaZZzR79mxt3LhRjzzyiB599FE9/fTT4W04/oCGKyoq0sCBAzVnzpxK76/NcTZhwgR98803Wrx4sd577z198sknuv7665trFwA0EcOseDobbdqwYcN06qmnavbs2ZIkv9+vrl276uabb9Zdd90V5dEBbdO+ffvUuXNnLV++XGeddZby8/PVqVMnvfLKK7r00kslSd9995369u2rlStX6vTTT4/yiIHWrbCwUKeccor+8pe/6A9/+IMGDRqkWbNmcewBTeSuu+7SihUr9Omnn1Z6v2mayszM1G9/+1tNmzZNkpSfn6+0tDTNmzdPV155ZXMOF2hTLrroIqWlpen5558Pr7vkkksUGxurv//97xx/QBMwDENvv/22xo8fL6l2/53buHGj+vXrp7Vr12ro0KGSpEWLFumCCy7Qjz/+qMzMzGjtDoAGomK9nfB4PFq3bp2ys7PD6ywWi7Kzs7Vy5coojgxo2/Lz8yVJKSkpkqR169aprKws4lg88cQT1a1bN45FoBHceOONuvDCCyOOMYljD2gq//73vzV06FBddtll6ty5swYPHqznnnsufH9ubq7y8vIijr2kpCQNGzaMYw9ooBEjRmjp0qXavHmzJOk///mPPvvsM40dO1YSxx/QHGpznK1cuVLJycnhUF2SsrOzZbFYtHr16mYfM4DGY4v2ANA89u/fL5/Pp7S0tIj1aWlp+u6776I0KqBt8/v9mjp1qkaOHKmTTjpJkpSXlye73a7k5OSIbdPS0pSXlxeFUQJtx6uvvqovv/xSa9euPeY+jj2gafzwww965plnlJOTo7vvvltr167VLbfcIrvdrkmTJoWPr8r+DcqxBzTMXXfdpYKCAp144omyWq3y+Xz64x//qAkTJkgSxx/QDGpznOXl5alz584R99tsNqWkpHAsAq0cwToANJEbb7xRGzZs0GeffRbtoQBt3s6dO3Xrrbdq8eLFcjqd0R4O0G74/X4NHTpUDz30kCRp8ODB2rBhg+bOnatJkyZFeXRA2/b666/rH//4h1555RX1799f69ev19SpU5WZmcnxBwBAM6AVTDvRsWNHWa1W7dmzJ2L9nj17lJ6eHqVRAW3XTTfdpPfee08ff/yxunTpEl6fnp4uj8ejw4cPR2zPsQg0zLp167R3716dcsopstlsstlsWr58uZ566inZbDalpaVx7AFNICMjQ/369YtY17dvX+3YsUOSwscX/wYFGt/tt9+uu+66S1deeaVOPvlk/epXv9Jtt92mmTNnSuL4A5pDbY6z9PR07d27N+J+r9ergwcPciwCrRzBejtht9s1ZMgQLV26NLzO7/dr6dKlGj58eBRHBrQtpmnqpptu0ttvv62PPvpIWVlZEfcPGTJEMTExEcfipk2btGPHDo5FoAHOPfdcff3111q/fn34MnToUE2YMCF8m2MPaHwjR47Upk2bItZt3rxZ3bt3lyRlZWUpPT094tgrKCjQ6tWrOfaABiouLpbFEvm/9FarVX6/XxLHH9AcanOcDR8+XIcPH9a6devC23z00Ufy+/0aNmxYs48ZQOOhFUw7kpOTo0mTJmno0KE67bTTNGvWLBUVFWnKlCnRHhrQZtx444165ZVX9M477yghISHcMy8pKUmxsbFKSkrSNddco5ycHKWkpCgxMVE333yzhg8frtNPPz3Kowdar4SEhPBcBiFxcXFKTU0Nr+fYAxrfbbfdphEjRuihhx7S5ZdfrjVr1ujZZ5/Vs88+K0kyDENTp07VH/7wB/Xu3VtZWVm69957lZmZqfHjx0d38EArN27cOP3xj39Ut27d1L9/f3311Vd64okndPXVV0vi+AMaS2FhobZs2RJezs3N1fr165WSkqJu3brVeJz17dtXY8aM0XXXXae5c+eqrKxMN910k6688kplZmZGaa8ANAoT7crTTz9tduvWzbTb7eZpp51mrlq1KtpDAtoUSZVeXnzxxfA2JSUl5m9+8xuzQ4cOpsvlMn/+85+bu3fvjt6ggTbq7LPPNm+99dbwMsce0DTeffdd86STTjIdDod54oknms8++2zE/X6/37z33nvNtLQ00+FwmOeee665adOmKI0WaDsKCgrMW2+91ezWrZvpdDrNnj17mr/73e9Mt9sd3objD2i4jz/+uNL/x5s0aZJpmrU7zg4cOGBeddVVZnx8vJmYmGhOmTLFPHLkSBT2BkBjMkzTNKOU6QMAAAAAAAAA0OrQYx0AAAAAAAAAgDogWAcAAAAAAAAAoA4I1gEAAAAAAAAAqAOCdQAAAAAAAAAA6oBgHQAAAAAAAACAOiBYBwAAAAAAAACgDgjWAQAAAAAAAACoA4J1AAAAAAAAAADqgGAdAAAArcayZctkGIYOHz4c7aEAAAAAaMcI1gEAAAAAAAAAqAOCdQAAAAAAAAAA6oBgHQAAALXm9/s1c+ZMZWVlKTY2VgMHDtS//vUvSeVtWhYsWKABAwbI6XTq9NNP14YNGyKe480331T//v3lcDjUo0cPPf744xH3u91u3XnnneratascDod69eql559/PmKbdevWaejQoXK5XBoxYoQ2bdoUvu8///mPRo0apYSEBCUmJmrIkCH64osvmugdAQAAANAeEawDAACg1mbOnKmXX35Zc+fO1TfffKPbbrtNv/zlL7V8+fLwNrfffrsef/xxrV27Vp06ddK4ceNUVlYmKRCIX3755bryyiv19ddf6/7779e9996refPmhR8/ceJE/fOf/9RTTz2ljRs36q9//avi4+MjxvG73/1Ojz/+uL744gvZbDZdffXV4fsmTJigLl26aO3atVq3bp3uuusuxcTENO0bAwAAAKBdMUzTNKM9CAAAALR8brdbKSkpWrJkiYYPHx5ef+2116q4uFjXX3+9Ro0apVdffVVXXHGFJOngwYPq0qWL5s2bp8svv1wTJkzQvn379OGHH4Yff8cdd2jBggX65ptvtHnzZvXp00eLFy9Wdnb2MWNYtmyZRo0apSVLlujcc8+VJC1cuFAXXnihSkpK5HQ6lZiYqKefflqTJk1q4ncEAAAAQHtFxToAAABqZcuWLSouLtZ5552n+Pj48OXll1/W1q1bw9tVDN1TUlLUp08fbdy4UZK0ceNGjRw5MuJ5R44cqe+//14+n0/r16+X1WrV2WefXe1YBgwYEL6dkZEhSdq7d68kKScnR9dee62ys7P18MMPR4wNAAAAABoDwToAAABqpbCwUJK0YMECrV+/Pnz59ttvw33WGyo2NrZW21Vs7WIYhqRA/3dJuv/++/XNN9/owgsv1EcffaR+/frp7bffbpTxAQAAAIBEsA4AAIBa6tevnxwOh3bs2KFevXpFXLp27RrebtWqVeHbhw4d0ubNm9W3b19JUt++fbVixYqI512xYoVOOOEEWa1WnXzyyfL7/RE92+vjhBNO0G233aYPP/xQF198sV588cUGPR8AAAAAVGSL9gAAAADQOiQkJGjatGm67bbb5Pf7dcYZZyg/P18rVqxQYmKiunfvLkn6/e9/r9TUVKWlpel3v/udOnbsqPHjx0uSfvvb3+rUU0/Vgw8+qCuuuEIrV67U7Nmz9Ze//EWS1KNHD02aNElXX321nnrqKQ0cOFDbt2/X3r17dfnll9c4xpKSEt1+++269NJLlZWVpR9//FFr167VJZdc0mTvCwAAAID2h2AdAAAAtfbggw+qU6dOmjlzpn744QclJyfrlFNO0d133x1uxfLwww/r1ltv1ffff69Bgwbp3Xffld1ulySdcsopev311zVjxgw9+OCDysjI0O9//3tNnjw5/BrPPPOM7r77bv3mN7/RgQMH1K1bN9199921Gp/VatWBAwc0ceJE7dmzRx07dtTFF1+sBx54oNHfCwAAAADtl2GaphntQQAAAKD1W7ZsmUaNGqVDhw4pOTk52sMBAAAAgCZDj3UAAAAAAAAAAOqAYB0AAAAAAAAAgDqgFQwAAAAAAAAAAHVAxToAAAAAAAAAAHVAsA4AAAAAAAAAQB0QrAMAAAAAAAAAUAcE6wAAAAAAAAAA1AHBOgAAAAAAAAAAdUCwDgAAAAAAAABAHRCsAwAAAAAAAABQBwTrAAAAAAAAAADUAcE6AAAAAAAAAAB1QLAOAAAAAAAAAEAdEKwDAAAAAAAAAFAHtmgPoCXy+/3atWuXEhISZBhGtIcDAAAAAAAANCrTNHXkyBFlZmbKYqH2FqgrgvVK7Nq1S127do32MAAAAAAAAIAmtXPnTnXp0iXawwBaHYL1SiT8f/buPDyq+m7/+D37ZF8IWYCwCbLIKgiCW61Y1ErF1p1WQItPq7hR+1Nbt9ZWamutVq089qmtbbVaW7cq1VrcFQFBFJBNBcKWsIRkss56fn+cySSBBBIykzNJ3q/rmmvmfOfMnM8sZyD3fOZ7MjIkmR8smZmZFlcDAAAAAAAAxJfP51NxcXEsBwPQPgTrLWiY/iUzM5NgHQAAAAAAAN0W0yADR4cJlAAAAAAAAAAAaAeCdQAAAAAAAAAA2oFgHQAAAAAAAACAdiBYBwAAAAAAAACgHQjWAQAAAAAAAABoB4J1AAAAAAAAAADagWAdAAAAAAAAAIB2IFgHAAAAAAAAAKAdCNYBAAAAAAAAAGgHgnUAAAAAAAAAANqBYB0AAAAAAAAAgHYgWAcAAAAAAAAAoB0I1gEAAAAAAAAAaAeCdQAAAAAAAAAA2oFgHQAAAAAAAACAdiBYBwAAAAAAAACgHZxWFwAAAAAAVgpHDIUiEYUjhsIRQ4Ykw5BkSIYM87K5KMNocn0SsdkkmySbzRY9l2wyBw++ri0OeaxNnouGh970esMwFIwYCocbn8tg2Gj23IYihkJhQxHDkMNuk91mk8Nuk8MuOex2OWw22e0yx2w22aPrhMIRBaP3GwwbseVgJKJQw3LEPE+khufUZmtYbv5cNz7PUsSIPi9NnjMj+qZpOm6zmY/daTefC6fdJqej+bLDbpPLYZfdFt1GC9tTs2WzLkPme9swDEUMKWKYz70Ru6zosiHJJrtNsttsse3YG14Pm002W+P1DfcTjpj3HzHM17bhcjhiKBIxFDbM19sfCssfiigQiigQNs9jy9ExfzAsQ1KKyyGvy6EUt0Op7uhll3k5JTqe4nbI63RE3zu25u+laM1Ouz1Wu91mi24jIn8orProuT8UUX3QPG863vi+NRQOR2KPreH9G440HzPf65HDLociRuz1b/6+ab7c9HVs/Jwxou+nJvtbk/dTYt/zNjlsavYc26PPc+z5j+2rLb03bdH7OfQ9G/s8aum6g56Hhv0+FH1uQ2HzvGG/b/h8CIWNxn3u4M+sFvbHsCHzvdrkvRuOvnebvo/DEWnigBwt+s6EhD/nALoWgnUAANDj1fhDWruzUmt2Vqo+GNZxfbI0qm+Wemd4rC6tRwmEIirz1avMV69SX71KK+u1t9qv+kC4MRRpFsqEmwU0wXBEXpdDWSkuZae6lJ3iVlaqK7aclWKONVx22G3NA55wQ8DSJPCJLhsy5HbY5XE5zHOnXW5n47nbaY9d77TbooFM80CxtfClYZuxgOeg88bgxwwVIk3+8I9E76tpINAQmDnttlgo5XXZ5Y0GVt7ockqTyzbZYiFlsCG8CDWGFk3DjEAoovpQWHWBsOpDEdUHwqoPhVUfDKsuaIZTDWPBsNFiQBhbdtjkbBIqup12uRx2uRw2uZ0OuR226LJ5Mp9nM3gMhpu8RtH3hxnahQ96XaPh62HC3mQLyQEAyeVAbcDqEgAkIYJ1AADQo/hDYW3YXaVPd1Tokx2V+nRHhT7fU61IC8FaQaZHo/uaIfvo6Ck/09uh7QfDEdX4Q6oJhM1zf0i1gbCq/SHVBkKq9odVHwgrxe1QhtepzBSXMr1OZXpdykxxKcPrVIrLEesCa2AYhqr9Ie2p8muPz689VfXaW+WPLtdrT5Vfe6v8qg+FD+kma+wYa945Z7fZYiGns+Hcbp67HHZzzG6TM3q9O9pl2Wxdp00uu73JOjZFDGmPz6/ShhC90jzfX8Mfrd1RKBpe+60uJAEO2g0tkwxfDNhtinVaN3wONP1CpfHcHu3obvgiSI1dok2+MGroGDUMmZ8f9obPoujnUpPPFZe94TMpsS9Isw7YFn7N0LQjttkvBnRQF26Ty0aTx3+kL+IavwRq3nXbtJtZTcbtTTqJ7Qd1nTvsjV3pDe/jiGEoEmnsbG+434bO9oYO3ob7behibnhtYx3iTTqanQ6bPE7zC0n3QV9Iepp8Iel2mO+Lhi/oagPRL+sCTS43Ga8PRhSKRBSJKPZeaQu7TfK6HPI47bFzj9P8gtHjdMjjsjf/8i92bo99GeiKLjvsauHXBeb79JDbRZ+fxvfOwa9Z09fUXG6pi7ulf6sT+TnU8Py29iVuONJ0X27aHW40eywHP7aGX7m0NK4mz03DvtR0/3fazffPIf/fiF5/8K9K7If5hUfD+7ThPdvwaweHvXE/aThP9xCfATgUnwwAAKDT1QZC2nGgTiX7a7X9QK22l9eprKpefbK8Gl6YqeFFGRqSny6P09Gh7VT7Q9q2v0brdvn06Y4KfbqjUht2VynQwnQBhZlejemXpVS3Q2t3+fTF3mqV+fwq8+3Rf9fvia2Xn9EYtg/MS1VdwAzKq6IheXV9SNUB87zGH1J19GSG6OEWt91eTrstFrqne5xmoO7zqy4Y7vB9W83tsKsgy6PCTK8KMr3Kz/AqzeNo7AxvEsK01DFeFwyrojaoirqgKmsDqqwLNi7XBVVZG1RFXUAVtUEZhg65j6bbcUcDF7fTLpsU65pvnMog3KyrvaFzOhSOtCFsiXZqO2yHDXiaXmcGCfaDfoqvWMDTfCoEm4LhiOqDEdUFw/IHGzrMw7Gx+mBjQGUYRuOXIweFFAeHGJ7o89QwHUOKO9oR73TIe9CY025vnCYiOnVHODadQvSn+9Flc5qPiAKh6E/8w42/RAhEO+kbloNhw+xqd9jlcTW8Xo4WX0tP9HlreCzNXgNH89eiIWSRWp7SoOkUBsmoeVDVGExFmkxB0h6N4VTXey7Q8zT9QqbhcycS/fKm4bPA5eAwcwCA+LEZnTExVxfj8/mUlZWlyspKZWZmWl0OAABd0t4qvzaXVamkvDE8Lymv1Y4DtdpXfeTOZIfdpsF5aRpelKnhhRkaUZShYYWZ6pPljQU5hmFof01A2/bXqqS8Rlv31aqkvFbb9tdo2/7aVjugs1NdGtMvW2P7ZcXOD+5Er/GH9Nlun9bsqIxNE/PF3pY724+G22lXmtuhNI9TaW6n0jyNl70uMyD21YVU5Q/KVxeSrz6oqvrQEbvy0j1O5Wd41DvDo/xMr/IzPI3LGV6luB3SQV2Wsc4xNXaNyVBsftymU2k0zm988JzH5uXYugfNfRpsch+GIeWle1SQ5VVhpldFWWaIXpjlVU6qi6AOAACgE5B/AR1DsN4CPlgAAEcjFI7o052V+rysWn1zUjQkP135GZ5ODQnLawJ67/N9+uDzfaoPhlWUnaI+2Snqk+VVUVaK+mR7lZUS3+CyPhjW53uqtX63TxtKq7Sh1KcNu6uOOK1Hhtep/rmpKs5JVXFuigoyvdpxoC52P5V1wVZvN6wgQ7WBsErKa1XtDx12O7lpbg3JT28SomerODflqJ6D2kBIn+3yaU00aN9dUa80j0PpHqfSPE6le51Kd5vnaR6nMjyNl2PruJ1KcZtd0O1lGIZqA+FYyO6rM8/TomF6fqZHqW5+kAgAAIAjI/8COoZgvQV8sAAA2iIcMbR+t09Lv9ivD77YpxVbDxwS8mZ4nRqSn66h+enRc3OKk77ZKbLbOx5u+0Nhrdx6QO9+vk/vbt6rdbt8R/ypf4rLoaJsr/pEg/airBTlprmj0yQcfn5Qp8OmGn9YG0t9Wl9apQ27fdqyr6bFLm6bTRqQm6oBvdJUnJui4pxUM0iPhulZqa5WazQMQ6W+ejOo390Y1n+xt1qhgzZms0lFmV7175Wqgb3S1L9XqgbkpmlAr1T175WqTG/r2wEAAAB6KvIvoGMI1lvABwsAoCWGYWhTWbU++GKfln6xX8u2lB/SVZ2V4tKovpnaXVGvrftbDpwlyeuy65jeZthemOlVr3S3eqV51Cvdrbx08zw3zX3IHOOGYWjznmq9s2mv3vt8n5Z9WX7IvNrDCzN06rG9lZvm1u6KOu2qrNfuyjrtqqhXeYIODpmT6orNjT68MEPDCzN1bEFGdNqR+AmEIvpib7U2lVUp3ePUgF6p6peTKq8rvtsBAAAAujvyL6Bj+K0wAAAtqPGHtG1/rbbur9HW6MEvP/xi/yHTm6R7nJo8KFdTjumlKcf00ojCzFgnuj8U1tZ9tdq8p0qf76nW5j3V+rysWlv21ag+GNG6XT6t2+U7bB0ZHqcZuqd7lJXi0rpdlSrz+Zut0zvDo1OG5OmUY/N00pA85Wd4W7k3c9qW3ZX1jYF79LyyLhA7oGAo0nhQQfPcaHawQZfTpmPzMzSsMEPDizI1ojBDvTtpyhu3064RRZkaUcR//AEAAAAA1qFjvQV8YwcAPUO1P6St+2oaA/To5S37a7S3yt/ibbwuu04YGA3SB/fS6L5ZcjraN1d2KBxRSXmtPt9TrS/21mhftV/7q/3aXxPQvuqA9lf7VV4TOGTKkwYep12TB/fSqUPzdPLQPA0ryOBgjwAAAADahfwL6Bg61gEAPcaeqnot/WK/efpyv7btrz3s+jmpLg3MS9PAXmkanJemyYN7aWxx1iHTs7SX02HX4N7pGtw7vdV1DMOQry6kfTV+7W8I22sDGpCbpokDc5j6BAAAAAAACxGsAwA6XbU/JK/T3u5O7/Yqrwnowy/3xw4u+sXemkPW6ZXm1oBeqbEAfUCvVA3KS9OA3LTDHlwz0Ww2m7JSXcpKdemY3paVAQAAAAAAWkCwDgDoNLsq6nTvqxv04updcjvsGtw7TUPy03VsQYaOLUjXkPwMDeyVelSBezAcUXlNQJ/uqIwF6RtKq5qtY7NJx/XJ1JTBvTT1mDwd3z/H0vAcAAAAAAB0TQTrAICEqwuE9dg7X+rRtz9XfTAiSQqEI9pQWhUNv3fH1j04cB+any6bTSqvCepAbUDlNQEdqA3oQE1A5bVBHagxL1f5Qy1ue1hBRuzAopMH5So71d0ZDxkAAAAAAHRjBOsAgIQxDEMvf7pbCxev167KeknSCQNzdPu5I5WT6tbmPVXaVFatzWXV2rynSpvLqlUXDLcYuLeFzSYN7JUWO7DoiYN7qXeGJwGPDAAAAAAA9GQE6wCAFtX4QyqvCahfTopsNlu7b79mR6V++vI6rdh6QJLUNztFt5w9XOeOKYrdX3Fuqr46vCB2m0jE0M6KuljIvqmsWl/srZbDblNOqlu5aS7lpLnNy6lu5aRFx1LNscwUlxz29tcKAAAAAADQHgTrAAAFwxFtLK3SJzsq9Mn2Cn2yvVKb91QpYkjZqS6NL87W8f1zdPyAHI0tzla6p/V/PvZW+XXfaxv195XbZRiS12XX1V8ZoqtOHSyvy3HYOux2m4pzUw8J3AEAAAAAAJIJwToA9DCGYaikvFarowH6JzsqtHZnpfyhyCHrOu02VdQG9ebGvXpz415Jkt0mDSvM1PH9G8P2gb1SFQhH9Kf3t+qhNz5XdXS+8/PG9dHNZw1Xn+yUTn2MAAAAAAAAiUSwDgA9xB5fvX79n036z2elOlAbPOT6DK9T44qzNbZftsYWZ2tsvyxlp7q1frdPK7cd0KqSA/q4pEI7K+q0frdP63f79OSyEklSbppbHqddu6PzqI/pl6U7Z4zUhAG5nfoYAQAAAAAAOgPBOgAkWCgc0d8/2qGnV5To9GH5+v5XjjnilCjx5A+F9Yf3tuiRNz5XTSAsSXI77BrRJ1Pj+mVpXH8zTB/YK032FuYnH1tsBu1XaJAkqcxXr1XbDsTC9rU7fSqvCUiSemd4dPNZw/XN8X1bvC8AAAAAAIDuwGYYhmF1EcnG5/MpKytLlZWVyszMtLocAF3Y25v26p5X1mtjWVVsbECvVP30vFE67djeCd22YRh6/bMy/Xzxem3bXytJGlecrf931jBNGJAjjzM+4b4/FNa6XT6VVtbr1GN7H3b+dQAAAABAciD/AjqGYL0FfLAA6KhNZVX6+Svr9fYmc17yrBSXLp3UX89/vENlPr8k6euji3T7uSNVmOWN+/Y3l1Xppy9/pnc375Mk5Wd4dMvZwzVzHJ3kAAAAAADyL6CjCNZbwAcLgKO1t8qv+1/fpGdWlChiSC6HTZdPGahrvzpE2aluVftD+s3rm/SnD7YqHDGU5nZowdeGafaUAXI67B3efmVtUL/57yb95cNtCkcMuR12ffeUQbrm9CFKo5McAAAAABBF/gV0DMF6C/hgAdBe9UFzHvPfvdk4j/nZowp181nDNTAv7ZD1P9vl049fWKOPSyokSSOLMvWz80fp+P45R7X9cMTQ35aX6Nf/2Rg7MOnXRhbox18foQG9Dt0+AAAAAKBnI/8COoZgvQV8sABoq0jE0Euf7NIvX92gXZX1kqQx/bJ029dHatKg3CPe9pmPtusX/96gyrqgbDbpkhP66+azhik71X3EbVfVB7Wrol5f7K3WQ298rvW7fZKkofnpunPGcTp5aF7HHyAAAAAAoFsi/wI6hmC9BXywAGiNYRjaWVGnTWVV2lharVfX7tYnOyolSX2yvPp/Zw3XN8b2adc85vur/Vr47w36x8odkqTcNLduPXu4ThqSp10VddoZPe2qqNOuivrYWFV9qNn9ZHqdWnDmsfr2ifGZVgYAAAAA0H2RfwEdQ7DeAj5YABiGob1Vfm0sq9LG0iptLqvWxrIqbS6rik310iDN7dDVpw/RlScPktflOOptLt9SrtteWKNNZdVtvk12qkt9slI0eXCurv3qUOWmHbnTHQAAAAAA8i+gYziSHQBEHagJ6OkV2/Xmhj3aWFalyrpgi+u5HDYd0ztdQwsyNKIoQxdOKFbvDE+Htz9pUK5eue4U/eG9LXr4jc9VHwyrKNurPlkp6pudoj7RU9+cFPXN9qooK4UDkgIAAAAAAFigS3SsP/LII/rVr36l0tJSjR07Vg899JAmTZrU6voPPPCAHn30UZWUlCgvL08XXHCBFi5cKK/X26bt8Y0d0LOs21WpJz7YqhdX75I/FImN223SwLw0DSvI0NCCDA0ryNCwwnQN6JUmV4KnWgmFI7LZbHK0Y0oZAAAAAADaivwL6Jikb3V85plntGDBAi1atEiTJ0/WAw88oOnTp2vjxo3Kz88/ZP2nnnpKt9xyix5//HFNnTpVmzZt0pw5c2Sz2XT//fdb8AgAJKNgOKLX1pXqiQ+2asXWA7HxUX0zddmkARpXnK3BvdM6NLVLRzBHOgAAAAAAQPJK+o71yZMn64QTTtDDDz8sSYpEIiouLta1116rW2655ZD158+fr/Xr12vJkiWxsR/84AdatmyZ3nvvvTZtk2/sgO5rX7Vff1tWoieXlajUVy9JctptOmtUoeaeNFDH98+RzUaXOAAAAACgeyP/AjomqTvWA4GAVq5cqVtvvTU2ZrfbNW3aNC1durTF20ydOlV//etftXz5ck2aNElffvmlFi9erO985zutbsfv98vv98eWfT5f/B4EgKTw6Y4K/emDrXr5k90KhM3pXvLS3bpsUn/NOnGACjLbNlUUAAAAAAAAkNTB+r59+xQOh1VQUNBsvKCgQBs2bGjxNpdddpn27dunk08+WYZhKBQK6Xvf+55+9KMftbqdhQsX6ic/+UlcawfQfvXBsPZV+1XjD6smEFKNP6Qaf1i1DZcD4dhYjT+k+lBYgVBE/lBEgejJHwqby+FI7Dp/MCxffSi2nbHF2ZozdYDOGV0kj9OaqV4AAAAAAADQdSV1sH403nrrLd1zzz363e9+p8mTJ+vzzz/X9ddfr7vvvlu33357i7e59dZbtWDBgtiyz+dTcXFxZ5UMQNKLq3fqR8+tUU0gnJD7dzlsOndMH82eOlDjirMTsg0AAAAAAAD0DEkdrOfl5cnhcKisrKzZeFlZmQoLC1u8ze23367vfOc7+u53vytJGj16tGpqanTVVVfpxz/+sez2Qw8I6PF45PF44v8AABxROGLol69u0P++86Ukye2wK93rVKrboXSPeZ7mcSrN7TTPPQ3LDnldDnmcdrmddnmcDrmddrkdDcvmecPl3hleZaW4LH60AAAAAAAA6A6SOlh3u92aMGGClixZopkzZ0oyD166ZMkSzZ8/v8Xb1NbWHhKeOxzmVA9JfpxWoMeprA1q/t9W6d3N+yRJ3//KMbrpa8PksHPwUAAAAAAAACSvpA7WJWnBggWaPXu2Jk6cqEmTJumBBx5QTU2N5s6dK0m6/PLL1bdvXy1cuFCSNGPGDN1///0aP358bCqY22+/XTNmzIgF7ACst6msSvP+/JG27a9VisuhX104RueO6WN1WQAAAAAAAMARJX2wfvHFF2vv3r264447VFpaqnHjxunVV1+NHdC0pKSkWYf6bbfdJpvNpttuu007d+5U7969NWPGDP385z+36iEAOMhr60q14JnVqgmE1Tc7Rb+/fKJG9sm0uiwAAAAAAACgTWwG86McwufzKSsrS5WVlcrMJOwD4iUSMfTgks16cMlmSdKUwb30yKzjlZvmtrgyAAAAAAB6FvIvoGOSvmMdQPdQ7Q9pwTOr9Z/PzIMRz5k6UD/++gi5HIceUBgAAAAAAABIZgTrABJu674azfvzR9q8p1puh10/O3+ULppYbHVZAAAAAAAAwFEhWAeQUG9v2qtrn1olX31IBZkeLfr2BI3vn2N1WQAAAAAAAMBRI1gHEHd7qur12tpSvbJmt5ZtKZdhSOP7Z+t/vz1B+Zleq8sDAAAAAAAAOoRgHUBcNITpL3+6W8u3mmF6g0snFeuubxwnj9NhXYEAAAAAAABAnBCsAzhqe6rq9eraUr3SQpg+rjhbXx9dpLNHF6pfTqp1RQIAAAAAAABxRrAOoF1qAyH9c+WOFjvTCdMBAAAAAADQExCsA2iz7eW1+u4TH2ljWVVsbFxxts4dU6SzRxepb3aKhdUBAAAAAAAAnYNgHUCbfPjlfn3/ryt1oDao3hke/c+pgwnTAQAAAAAA0CMRrAM4oqeWleiOF9cqFDE0um+WHrt8goqyCNQBAAAAAADQMxGsA2hVKBzR3S9/pieWbpMkzRjbR7/81hiluB0WVwYAAAAAAABYh2Ad6CL2Vfv1P39ZqU+2V8hht8lpt5nnDnuzZVeT5aIsry4+oVjTRhTI6bC3a3sVtQFd89Qqvf/5fknSTV87VtecPkQ2my0RDw8AAAAAAADoMgjWgS6gNhDSlX9aoU92VEqSQhFD/jbcbkNpld7cuFdFWV5dNqm/LpnUX70zPEe83ed7qvTdJz7S1v21SnU79JuLx2n6cYUdfBQAAAAAAABA92AzDMOwuohk4/P5lJWVpcrKSmVmZlpdDnq4UDiiq/6yUm9s2KOcVJeeuGKSeqV7FA4bCkUiCkcMhSKGwhFDwXDjcihsaOmX+/T08u3aXxOQJLkcNp09qkiXTxmgCQNyWuw+f3PDHl33t49V5Q+pb3aK/m/2RI0oYj8AAAAAAKA7If8COoZgvQV8sCBZGIahHz2/Rn9bvl0ep11/u+pEHd8/p1334Q+FtXjNbv156TZ9XFIRGx9RlKnLpwzQeeP6KNXtlGEY+v27X2rhvzfIMKRJA3P16LePV6/0I3e4AwAAAACAroX8C+gYgvUW8MGCZPHwG5t13382yWaTFn17QoenY1m7s1J/XrpVL67eJX8oIknK8Dp14YRiVdQG9NzHOyVJl5xQrJ+eN0puZ/vmZQcAAAAAAF0D+RfQMQTrLeCDBcngHyt36KZnP5Ek/fS843T5lIFxu++K2oCe/WiH/vLhNpWU18bGHXabbv/6CM2eOpCDlAIAAAAA0I2RfwEdQ7DeAj5YYLV3N+/V3D+uUChi6H9OG6xbzx6RkO1EIobe3rxXf1m6TVv21ein5x2nU4b2Tsi2AAAAAABA8iD/AjrGaXUBAJpbt6tS3//rKoUihr4xto9unj48Yduy2206fVi+Th+Wn7BtAAAAAAAAAN0NEygDSWRnRZ3m/nGFqv0hnTg4V7+6cIzsdqZkAQAAAAAAAJIJHetAkqisDWrO48u1p8qvYwvS9b/fmSiP02F1WQDaIxySNvxL2vqeVDRWGvo1KaNjBx1GJ6rYLn36tBQKSMdfLmUXW10RAAAAACBJEawDScAfCmveXz7S5j3VKsj06E9zJykrxWV1WQDayl8lffxX6cPfSRUlza8rGisNnS4dO13qc7xk58dizRiGVP6ltHOltP9zaeAp0sCTpc46gHIoIG1cLK36s/TFG5Kih555735p9IXSSTdI+YmbkgsAAAAA0DVx8NIWcPAGdKZIxNB1T3+slz/drXSPU89+b4pGFPG+A7oE3y5p2f9KK/8o1VeaYym50sjzpN2fSLtWNV8/NU8aMk069mvSMWdIKdmJq+3ANsnvk3IGSZ70xG2nvWrLpZ2rpJ0fSTs+MgP1uvLm6/Q7QTr5RunYsxP3RcSeDdLHf5E++ZtUu79xfNCp5vmWdxrHhp9r1tNvYmJqQdtEwlLNXqlqt1RV2vJ59V4pZ4D5Bc2gU6XiSZIrxerKAQAAkhL5F9AxBOst4IMFiWYYhipqg9pZUadnVmzXXz7cJqfdpieumKSThuRZXR7izbdLeuc+ac2z0pT50ldutroidFTpWmnpw9Kaf0iRoDmWe4w05Rpp7KWSO9Ucq94jbX5d2vya9MWbZtDdwOaQ+p9oThdz3EwpZ2DH6zIMMxBe+rC0+T+N42n5Uu4gKXewGbQ3vZyam7ju8HBIKv3UDM93fGSG6fs/P3Q9h9vs7M/qJ21YLIX95njvEdLJN0ijviU54vArHn+1tO55szt9x/LG8Ywiadwsafws83mRzJrf+420/mXFutgHnmIG7Md8tfM66uOhem/jFxm7Vpnvk5yBjafcQea5N8vaOpvy7ZZKlkolH5qvhW+nVF0mGZH23Y/DY4brDUF73wmS092220YiZpBfuUOq3C6Fg1J6vjm9U3qB+Xx1xvvAMKRQvfnlXb1PCtZIwTopUCsFm5wCteZ4w/XBWsmdLqX2ktLyzPPUvMbL3uyOfXEV8pv11FdK/srG+uorzc+6+uhYsM7cf+2u6LnTPHe4Gy/bXZLDab5e3kzJk2k+vw0nT2bbXrdwqPm2Y5d95nMYCZmvYzjQeDkSjI41uaz2/Hlka+XxuA593HbHodtqraZIuIXnx914+eBttOe96PCYXzi5UyVX01OK5E4zz53eQ+8zEm6h9kDj5VB943sv9v6sa/I+rZMC0fdnONDC+8Ld/D0Se6zO5s9VJGi+1uFA4+VIk+c1UWz2aE3u5q/BIa+3U5LtoOenaZ2B5jXLaP46uKOvRUtjDo95vwc/7pbeU5GQ+Z473Punpdpbe4ytfV4c/L44uJYW3xe1LY9JUnqh+TmbUdR4ntqrc37x568y/+8WCbVtfcOQjHALj7uV92h7/w1LBt7sxn/3Mgolp8fqig5VXxn9t3qH+X+FkP8wn/FN9sO2vs4N8o6VTro+MY/BQuRfQMcQrLeADxZ0VCRiaF+1Xzsq6rTjQJ12HqjTzora6Lk5VhsIN7vN/ReN1TeP72dRxUiIqjIzmPvo8cagUJJmPiqNu8y6upJBKND2cCtZGIb0xRLpg4elL99sHO8/VZo6/8jd1eGgGRJufk3a9B9p38bm1w88xQzlR57X/g7zcNAMjD94yAyyJUk2M5Cqrzj8bT2ZZrCad6wZOvY7QSocfXR/OEUiUtkaM9zf8q607QMpUHXoermDpb4TzW31myAVjG58P1TvMafUWfGHxi8isvpLU6+Vxn+78UuLtqo7IO3+VFr7T/MUqDbHbQ7p2LPMudSHTIsGIS3Yu0l6/0Fz7vWGP8CKxpkB+4gZZmhxNCJhs8u6IbRt+IMw9odhaZMvRAZFvxAZbF7O7Nd6vcF689cSsV8EfHTo9EStScmJhu2DGkP37P6N4UaigmTDkPZtlko+MPeRkqXSga0tr2uzN/5x3zR0iYUvudKe9dH34DtmF3tTrlTzC62Bp0iDTjMfU+X2g16Dhss7m392H8zpbawlvSB6ucAMhVJyzLAlEjJf60io8RQOHbQcjAbAFS0E09HzRISFNof5fDWE7Q7X4YPepsFEsM4MyzqTK7V54O5OM+toGqI37N+IA5v5nNtsja97VwwFET82R2PILqPxc6JdX0QdJbsr+jnfJHRPyzf/T+D0Rr+A8ErOFPPc1TCeYp7sLql2n/nvbvUe89/YqrJDz4M1iX8sXd3BQXvTc6e3yb9v4eiXhKGW/y1s+IyJvW5NTrHXMXo5EjT/TT7k/0zRy00bVxJp4CnSnJc7Z1udiPwL6BiC9RbwwYKOWL6lXFc/uVL7qgNHXDcv3aO+OSn69uT+unAiB8nrNmr2S+8/IC3/vRSqM8f6T5V6DTbn4Xa4pdn/MsOdZBWJSJ89bwaSJ15thkXxEPJLb/7cDKd7D5dGX2CesvvH5/7bWsOOFWbwW7mj7bfb9bG0Z5152WY3A/Ap15rB8NE4sNXsZl//r+i0I9F/jl1p0shvmF++DDj58GF9faW08glp2SKzo1cy/wAZP8t83XodI9VVSAe2mPOYl2+JXo6eqna1fL8Otxmu951oTn/Sd4IZ6h4cqhqGtHdDY4i59b1Dg3xvlhmgN72v1NwjPz/1lWa4/uHvzK5hyQwAT/y+dMJ3D51GJxw0u+HL1klla6Pn6xqflwa5g80wfeyl7TuwbOUO83276onGrrpeQ6TjZ5t/+MWCyMN0JQZrzV+wVG43z9vbKdXA7pSyBzQG7tn9zfB850fmrykOCWFtUu9h5mvQ93jzD98DW833woGt5qnhOT4cZ0rrgXZmkTkNkq2NHYX1leavBrYtlbZ/2Hw6Hsm8n4JRUv8pZsd5r2OiQUrvtn+ZYRjS/i+kre80ftlTu69tt21aR0aR+WsKh9vshKsqMzu0O5vNLnkyzC70hu7iWCdr6qFjTq8ZNNfsNx93zT7zea7dH98QwtO0wzyzscO8YdmV0tjRekhY37TjNmCe/FWNYXm9r+Uv547ElXZQLZnm83G4bvKm3dJtfR9LZth8cOd2s8tNHnckfJju4IM6te32gzqBD+rKPvhzpu0Fm/8ONv11Q9OO4fCR///aTNPuZoenyfsxRS12wrtSzNfH4Yw+voOeqxa7nsOtd1K39MsHJeiXJEb40C7k1j73ZbTxlxrRX2MF61p5TQ4aC/ujv344XHe5s3E7LXW3t/rlWQc6eQ/WtDu+2fviCJ9dRiQacjeZ5qst/z7Fkyu1fc0FNsdBr8Fh3qO2o/wy3ipGxJyur+HLh/Z+PnSmlFzz3+qMwoM+75v+WuPgfdGhdn1eZBVLYy5M2EOwCvkX0DEE6y3ggwVHq6o+qLMeeFc7K+pkt0mFmV71zUlRv5xU9c1OUd+cFPXNTlG/nBT1yU6R19XF/nOFw6s7YAZvyxY1ds31nSh99cfS4NPNkOfZy80gNTVPmveGORdwMjEM6fP/Sv/9idl5LJmdKdPvMYPejnSrlq6Vnv8fM/Q8WP8p5oEiR86U0nod/TZaEg6aoXhD+Lt92dF3WrrSzFD2xO/FZ+qWBpU7pE+ellY/JZV/0Tie1V8ae4l56nVM43jFdvN9tvKJxtApLV+afJU08cq2BdeS+Yf6ga1m6F62rrHD+eCQUzL/YOk7wQzHU3uZ3ehb3z30D153hjRgqjQoOvVGwaij7+puqHH1k9L7v5UqtjVuY+JcM2RtCND3bWz9D76sYmnASdLx3zHPO/I+rtkvLf9fc279I/0a4EjsTimzj1lfVr8mp2JzypHqPQd9IfKl+Xod6Q/btN7RLzImNIbpR5rmxV/dGLLHTlvMDrGqXY3HEEgUp9esdcAU80vHfifEf2oaw2jsZt/6rvlFUCTU8vPfcDmzT8vTEAVqzZC9uizaAXnQeX1l8+k/7M7WTw5X69OfxJYzzUA9Xr8YCPnN/bxp2B4OHho4txY+O73RGjM6tn+3RcP0Lgd38QeqzQDM2+R58kSfq3hMHdVThUPNA13DaBL8HxRS2R1da1ostJ1htBzANw3dO+t9EQ42frbGjqlRKtXsafKFRF2TKWfqzMaWYJ35K65QdOqhlJzoNDMFrZxHu66T6bg0ycQwzL91Wvt3r7rMfK0O/nfP4Wrh30GH2cQTavIatfg61pufRzaHlNW3lX+vi83r3GlWP0NdFvkX0DEE6y3ggwVH69bn1uhvy0tUnJuixdedogwvf9j1CPU+6cNHpaWPNHYxFo6RvnqbOX920z8uAjXS42eZ03Xkj5Su/I8ZTCSD7cul/94lbXvfXPZkSpl9pb3rzeVjviqd+0D7vwyIhKUPfiu98XOzEyo1Tzrnl2aQt+ZZM9xq6Na2O81pOUZfKA07++j+kxwJm8/vlnfNAK1k6aHTA6Tlm8FvwXFt70z0ZkvHnZ/YA44ahtlNv/pJae3zzbti+08x5xovWSqte8HsnJPMzv8p883nzOWNTw0HtjaG7Ds+Mp/P1sJcZ4oZhA461TwVjWt9mpKOCIfM6W7e+03jLwcO5k43X9P8keZ5wSgpf0RiXjN/lfnFRslS8w/Ew8193HDu9DQP0jMK2x9KRsJmt3vslwdfmt3qGUWNQXp2//iHXYHaQ7sImwYcVbvNP7jbyuGW+ow339f9p5hz7FsxPZRhEAwCQHfHZ33X1RBX8folDPkX0DEE6y3ggwVH493Ne/WdP5gHw/vbvBM15Zg4d90i+VSVmQHoB79tDJTyR0qn/0gafm7r/wGs3Cn9/nSzs+PYs6RLnkp8x9/h7FkvLblb2viKuezwSJPmSScvMDsAlz4kvbnQ/PmxK02adqd0wry2HUSqfIv0/PfMqR4kadjXpRkPSum9G9ep3Cmte0769O9N5geXua3hXzcD46x+0Y7F6BQB/qpo96Kv8XLDdaWfHtpdm5IjDTzZnFN54CnmtBjJ/h/0YJ204RWzi/3LNw+d23bQqdLU66Rjzkj8Ab1CfvMXBw1Be+0+qfhE88uJvhM690BWhmEemHXlE2ZgXTBKKogG6Vn9O+fgZgAAAEA3QP4FdAzBegv4YEF7VdUHNf0372hXZb1mTxmgn5w3yuqSuqYDW83pJrxJut8F68zpL754Q/rizeZds3nHSl+5RRp5ftuCvR0rpT+dY/7Ecep10tfuTlzdrakoMQPzT/4myTA7t8fNMh9H1kEH0t33ufTStebBBSUzVP3GQ1LvY1u+b8Mw56J+9UfmT8ndGdLZvzDv/3CB9t6NZhf7mmdbP3hhW3gyo9ORnGoG6QWjunbg6tslffqMtPFVcwqaKVebHb4AAAAAcJTIv4COIVhvAR8saK9b/vmpnl6xXf1zU/XqDaco1Z2AaRC6s7oD0ut3mkGsM8WcbmPCbKl4srVdxZGIOR/4l2+aYfq2pWbXdoxN6jNOmvw9s6u6vV3na/4h/fNK8/J5j0jjvx2vyg+vZp/0zn3SR39onN5jxDfMqWt6D2v9dpGIeZv/3mVOreJwS6fdLJ10ffP5bKvKzBB+82vm8oCTpJmPtm8KGcMwO6PXPCtteNnsmPZkNM5D7Mk8aLnJ5ZxBZuiciOlIAAAAAKCbIP8COoZgvQV8sKA93t60V7MfN6eAefqqE3XiYKaAaTPDkNb+U3r1VvMARAfLG2YeKHLspfE/oGVLAjVml/TuT8yO9C/fPPTAjJl9pWNON+cbH/SVjtf15j3S2/ea8y/Pfsnssk6U8i+lVX+Wlv++cc7xQadKZ9xlzs3cVhXbpZdvMA9yKkmFo80vBorGSp+9KP3rBqmu3Azez7hDOvGart0tDgAAAADdEPkX0DEE6y3ggwVt5YtOAbO7sl5zpg7UXd84zuqSuo4DW6VXftAYzuYNk2Y8YB68cuUT5pzbwVrzOofbnLN8wmxp4KkdC2lry815vxsO/HcgevC/8i3mgfkO5koz5+Y+5qvmKW9ofLvoIxHpH3PMQDq1lzTvDXOqj3gJ1kvr/2X+GmDru43jReOkaXeZXxIcDcMwpyZ59RbzFwc2h3kAwm3vmdcXjpbOf8yc+xoAAAAAkHTIv4COIVhvAR8saKub//Gpnvlouwb0StW/r0/wFDAHtkr/uEIKBaQxF0ljLpYyChK3vUQJB6Wlj0hv/UIK1Zmh+ak/NKcTaXoAxPpKc6qUVU+YHeQNcgaaXezjZkkZheZYoFaq3W8eULEmel6735zypHafGab7dpoB+sEHtTyYN8ucL33QqWaQ3m+S5HTH/WloJlAr/fFsafdqqfcI6cr/dHye+dI1Znf6p3+X6iuigzZpyBnShLnSsHPi00VevUf69/+T1j0f3YRdOvlG6bRbEv+8AQAAAACOGvkX0DEE6y3ggwVt8ebGPZr7xxWy2aRnrpqiSYNyE7exHR9Jf7uk+bQkNoc0ZJo07lLp2LMllzdx24+XHSulf11nzlsumQeVPPcBKW/I4W+3a7UZEq95VvL7zDGbQ8rsYwboDZ3tbZVRZM7DnRs95TQ5T03g63g4vl3SY6ebXfNDvyZd+nT752yvrzSn1ln1Z2nXx43jWcXm/O3jZknZxfGtu8GGV6R1L0gnfFfqPzkx2wAAAAAAxA35F9AxBOst4IMFR1JZZ04BU+qr1xUnDdIdMxI43cVnL0nPzZNC9eb0GsfPNruQdyxvXMebLY2+QBp7mdT3eGsP+NmSep/0xt3m3N4ypJQc6Ws/l8Zd1r5aAzVmeLvqz9L2D5tfZ3dJaXlSap4573lqnjm1Slr0PKPQDM5zBkru1Dg+uDjaucrsXA/VS1PmS9N/fug6kYj55YLfZwbp9ZVmR/7Gf5td46E6cz27Sxr+den470iDT29/SA8AAAAA6NbIv4COIVhvAR8sOJIfPvuJnl25Q4Py0rT4ulOU4k5AaGkY5pQp/7lNkmF2MV/wuOTJMK/ft1la/ZT0ydNS1a7G2+UNMwPrMRdLmUXxr6utDMOce3vL29KrP2qsccwlZmCcltex+9//hRkop/Uyg3NPZvJ9oXA01j4n/WOueXnwV8ypcxoC9PpooK7DfGzHDvh6ScefYwAAAABAt0X+BXQMwXoL+GDB4byxoUxX/Okj2WzSs/8zRRMHJmDqkHBIevVmacX/mcsTr5TO/qXkaGEO90jYDK9XP2UepDJUb47b7NKxZ0ln/ULKGRD/GkMBMyyv3CFVbDfPKxvOo6dgTeP6OYOkc39z9AfL7Ene+oX01sLDr+P0ml8meLPM+djzR0rjvyMVT+oeXzAAAAAAABKK/AvomAQeaRHofiprg7r1uTWSpCtOGpSYUN1fbR6kdPNrkmzS1+42pwVpLSy1O8yDbB7zVbOred0LZsi+/UNp42JpyzvmfUyY2/HANVgnffCQtPIJ82Cgh+ucbpBeYM7vfeoPJVdKx7bfU5x2s3kAVX+VGZp7syRPVjREjwbpTQ/0CgAAAAAAgE5Fx3oL+MYOrfnB3z/RP1ft0OC8NL2SiClgfLulpy6SSj81O5K/+Zg08ryju689G6SXb5BKlprLg78ifeMhKbt/++/LMMyDU752q1RR0jju8EhZ/aKnYvPAmE2XM/sQpgMAAAAAkITIv4COoWMdaKMl68v0z1U7ZLNJv7pwTPxD9bJ10pMXmp3gqXnSpU9LxScc/f3lD5fmLJaWLZKW/FT68i3pd1Ol6T8zD4Da1u71vZvMaWm+eMNczugjnfkT84CYaXlMOwIAAAAAAIAeh2AdaIOmU8B89+RBmjDgMFPA7FkvvXSd2amdO0jKHWzOL547yDz3pB96m8+XSH+fLQWqpF5DpVnPmut3lN0uTbnaPPDpi1dL25dJ/7pe+uwl6Ru/NTvLW1Pvk96+1wzmIyHJ4ZamXiudvKDlxwAAAAAAAAD0EEwF0wJ+CoOD3fjMaj3/8U4N7p2mxdedIq+rlW71kF967HRpz7rW7ywtv3ngLkN6+5eSEZYGnCxd/BcpNQFzt0fC0oePSm/cbR7g1JMpTb/HnP+8add5JCJ9+oz03zul6jJz7NizzHV7HRP/ugAAAAAAQKcj/wI6ho514AgWr9mt5z/eKbtNuu/Csa2H6pL05j1mqJ7WWzrjDnM+8vIt0oEtUvmXUt0BqWaPedq+rPltx1xszoGeqINS2h3S1PnSsdOlF74v7VghvTRf+uxFs3s9s4+062Np8f+Tdiw3b5M7WDrrXunYryWmJgAAAAAAAKALIlgHDmOPr14/ft6cAubqrwzR8f1zWl9521Lp/QfNyzMelIZ//dB16ioaQ/aGwL1ypzT0TOnEqztnvvK8odIVr0lLH5He+Jn0+evSIydKx5xuhuwyJFeadNoPzZoSFfQDAAAAAAAAXRTBOtAKwzB0y3NrdKA2qJFFmbrujKGtr+yvll74niRDGjer5VBdklKypZTxUp/xiSi57ewO6aTrot3rV0s7P5I+e8G8bvSF0pk/NTvYAQAAAAAAAByCYB1oxTMrtuuNDXvkdtj1m4vHye20t77y67dLB7ZKWcXSWQs7rcYO6z3M7F5f9qi07QPz4KQDplpdFQAAAAAAAJDUCNaBFpTsr9XdL38mSbpp+rEaVpjR+sqbX5c+ety8PPN3kjerEyqMI4fTDNSnXmt1JQAAAAAAAECXcJgWXKBnCkcM3fTsJ6oJhDVpUK6uPHlw6yvXlksvzjcvT/6+NOjUzikSAAAAAAAAgGUI1oGD/OG9L7V8a7nS3A79+sKxctgPc0DRxT+UqkulXkOlaXd2XpEAAAAAAAAALEOwDjSxsbRK9722SZJ0x4yRKs5NbX3ltf+U1v5Dsjmk8/9XcqV0UpUAAAAAAAAArESwDkQFQhHd8MxqBcIRnTE8XxdNLG595apS6ZUfmJdPvUnqN6FzigQAAAAAAABgOYJ1IOrBJZu0frdPOakuLfzWaNlsrUwBYxjSS9dKdQekorHSqT/s3EIBAAAAAAAAWIpgHZC0ctsBPfrWF5Kke84frfwMb+srr3pC2vwfyeExp4BxuDqpSgAAAAAAAADJgGAdPV5tIKQf/H21IoZ0/vi+Ont0Uesrl2+RXv2RefmM26X8EZ1TJAAAAAAAAICkQbCOHm/h4g3aur9WRVle3fWN41pfMRKWXrhaCtZIA06STry684oEAAAAAAAAkDQI1tGjvb1pr/7y4TZJ0q8uGKuslMNM67L0EankA8mdLs38nWR3dFKVAAAAAAAAAJIJwTp6rMraoP7fPz6RJM2ZOlAnD81rfeWyz6Q37jYvT79HyhmY+AIBAAAAAAAAJCWCdfRY9yxerzKfX4Pz0nTzWcNbXzFYJ/3zu1I4IA2dLh1/eecVCQAAAAAAACDpEKyjR9pTVa/nPt4hSbr3gjFKcR9mWpf/3CbtWSel9Za+8ZBks3VSlQAAAAAAAACSEcE6eqS/LduuYNjQ8f2zdcLA3NZXXP+ytOL/zMvnL5IyCjqnQAAAAAAAAABJi2AdPU4gFNFfl5kHLJ09dWDrK1bukF68xrw89VppyLTEFwcAAAAAAAAg6RGso8f599rd2lvlV36GR2ePKmp5pUhY+uc8qb5C6jNe+uodnVojAAAAAAAAgORFsI4e508fbJUkzZo8QG5nK7vAO7+SSj6Q3OnSt/4gOd2dVyAAAAAAAACApEawjh7lk+0V+rikQi6HTZdN7t/ySts+kN6+17x87m+kXsd0XoEAAAAAAAAAkh7BOnqUJ6Ld6ueO6aPeGZ5DV6gtN6eAMSLS2EulMRd1boEAAAAAAAAAkh7BOnqMvVV+vfzpbknSnJYOWmoY0kvXSr4dUu4x0jm/6twCAQAAAAAAAHQJBOvoMf62vESBcETjirM1tjj70BU++oO04WXJ7pIu+IPkyej0GgEAAAAAAAAkP4J19AjBcER//XCbpFa61cvWSa/+yLx85k+kPuM7rzgAAAAAAAAAXQrBOnqEf68t1Z4qv3pneHTO6KLmVwZqpX9cIYX90pAzpcnft6ZIAAAAAAAAAF0CwTp6hIaDll42qb/czoPe9q/9SNq7QUovkGY+KtnZLQAAAAAAAAC0jgQR3d6aHZVaue2AXA6bZk3u3/zKz16UVv5Rkk06/3+l9N6W1AgAAAAAAACg6yBYR7f3p2i3+jmji5Sf6W28oqJEeula8/LJN0jHnN7ptQEAAAAAAADoegjW0a3tq/brX5/sktTCQUtf+YFUXyn1nSid/uPOLw4AAAAAAABAl0Swjm7t6eUlCoQjGtsvS+P75zResfsTafN/JJtD+uZjksNlXZEAAAAAAAAAuhSCdXRbwXBEf/lwmyRpzkkDm1/53m/M81Hfknod07mFAQAAAAAAAOjSCNbRbb22rlRlPr/y0t06Z3RR4xX7vzAPWiqZc6sDAAAAAAAAQDsQrKPb+tP7WyVJl00eII/T0XjF+w9KRkQ69iyp4DhrigMAAAAAAADQZRGso1tau7NSH207IKfdplmT+zde4dstffI38/LJN1pTHAAAAAAAAIAujWAd3dKfPtgqSTp7dJEKMr2NV3z4iBQOSP2nSv1PtKY4AAAAAAAAAF1alwjWH3nkEQ0cOFBer1eTJ0/W8uXLD7t+RUWFrrnmGhUVFcnj8ejYY4/V4sWLO6laWG1/tV8vfbJLkjRn6sDGK+oOSB/90bxMtzoAAAAAAACAo+S0uoAjeeaZZ7RgwQItWrRIkydP1gMPPKDp06dr48aNys/PP2T9QCCgM888U/n5+frHP/6hvn37atu2bcrOzu784mGJp1dsVyAU0ei+WTq+f3bjFcv/TwpUSwWjpKFnWlYfAAAAAAAAgK4t6YP1+++/X/PmzdPcuXMlSYsWLdIrr7yixx9/XLfccssh6z/++OMqLy/XBx98IJfLJUkaOHBgZ5YMCwXDEf31w22SzG51m81mXhGolZY9al4++UapYRwAAAAAAAAA2impp4IJBAJauXKlpk2bFhuz2+2aNm2ali5d2uJtXnrpJU2ZMkXXXHONCgoKNGrUKN1zzz0Kh8Otbsfv98vn8zU7oWv6z7oy7a6sV680t84dW9R4xao/S7X7pZyB0siZVpUHAAAAAAAAoBtI6mB93759CofDKigoaDZeUFCg0tLSFm/z5Zdf6h//+IfC4bAWL16s22+/Xb/+9a/1s5/9rNXtLFy4UFlZWbFTcXFxXB8HOs8TS7dKki6b3F8ep8McDAelDx4yL0+9TnIk/Q81AAAAAAAAACSxpA7Wj0YkElF+fr4ee+wxTZgwQRdffLF+/OMfa9GiRa3e5tZbb1VlZWXstH379k6sGPGyt8qv5VvKJUmXTurfeMWaZyXfDiktXxo3y6LqAAAAAAAAAHQXSd26m5eXJ4fDobKysmbjZWVlKiwsbPE2RUVFcrlccjgcsbERI0aotLRUgUBAbrf7kNt4PB55PJ74Fo9O99bGPZKk0X2z1Cc7xRyMRKT3HjAvT7lacnmtKQ4AAAAAAABAt5HUHetut1sTJkzQkiVLYmORSERLlizRlClTWrzNSSedpM8//1yRSCQ2tmnTJhUVFbUYqqP7eDMarJ8+PL9xcONiad9GyZMlTbzSosoAAAAAAAAAdCdJHaxL0oIFC/T73/9eTzzxhNavX6/vf//7qqmp0dy5cyVJl19+uW699dbY+t///vdVXl6u66+/Xps2bdIrr7yie+65R9dcc41VDwGdIBCK6J1N+yRJX20I1g1Deu9+8/IJV0reTIuqAwAAAAAAANCdJPVUMJJ08cUXa+/evbrjjjtUWlqqcePG6dVXX40d0LSkpER2e+P3A8XFxXrttdd04403asyYMerbt6+uv/563XzzzVY9BHSCj7aWq9ofUl66W2P6ZpmDW9+Vdq6UnF7pxO9bWyAAAAAAAACAbiPpg3VJmj9/vubPn9/idW+99dYhY1OmTNGHH36Y4KqQTJZsMKeB+cqwfNntNnPw3Wi3+vhvS+n5rdwSAAAAAAAAANon6aeCAdrizWiwHpsGZtfH0pdvSjaHNPU6CysDAAAAAAAA0N0QrKPL27KvRl/uq5HTbtMpQ/PMwfd+Y56PvkDKGWBdcQAAAAAAAAC6HYJ1dHlvRLvVJw3KVYbXJe37XPrsJfPKk26wrjAAAAAAAAAA3RLBOrq8Q6aBef8BSYZ07NlSwUjL6gIAAAAAAADQPRGso0ur9oe0bMt+SdFg3bdL+uRp88qTb7SwMgAAAAAAAADdFcE6urT3Nu9VMGxoYK9UDe6dLi19RIoEpQEnSf0nW10eAAAAAAAAgG6IYB1dWsP86qcPz5dCfmnVn80rmFsdAAAAAAAAQIIQrKPLikQMvblxryTpjOEF0hdvSH6flNFHGjLN4uoAAAAAAAAAdFcE6+iy1u6q1N4qv9LcDk0alCt99qJ5xchvSHbe2gAAAAAAAAASg/QRXVbDNDAnD82TW0Fpw2LzipHnWVgVAAAAAAAAgO6OYB1d1pvRYP2M4QXSl29L/kopvVAqPtHiygAAAAAAAAB0ZwTr6JL2VNXrkx2VkqSvDO8tffaCecWIGUwDAwAAAAAAACChSCDRJb0VPWjp6L5Zyk+xSxteNq84bqZ1RQEAAAAAAADoEQjW0SU1TAPz1eH50pZ3pPpKKS1f6j/F4soAAAAAAAAAdHcE6+hyAqGI3t28T1I0WG82DYzDusIAAAAAAAAA9AgE6+hyVmwtV7U/pLx0j0YXpjZOAzPyPGsLAwAAAAAAANAjEKyjy3kjOg3M6cN6y17ynlR3QErNkwacZHFlAAAAAAAAAHoCgnV0OW80nV993Qvm4IgZksNpXVEAAAAAAAAAegyCdXQpW/bVaMu+GrkcNp18TDbTwAAAAAAAAADodATr6FIautUnDcpVRukyqXa/lJIrDTzF4soAAAAAAAAA9BQE6+hS3thQJkk6fVi+9NmL5uCIc5kGBgAAAAAAAECnIVhHl1HtD2n5lnJJ0hnD8qT1/zKvGDnTuqIAAAAAAAAA9DgE6+gy3tu8V8GwoUF5aRpU84lUs1dKyZEGnWp1aQAAAAAAAAB6EIJ1dBlL1pvzq5vTwLxgDg7/uuRwWVcUAAAAAAAAgB6HYB1dQiRi6M2NeyVJZwzrxTQwAAAAAAAAACxDsI4uYc3OSu2r9ivd49QkxyapukzyZkmDTrO6NAAAAAAAAAA9DME6uoQ3NpjTwJw8JE+ujdFu9WFfl5xuC6sCAAAAAAAA0BMRrKNLeHOjGax/dXietP4lc/C4mdYVBAAAAAAAAKDHIlhH0tvjq9enOyolSWdmbJOqdkueTGnwV6wtDAAAAAAAAECPRLCOpPdW9KClY/plKWfLYnNw2DmS02NhVQAAAAAAAAB6KoJ1JL2G+dW/OoxpYAAAAAAAAABYj2AdSS0QiujdzWbH+rm5OyXfTsmdIQ0+3eLKAAAAAAAAAPRUBOtIapvKqlQTCCsrxaVj9i4xB4edLbm81hYGAAAAAAAAoMciWEdSW7fLPGjpqD4ZsjVMAzPyPAsrAgAAAAAAANDTEawjqa3d6ZMknZm1U6rcLrnTpSFnWFwVAAAAAAAAgJ6MYB1JbW20Y/2kwHvmwLHTJVeKhRUBAAAAAAAA6OkI1pG0whFD63f7JBkaWPZfc3DkTCtLAgAAAAAAAACCdSSvL/dWqz4Y0ST3NrmqtkuuVGnINKvLAgAAAAAAANDDEawjaTVMA3NZxsfmwLHTJXeqhRUBAAAAAAAAAME6kph54FJDpwbfNwdGnmdpPQAAAAAAAAAgEawjia3dWan+tj3KDeySHG5p6NesLgkAAAAAAAAACNaRnCIRQ5/t8mmUbYs5UDBKcqdZWxQAAAAAAAAAiGAdSWr7gVpV+UMa59xqDhSNtbQeAAAAAAAAAGhAsI6kZM6vLp3g2W4OEKwDAAAAAAAASBIE60hKa3dVSjJ0bORLc6DPOCvLAQAAAAAAAIAYgnUkpbU7K9VX+5QWrpTsTil/pNUlAQAAAAAAAIAkgnUkIcMwtG6XT6PsW82B/BGS02NpTQAAAAAAAADQgGAdSafUV6/ymoBGO7aaA0XjrCwHAAAAAAAAAJohWEfSaThw6SRPiTnAgUsBAAAAAAAAJBGCdSSdtTvNA5eOMKIHLqVjHQAAAAAAAEASIVhH0lm3q1L5qlBG+IBkc0iFo6wuCQAAAAAAAABiCNaRdNbu9Gm0Pdqt3nuY5EqxtiAAAAAAAAAAaIJgHUllX7Vfpb56jbJvNQeYXx0AAAAAAABAkiFYR1JZtyt64FLvdnOA+dUBAAAAAAAAJBmCdSQV88Cl0khtMQfoWAcAAAAAAACQZAjWkVTW7apUL1UqJ7RXkk0qHG11SQAAAAAAAADQDME6ksq6Xb7G+dXzhkqedEvrAQAAAAAAAICDEawjaVTWBbVtf62OszENDAAAAAAAAIDkRbCOpPFZw4FLPSXmAME6AAAAAAAAgCREsI6ksW6XeeDS2FQwBOsAAAAAAAAAkhDBOpLG2p2VylK18kKl5kDhGGsLAgAAAAAAAIAWEKwjaZgHLo3Or54zSErJtrQeAAAAAAAAAGgJwTqSQm0gpC/2VmuUbas5wDQwAAAAAAAAAJIUwTqSwvrdVYoY0gR39MClfcZZWg8AAAAAAAAAtKbTg/Xt27friiuu6OzNIsk1HLh0rCM6FQwd6wAAAAAAAACSVKcH6+Xl5XriiSc6e7NIcmt3VipDtSoI7TIHisZZWg8AAAAAAAAAtMYZ7zt86aWXDnv9l19+Ge9NohtYt8unkbZt5kJWfyk119qCAAAAAAAAAKAVcQ/WZ86cKZvNJsMwWl3HZrPFe7PowvyhsDaVVelye8M0MGOsLQgAAAAAAAAADiPuU8EUFRXpueeeUyQSafG0atWqeG8SXdzmsmoFw4aOd0U71pkGBgAAAAAAAEASi3uwPmHCBK1cubLV64/UzY6eZ+1O88Cl45zRYL3POOuKAQAAAAAAAIAjiPtUMD/84Q9VU1PT6vVDhgzRm2++Ge/Nogtbu6tSKapXn9B2c6BorLUFAQAAAAAAAMBhxD1Y79u3rwYNGtTq9WlpaTrttNPivVl0YQ0HLrXJkDKKpPR8q0sCAAAAAAAAgFbFfSqYoUOHau/evbHliy++WGVlZfHeDLqJUDii9bt9GmXfag4wvzoAAAAAAACAJBf3YP3g+dMXL1582Klh0LN9ua9G9cGIxjm3mgNMAwMAAAAAAAAgycU9WAfao+HApce7ogcuJVgHAAAAAAAAkOTiHqzbbDbZbLZDxoCWrN3pk0cBFTccuLTPOEvrAQAAAAAAAIAjifvBSw3D0Jw5c+TxeCRJ9fX1+t73vqe0tLRm6z333HPx3jS6oHW7KjXcViK7wlJab/PgpQAAAAAAAACQxOIerM+ePbvZ8re//e14bwLdRCRi6LNdPp1n32IOFI2V+HUDAAAAAAAAgCQX92D9j3/8Y7zvEt1USXmtqvwhjXE3zK8+ztJ6AAAAAAAAAKAtusTBSx955BENHDhQXq9XkydP1vLly9t0u6efflo2m00zZ85MbIE4Kmt3ceBSAAAAAAAAAF1P0gfrzzzzjBYsWKA777xTq1at0tixYzV9+nTt2bPnsLfbunWrbrrpJp1yyimdVCnaa+1On1wKaWCEYB0AAAAAAABA15H0wfr999+vefPmae7cuRo5cqQWLVqk1NRUPf74463eJhwOa9asWfrJT36iwYMHd2K1aI91uyp1rG27nEZI8mZL2f2tLgkAAAAAAAAAjiipg/VAIKCVK1dq2rRpsTG73a5p06Zp6dKlrd7upz/9qfLz83XllVd2Rpk4CoZhaN0un0bZt5oDfcZx4FIAAAAAAAAAXULcD14aT/v27VM4HFZBQUGz8YKCAm3YsKHF27z33nv6wx/+oNWrV7d5O36/X36/P7bs8/mOql603e7KepXXBDTatcUcYBoYAAAAAAAAAF1EUnest1dVVZW+853v6Pe//73y8vLafLuFCxcqKysrdiouLk5glZCktTvNA5dOdJeYAwTrAAAAAAAAALqIpO5Yz8vLk8PhUFlZWbPxsrIyFRYWHrL+F198oa1bt2rGjBmxsUgkIklyOp3auHGjjjnmmENud+utt2rBggWxZZ/PR7ieYOt2+eRUSMdEtpoDReOsLAcAAAAAAAAA2iypg3W3260JEyZoyZIlmjlzpiQzKF+yZInmz59/yPrDhw/XmjVrmo3ddtttqqqq0oMPPthqWO7xeOTxeOJeP1q3bleljrHtkssISJ5MKWeQ1SUBAAAAAAAAQJskdbAuSQsWLNDs2bM1ceJETZo0SQ888IBqamo0d+5cSdLll1+uvn37auHChfJ6vRo1alSz22dnZ0vSIeOw1tqdPp1sj86vXjhGsnerWYkAAAAAAAAAdGNJH6xffPHF2rt3r+644w6VlpZq3LhxevXVV2MHNC0pKZGdULZL2VvlV6mvXqOcW82BPuOsLAcAAAAAAAAA2sVmGIZhdRHJxufzKSsrS5WVlcrMzLS6nG5n6Rf7denvP9RLqXdrTGS99M3fS2MusrosAAAAAACAHoP8C+gYWr3R6SrrArIromMjX5oDRWOtLQgAAAAAAAAA2oFgHZ3OVxfSINtueeWXXGlSryFWlwQAAAAAAAAAbUawjk5XWRfUKFvDgUtHS3aHtQUBAAAAAAAAQDsQrKPT+eqDGm2PButMAwMAAAAAAACgiyFYR6fz1QU1yr7VXOgzzspSAAAAAAAAAKDdCNbR6arqAhpp22ou0LEOAAAAAAAAoIshWEenc1eVKNNWp5DdI+UNs7ocAAAAAAAAAGgXgnV0urSa7ZKk2vQBksNpcTUAAAAAAAAA0D4E6+h0Dn+FJMlIybG2EAAAAAAAAAA4CgTr6HSOQKV5wUuwDgAAAAAAAKDrIVhHp3MHqyRJjrRsawsBAAAAAAAAgKNAsI5OFQhFlBoxg3VnWq7F1QAAAAAAAABA+xGso1NV1QeVpRpJkiuNqWAAAAAAAAAAdD0E6+hUlXVBZdnMYN2RSrAOAAAAAAAAoOshWEen8tWHYh3rSiFYBwAAAAAAAND1EKyjU/madKzLm21pLQAAAAAAAABwNAjW0al89U2C9ZRsS2sBAAAAAAAAgKNBsI5OVVkXVKboWAcAAAAAAADQdRGso1NV1fqVaaszF+hYBwAAAAAAANAFEayjU/mrDzQueLOsKwQAAAAAAAAAjhLBOjpVqGa/JClgT5UcLourAQAAAAAAAID2I1hHp4rUmB3rAVemxZUAAAAAAAAAwNEhWEfnqq+QJIU8BOsAAAAAAAAAuiaCdXQqW32lJCniyba2EAAAAAAAAAA4SgTr6FTOgBmsy5ttaR0AAAAAAAAAcLQI1tGpXEEzWLenZltbCAAAAAAAAAAcJYJ1dBrDMOQJVkmSHGk5FlcDAAAAAAAAAEeHYB2dpj4YUbpRLUlyp+VaXA0AAAAAAAAAHB2CdXQaX31QWbYaSZI7nWAdAAAAAAAAQNdEsI5O46sLKktmsG5LYSoYAAAAAAAAAF0TwTo6TdOOdaVkW1oLAAAAAAAAABwtgnV0msq6JsG6N9vSWgAAAAAAAADgaBGso9P46kLKFB3rAAAAAAAAALo2gnV0mqq6emXa6swFOtYBAAAAAAAAdFEE6+g09b7yxgU61gEAAAAAAAB0UQTr6DTBGjNYD9hTJIfL4moAAAAAAAAA4OgQrKPThGsqJEl+V6a1hQAAAAAAAABABxCso9MY9QckSUFXlsWVAAAAAAAAAMDRI1hH56mrkCRFPATrAAAAAAAAALougnV0GkegUpJkeLOtLQQAAAAAAAAAOoBgHZ3GGfBJkmyp2dYWAgAAAAAAAAAdQLCOTuMJmsG6IzXH4koAAAAAAAAA4OgRrKNTRCKGvOEqSZIrjWAdAAAAAAAAQNdFsI5OURMIKVM1kiRPRi+LqwEAAAAAAACAo0ewjk7hqw8pKxqsO+lYBwAAAAAAANCFEayjU1TWBpVlM4N1WwrBOgAAAAAAAICui2AdncJXH4x1rMubbWktAAAAAAAAANARBOvoFL66xo51pWRbWgsAAAAAAAAAdATBOjqFr86vTFutuUDHOgAAAAAAAIAujGAdnaKu6kDjAh3rAAAAAAAAALowgnV0imBVuSTJb0+RHC6LqwEAAAAAAACAo0ewjk4Rqo0G684MiysBAAAAAAAAgI4hWEeniNSaU8EEXZkWVwIAAAAAAAAAHUOwjs5RVylJCrmzLC4EAAAAAAAAADqGYB2dwu6vkCQZ3mxL6wAAAAAAAACAjiJYR6dwBsyOdaVkW1oHAAAAAAAAAHQUwTo6hTvokyQ5CNYBAAAAAAAAdHEE6+gUnlCVJMmZnmtxJQAAAAAAAADQMQTrSLhQOKLUiBmsu9MI1gEAAAAAAAB0bQTrSLiq+pCyVCNJ8mQSrAMAAAAAAADo2gjWkXC++qCybGaw7kwlWAcAAAAAAADQtRGsI+F8dY0d6+LgpQAAAAAAAAC6OIJ1JFzTjnV5sy2tBQAAAAAAAAA6imAdCVdZ61eG6swFOtYBAAAAAAAAdHEE60i4uqpy2W2GuUDHOgAAAAAAAIAujmAdCReoLpck+W1eyem2uBoAAAAAAAAA6BiCdSRcqPqAJKnemWFxJQAAAAAAAADQcQTrSLhwrdmxHnRlWlwJAAAAAAAAAHQcwToSzqirkCQF3QTrAAAAAAAAALo+gnUknL2+QpIU8WRbWgcAAAAAAAAAxAPBOhLO7q80L3izLa0DAAAAAAAAAOKBYB0J5wr6JEm2lGxrCwEAAAAAAACAOCBYR8J5osG6Iy3H4koAAAAAAAAAoOMI1pFwKZEqSZI7PdfiSgAAAAAAAACg4wjWkVD+UFjpkWpJkieDYB0AAAAAAABA10ewjoTy1YWUZauRJHnTe1lcDQAAAAAAAAB0HME6EspXH1SWzGDdnsoc6wAAAAAAAAC6PoJ1JFRlXTDWsa6UbEtrAQAAAAAAAIB4IFhHQvlq/cpQnbmQQsc6AAAAAAAAgK6PYB0JVVd1QHabYS54sy2tBQAAAAAAAADioUsE64888ogGDhwor9eryZMna/ny5a2u+/vf/16nnHKKcnJylJOTo2nTph12fSSWv6rcPLd5Jafb4moAAAAAAAAAoOOSPlh/5plntGDBAt15551atWqVxo4dq+nTp2vPnj0trv/WW2/p0ksv1ZtvvqmlS5equLhYX/va17Rz585OrhySFKg2g/U6R4bFlQAAAAAAAABAfCR9sH7//fdr3rx5mjt3rkaOHKlFixYpNTVVjz/+eIvrP/nkk7r66qs1btw4DR8+XP/3f/+nSCSiJUuWdHLlkKRwrRmsB1wE6wAAAAAAAAC6h6QO1gOBgFauXKlp06bFxux2u6ZNm6alS5e26T5qa2sVDAaVm5vb6jp+v18+n6/ZCfERqa2QJAVdWdYWAgAAAAAAAABxktTB+r59+xQOh1VQUNBsvKCgQKWlpW26j5tvvll9+vRpFs4fbOHChcrKyoqdiouLO1Q3GtnqKyRJYQ/BOgAAAAAAAIDuIamD9Y76xS9+oaefflrPP/+8vF5vq+vdeuutqqysjJ22b9/eiVV2b3Z/pSTJ8BKsAwAAAAAAAOgenFYXcDh5eXlyOBwqKytrNl5WVqbCwsLD3va+++7TL37xC/33v//VmDFjDruux+ORx+PpcL04lCtgBus2b47FlQAAAAAAAABAfCR1x7rb7daECROaHXi04UCkU6ZMafV2v/zlL3X33Xfr1Vdf1cSJEzujVLTCHaySJNnTCNYBAAAAAAAAdA9J3bEuSQsWLNDs2bM1ceJETZo0SQ888IBqamo0d+5cSdLll1+uvn37auHChZKke++9V3fccYeeeuopDRw4MDYXe3p6utLT0y17HD2VJ2weCNZFsA4AAAAAAACgm0j6YP3iiy/W3r17dccdd6i0tFTjxo3Tq6++GjugaUlJiez2xsb7Rx99VIFAQBdccEGz+7nzzjt11113dWbpPZ5hGEoNV0l2yZORa3U5AAAAAAAAABAXNsMwDKuLSDY+n09ZWVmqrKxUZmam1eV0WbWBkD7/2USNsW9R/YV/k/e4c6wuCQAAAAAAACL/AjoqqedYR9fmqwspSzWS6FgHAAAAAAAA0H0QrCNhKuuCyrKZwbothTnWAQAAAAAAAHQPBOtIGF+dX5mqNRe82ZbWAgAAAAAAAADxQrCOhKn1HZDdFp3CPyXb0loAAAAAAAAAIF4I1pEwdb79kiS/zSM5PRZXAwAAAAAAAADxQbCOhAlUl0uS6uwZFlcCAAAAAAAAAPFDsI6ECdUckCTVOwnWAQAAAAAAAHQfBOtImEitGawH3ZkWVwIAAAAAAAAA8UOwjoQx6iokSSF3lrWFAAAAAAAAAEAcEawjYez+CklSxEOwDgAAAAAAAKD7IFhHwjgDPvOCN9vSOgAAAAAAAAAgngjWkTCuQKUkyZ6aY3ElAAAAAAAAABA/BOtIGE+4SpLkTCNYBwAAAAAAANB9EKwjYVKiwbo7PdfiSgAAAAAAAAAgfgjWkRCRiKG0SLUkyZvRy+JqAAAAAAAAACB+CNaREFX+kLJUI0nyZtKxDgAAAAAAAKD7IFhHQvjqgsqymcG6J52OdQAAAAAAAADdB8E6EsJX51dmtGNdKdmW1gIAAAAAAAAA8USwjoSo8VXIYTPMBW+2pbUAAAAAAAAAQDwRrCMh6nz7JUl+uSWX1+JqAAAAAAAAACB+CNaREP7qcklSrSPd4koAAAAAAAAAIL4I1pEQwWiwXu/ItLgSAAAAAAAAAIgvgnUkRKT2gCQp4CJYBwAAAAAAANC9EKwjIYy6CklS0E2wDgAAAAAAAKB7IVhHQtjqzY71iCfL4koAAAAAAAAAIL4I1pEQDn+lJMnwZltbCAAAAAAAAADEGcE6EsIZ8EmS7CnZ1hYCAAAAAAAAAHFGsI6E8ITMYN2RmmNxJQAAAAAAAAAQXwTrSAhvqEqS5ErvZXElAAAAAAAAABBfBOtIiNSIGax7MnItrgQAAAAAAAAA4otgHXEXDEeUYdRIkrwZdKwDAAAAAAAA6F4I1hF3VfUhZdnMYD01i2AdAAAAAAAAQPdCsI6489X6lSkzWHemMRUMAAAAAAAAgO6FYB1xV+U7IIfNMBe82ZbWAgAAAAAAAADxRrCOuKvz7Zck+eWWXF6LqwEAAAAAAACA+CJYR9z5q8olSTX2dIsrAQAAAAAAAID4I1hH3AWqzWC9zpFhcSUAAAAAAAAAEH8E64i7UM0BSZLfmWlxJQAAAAAAAAAQfwTriDujzgzWg26CdQAAAAAAAADdD8E64q++QpIUdmdZWwcAAAAAAAAAJADBOuLOXl8pSTK8BOsAAAAAAAAAuh+CdcSdM2AG6/JmW1oHAAAAAAAAACQCwTrizhX0SZIcaTkWVwIAAAAAAAAA8Uewjrjzhsxg3ZmWa3ElAAAAAAAAABB/BOuIu9RwtSTJnU6wDgAAAAAAAKD7IVhH3KUZZrCektnL4koAAAAAAAAAIP4I1hFX9cGwMmUG66kE6wAAAAAAAAC6IYJ1xJWvNqBM1UqiYx0AAAAAAABA90Swjriqqjogpy0iSbKn5lhcDQAAAAAAAADEH8E64qq2cr8kyS+X5EqxuBoAAAAAAAAAiD+CdcRVfZUZrNfY0i2uBAAAAAAAAAASg2AdcRWoKpck1ToyLK4EAAAAAAAAABKDYB1xFawxg3W/k2AdAAAAAAAAQPdEsI64itRWSJICrkxrCwEAAAAAAACABCFYR1wZdRWSpJA7y9pCAAAAAAAAACBBCNYRV7b6CklSxEOwDgAAAAAAAKB7IlhHXDkDleaFlGxL6wAAAAAAAACARHFaXQC6F1fAJ0myp+RYXAkAAAAAAAB6gnA4rGAwaHUZSGIOh0NOp1M2my1u90mwjrhyh8xg3ZGWbW0hAAAAAAAA6Paqq6u1Y8cOGYZhdSlIcqmpqSoqKpLb7Y7L/RGsI65SwlWSJHd6rsWVAAAAAAAAoDsLh8PasWOHUlNT1bt377h2I6P7MAxDgUBAe/fu1ZYtWzR06FDZ7R2fIZ1gHXGVFqmWJHnSe1lcCQAAAAAAALqzYDAowzDUu3dvpaSkWF0OklhKSopcLpe2bdumQCAgr9fb4fvk4KWIG8MwlG6YwXpKFsE6AAAAAAAAEo9OdbRFPLrUm91fXO8NPVqNP6Qs1UiS0rLyLK4GAAAAAAAAABKDYB1xU+WrkNMWkSR5M5hjHQAAAAAAAEi0gQMH6oEHHmjz+m+99ZZsNpsqKioSVlNPQLCOuKmp3CdJCsgpmyvV4moAAAAAAACA5POVr3xFN9xwQ9zub8WKFbrqqqvavP7UqVO1e/duZWVlxa2GljQE+AefbrvtNklSfX295syZo9GjR8vpdGrmzJkJrSfeOHgp4qaucr8kqdqWrlzmtgIAAAAAAACOimEYCofDcjqPHN/27t27XfftdrtVWFh4tKW128aNG5WZmRlbTk9PlySFw2GlpKTouuuu0z//+c9Oqyde6FhH3PiryyVJNfYMiysBAAAAAAAAks+cOXP09ttv68EHH4x1cG/dujXW3f3vf/9bEyZMkMfj0XvvvacvvvhC5513ngoKCpSenq4TTjhB//3vf5vd58FTwdhsNv3f//2fzj//fKWmpmro0KF66aWXYtcfPBXMn/70J2VnZ+u1117TiBEjlJ6errPOOku7d++O3SYUCum6665Tdna2evXqpZtvvlmzZ89uU5d5fn6+CgsLY6eGYD0tLU2PPvqo5s2b16lBf7wQrCNugtFgvd5BsA4AAAAAAIDOZRiGagMhS06GYbSpxgcffFBTpkzRvHnztHv3bu3evVvFxcWx62+55Rb94he/0Pr16zVmzBhVV1frnHPO0ZIlS/Txxx/rrLPO0owZM1RSUnLY7fzkJz/RRRddpE8//VTnnHOOZs2apfLy8lbXr62t1X333ae//OUveuedd1RSUqKbbropdv29996rJ598Un/84x/1/vvvy+fz6YUXXmjTY+6umAoGcROuPSBJ8rsI1gEAAAAAANC56oJhjbzjNUu2/dlPpyvVfeSoNSsrS263W6mpqS12af/0pz/VmWeeGVvOzc3V2LFjY8t33323nn/+eb300kuaP39+q9uZM2eOLr30UknSPffco9/+9rdavny5zjrrrBbXDwaDWrRokY455hhJ0vz58/XTn/40dv1DDz2kW2+9Veeff74k6eGHH9bixYuP+HglqV+/fs2Wt23bpl69erXptsmMYB1xE4kG60FXYg98AAAAAAAAAHRHEydObLZcXV2tu+66S6+88op2796tUCikurq6I3asjxkzJnY5LS1NmZmZ2rNnT6vrp6amxkJ1SSoqKoqtX1lZqbKyMk2aNCl2vcPh0IQJExSJRI74mN59911lZDQ24ubk5BzxNl0BwTrixlZfKUkKe7OtLQQAAAAAAAA9TorLoc9+Ot2ybcdDWlpas+WbbrpJr7/+uu677z4NGTJEKSkpuuCCCxQIBA57Py6Xq9myzWY7bAje0vptnd7mSAYNGqTs7Oy43FcyIVhH3Dj8FeYFDx3rAAAAAAAA6Fw2m61N07FYze12KxwOt2nd999/X3PmzIlNwVJdXa2tW7cmsLpDZWVlqaCgQCtWrNCpp54qSQqHw1q1apXGjRvXqbUkk+R/p6HLcAbMjnWlZFtaBwAAAAAAAJCsBg4cqGXLlmnr1q1KT09Xbm5uq+sOHTpUzz33nGbMmCGbzabbb7+9TdOvxNu1116rhQsXasiQIRo+fLgeeughHThwQDabrUP3+9lnnykQCKi8vFxVVVVavXq1JHWJwJ5gHXHjDvkkSc601j8MAAAAAAAAgJ7spptu0uzZszVy5EjV1dVpy5Ytra57//3364orrtDUqVOVl5enm2++WT6frxOrNd18880qLS3V5ZdfLofDoauuukrTp0+Xw9GxKXDOOeccbdu2LbY8fvx4SYrbNDSJZDO6QpWdzOfzKSsrS5WVlcrMzLS6nC5jw90naHh4k9aeukijvnqp1eUAAAAAAACgFd0h/6qvr9eWLVs0aNAgeb1eq8vpUSKRiEaMGKGLLrpId999t9XltEm83y90rCNuUiNVkiRPRi+LKwEAAAAAAAAQL9u2bdN//vMfnXbaafL7/Xr44Ye1ZcsWXXbZZVaXZhm71QWg+0iPVEuSvBlMBQMAAAAAAAB0F3a7XX/60590wgkn6KSTTtKaNWv03//+VyNGjLC6NMvQsY64CIcjylSNJCk9K8/iagAAAAAAAADES3Fxsd5//32ry0gqdKwjLqp9FXLazCMSp2UTrAMAAAAAAADovgjWERfVlfskSQHDKbc3zeJqAAAAAAAAACBxukSw/sgjj2jgwIHyer2aPHmyli9fftj1n332WQ0fPlxer1ejR4/W4sWLO6nSnqvOt1+SVGVLl2w2i6sBAAAAAAAAgMRJ+mD9mWee0YIFC3TnnXdq1apVGjt2rKZPn649e/a0uP4HH3ygSy+9VFdeeaU+/vhjzZw5UzNnztTatWs7ufKepT4arNfY0y2uBAAAAAAAAAASK+mD9fvvv1/z5s3T3LlzNXLkSC1atEipqal6/PHHW1z/wQcf1FlnnaUf/vCHGjFihO6++24df/zxevjhhzu58uQUDoUUCgbafAoGAwoEAvIH/Kr3m6e6er9q6+tVU1ev6rp6VdXWqabS/KKjzpFh8SMEAAAAAAAAgMRyWl3A4QQCAa1cuVK33nprbMxut2vatGlaunRpi7dZunSpFixY0Gxs+vTpeuGFF1rdjt/vl9/vjy37fL6OFZ7ENvzyqzou8Enc73dy9LzeSbAOAAAAAAAAoHtL6o71ffv2KRwOq6CgoNl4QUGBSktLW7xNaWlpu9aXpIULFyorKyt2Ki4u7njxPVRwwGlWlwAAAAAAAAB0awMHDtQDDzwQW7bZbIdtLN66datsNptWr17doe3G6366g6TuWO8st956a7Mud5/P123D9f7XvKDKULBdt7HJJtmk2JnNJrvNHG84TqnNJjkcLk1MzYp7zQAAAAAAAABat3v3buXk5MT1PufMmaOKiopmgX1xcbF2796tvLy8uG7rYHfddZd+8pOfHDL++uuva9q0aVq3bp3uuOMOrVy5Utu2bdNvfvMb3XDDDQmt6WBJHazn5eXJ4XCorKys2XhZWZkKCwtbvE1hYWG71pckj8cjj8fT8YK7gIysXKtLAAAAAAAAABBHh8s+48nhcHTato477jj997//bTaWm2tmm7W1tRo8eLAuvPBC3XjjjZ1Sz8GSeioYt9utCRMmaMmSJbGxSCSiJUuWaMqUKS3eZsqUKc3Wl8xvMlpbHwAAAAAAAAA6w2OPPaY+ffooEok0Gz/vvPN0xRVXSJK++OILnXfeeSooKFB6erpOOOGEQwLmgx08Fczy5cs1fvx4eb1eTZw4UR9//HGz9cPhsK688koNGjRIKSkpGjZsmB588MHY9XfddZeeeOIJvfjii7LZbLLZbHrrrbdanArm7bff1qRJk+TxeFRUVKRbbrlFoVAodv1XvvIVXXfddfp//+//KTc3V4WFhbrrrruO+Fw5nU4VFhY2O7ndbknSCSecoF/96le65JJLLGuYTuqOdUlasGCBZs+erYkTJ2rSpEl64IEHVFNTo7lz50qSLr/8cvXt21cLFy6UJF1//fU67bTT9Otf/1pf//rX9fTTT+ujjz7SY489ZuXDAAAAAAAAAJBIhiEFa63ZtitVsTmTD+PCCy/UtddeqzfffFNnnHGGJKm8vFyvvvqqFi9eLEmqrq7WOeeco5///OfyeDz685//rBkzZmjjxo3q37//EbdRXV2tc889V2eeeab++te/asuWLbr++uubrROJRNSvXz89++yz6tWrlz744ANdddVVKioq0kUXXaSbbrpJ69evl8/n0x//+EdJZrf4rl27mt3Pzp07dc4552jOnDn685//rA0bNmjevHnyer3NwvMnnnhCCxYs0LJly7R06VLNmTNHJ510ks4888wjPp5klfTB+sUXX6y9e/fqjjvuUGlpqcaNG6dXX301doDSkpIS2e2NjfdTp07VU089pdtuu00/+tGPNHToUL3wwgsaNWqUVQ8BAAAAAAAAQKIFa6V7+liz7R/tktxpR1wtJydHZ599tp566qlYsP6Pf/xDeXl5Ov300yVJY8eO1dixY2O3ufvuu/X888/rpZde0vz584+4jaeeekqRSER/+MMf5PV6ddxxx2nHjh36/ve/H1vH5XI1m8N80KBBWrp0qf7+97/roosuUnp6ulJSUuT3+w879cvvfvc7FRcX6+GHH5bNZtPw4cO1a9cu3Xzzzbrjjjtiue2YMWN05513SpKGDh2qhx9+WEuWLDlssL5mzRqlp6fHlkeOHKnly5cf8fF3lqQP1iVp/vz5rb5p3nrrrUPGLrzwQl144YUJrgoAAAAAAAAA2mfWrFmaN2+efve738nj8ejJJ5/UJZdcEguhq6urddddd+mVV17R7t27FQqFVFdXp5KSkjbd//r16zVmzBh5vd7YWEvTZD/yyCN6/PHHVVJSorq6OgUCAY0bN65dj2X9+vWaMmWKbE269U866SRVV1drx44dsQ77MWPGNLtdUVGR9uzZc9j7HjZsmF566aXYcrIdI7NLBOsAAAAAAAAAcFiuVLNz3Kptt9GMGTNkGIZeeeUVnXDCCXr33Xf1m9/8Jnb9TTfdpNdff1333XefhgwZopSUFF1wwQUKBAJxK/fpp5/WTTfdpF//+teaMmWKMjIy9Ktf/UrLli2L2zaacrlczZZtNtsh88wfzO12a8iQIQmpJx4I1gEAAAAAAAB0fTZbm6ZjsZrX69U3v/lNPfnkk/r88881bNgwHX/88bHr33//fc2ZM0fnn3++JLODfevWrW2+/xEjRugvf/mL6uvrY13rH374YbN13n//fU2dOlVXX311bOyLL75oto7b7VY4HD7itv75z3/KMIxY1/r777+vjIwM9evXr801d0X2I68CAAAAAAAAAIiXWbNm6ZVXXtHjjz+uWbNmNbtu6NCheu6557R69Wp98sknuuyyy47Y3d3UZZddJpvNpnnz5umzzz7T4sWLdd999x2yjY8++kivvfaaNm3apNtvv10rVqxots7AgQP16aefauPGjdq3b5+CweAh27r66qu1fft2XXvttdqwYYNefPFF3XnnnVqwYEGz42LGWyAQ0OrVq7V69WoFAgHt3LlTq1ev1ueff56wbR6MYB0AAAAAAAAAOtFXv/pV5ebmauPGjbrsssuaXXf//fcrJydHU6dO1YwZMzR9+vRmHe1Hkp6ern/9619as2aNxo8frx//+Me69957m63zP//zP/rmN7+piy++WJMnT9b+/fubda9L0rx58zRs2DBNnDhRvXv31vvvv3/Itvr27avFixdr+fLlGjt2rL73ve/pyiuv1G233daOZ6P9du3apfHjx2v8+PHavXu37rvvPo0fP17f/e53E7rdpmyGYRidtrUuwufzKSsrS5WVlcrMzLS6HAAAAAAAACCuukP+VV9fry1btmjQoEHNDtQJtCTe7xc61gEAAAAAAAAAaAeCdQAAAAAAAAAA2oFgHQAAAAAAAACAdiBYBwAAAAAAAACgHQjWAQAAAAAAAABoB4J1AAAAAAAAAF2WYRhWl4AuIN7vE4J1AAAAAAAAAF2Ow+GQJAUCAYsrQVdQW1srSXK5XHG5P2dc7qWbafj2wufzWVwJAAAAAAAAEH8NuVdX7vZ2Op1KTU3V3r175XK5ZLfTQ4xDGYah2tpa7dmzR9nZ2bEvZDrKZnTlvSdBduzYoeLiYqvLAAAAAAAAABJq+/bt6tevn9VlHLVAIKAtW7YoEolYXQqSXHZ2tgoLC2Wz2eJyfwTrLYhEItq1a5cyMjLi9kQnC5/Pp+LiYm3fvl2ZmZlWlwP0KOx/gDXY9wBrsO8B1mH/A6zR1fY9wzBUVVWlPn36dPlO70gkwnQwOCyXyxW3TvUGTAXTArvd3qW/qWuLzMzMLvEhD3RH7H+ANdj3AGuw7wHWYf8DrNGV9r2srCyrS4gLu90ur9drdRnoYbr211EAAAAAAAAAAHQygnUAAAAAAAAAANqBYL2H8Xg8uvPOO+XxeKwuBehx2P8Aa7DvAdZg3wOsw/4HWIN9D+hZOHgpAAAAAAAAAADtQMc6AAAAAAAAAADtQLAOAAAAAAAAAEA7EKwDAAAAAAAAANAOBOsAAAAAAAAAALQDwXoP88gjj2jgwIHyer2aPHmyli9fbnVJQLeycOFCnXDCCcrIyFB+fr5mzpypjRs3Nlunvr5e11xzjXr16qX09HR961vfUllZmUUVA93TL37xC9lsNt1www2xMfY9IDF27typb3/72+rVq5dSUlI0evRoffTRR7HrDcPQHXfcoaKiIqWkpGjatGnavHmzhRUD3UM4HNbtt9+uQYMGKSUlRcccc4zuvvtuGYYRW4f9D+i4d955RzNmzFCfPn1ks9n0wgsvNLu+LftZeXm5Zs2apczMTGVnZ+vKK69UdXV1Jz4KAIlAsN6DPPPMM1qwYIHuvPNOrVq1SmPHjtX06dO1Z88eq0sDuo23335b11xzjT788EO9/vrrCgaD+trXvqaamprYOjfeeKP+9a9/6dlnn9Xbb7+tXbt26Zvf/KaFVQPdy4oVK/S///u/GjNmTLNx9j0g/g4cOKCTTjpJLpdL//73v/XZZ5/p17/+tXJycmLr/PKXv9Rvf/tbLVq0SMuWLVNaWpqmT5+u+vp6CysHur57771Xjz76qB5++GGtX79e9957r375y1/qoYceiq3D/gd0XE1NjcaOHatHHnmkxevbsp/NmjVL69at0+uvv66XX35Z77zzjq666qrOeggAEsRmNP06G93a5MmTdcIJJ+jhhx+WJEUiERUXF+vaa6/VLbfcYnF1QPe0d+9e5efn6+2339app56qyspK9e7dW0899ZQuuOACSdKGDRs0YsQILV26VCeeeKLFFQNdW3V1tY4//nj97ne/089+9jONGzdODzzwAPsekCC33HKL3n//fb377rstXm8Yhvr06aMf/OAHuummmyRJlZWVKigo0J/+9CddcsklnVku0K2ce+65Kigo0B/+8IfY2Le+9S2lpKTor3/9K/sfkAA2m03PP/+8Zs6cKalt/86tX79eI0eO1IoVKzRx4kRJ0quvvqpzzjlHO3bsUJ8+fax6OAA6iI71HiIQCGjlypWaNm1abMxut2vatGlaunSphZUB3VtlZaUkKTc3V5K0cuVKBYPBZvvi8OHD1b9/f/ZFIA6uueYaff3rX2+2j0nse0CivPTSS5o4caIuvPBC5efna/z48fr9738fu37Lli0qLS1ttu9lZWVp8uTJ7HtAB02dOlVLlizRpk2bJEmffPKJ3nvvPZ199tmS2P+AztCW/Wzp0qXKzs6OheqSNG3aNNntdi1btqzTawYQP06rC0Dn2Ldvn8LhsAoKCpqNFxQUaMOGDRZVBXRvkUhEN9xwg0466SSNGjVKklRaWiq3263s7Oxm6xYUFKi0tNSCKoHu4+mnn9aqVau0YsWKQ65j3wMS48svv9Sjjz6qBQsW6Ec/+pFWrFih6667Tm63W7Nnz47tXy39H5R9D+iYW265RT6fT8OHD5fD4VA4HNbPf/5zzZo1S5LY/4BO0Jb9rLS0VPn5+c2udzqdys3NZV8EujiCdQBIkGuuuUZr167Ve++9Z3UpQLe3fft2XX/99Xr99dfl9XqtLgfoMSKRiCZOnKh77rlHkjR+/HitXbtWixYt0uzZsy2uDuje/v73v+vJJ5/UU089peOOO06rV6/WDTfcoD59+rD/AQDQCZgKpofIy8uTw+FQWVlZs/GysjIVFhZaVBXQfc2fP18vv/yy3nzzTfXr1y82XlhYqEAgoIqKimbrsy8CHbNy5Urt2bNHxx9/vJxOp5xOp95++2399re/ldPpVEFBAfsekABFRUUaOXJks7ERI0aopKREkmL7F/8HBeLvhz/8oW655RZdcsklGj16tL7zne/oxhtv1MKFCyWx/wGdoS37WWFhofbs2dPs+lAopPLycvZFoIsjWO8h3G63JkyYoCVLlsTGIpGIlixZoilTplhYGdC9GIah+fPn6/nnn9cbb7yhQYMGNbt+woQJcrlczfbFjRs3qqSkhH0R6IAzzjhDa9as0erVq2OniRMnatasWbHL7HtA/J100knauHFjs7FNmzZpwIABkqRBgwapsLCw2b7n8/m0bNky9j2gg2pra2W3N/+T3uFwKBKJSGL/AzpDW/azKVOmqKKiQitXroyt88YbbygSiWjy5MmdXjOA+GEqmB5kwYIFmj17tiZOnKhJkybpgQceUE1NjebOnWt1aUC3cc011+ipp57Siy++qIyMjNiceVlZWUpJSVFWVpauvPJKLViwQLm5ucrMzNS1116rKVOm6MQTT7S4eqDrysjIiB3LoEFaWpp69eoVG2ffA+Lvxhtv1NSpU3XPPffooosu0vLly/XYY4/psccekyTZbDbdcMMN+tnPfqahQ4dq0KBBuv3229WnTx/NnDnT2uKBLm7GjBn6+c9/rv79++u4447Txx9/rPvvv19XXHGFJPY/IF6qq6v1+eefx5a3bNmi1atXKzc3V/379z/ifjZixAidddZZmjdvnhYtWqRgMKj58+frkksuUZ8+fSx6VADiwkCP8tBDDxn9+/c33G63MWnSJOPDDz+0uiSgW5HU4umPf/xjbJ26ujrj6quvNnJycozU1FTj/PPPN3bv3m1d0UA3ddpppxnXX399bJl9D0iMf/3rX8aoUaMMj8djDB8+3HjssceaXR+JRIzbb7/dKCgoMDwej3HGGWcYGzdutKhaoPvw+XzG9ddfb/Tv39/wer3G4MGDjR//+MeG3++PrcP+B3Tcm2++2eLfeLNnzzYMo2372f79+41LL73USE9PNzIzM425c+caVVVVFjwaAPFkMwzDsCjTBwAAAAAAAACgy2GOdQAAAAAAAAAA2oFgHQAAAAAAAACAdiBYBwAAAAAAAACgHQjWAQAAAAAAAABoB4J1AAAAAAAAAADagWAdAAAAAAAAAIB2IFgHAAAA8P/buZ+QqPo9juNvsRoNFTElBC0FUTTSsILKIIZsJUFYaGBoiLRoE1ZWKIU5C93MxqI/i0jcFFG0iGlhEraQAhWMMkkrKNpkmAWhSKTP4oG5yYV7nfsk3vD9ggOHOb/zmw+/5YfDV5IkSVIMLNYlSZL0x+jr6yMuLo6vX78udxRJkiRJK5jFuiRJkiRJkiRJMbBYlyRJkiRJkiQpBhbrkiRJWrS5uTna29vJzc0lMTGRkpIS7t69C/xrTEskEqG4uJiEhAR27NjBy5cvF+xx7949Nm3aRCAQICcnh3A4vOD57OwsZ8+eJTs7m0AgQF5eHjdu3FiwZmhoiG3btrF27Vp27drF69evo8+eP39OMBgkOTmZlJQUtm7dyuDg4BKdiCRJkqSVyGJdkiRJi9be3k53dzfXrl1jZGSExsZGjhw5wpMnT6JrmpqaCIfDDAwMkJGRwf79+/nx4wfwdyFeVVXF4cOHefHiBa2trZw/f56urq7o+7W1tdy6dYvOzk5GR0e5fv06SUlJC3K0tLQQDocZHBxk1apV1NfXR5/V1NSQlZXFwMAAQ0NDnDt3jtWrVy/twUiSJElaUeLm5+fnlzuEJEmS/v/Nzs6SlpZGb28vO3fujP7e0NDA9PQ0x44dIxgMcvv2baqrqwH48uULWVlZdHV1UVVVRU1NDZ8/f6anpyf6/pkzZ4hEIoyMjDA2NkZBQQGPHj2ivLz83zL09fURDAbp7e1l7969ADx8+JCKigpmZmZISEggJSWFS5cuUVdXt8QnIkmSJGml8ot1SZIkLcqbN2+Ynp5m3759JCUlRa/u7m7evn0bXfdr6Z6WlkZBQQGjo6MAjI6OUlZWtmDfsrIyxsfH+fnzJ8PDw8THx7Nnz57/mKW4uDh6n5mZCcDExAQAJ0+epKGhgfLycjo6OhZkkyRJkqTfwWJdkiRJi/L9+3cAIpEIw8PD0evVq1fROev/VGJi4qLW/TraJS4uDvh7/jtAa2srIyMjVFRU8PjxY4qKirh///5vySdJkiRJYLEuSZKkRSoqKiIQCPDhwwfy8vIWXNnZ2dF1z549i95PTU0xNjZGYWEhAIWFhfT39y/Yt7+/n/z8fOLj49m8eTNzc3MLZrb/L/Lz82lsbKSnp4fKykpu3rz5j/aTJEmSpF+tWu4AkiRJ+jMkJydz+vRpGhsbmZubY/fu3Xz79o3+/n5SUlLYuHEjAG1tbaxbt47169fT0tJCeno6Bw4cAODUqVNs376dUChEdXU1T58+5fLly1y5cgWAnJwc6urqqK+vp7Ozk5KSEt6/f8/ExARVVVX/NePMzAxNTU0cOnSI3NxcPn78yMDAAAcPHlyyc5EkSZK08lisS5IkadFCoRAZGRm0t7fz7t07UlNTKS0tpbm5OTqKpaOjgxMnTjA+Ps6WLVt48OABa9asAaC0tJQ7d+5w4cIFQqEQmZmZtLW1cfTo0eh/XL16lebmZo4fP87k5CQbNmygubl5Ufni4+OZnJyktraWT58+kZ6eTmVlJRcvXvztZyFJkiRp5Yqbn5+fX+4QkiRJ+vP19fURDAaZmpoiNTV1ueNIkiRJ0pJxxrokSZIkSZIkSTGwWJckSZIkSZIkKQaOgpEkSZIkSZIkKQZ+sS5JkiRJkiRJUgws1iVJkiRJkiRJioHFuiRJkiRJkiRJMbBYlyRJkiRJkiQpBhbrkiRJkiRJkiTFwGJdkiRJkiRJkqQYWKxLkiRJkiRJkhQDi3VJkiRJkiRJkmJgsS5JkiRJkiRJUgz+AvWLeSrdpqGNAAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 1500x1000 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 24/24 [00:10<00:00,  2.34it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:56:11,419 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n",
            "100%|██████████| 3/3 [00:01<00:00,  2.24it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:56:15,864 Evaluating as a multi-label problem: False\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n",
            "100%|██████████| 3/3 [00:01<00:00,  2.36it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-21 17:56:17,232 Evaluating as a multi-label problem: True\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-0f52dfd3-0601-4472-907c-d27c4158a3dd\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>split</th>\n",
              "      <th>index</th>\n",
              "      <th>I-ADR</th>\n",
              "      <th>B-ADR</th>\n",
              "      <th>B-Drug</th>\n",
              "      <th>B-Symptom</th>\n",
              "      <th>B-Finding</th>\n",
              "      <th>I-Finding</th>\n",
              "      <th>B-Disease</th>\n",
              "      <th>I-Drug</th>\n",
              "      <th>I-Symptom</th>\n",
              "      <th>I-Disease</th>\n",
              "      <th>micro avg</th>\n",
              "      <th>macro avg</th>\n",
              "      <th>weighted avg</th>\n",
              "      <th>samples avg</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>train</td>\n",
              "      <td>precision</td>\n",
              "      <td>0.799583</td>\n",
              "      <td>0.811342</td>\n",
              "      <td>0.931601</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.389189</td>\n",
              "      <td>0.347222</td>\n",
              "      <td>0.407407</td>\n",
              "      <td>0.957143</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.666667</td>\n",
              "      <td>0.780802</td>\n",
              "      <td>0.531015</td>\n",
              "      <td>0.768763</td>\n",
              "      <td>0.138356</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>train</td>\n",
              "      <td>recall</td>\n",
              "      <td>0.915504</td>\n",
              "      <td>0.910866</td>\n",
              "      <td>0.975645</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.233010</td>\n",
              "      <td>0.071633</td>\n",
              "      <td>0.047619</td>\n",
              "      <td>0.416149</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.011834</td>\n",
              "      <td>0.834097</td>\n",
              "      <td>0.358226</td>\n",
              "      <td>0.834097</td>\n",
              "      <td>0.138324</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>train</td>\n",
              "      <td>f1-score</td>\n",
              "      <td>0.853626</td>\n",
              "      <td>0.858228</td>\n",
              "      <td>0.953114</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.291498</td>\n",
              "      <td>0.118765</td>\n",
              "      <td>0.085271</td>\n",
              "      <td>0.580087</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.023256</td>\n",
              "      <td>0.806570</td>\n",
              "      <td>0.376384</td>\n",
              "      <td>0.789564</td>\n",
              "      <td>0.138335</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>train</td>\n",
              "      <td>support</td>\n",
              "      <td>7953.000000</td>\n",
              "      <td>4712.000000</td>\n",
              "      <td>1396.000000</td>\n",
              "      <td>222.0</td>\n",
              "      <td>309.000000</td>\n",
              "      <td>349.000000</td>\n",
              "      <td>231.000000</td>\n",
              "      <td>161.000000</td>\n",
              "      <td>218.0</td>\n",
              "      <td>169.000000</td>\n",
              "      <td>15720.000000</td>\n",
              "      <td>15720.000000</td>\n",
              "      <td>15720.000000</td>\n",
              "      <td>15720.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>dev</td>\n",
              "      <td>precision</td>\n",
              "      <td>0.668464</td>\n",
              "      <td>0.747956</td>\n",
              "      <td>0.916667</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.466667</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>0.375000</td>\n",
              "      <td>0.785714</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.679946</td>\n",
              "      <td>0.446047</td>\n",
              "      <td>0.673295</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>dev</td>\n",
              "      <td>recall</td>\n",
              "      <td>0.770186</td>\n",
              "      <td>0.843318</td>\n",
              "      <td>0.956522</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.218750</td>\n",
              "      <td>0.092593</td>\n",
              "      <td>0.096774</td>\n",
              "      <td>0.407407</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.727009</td>\n",
              "      <td>0.338555</td>\n",
              "      <td>0.727009</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>dev</td>\n",
              "      <td>f1-score</td>\n",
              "      <td>0.715729</td>\n",
              "      <td>0.792780</td>\n",
              "      <td>0.936170</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.297872</td>\n",
              "      <td>0.156250</td>\n",
              "      <td>0.153846</td>\n",
              "      <td>0.536585</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.702690</td>\n",
              "      <td>0.358923</td>\n",
              "      <td>0.690468</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>dev</td>\n",
              "      <td>support</td>\n",
              "      <td>966.000000</td>\n",
              "      <td>651.000000</td>\n",
              "      <td>184.000000</td>\n",
              "      <td>25.0</td>\n",
              "      <td>64.000000</td>\n",
              "      <td>54.000000</td>\n",
              "      <td>31.000000</td>\n",
              "      <td>27.000000</td>\n",
              "      <td>36.0</td>\n",
              "      <td>28.000000</td>\n",
              "      <td>2066.000000</td>\n",
              "      <td>2066.000000</td>\n",
              "      <td>2066.000000</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>test</td>\n",
              "      <td>precision</td>\n",
              "      <td>0.713499</td>\n",
              "      <td>0.747678</td>\n",
              "      <td>0.886555</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.318182</td>\n",
              "      <td>0.333333</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>0.900000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.740247</td>\n",
              "      <td>0.439925</td>\n",
              "      <td>0.703930</td>\n",
              "      <td>0.126669</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>test</td>\n",
              "      <td>recall</td>\n",
              "      <td>0.803516</td>\n",
              "      <td>0.838542</td>\n",
              "      <td>0.963470</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.134615</td>\n",
              "      <td>0.055556</td>\n",
              "      <td>0.050000</td>\n",
              "      <td>0.346154</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.753645</td>\n",
              "      <td>0.319185</td>\n",
              "      <td>0.753645</td>\n",
              "      <td>0.126584</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>test</td>\n",
              "      <td>f1-score</td>\n",
              "      <td>0.755837</td>\n",
              "      <td>0.790507</td>\n",
              "      <td>0.923414</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.189189</td>\n",
              "      <td>0.095238</td>\n",
              "      <td>0.090909</td>\n",
              "      <td>0.500000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.746886</td>\n",
              "      <td>0.334509</td>\n",
              "      <td>0.718722</td>\n",
              "      <td>0.126613</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>test</td>\n",
              "      <td>support</td>\n",
              "      <td>967.000000</td>\n",
              "      <td>576.000000</td>\n",
              "      <td>219.000000</td>\n",
              "      <td>22.0</td>\n",
              "      <td>52.000000</td>\n",
              "      <td>36.000000</td>\n",
              "      <td>20.000000</td>\n",
              "      <td>52.000000</td>\n",
              "      <td>33.0</td>\n",
              "      <td>12.000000</td>\n",
              "      <td>1989.000000</td>\n",
              "      <td>1989.000000</td>\n",
              "      <td>1989.000000</td>\n",
              "      <td>1989.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0f52dfd3-0601-4472-907c-d27c4158a3dd')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-0f52dfd3-0601-4472-907c-d27c4158a3dd button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-0f52dfd3-0601-4472-907c-d27c4158a3dd');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "    split      index        I-ADR        B-ADR       B-Drug  B-Symptom  \\\n",
              "0   train  precision     0.799583     0.811342     0.931601        0.0   \n",
              "1   train     recall     0.915504     0.910866     0.975645        0.0   \n",
              "2   train   f1-score     0.853626     0.858228     0.953114        0.0   \n",
              "3   train    support  7953.000000  4712.000000  1396.000000      222.0   \n",
              "4     dev  precision     0.668464     0.747956     0.916667        0.0   \n",
              "5     dev     recall     0.770186     0.843318     0.956522        0.0   \n",
              "6     dev   f1-score     0.715729     0.792780     0.936170        0.0   \n",
              "7     dev    support   966.000000   651.000000   184.000000       25.0   \n",
              "8    test  precision     0.713499     0.747678     0.886555        0.0   \n",
              "9    test     recall     0.803516     0.838542     0.963470        0.0   \n",
              "10   test   f1-score     0.755837     0.790507     0.923414        0.0   \n",
              "11   test    support   967.000000   576.000000   219.000000       22.0   \n",
              "\n",
              "     B-Finding   I-Finding   B-Disease      I-Drug  I-Symptom   I-Disease  \\\n",
              "0     0.389189    0.347222    0.407407    0.957143        0.0    0.666667   \n",
              "1     0.233010    0.071633    0.047619    0.416149        0.0    0.011834   \n",
              "2     0.291498    0.118765    0.085271    0.580087        0.0    0.023256   \n",
              "3   309.000000  349.000000  231.000000  161.000000      218.0  169.000000   \n",
              "4     0.466667    0.500000    0.375000    0.785714        0.0    0.000000   \n",
              "5     0.218750    0.092593    0.096774    0.407407        0.0    0.000000   \n",
              "6     0.297872    0.156250    0.153846    0.536585        0.0    0.000000   \n",
              "7    64.000000   54.000000   31.000000   27.000000       36.0   28.000000   \n",
              "8     0.318182    0.333333    0.500000    0.900000        0.0    0.000000   \n",
              "9     0.134615    0.055556    0.050000    0.346154        0.0    0.000000   \n",
              "10    0.189189    0.095238    0.090909    0.500000        0.0    0.000000   \n",
              "11   52.000000   36.000000   20.000000   52.000000       33.0   12.000000   \n",
              "\n",
              "       micro avg     macro avg  weighted avg   samples avg  \n",
              "0       0.780802      0.531015      0.768763      0.138356  \n",
              "1       0.834097      0.358226      0.834097      0.138324  \n",
              "2       0.806570      0.376384      0.789564      0.138335  \n",
              "3   15720.000000  15720.000000  15720.000000  15720.000000  \n",
              "4       0.679946      0.446047      0.673295           NaN  \n",
              "5       0.727009      0.338555      0.727009           NaN  \n",
              "6       0.702690      0.358923      0.690468           NaN  \n",
              "7    2066.000000   2066.000000   2066.000000           NaN  \n",
              "8       0.740247      0.439925      0.703930      0.126669  \n",
              "9       0.753645      0.319185      0.753645      0.126584  \n",
              "10      0.746886      0.334509      0.718722      0.126613  \n",
              "11   1989.000000   1989.000000   1989.000000   1989.000000  "
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "print(suf)\n",
        "\n",
        "os.mkdir('flair_ner_model'+suf+'/classification_report')\n",
        "plotter.plot_training_curves('flair_ner_model'+suf+'/loss.tsv')\n",
        "\n",
        "# evaluate the tagger on the test set\n",
        "result = tagger.evaluate(corpus.train, \n",
        "                         embeddings_storage_mode='gpu',\n",
        "                         mini_batch_size=40,\n",
        "                         main_evaluation_metric = (\"weighted avg\", \"f1-score\"),\n",
        "                         gold_label_type='ner',\n",
        "                         gold_label_dictionary=tag_dictionary)\n",
        "open('flair_ner_model'+suf+'/classification_report/train.txt', 'w').write(str(result))\n",
        "res1 = pd.DataFrame.from_dict(result.classification_report).reset_index()\n",
        "res1.insert(loc=0, column='split', value='train')\n",
        "\n",
        "result = tagger.evaluate(corpus.dev, \n",
        "                         embeddings_storage_mode='gpu',\n",
        "                         mini_batch_size=40,\n",
        "                         main_evaluation_metric = (\"weighted avg\", \"f1-score\"),\n",
        "                         gold_label_type='ner',\n",
        "                         gold_label_dictionary=tag_dictionary)\n",
        "open('flair_ner_model'+suf+'/classification_report/dev.txt', 'w').write(str(result))\n",
        "res2 = pd.DataFrame.from_dict(result.classification_report).reset_index()\n",
        "res2.insert(loc=0, column='split', value='dev')\n",
        "\n",
        "result = tagger.evaluate(corpus.test, \n",
        "                         embeddings_storage_mode='gpu',\n",
        "                         mini_batch_size=40,\n",
        "                         main_evaluation_metric = (\"weighted avg\", \"f1-score\"),\n",
        "                         gold_label_type='ner',\n",
        "                         gold_label_dictionary=tag_dictionary)\n",
        "open('flair_ner_model'+suf+'/classification_report/test.txt', 'w').write(str(result))\n",
        "res3 = pd.DataFrame.from_dict(result.classification_report).reset_index()\n",
        "res3.insert(loc=0, column='split', value='test')\n",
        "\n",
        "res = pd.concat([res1,res2,res3],ignore_index=True)\n",
        "res.to_csv('flair_ner_model'+suf+'/classification_report.csv',index=False)\n",
        "res"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "VvIDCtYJkY-G"
      },
      "outputs": [],
      "source": [
        "# import time\n",
        "# start_time = time.time()\n",
        "# while time.time() - start_time < 1:\n",
        "#     time.sleep(1)\n",
        "# print('Session has timed out (2 minutes). Disconnecting...')\n",
        "# !kill -9 -1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tGILL4RFtGil"
      },
      "source": [
        "Layering can be a good idea if the NER tags in the first layer have their own set of dependent values in the second layer. This can help capture more information about the entities being recognized and provide more context for downstream tasks. However, it also increases the complexity of the annotation and the training process, so it should be used judiciously. In general, it's a good idea to start with a simple tag scheme and gradually increase complexity as needed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JoZ5D6aEtGin"
      },
      "source": [
        "    does it really matter if u use the E header\n",
        "\n",
        "It depends on the specific task and the requirements of the downstream model. In some cases, using the E tag can help disambiguate between overlapping entities, while in other cases it may not be necessary or even desired. Ultimately, it's up to the developer to determine the appropriate tagging scheme for their specific use case."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BGt2RhM6tGin"
      },
      "source": [
        "    would adding POS as an extra NER layer help model performance or not\n",
        "\n",
        "Adding POS as an extra NER layer may or may not help model performance, depending on the specific use case and data being analyzed.\n",
        "\n",
        "On one hand, incorporating POS tags could provide additional context to the model and help it better distinguish between named entities and other parts of speech. For example, knowing that a word is a noun or verb could help the model determine if it is likely to be a named entity or not.\n",
        "\n",
        "On the other hand, adding an additional layer could also increase model complexity and training time without necessarily improving performance. It may also require additional manual annotation efforts to create a labeled dataset with both NER and POS tags.\n",
        "\n",
        "Ultimately, whether to include POS as an additional layer will depend on the specific requirements and goals of the project, as well as the available data and resources."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6zpEn1bgtGin"
      },
      "source": [
        "    besides bioes ordering (which is up-to-down), it doesn't matter what order the tags in the same domain appear in left-to-right?\n",
        "\n",
        "Yes, that is correct. As long as the BIOES order is maintained vertically, the order of tags in the same domain from left-to-right should not impact training in flai"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yrmL98VatGin"
      },
      "source": [
        "    different typenames must be treated as separate tasks?\n",
        "\n",
        "Yes, that's correct. Different typenames should be treated as separate tasks because they represent different types of annotations that you want to predict. For example, in a biomedical NER task, you might want to predict the entity types of both drugs and diseases, which would be represented by two different typenames in your annotation data. Predicting both types of entities would require different models and possibly different evaluation metrics."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "A100",
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "flair",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.16"
    },
    "vscode": {
      "interpreter": {
        "hash": "381c8abc3392c5f20473858f182d9ac4a53ef053fb1f276fcb4936938616c3f3"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}