dataperf_speech_colab.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "private_outputs": true,
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "This notebook works through running the [DataPerf Speech](https://www.dataperf.org/training-set-selection-speech) challenge evaluation with a [baseline selection algorithm](https://github.com/harvard-edge/dataperf-speech-example/blob/main/selection/implementations/baseline_selection.py).\n",
        "\n",
        "We start by cloning our example selection algorithm repository and installing some additional dependencies not preinstalled in Colab environments:"
      ],
      "metadata": {
        "id": "E5uSni11iDCb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q fire wget\n",
        "!git clone https://github.com/harvard-edge/dataperf-speech-example/\n",
        "import sys\n",
        "sys.path.append(\"/content/dataperf-speech-example/\")\n",
        "import os\n",
        "os.chdir(\"/content/dataperf-speech-example/\")"
      ],
      "metadata": {
        "id": "LErPly0cbAsO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, we download the spoken word embeddings which we will use for training coreset selection and evaluation."
      ],
      "metadata": {
        "id": "aSzmJSd-liEX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python utils/download_data.py --output_path workspace/data 1> /dev/null"
      ],
      "metadata": {
        "id": "TiJKEor_fo4k"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below, we generate a set of 25 training samples from the available embeddings for each language, using our default selection algorithm (which simply performs crossfold-validation). The evaluation strategy can be changed by editing `dataperf-speech-example/workspace/dataperf_speech_config.yaml` \n",
        "\n",
        "The goal of this challenge is to add your own selection algorithm and outperform the provided baselines' macro F1 scores.\n",
        "\n",
        "The selection algorithm will output a training file for each language, `en_25_train.json`, `id_25_train.json`, and `pt_25_train.json`.\n",
        "\n",
        "These are the files you would upload to Dynabench for official evaluation, but in the next cell, we will run local unofficial evaluation using our provided evaluation data."
      ],
      "metadata": {
        "id": "wXWqcH_PfH_f"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "TRAIN_SIZE = 25 # or 60\n",
        "for lang in [\"en\", \"id\", \"pt\"]:\n",
        "  !python -m selection.main \\\n",
        "     --language \"{lang}\" \\\n",
        "     --allowed_training_set \"workspace/data/dataperf_{lang}_data/allowed_training_set.yaml\" \\\n",
        "     --train_embeddings_dir \"workspace/data//dataperf_{lang}_data/train_embeddings/\" \\\n",
        "     --train_size {TRAIN_SIZE} \\\n",
        "     --outdir \"/content/\""
      ],
      "metadata": {
        "id": "hWbNznXKdic7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally, let's run a local unofficial evaluation on the results of the training set selection algorithm (`en_25_train.json`, `id_25_train.json`, and `pt_25_train.json`). \n",
        "\n",
        "For each language, we load the coreset training samples specified in the  JSON file, along with evaluation samples specified in `eval.yaml`. We then train an ensemble classifier and [average the macro F1 score across ten random seeds](https://github.com/harvard-edge/dataperf-speech-example/blob/main/eval.py#L139-L154), and display the score (which should match the scores on the DynaBench leaderboard for the coreset sizes of 25 and 60). \n",
        "\n",
        "Here is the expected output for English with a coreset size of 25, using the input of `en_25_train.json` produced by the previous cell:\n",
        "\n",
        "```\n",
        "validating selected IDs\n",
        "loading selected training data\n",
        "Loading targets: 100% 5/5 [00:00<00:00, 17.97it/s]\n",
        "Loading nontargets: 100% 9/9 [00:00<00:00, 140.54it/s]\n",
        "loading eval data\n",
        "Loading targets: 100% 5/5 [00:00<00:00, 119.50it/s]\n",
        "Loading nontargets: 100% 200/200 [00:12<00:00, 16.11it/s]\n",
        "\n",
        "Score:  0.3524448610675314\n",
        "```\n"
      ],
      "metadata": {
        "id": "s9SoqEJGg-u_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for lang in [\"en\", \"id\", \"pt\"]:\n",
        "  !python eval.py \\\n",
        "    --language \"{lang}\" \\\n",
        "    --eval_embeddings_dir \"workspace/data/dataperf_{lang}_data/eval_embeddings/\" \\\n",
        "    --train_embeddings_dir \"workspace/data/dataperf_{lang}_data/train_embeddings/\" \\\n",
        "    --allowed_training_set \"workspace/data/dataperf_{lang}_data/allowed_training_set.yaml\" \\\n",
        "    --eval_file \"workspace/data/dataperf_{lang}_data/eval.yaml\" \\\n",
        "    --train_file \"/content/{lang}_{TRAIN_SIZE}_train.json\" \\\n",
        "    --train_size {TRAIN_SIZE} \\\n",
        "    --config_file workspace/dataperf_speech_config.yaml"
      ],
      "metadata": {
        "id": "6iwATQs1fGXR"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}