02: BERT_Word_Embeddings_v2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "02: BERT_Word_Embeddings_v2.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "6133fab744974b3d809204612d74f0a2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_1443ef86cd3c4ed3941bbe17087ad0df",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_5f8d79903842488692be36eb54426eac",
              "IPY_MODEL_dfdb93781b0549fd95b33fb33e5903fd"
            ]
          }
        },
        "1443ef86cd3c4ed3941bbe17087ad0df": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "5f8d79903842488692be36eb54426eac": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_844011418db34ee7bbc95c8e8be54015",
            "_dom_classes": [],
            "description": "Downloading: 100%",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 231508,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 231508,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_312cf05fc2bc4632ae55cd14d7c5f4d0"
          }
        },
        "dfdb93781b0549fd95b33fb33e5903fd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_cc54387464654666bf0de8b8c59c4964",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 232k/232k [00:00&lt;00:00, 615kB/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_7a6cee8ec01a44338329f7143740c6de"
          }
        },
        "844011418db34ee7bbc95c8e8be54015": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "initial",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "312cf05fc2bc4632ae55cd14d7c5f4d0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "cc54387464654666bf0de8b8c59c4964": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "7a6cee8ec01a44338329f7143740c6de": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "6e2ae067ffc04984b442f95b5f0441f6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_5c76f0ea92dd4a39817b68dfa9aaf0a8",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_1edbae9631684961ad77d6b32e332293",
              "IPY_MODEL_ab6057a7bb08466a883314cdcb7a542d"
            ]
          }
        },
        "5c76f0ea92dd4a39817b68dfa9aaf0a8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "1edbae9631684961ad77d6b32e332293": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_3381d947e661406fb26ec95e7974d68e",
            "_dom_classes": [],
            "description": "Downloading: 100%",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 433,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 433,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_af57a5f70f574bf1a2a1c5a01557732e"
          }
        },
        "ab6057a7bb08466a883314cdcb7a542d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_3557cddbb2a74bd9b6d39332c3adebf1",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 433/433 [00:09&lt;00:00, 44.2B/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_0028d4de549347deb60c177654162f5d"
          }
        },
        "3381d947e661406fb26ec95e7974d68e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "initial",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "af57a5f70f574bf1a2a1c5a01557732e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "3557cddbb2a74bd9b6d39332c3adebf1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "0028d4de549347deb60c177654162f5d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "412e1f72fe1f48dfa7e25db50a0514b6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_f30dc9ad3a3a4778971d4c464b848603",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_9f7723b2217447309eee0fae5e6cc903",
              "IPY_MODEL_7dda77ecb5a84aeea9fb802aade55193"
            ]
          }
        },
        "f30dc9ad3a3a4778971d4c464b848603": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "9f7723b2217447309eee0fae5e6cc903": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_ed39302ee90f43cfa3905abf3dac1666",
            "_dom_classes": [],
            "description": "Downloading: 100%",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 440473133,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 440473133,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_5147c6c0f0d746adba86e83cd62e95e1"
          }
        },
        "7dda77ecb5a84aeea9fb802aade55193": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_e005cd31461847ef8c955ad8b079a80c",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 440M/440M [00:09&lt;00:00, 47.4MB/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_ab5988e6a3814895b6b750039c34bbd4"
          }
        },
        "ed39302ee90f43cfa3905abf3dac1666": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "initial",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "5147c6c0f0d746adba86e83cd62e95e1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "e005cd31461847ef8c955ad8b079a80c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "ab5988e6a3814895b6b750039c34bbd4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        }
      }
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/gulabpatel/NLP_BERT/blob/main/02%3A%20BERT_Word_Embeddings_v2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "78HE8FLsKN9Q"
      },
      "source": [
        "In this post, I take an in-depth look at word embeddings produced by Google's BERT and show you how to get started with BERT by producing your own word embeddings.\n",
        "\n",
        "This post is presented in two forms--as a blog post [here](http://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/) and as a Colab notebook [here](https://colab.research.google.com/drive/1yFphU6PW9Uo6lmDly_ud9a6c4RCYlwdX). \n",
        "The content is identical in both, but: \n",
        "\n",
        "* The blog post format may be easier to read, and includes a comments section for discussion. \n",
        "* The Colab Notebook will allow you to run the code and inspect it as you read through.\n",
        "\n",
        "*UPDATE: May 27, 2020 - I've updated this post to use the new `transformers` library  from huggingface in place of the old `pytorch-pretrained-bert` library. You can still find the old post / Notebook [here](https://colab.research.google.com/drive/1ZQvuAVwA3IjybezQOXnrXMGAnMyZRuPU) if you need it.* \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dYapTjoYa0kO"
      },
      "source": [
        "# Introduction\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c8HDKzBai5dL"
      },
      "source": [
        "### History\n",
        "\n",
        "2018 was a breakthrough year in NLP. Transfer learning, particularly models like Allen AI's ELMO, OpenAI's Open-GPT, and Google's BERT allowed researchers to smash multiple benchmarks with minimal task-specific fine-tuning and provided the rest of the NLP community with pretrained models that could easily (with less data and less compute time) be fine-tuned and implemented to produce state of the art results. Unfortunately, for many starting out in NLP and even for some experienced practicioners, the theory and practical application of these powerful models is still not well understood.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WoitNQMWA1bt"
      },
      "source": [
        "\n",
        "### What is BERT?\n",
        "\n",
        "BERT (Bidirectional Encoder Representations from Transformers), released in late 2018, is the model we will use in this tutorial to provide readers with a better understanding of and practical guidance for using transfer learning models in NLP. BERT is a method of pretraining language representations that was used to create models that NLP practicioners can then download and use for free. You can either use these models to extract high quality language features from your text data, or you can fine-tune these models on a specific task (classification, entity recognition, question answering, etc.) with your own data to produce state of the art predictions.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q-dDVmXAA3At"
      },
      "source": [
        "\n",
        "### Why BERT embeddings?\n",
        "\n",
        "In this tutorial, we will use BERT to extract features, namely word and sentence embedding vectors, from text data. What can we do with these word and sentence embedding vectors? First, these embeddings are useful for keyword/search expansion, semantic search and information retrieval. For example, if you want to match customer questions or searches against already answered questions or well documented searches, these representations will help you accuratley retrieve results matching the customer's intent and contextual meaning, even if there's no keyword  or phrase overlap.\n",
        "\n",
        "Second, and perhaps more importantly, these vectors are used as high-quality feature inputs to downstream models. NLP models such as LSTMs or CNNs require inputs in the form of numerical vectors, and this typically means translating features like the vocabulary and parts of speech into numerical representations. In the past, words have been represented either as uniquely indexed values (one-hot encoding), or more helpfully as neural word embeddings where vocabulary words are matched against the fixed-length feature embeddings that result from models like Word2Vec or Fasttext. BERT offers an advantage over models like Word2Vec, because while each word has a fixed representation under Word2Vec regardless of the context within which the word appears, BERT produces word representations that are dynamically informed by the words around them. For example, given two sentences:\n",
        "\n",
        "\"The man was accused of robbing a bank.\"\n",
        "\"The man went fishing by the bank of the river.\"\n",
        "\n",
        "Word2Vec would produce the same word embedding for the word \"bank\" in both sentences, while under BERT the word embedding for \"bank\" would be different for each sentence. Aside from capturing obvious differences like polysemy, the context-informed word embeddings capture other forms of information that result in more accurate feature representations, which in turn results in better model performance.\n",
        "\n",
        "From an educational standpoint, a close examination of BERT word embeddings is a good way to get  your feet wet with BERT and its family of transfer learning models, and sets us up with some practical knowledge and context to better understand the inner details of the model in later tutorials.\n",
        "\n",
        "Onward!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lkfoKeUSsgEH"
      },
      "source": [
        "[![BERT eBook Display Ad](http://www.mccormickml.com/assets/BERT/BERT_eBook_Display_Ad.png)](https://bit.ly/2F1zTXB)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pqa-7WXBAw8q"
      },
      "source": [
        "# 1. Loading Pre-Trained BERT"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eCdqJCtQN52l"
      },
      "source": [
        "Install the pytorch interface for BERT by Hugging Face. (This library contains interfaces for other pretrained language models like OpenAI's GPT and GPT-2.) \n",
        "\n",
        "We've selected the pytorch interface because it strikes a nice balance between the high-level APIs (which are easy to use but don't provide insight into how things work) and tensorflow code (which contains lots of details but often sidetracks us into lessons about tensorflow, when the purpose here is BERT!).\n",
        "\n",
        "If you're running this code on Google Colab, you will have to install this library each time you reconnect; the following cell will take care of that for you."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1RfUN_KolV-f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 615
        },
        "outputId": "ec4a6f38-54e4-4a13-b90f-2a7f9d4238a8"
      },
      "source": [
        "!pip install transformers"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting transformers\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/19/22/aff234f4a841f8999e68a7a94bdd4b60b4cebcfeca5d67d61cd08c9179de/transformers-3.3.1-py3-none-any.whl (1.1MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1MB 8.1MB/s \n",
            "\u001b[?25hCollecting sentencepiece!=0.1.92\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d4/a4/d0a884c4300004a78cca907a6ff9a5e9fe4f090f5d95ab341c53d28cbc58/sentencepiece-0.1.91-cp36-cp36m-manylinux1_x86_64.whl (1.1MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1MB 29.6MB/s \n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers) (3.0.12)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers) (2019.12.20)\n",
            "Requirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from transformers) (0.7)\n",
            "Collecting sacremoses\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz (883kB)\n",
            "\u001b[K     |████████████████████████████████| 890kB 53.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from transformers) (20.4)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers) (2.23.0)\n",
            "Collecting tokenizers==0.8.1.rc2\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/80/83/8b9fccb9e48eeb575ee19179e2bdde0ee9a1904f97de5f02d19016b8804f/tokenizers-0.8.1rc2-cp36-cp36m-manylinux1_x86_64.whl (3.0MB)\n",
            "\u001b[K     |████████████████████████████████| 3.0MB 51.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers) (4.41.1)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers) (1.18.5)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (1.15.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (0.16.0)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers) (2.4.7)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2020.6.20)\n",
            "Building wheels for collected packages: sacremoses\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893257 sha256=9d2d937512fdcc73d6285ccb13a4975364a8fdbfc9cb9c174e405f4779da8875\n",
            "  Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45\n",
            "Successfully built sacremoses\n",
            "Installing collected packages: sentencepiece, sacremoses, tokenizers, transformers\n",
            "Successfully installed sacremoses-0.0.43 sentencepiece-0.1.91 tokenizers-0.8.1rc2 transformers-3.3.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JSXImOxMPdNg"
      },
      "source": [
        "Now let's import pytorch, the pretrained BERT model, and a BERT tokenizer. \n",
        "\n",
        "We'll explain the BERT model in detail in a later tutorial, but this is the pre-trained model released by Google that ran for many, many hours on Wikipedia and [Book Corpus](https://arxiv.org/pdf/1506.06724.pdf), a dataset containing +10,000 books of different genres. This model is responsible (with a little modification) for beating NLP benchmarks across a range of tasks. Google released a few variations of BERT models, but the one we'll use here is the smaller of the two available sizes (\"base\" and \"large\") and ignores casing, hence \"uncased.\"\"\n",
        "\n",
        "`transformers` provides a number of classes for applying BERT to different tasks (token classification, text classification, ...). Here, we're using the basic `BertModel` which has no specific output task--it's a good choice for using BERT just to extract embeddings."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lJEnBJ3gHTsQ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 66,
          "referenced_widgets": [
            "6133fab744974b3d809204612d74f0a2",
            "1443ef86cd3c4ed3941bbe17087ad0df",
            "5f8d79903842488692be36eb54426eac",
            "dfdb93781b0549fd95b33fb33e5903fd",
            "844011418db34ee7bbc95c8e8be54015",
            "312cf05fc2bc4632ae55cd14d7c5f4d0",
            "cc54387464654666bf0de8b8c59c4964",
            "7a6cee8ec01a44338329f7143740c6de"
          ]
        },
        "outputId": "7f2fd7f6-5480-4d33-f82a-0498dd58863d"
      },
      "source": [
        "import torch\n",
        "from transformers import BertTokenizer, BertModel\n",
        "\n",
        "# OPTIONAL: if you want to have more information on what's happening, activate the logger as follows\n",
        "import logging\n",
        "#logging.basicConfig(level=logging.INFO)\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "% matplotlib inline\n",
        "\n",
        "# Load pre-trained model tokenizer (vocabulary)\n",
        "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6133fab744974b3d809204612d74f0a2",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=231508.0, style=ProgressStyle(descripti…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tlv3VlPnKKHN"
      },
      "source": [
        "# 2. Input Formatting\n",
        "Because BERT is a pretrained model that expects input data in a specific format, we will need:\n",
        "\n",
        "1. A **special token, `[SEP]`,** to mark the end of a sentence, or the separation between two sentences\n",
        "2. A **special token, `[CLS]`,** at the beginning of our text. This token is used for classification tasks, but BERT expects it no matter what your application is. \n",
        "3. Tokens that conform with the fixed vocabulary used in BERT\n",
        "4. The **Token IDs** for the tokens, from BERT's tokenizer\n",
        "5. **Mask IDs** to indicate which elements in the sequence are tokens and which are padding elements\n",
        "6. **Segment IDs** used to distinguish different sentences\n",
        "7. **Positional Embeddings** used to show token position within the sequence\n",
        "\n",
        "Luckily, the `transformers` interface takes care of all of the above requirements (using the `tokenizer.encode_plus` function). \n",
        "\n",
        "Since this is intended as an introduction to working with BERT, though, we're going to perform these steps in a (mostly) manual way. \n",
        "\n",
        "> *For an example of using `tokenizer.encode_plus`, see the next post on Sentence Classification [here](http://mccormickml.com/2019/07/22/BERT-fine-tuning/).*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "diVtyCJCurxJ"
      },
      "source": [
        "## 2.1. Special Tokens\n",
        "BERT can take as input either one or two sentences, and uses the special token `[SEP]` to differentiate them. The `[CLS]` token always appears at the start of the text, and is specific to classification tasks. \n",
        "\n",
        "Both tokens are *always required*, however, even if we only have one sentence, and even if we are not using BERT for classification. That's how BERT was pre-trained, and so that's what BERT expects to see.\n",
        "\n",
        "**2 Sentence Input**:\n",
        "\n",
        "`[CLS] The man went to the store. [SEP] He bought a gallon of milk.`\n",
        "\n",
        "**1 Sentence Input**:\n",
        "\n",
        "`[CLS] The man went to the store. [SEP]`\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3gsyrAwYvBfC"
      },
      "source": [
        "## 2.2. Tokenization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2WafgQPLAWmo"
      },
      "source": [
        "BERT provides its own tokenizer, which we imported above. Let's see how it handles the below sentence."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pg0P9rFxJwwp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "c32fde04-3fb2-4a9b-ec6d-10952ddfbb25"
      },
      "source": [
        "text = \"Here is the sentence I want embeddings for.\"\n",
        "marked_text = \"[CLS] \" + text + \" [SEP]\"\n",
        "\n",
        "# Tokenize our sentence with the BERT tokenizer.\n",
        "tokenized_text = tokenizer.tokenize(marked_text)\n",
        "\n",
        "# Print out the tokens.\n",
        "print (tokenized_text)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['[CLS]', 'here', 'is', 'the', 'sentence', 'i', 'want', 'em', '##bed', '##ding', '##s', 'for', '.', '[SEP]']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q51eN4KAkbIJ"
      },
      "source": [
        "Notice how the word \"embeddings\" is represented:\n",
        "\n",
        "`['em', '##bed', '##ding', '##s']`\n",
        "\n",
        "The original word has been split into smaller subwords and characters. The two hash signs preceding some of these subwords are just our tokenizer's way to denote that this subword or character is part of a larger word and preceded by another subword. So, for example, the '##bed' token is separate from the 'bed' token; the first is used whenever the subword 'bed' occurs within a larger word and the second is used explicitly for when the standalone token 'thing you sleep on' occurs.\n",
        "\n",
        "Why does it look this way? This is because the BERT tokenizer was created with a WordPiece model. This model greedily creates a fixed-size vocabulary of individual characters, subwords, and words that best fits our language data. Since the vocabulary limit size of our BERT tokenizer model is 30,000, the WordPiece model generated a vocabulary that contains all English characters plus the ~30,000 most common words and subwords found in the English language corpus the model is trained on. This vocabulary contains four things:\n",
        "\n",
        "1. Whole words\n",
        "2. Subwords occuring at the front of a word or in isolation (\"em\" as in \"embeddings\" is assigned the same vector as the standalone sequence of characters \"em\" as in \"go get em\" )\n",
        "3. Subwords not at the front of a word, which are preceded by '##' to denote this case\n",
        "4. Individual characters\n",
        "\n",
        "To tokenize a word under this model, the tokenizer first checks if the whole word is in the vocabulary. If not, it tries to break the word into the largest possible subwords contained in the vocabulary, and as a last resort will decompose the word into individual characters. Note that because of this, we can always represent a word as, at the very least, the collection of its individual characters.\n",
        "\n",
        "As a result, rather than assigning out of vocabulary words to a catch-all token like 'OOV' or 'UNK,' words that are not in the vocabulary are decomposed into subword and character tokens that we can then generate embeddings for. \n",
        "\n",
        "So, rather than assigning \"embeddings\" and every other out of vocabulary word to an overloaded unknown vocabulary token, we split it into subword tokens ['em', '##bed', '##ding', '##s'] that will retain some of the contextual meaning of the original word.  We can even average these subword embedding vectors to generate an approximate vector for the original word.\n",
        "\n",
        "\n",
        "(For more information about WordPiece, see the [original paper](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37842.pdf) and further disucssion in Google's [Neural Machine Translation System](https://arxiv.org/pdf/1609.08144.pdf).)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jp5zXAPBVp82"
      },
      "source": [
        "Here are some examples of the tokens contained in our vocabulary. Tokens beginning with two hashes are subwords or individual characters.\n",
        "\n",
        "*For an exploration of the contents of BERT's vocabulary, see [this notebook](https://colab.research.google.com/drive/1fCKIBJ6fgWQ-f6UKs7wDTpNTL9N-Cq9X) I created and the accompanying YouTube video [here](https://youtu.be/zJW57aCBCTk).*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1z1SzuTrqx-7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 357
        },
        "outputId": "474de216-dd41-4d4f-f867-10b633d3d400"
      },
      "source": [
        "list(tokenizer.vocab.keys())[5000:5020]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['knight',\n",
              " 'lap',\n",
              " 'survey',\n",
              " 'ma',\n",
              " '##ow',\n",
              " 'noise',\n",
              " 'billy',\n",
              " '##ium',\n",
              " 'shooting',\n",
              " 'guide',\n",
              " 'bedroom',\n",
              " 'priest',\n",
              " 'resistance',\n",
              " 'motor',\n",
              " 'homes',\n",
              " 'sounded',\n",
              " 'giant',\n",
              " '##mer',\n",
              " '150',\n",
              " 'scenes']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HoF3LC47VgBb"
      },
      "source": [
        "After breaking the text into tokens, we then have to convert the sentence from a list of strings to a list of vocabulary indeces.\n",
        "\n",
        "From here on, we'll use the below example sentence, which contains two instances of the word \"bank\" with different meanings."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XYjcYJuXoAQx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 391
        },
        "outputId": "bca91267-37a4-4e1e-cba6-f4f3f4f502e1"
      },
      "source": [
        "# Define a new example sentence with multiple meanings of the word \"bank\"\n",
        "text = \"After stealing money from the bank vault, the bank robber was seen \" \\\n",
        "       \"fishing on the Mississippi river bank.\"\n",
        "\n",
        "# Add the special tokens.\n",
        "marked_text = \"[CLS] \" + text + \" [SEP]\"\n",
        "\n",
        "# Split the sentence into tokens.\n",
        "tokenized_text = tokenizer.tokenize(marked_text)\n",
        "\n",
        "# Map the token strings to their vocabulary indeces.\n",
        "indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)\n",
        "\n",
        "# Display the words with their indeces.\n",
        "for tup in zip(tokenized_text, indexed_tokens):\n",
        "    print('{:<12} {:>6,}'.format(tup[0], tup[1]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[CLS]           101\n",
            "after         2,044\n",
            "stealing     11,065\n",
            "money         2,769\n",
            "from          2,013\n",
            "the           1,996\n",
            "bank          2,924\n",
            "vault        11,632\n",
            ",             1,010\n",
            "the           1,996\n",
            "bank          2,924\n",
            "robber       27,307\n",
            "was           2,001\n",
            "seen          2,464\n",
            "fishing       5,645\n",
            "on            2,006\n",
            "the           1,996\n",
            "mississippi   5,900\n",
            "river         2,314\n",
            "bank          2,924\n",
            ".             1,012\n",
            "[SEP]           102\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "if6C_iCULU60"
      },
      "source": [
        "## 2.3. Segment ID\n",
        "BERT is trained on and expects sentence pairs, using 1s and 0s to distinguish between the two sentences. That is, for each token in \"tokenized_text,\" we must specify which sentence it belongs to: sentence 0 (a series of 0s) or sentence 1 (a series of 1s). For our purposes, single-sentence inputs only require a series of 1s, so we will create a vector of 1s for each token in our input sentence. \n",
        "\n",
        "If you want to process two sentences, assign each word in the first sentence plus the '[SEP]' token a 0, and all tokens of the second sentence a 1."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u_jEkVKxJMc0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "ee4539c3-670d-4e5e-a4c6-547c1f2c2834"
      },
      "source": [
        "# Mark each of the 22 tokens as belonging to sentence \"1\".\n",
        "segments_ids = [1] * len(tokenized_text)\n",
        "\n",
        "print (segments_ids)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c-nY9LASLr2L"
      },
      "source": [
        "# 3. Extracting Embeddings \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sl-iCj8wMEd5"
      },
      "source": [
        "## 3.1. Running BERT on our text\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Nvaw46mfc8M"
      },
      "source": [
        "\n",
        "Next we need to convert our data to torch tensors and call the BERT model. The BERT PyTorch interface requires that the data be in torch tensors rather than Python lists, so we convert the lists here - this does not change the shape or the data.\n",
        " "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E_t4cM6KLc98"
      },
      "source": [
        "# Convert inputs to PyTorch tensors\n",
        "tokens_tensor = torch.tensor([indexed_tokens])\n",
        "segments_tensors = torch.tensor([segments_ids])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UCIGe0AXfg4Z"
      },
      "source": [
        "Calling `from_pretrained` will fetch the model from the internet. When we load the `bert-base-uncased`, we see the definition of the model printed in the logging. The model is a deep neural network with 12 layers! Explaining the layers and their functions is outside the scope of this post, and you can skip over this output for now.\n",
        "\n",
        "model.eval() puts our model in evaluation mode as opposed to training mode. In this case, evaluation mode turns off dropout regularization which is used in training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mq2PKplWfbFv",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000,
          "referenced_widgets": [
            "6e2ae067ffc04984b442f95b5f0441f6",
            "5c76f0ea92dd4a39817b68dfa9aaf0a8",
            "1edbae9631684961ad77d6b32e332293",
            "ab6057a7bb08466a883314cdcb7a542d",
            "3381d947e661406fb26ec95e7974d68e",
            "af57a5f70f574bf1a2a1c5a01557732e",
            "3557cddbb2a74bd9b6d39332c3adebf1",
            "0028d4de549347deb60c177654162f5d",
            "412e1f72fe1f48dfa7e25db50a0514b6",
            "f30dc9ad3a3a4778971d4c464b848603",
            "9f7723b2217447309eee0fae5e6cc903",
            "7dda77ecb5a84aeea9fb802aade55193",
            "ed39302ee90f43cfa3905abf3dac1666",
            "5147c6c0f0d746adba86e83cd62e95e1",
            "e005cd31461847ef8c955ad8b079a80c",
            "ab5988e6a3814895b6b750039c34bbd4"
          ]
        },
        "outputId": "ad90a1b6-82ed-4b0a-cd37-1ff93c0ff103"
      },
      "source": [
        "# Load pre-trained model (weights)\n",
        "model = BertModel.from_pretrained('bert-base-uncased',\n",
        "                                  output_hidden_states = True, # Whether the model returns all hidden-states.\n",
        "                                  )\n",
        "\n",
        "# Put the model in \"evaluation\" mode, meaning feed-forward operation.\n",
        "model.eval()\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6e2ae067ffc04984b442f95b5f0441f6",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=433.0, style=ProgressStyle(description_…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "412e1f72fe1f48dfa7e25db50a0514b6",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=440473133.0, style=ProgressStyle(descri…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BertModel(\n",
              "  (embeddings): BertEmbeddings(\n",
              "    (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
              "    (position_embeddings): Embedding(512, 768)\n",
              "    (token_type_embeddings): Embedding(2, 768)\n",
              "    (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "    (dropout): Dropout(p=0.1, inplace=False)\n",
              "  )\n",
              "  (encoder): BertEncoder(\n",
              "    (layer): ModuleList(\n",
              "      (0): BertLayer(\n",
              "        (attention): BertAttention(\n",
              "          (self): BertSelfAttention(\n",
              "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "          (output): BertSelfOutput(\n",
              "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (intermediate): BertIntermediate(\n",
              "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "        )\n",
              "        (output): BertOutput(\n",
              "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (dropout): Dropout(p=0.1, inplace=False)\n",
              "        )\n",
              "      )\n",
              "      (1): BertLayer(\n",
              "        (attention): BertAttention(\n",
              "          (self): BertSelfAttention(\n",
              "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "          (output): BertSelfOutput(\n",
              "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (intermediate): BertIntermediate(\n",
              "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "        )\n",
              "        (output): BertOutput(\n",
              "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (dropout): Dropout(p=0.1, inplace=False)\n",
              "        )\n",
              "      )\n",
              "      (2): BertLayer(\n",
              "        (attention): BertAttention(\n",
              "          (self): BertSelfAttention(\n",
              "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "          (output): BertSelfOutput(\n",
              "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (intermediate): BertIntermediate(\n",
              "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "        )\n",
              "        (output): BertOutput(\n",
              "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (dropout): Dropout(p=0.1, inplace=False)\n",
              "        )\n",
              "      )\n",
              "      (3): BertLayer(\n",
              "        (attention): BertAttention(\n",
              "          (self): BertSelfAttention(\n",
              "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "          (output): BertSelfOutput(\n",
              "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (intermediate): BertIntermediate(\n",
              "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "        )\n",
              "        (output): BertOutput(\n",
              "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (dropout): Dropout(p=0.1, inplace=False)\n",
              "        )\n",
              "      )\n",
              "      (4): BertLayer(\n",
              "        (attention): BertAttention(\n",
              "          (self): BertSelfAttention(\n",
              "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "          (output): BertSelfOutput(\n",
              "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (intermediate): BertIntermediate(\n",
              "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "        )\n",
              "        (output): BertOutput(\n",
              "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (dropout): Dropout(p=0.1, inplace=False)\n",
              "        )\n",
              "      )\n",
              "      (5): BertLayer(\n",
              "        (attention): BertAttention(\n",
              "          (self): BertSelfAttention(\n",
              "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "          (output): BertSelfOutput(\n",
              "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (intermediate): BertIntermediate(\n",
              "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "        )\n",
              "        (output): BertOutput(\n",
              "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (dropout): Dropout(p=0.1, inplace=False)\n",
              "        )\n",
              "      )\n",
              "      (6): BertLayer(\n",
              "        (attention): BertAttention(\n",
              "          (self): BertSelfAttention(\n",
              "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "          (output): BertSelfOutput(\n",
              "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (intermediate): BertIntermediate(\n",
              "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "        )\n",
              "        (output): BertOutput(\n",
              "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (dropout): Dropout(p=0.1, inplace=False)\n",
              "        )\n",
              "      )\n",
              "      (7): BertLayer(\n",
              "        (attention): BertAttention(\n",
              "          (self): BertSelfAttention(\n",
              "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "          (output): BertSelfOutput(\n",
              "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (intermediate): BertIntermediate(\n",
              "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "        )\n",
              "        (output): BertOutput(\n",
              "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (dropout): Dropout(p=0.1, inplace=False)\n",
              "        )\n",
              "      )\n",
              "      (8): BertLayer(\n",
              "        (attention): BertAttention(\n",
              "          (self): BertSelfAttention(\n",
              "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "          (output): BertSelfOutput(\n",
              "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (intermediate): BertIntermediate(\n",
              "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "        )\n",
              "        (output): BertOutput(\n",
              "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (dropout): Dropout(p=0.1, inplace=False)\n",
              "        )\n",
              "      )\n",
              "      (9): BertLayer(\n",
              "        (attention): BertAttention(\n",
              "          (self): BertSelfAttention(\n",
              "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "          (output): BertSelfOutput(\n",
              "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (intermediate): BertIntermediate(\n",
              "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "        )\n",
              "        (output): BertOutput(\n",
              "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (dropout): Dropout(p=0.1, inplace=False)\n",
              "        )\n",
              "      )\n",
              "      (10): BertLayer(\n",
              "        (attention): BertAttention(\n",
              "          (self): BertSelfAttention(\n",
              "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "          (output): BertSelfOutput(\n",
              "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (intermediate): BertIntermediate(\n",
              "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "        )\n",
              "        (output): BertOutput(\n",
              "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (dropout): Dropout(p=0.1, inplace=False)\n",
              "        )\n",
              "      )\n",
              "      (11): BertLayer(\n",
              "        (attention): BertAttention(\n",
              "          (self): BertSelfAttention(\n",
              "            (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "          (output): BertSelfOutput(\n",
              "            (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (intermediate): BertIntermediate(\n",
              "          (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "        )\n",
              "        (output): BertOutput(\n",
              "          (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "          (dropout): Dropout(p=0.1, inplace=False)\n",
              "        )\n",
              "      )\n",
              "    )\n",
              "  )\n",
              "  (pooler): BertPooler(\n",
              "    (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "    (activation): Tanh()\n",
              "  )\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G4Qa5KkkM2Aq"
      },
      "source": [
        "Next, let's evaluate BERT on our example text, and fetch the hidden states of the network!\n",
        "\n",
        "*Side note: `torch.no_grad` tells PyTorch not to construct the compute graph during this forward pass (since we won't be running backprop here)--this just reduces memory consumption and speeds things up a little.*\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nN0QTZwiMzeq"
      },
      "source": [
        "# Run the text through BERT, and collect all of the hidden states produced\n",
        "# from all 12 layers. \n",
        "with torch.no_grad():\n",
        "\n",
        "    outputs = model(tokens_tensor, segments_tensors)\n",
        "\n",
        "    # Evaluating the model will return a different number of objects based on \n",
        "    # how it's  configured in the `from_pretrained` call earlier. In this case, \n",
        "    # becase we set `output_hidden_states = True`, the third item will be the \n",
        "    # hidden states from all layers. See the documentation for more details:\n",
        "    # https://huggingface.co/transformers/model_doc/bert.html#bertmodel\n",
        "    hidden_states = outputs[2]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UeQNEFbUgMSf"
      },
      "source": [
        "## 3.2. Understanding the Output\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HKTlTS_sfuAe"
      },
      "source": [
        "\n",
        "The full set of hidden states for this model, stored in the object `hidden_states`, is a little dizzying. This object has four dimensions, in the following order:\n",
        "\n",
        "1. The layer number (13 layers)\n",
        "2. The batch number (1 sentence)\n",
        "3. The word / token number (22 tokens in our sentence)\n",
        "4. The hidden unit / feature number (768 features)\n",
        "\n",
        "Wait, 13 layers? Doesn't BERT only have 12? It's 13 because the first element is the input embeddings, the rest is the outputs of each of BERT's 12 layers. \n",
        "\n",
        "That’s 219,648 unique values just to represent our one sentence! \n",
        "\n",
        "The second dimension, the batch size, is used when submitting multiple sentences to the model at once; here, though, we just have one example sentence."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eI_uxiW7eRWA",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "41a67dac-1300-4b94-cc6c-a6e38c3b9495"
      },
      "source": [
        "print (\"Number of layers:\", len(hidden_states), \"  (initial embeddings + 12 BERT layers)\")\n",
        "layer_i = 0\n",
        "\n",
        "print (\"Number of batches:\", len(hidden_states[layer_i]))\n",
        "batch_i = 0\n",
        "\n",
        "print (\"Number of tokens:\", len(hidden_states[layer_i][batch_i]))\n",
        "token_i = 0\n",
        "\n",
        "print (\"Number of hidden units:\", len(hidden_states[layer_i][batch_i][token_i]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of layers: 13   (initial embeddings + 12 BERT layers)\n",
            "Number of batches: 1\n",
            "Number of tokens: 22\n",
            "Number of hidden units: 768\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Uc_S_hmOWe7"
      },
      "source": [
        "Let's take a quick look at the range of values for a given layer and token.\n",
        "\n",
        "You'll find that the range is fairly similar for all layers and tokens, with the majority of values falling between \\[-2, 2\\], and a small smattering of values around -10."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-UF_OAO-S1sP",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 592
        },
        "outputId": "9f1a8a16-cfdb-4b84-a1f0-8778c7b43892"
      },
      "source": [
        "# For the 5th token in our sentence, select its feature values from layer 5.\n",
        "token_i = 5\n",
        "layer_i = 5\n",
        "vec = hidden_states[layer_i][batch_i][token_i]\n",
        "\n",
        "# Plot the values as a histogram to show their distribution.\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.hist(vec, bins=200)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlAAAAI/CAYAAAC4QOfKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAVxElEQVR4nO3df4xl93nX8c+DJwkVBdLgqbHimHFVl8qhxEFbK6hFKHaTurjUBtooFYJFWFpRCkogqEwShFQJpE2LmlYI/rDqqAtKSdImwVanQI2bUkDE6To/mjhuiBs2NI4Tb0qipkKkcvPwx1yna++uZ56dH/fuzuslWXPPuXfmPv5qd+a95945p7o7AADs3h9a9gAAAJcbAQUAMCSgAACGBBQAwJCAAgAYElAAAENrh/lkV199dW9sbBzmUwIAXJKHH374C929fqH7DjWgNjY2cvr06cN8SgCAS1JVn77YfV7CAwAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAoDL3MbmVjY2t5Y9xpEioAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABhaW/YAAMD+2Njc+trtMyfvWOIkVz5HoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAobXdPKiqziT5cpLfT/JUdx+rqhcleWeSjSRnkrymu794MGMCAKyOyRGoV3b3zd19bLG9meTB7r4xyYOLbQCAK95eXsK7M8mpxe1TSe7a+zgAAKtvtwHVSX6pqh6uqhOLfdd09xOL259Lcs2+TwcAsIJ29R6oJN/Z3Y9X1TcmeaCqfuPcO7u7q6ov9ImL4DqRJNdff/2ehgUAWAW7OgLV3Y8vPj6Z5L1Jbkny+aq6NkkWH5+8yOfe093HuvvY+vr6/kwNALBEOwZUVf2RqvqjT99O8uokH0tyf5Lji4cdT3LfQQ0JALBKdvMS3jVJ3ltVTz/+Z7v7P1bVryV5V1XdneTTSV5zcGMCAKyOHQOquz+V5GUX2P/bSW47iKEAAFaZM5EDAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQBXoI3NrWxsbu24j0sjoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwtLbsAQCAg+PEmQfDESgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoADjCNja3nGzzEggoAIAhAQUAMCSgAACGBBQAwNCuA6qqrqqqD1XVLyy2b6iqh6rqsap6Z1U9/+DGBABYHZMjUK9L8ug5229J8tbu/uYkX0xy934OBgCwqnYVUFV1XZI7kvz0YruS3Jrk5xcPOZXkroMYEABg1ez2CNRPJvmRJF9dbP+JJF/q7qcW259J8uJ9ng0AYCXtGFBV9b1Jnuzuhy/lCarqRFWdrqrTZ8+evZQvAQAsOPHlatjNEajvSPJ9VXUmyTuy/dLdTyV5YVWtLR5zXZLHL/TJ3X1Pdx/r7mPr6+v7MDIAwHLtGFDd/cbuvq67N5K8Nskvd/dfT/K+JN+/eNjxJPcd2JQAACtkL+eB+sdJ/mFVPZbt90Tduz8jAQCstrWdH/IHuvtXkvzK4vanktyy/yMBAKw2ZyIHABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADC0tuwBAIC5jc2tZY9wpDkCBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhJ9IEgCPmQifhPHffmZN3HOY4lyVHoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACG1pY9AADw3DY2t5Y9As/iCBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIZ2DKiq+sNV9YGq+khVPVJVP7rYf0NVPVRVj1XVO6vq+Qc/LgDA8u3mCNRXktza3S9LcnOS26vqFUnekuSt3f3NSb6Y5O6DGxMAYHXsGFC97XcXm89b/NdJbk3y84v9p5LcdSATAgCsmF29B6qqrqqqDyd5MskDSX4zyZe6+6nFQz6T5MUHMyIAwGrZVUB19+93981JrktyS5Jv3e0TVNWJqjpdVafPnj17iWMCAIdlY3MrG5tbyx5jpY1+C6+7v5TkfUn+fJIXVtXa4q7rkjx+kc+5p7uPdfex9fX1PQ0LALAKdvNbeOtV9cLF7a9L8qokj2Y7pL5/8bDjSe47qCEBAFbJ2s4PybVJTlXVVdkOrnd19y9U1ceTvKOq/lmSDyW59wDnBABYGTsGVHf/epKXX2D/p7L9figAgCPFmcgBAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAKyQjc2tbGxuLXsMdiCgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADC0tuwBAIDzOZnmanMECgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBobdkDAMBRt7G5tewRGHIECgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIZ2DKiqeklVva+qPl5Vj1TV6xb7X1RVD1TVJxcfv+HgxwUAWL7dHIF6KskbuvumJK9I8sNVdVOSzSQPdveNSR5cbAMAXPF2DKjufqK7P7i4/eUkjyZ5cZI7k5xaPOxUkrsOakgAgFUyeg9UVW0keXmSh5Jc091PLO76XJJr9nUyAIAVtbbbB1bV1yd5d5LXd/fvVNXX7uvurqq+yOedSHIiSa6//vq9TQsAHJqNza2v3T5z8o4lTrJ6dnUEqqqel+14ent3v2ex+/NVde3i/muTPHmhz+3ue7r7WHcfW19f34+ZAQCWaje/hVdJ7k3yaHf/xDl33Z/k+OL28ST37f94AACrZzcv4X1Hkr+R5KNV9eHFvjclOZnkXVV1d5JPJ3nNwYwIALBadgyo7v5vSeoid9+2v+MAAKw+ZyIHABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAobVlDwAArL6Nza3z9p05eccSJlkNjkABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwNDasgcAgKNqY3Nr2SPsybnznzl5xxInOXyOQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgyIk0AeCAXO4nypy40P/rlXxyTUegAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGBJQAABDAgoAYGjHgKqqt1XVk1X1sXP2vaiqHqiqTy4+fsPBjgkAsDp2cwTqZ5Lc/qx9m0ke7O4bkzy42AYAOBJ2DKju/tUk/+dZu+9Mcmpx+1SSu/Z5LgCAlXWp74G6prufWNz+XJJr9mkeAICVt+c3kXd3J+mL3V9VJ6rqdFWdPnv27F6fDgBg6S41oD5fVdcmyeLjkxd7YHff093HuvvY+vr6JT4dAMDquNSAuj/J8cXt40nu259xAABW325OY/DvkvyPJH+6qj5TVXcnOZnkVVX1ySTftdgGADgS1nZ6QHf/4EXuum2fZwEAuCw4EzkAwJCAAgAYElAAAEMCCgBgaMc3kQMAMxubW8segQPmCBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAHAgNja3rtiTigooAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAytLXsAALjcnHtyyDMn71jiJCyLI1AAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhgQUAMCQgAIAGHIiTQDgQF2JJx51BAoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQ06kCQAcusv95JqOQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAYElAAAEMCCgBgSEABwB5sbG4946SQzF2OayigAACGBBQAwJCAAgAYElAAAEMCCgBgSEABAAwJKACAIQEFADAkoAAAhtaWPQAAcHRMzzh+7uPPnLxjv8e5ZI5AAQAMCSgAgCEBBQAwJKAAAIYEFADAkIACABgSUAAAQwIKAGDIiTQB4Dns9kSO0xNE8txWfT0dgQIAGBJQAABDAgoAYEhAAQAMCSgAgCEBBQAwJKAAAIYEFADA0BV3Is3dnvAMAC7mYidxXPWTO17pnl7/Vfj57ggUAMCQgAIAGBJQAABDAgoAYGhPAVVVt1fVJ6rqsara3K+hAABW2SUHVFVdleRfJfmeJDcl+cGqumm/BgMAWFV7OQJ1S5LHuvtT3f17Sd6R5M79GQsAYHXtJaBenOS3ztn+zGIfAMAV7cBPpFlVJ5KcWGz+blV94qCf82vP/ZbDeqaRq5N8YdlDrBhrcj5rcj5rcj5rcj5r8kyX1Xrs9uf2Hn++T9bkT13sjr0E1ONJXnLO9nWLfc/Q3fckuWcPz3NFqarT3X1s2XOsEmtyPmtyPmtyPmtyPmvyTNbjfPu1Jnt5Ce/XktxYVTdU1fOTvDbJ/XsdCABg1V3yEajufqqq/l6S/5TkqiRv6+5H9m0yAIAVtaf3QHX3Lyb5xX2a5ajwcub5rMn5rMn5rMn5rMn5rMkzWY/z7cuaVHfvx9cBADgyXMoFAGBIQB2SqvqBqnqkqr5aVcfO2f+qqnq4qj66+HjrMuc8TBdbk8V9b1xcIugTVfXdy5pxmarq5qp6f1V9uKpOV9Uty55p2arq71fVbyz+3PzYsudZFVX1hqrqqrp62bMsW1X9+OLPyK9X1Xur6oXLnmlZXG7tmarqJVX1vqr6+OJ7yOv28vUE1OH5WJK/muRXn7X/C0n+cnd/W5LjSf7tYQ+2RBdck8UlgV6b5KVJbk/yrxeXDjpqfizJj3b3zUn+6WL7yKqqV2b7agcv6+6XJvkXSx5pJVTVS5K8Osn/XvYsK+KBJH+mu/9skv+Z5I1LnmcpXG7tgp5K8obuvinJK5L88F7WREAdku5+tLvPO4lod3+ouz+72HwkyddV1QsOd7rluNiaZPuH5Du6+yvd/b+SPJbtSwcdNZ3kjy1u//Ekn32Oxx4FP5TkZHd/JUm6+8klz7Mq3prkR7L95+XI6+5f6u6nFpvvz/Y5Co8il1t7lu5+ors/uLj95SSPZg9XUBFQq+WvJfng0z8gjjCXCdr2+iQ/XlW/le2jLUfyX9Ln+JYkf6GqHqqq/1JV377sgZatqu5M8nh3f2TZs6yov53kPyx7iCXxffQ5VNVGkpcneehSv8aBX8rlKKmq/5zkT17grjd39307fO5Lk7wl24firxh7WZOj4LnWJ8ltSf5Bd7+7ql6T5N4k33WY8x22HdZjLcmLsn3o/duTvKuqvqmv8F8l3mFN3pQr7HvGbuzm+0pVvTnbL9m8/TBnY/VV1dcneXeS13f371zq1xFQ+6i7L+mHW1Vdl+S9Sf5md//m/k61XJe4Jru6TNCV4LnWp6r+TZKn3+T4c0l++lCGWqId1uOHkrxnEUwfqKqvZvuaVmcPa75luNiaVNW3JbkhyUeqKtn+e/LBqrqluz93iCMeup2+r1TV30ryvUluu9ID+zkcme+jE1X1vGzH09u7+z17+VpewluyxW+IbCXZ7O7/vux5VsT9SV5bVS+oqhuS3JjkA0ueaRk+m+QvLm7fmuSTS5xlFfz7JK9Mkqr6liTPz2V0kdT91t0f7e5v7O6N7t7I9ks0f+5Kj6edVNXt2X5P2Pd19/9d9jxL5HJrz1Lb/9K4N8mj3f0Te/56RzfOD1dV/ZUk/zLJepIvJflwd393Vf2TbL+35dwfjq8+Cm+QvdiaLO57c7bfv/BUtg+zHrn3MVTVdyb5qWwfKf5/Sf5udz+83KmWZ/FD4G1Jbk7ye0n+UXf/8nKnWh1VdSbJse4+slGZJFX1WJIXJPntxa73d/ffWeJIS1NVfynJT+YPLrf2z5c80lItvqf+1yQfTfLVxe43La6qMv96AgoAYMZLeAAAQwIKAGBIQAEADAkoAIAhAQUAMCSgAACGBBQAwJCAAgAY+v8yrajMccmJFwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 720x720 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n194RcReDYfw"
      },
      "source": [
        "Grouping the values by layer makes sense for the model, but for our purposes we want it grouped by token. \n",
        "\n",
        "Current dimensions:\n",
        "\n",
        "`[# layers, # batches, # tokens, # features]`\n",
        "\n",
        "Desired dimensions:\n",
        "\n",
        "`[# tokens, # layers, # features]`\n",
        "\n",
        "Luckily, PyTorch includes the `permute` function for easily rearranging the dimensions of a tensor. \n",
        "\n",
        "However, the first dimension is currently a Python list! "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0CcY_oRwcHlS",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "414db21b-bf34-4374-c8be-b2cc0b04ea10"
      },
      "source": [
        "# `hidden_states` is a Python list.\n",
        "print('      Type of hidden_states: ', type(hidden_states))\n",
        "\n",
        "# Each layer in the list is a torch tensor.\n",
        "print('Tensor shape for each layer: ', hidden_states[0].size())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "      Type of hidden_states:  <class 'tuple'>\n",
            "Tensor shape for each layer:  torch.Size([1, 22, 768])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1yXZjLSke3F0"
      },
      "source": [
        "Let's combine the layers to make this one whole big tensor."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pTJV8AFFcLbL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "ee719c26-ce72-4469-eb5b-2c45c8979564"
      },
      "source": [
        "# Concatenate the tensors for all layers. We use `stack` here to\n",
        "# create a new dimension in the tensor.\n",
        "token_embeddings = torch.stack(hidden_states, dim=0)\n",
        "\n",
        "token_embeddings.size()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([13, 1, 22, 768])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rnBv2TUNhzf4"
      },
      "source": [
        "Let's get rid of the \"batches\" dimension since we don't need it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "En4JZ41fh6CI",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "2a691913-7ea6-4774-ccaa-45dd2fdb9ac8"
      },
      "source": [
        "# Remove dimension 1, the \"batches\".\n",
        "token_embeddings = torch.squeeze(token_embeddings, dim=1)\n",
        "\n",
        "token_embeddings.size()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([13, 22, 768])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YVzRfvkbe-Yp"
      },
      "source": [
        "Finally, we can switch around the \"layers\" and \"tokens\" dimensions with `permute`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AtDVE58cdeYp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "9ac9b020-0a41-4aed-bcaa-a4185e89a60f"
      },
      "source": [
        "# Swap dimensions 0 and 1.\n",
        "token_embeddings = token_embeddings.permute(1,0,2)\n",
        "\n",
        "token_embeddings.size()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([22, 13, 768])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ey5RhOQ7NGtz"
      },
      "source": [
        "## 3.3. Creating word and sentence vectors from hidden states\n",
        "\n",
        "Now, what do we do with these hidden states? We would like to get individual vectors for each of our tokens, or perhaps a single vector representation of the whole sentence, but for each token of our input we have 13 separate vectors each of length 768.\n",
        "\n",
        "In order to get the individual vectors we will need to combine some of the layer vectors...but which layer or combination of layers provides the best representation? \n",
        "\n",
        "Unfortunately, there's no single easy answer... Let's try a couple reasonable approaches, though. Afterwards, I'll point you to some helpful resources which look into this question further.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "76TdtFH8NM9q"
      },
      "source": [
        "### Word Vectors\n",
        "\n",
        "To give you some examples, let's create word vectors two ways. \n",
        "\n",
        "First, let's **concatenate** the last four layers, giving us a single word vector per token. Each vector will have length `4 x 768 = 3,072`. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pv42h9jANMRf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "dea3396c-1960-4b22-d4bd-e9ef17aba218"
      },
      "source": [
        "# Stores the token vectors, with shape [22 x 3,072]\n",
        "token_vecs_cat = []\n",
        "\n",
        "# `token_embeddings` is a [22 x 12 x 768] tensor.\n",
        "\n",
        "# For each token in the sentence...\n",
        "for token in token_embeddings:\n",
        "    \n",
        "    # `token` is a [12 x 768] tensor\n",
        "\n",
        "    # Concatenate the vectors (that is, append them together) from the last \n",
        "    # four layers.\n",
        "    # Each layer vector is 768 values, so `cat_vec` is length 3,072.\n",
        "    cat_vec = torch.cat((token[-1], token[-2], token[-3], token[-4]), dim=0)\n",
        "    \n",
        "    # Use `cat_vec` to represent `token`.\n",
        "    token_vecs_cat.append(cat_vec)\n",
        "\n",
        "print ('Shape is: %d x %d' % (len(token_vecs_cat), len(token_vecs_cat[0])))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Shape is: 22 x 3072\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VnWaByfelM-e"
      },
      "source": [
        "As an alternative method, let's try creating the word vectors by **summing** together the last four layers."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j4DKDtFwiF0S",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "17646453-f724-4063-9ba8-e15056e0bcdc"
      },
      "source": [
        "# Stores the token vectors, with shape [22 x 768]\n",
        "token_vecs_sum = []\n",
        "\n",
        "# `token_embeddings` is a [22 x 12 x 768] tensor.\n",
        "\n",
        "# For each token in the sentence...\n",
        "for token in token_embeddings:\n",
        "\n",
        "    # `token` is a [12 x 768] tensor\n",
        "\n",
        "    # Sum the vectors from the last four layers.\n",
        "    sum_vec = torch.sum(token[-4:], dim=0)\n",
        "    \n",
        "    # Use `sum_vec` to represent `token`.\n",
        "    token_vecs_sum.append(sum_vec)\n",
        "\n",
        "print ('Shape is: %d x %d' % (len(token_vecs_sum), len(token_vecs_sum[0])))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Shape is: 22 x 768\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mQaco6jRLkXn"
      },
      "source": [
        "### Sentence Vectors\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uuul6iQqnXT2"
      },
      "source": [
        "\n",
        "To get a single vector for our entire sentence we have multiple application-dependent strategies, but a simple approach is to average the second to last hiden layer of each token producing a single 768 length vector."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zn0n2S-FWZih"
      },
      "source": [
        "# `hidden_states` has shape [13 x 1 x 22 x 768]\n",
        "\n",
        "# `token_vecs` is a tensor with shape [22 x 768]\n",
        "token_vecs = hidden_states[-2][0]\n",
        "\n",
        "# Calculate the average of all 22 token vectors.\n",
        "sentence_embedding = torch.mean(token_vecs, dim=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MQv0FL8VWadn",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "f5f78a03-971f-47e6-85be-aff7bf956ad9"
      },
      "source": [
        "print (\"Our final sentence embedding vector of shape:\", sentence_embedding.size())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Our final sentence embedding vector of shape: torch.Size([768])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TqYcrAipfE3E"
      },
      "source": [
        "## 3.4. Confirming contextually dependent vectors\n",
        "\n",
        "To confirm that the value of these vectors are in fact contextually dependent, let's look at the different instances of the word \"bank\" in our example sentence:\n",
        "\n",
        "\"After stealing money from the **bank vault**, the **bank robber** was seen fishing on the Mississippi **river bank**.\"\n",
        "\n",
        "Let's find the index of those three instances of the word \"bank\" in the example sentence."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DNiRsEh9cmWz",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 391
        },
        "outputId": "e1149ca7-b485-4e1e-cf0a-8bbedcb051a1"
      },
      "source": [
        "for i, token_str in enumerate(tokenized_text):\n",
        "  print (i, token_str)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0 [CLS]\n",
            "1 after\n",
            "2 stealing\n",
            "3 money\n",
            "4 from\n",
            "5 the\n",
            "6 bank\n",
            "7 vault\n",
            "8 ,\n",
            "9 the\n",
            "10 bank\n",
            "11 robber\n",
            "12 was\n",
            "13 seen\n",
            "14 fishing\n",
            "15 on\n",
            "16 the\n",
            "17 mississippi\n",
            "18 river\n",
            "19 bank\n",
            "20 .\n",
            "21 [SEP]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AEhBIA5RlS8-"
      },
      "source": [
        "They are at 6, 10, and 19.\n",
        "\n",
        "For this analysis, we'll use the word vectors that we created by summing the last four layers.\n",
        "\n",
        "We can try printing out their vectors to compare them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tBa6vRHknSkv",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "2058a07b-cb3a-4e23-a39d-8124783261a8"
      },
      "source": [
        "print('First 5 vector values for each instance of \"bank\".')\n",
        "print('')\n",
        "print(\"bank vault   \", str(token_vecs_sum[6][:5]))\n",
        "print(\"bank robber  \", str(token_vecs_sum[10][:5]))\n",
        "print(\"river bank   \", str(token_vecs_sum[19][:5]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "First 5 vector values for each instance of \"bank\".\n",
            "\n",
            "bank vault    tensor([ 3.3596, -2.9805, -1.5421,  0.7065,  2.0031])\n",
            "bank robber   tensor([ 2.7359, -2.5577, -1.3094,  0.6797,  1.6633])\n",
            "river bank    tensor([ 1.5266, -0.8895, -0.5152, -0.9298,  2.8334])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ca2TCQ_G7SM3"
      },
      "source": [
        "We can see that the values differ, but let's calculate the cosine similarity between the vectors to make a more precise comparison."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eYXUwiG0yhBS",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "c9eb901f-8c6f-4961-c496-6491f99d7bc8"
      },
      "source": [
        "from scipy.spatial.distance import cosine\n",
        "\n",
        "# Calculate the cosine similarity between the word bank \n",
        "# in \"bank robber\" vs \"river bank\" (different meanings).\n",
        "diff_bank = 1 - cosine(token_vecs_sum[10], token_vecs_sum[19])\n",
        "\n",
        "# Calculate the cosine similarity between the word bank\n",
        "# in \"bank robber\" vs \"bank vault\" (same meaning).\n",
        "same_bank = 1 - cosine(token_vecs_sum[10], token_vecs_sum[6])\n",
        "\n",
        "print('Vector similarity for  *similar*  meanings:  %.2f' % same_bank)\n",
        "print('Vector similarity for *different* meanings:  %.2f' % diff_bank)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Vector similarity for  *similar*  meanings:  0.94\n",
            "Vector similarity for *different* meanings:  0.69\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N7jroXfKspe_"
      },
      "source": [
        "This looks pretty good!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "orjhWUJgmxo5"
      },
      "source": [
        "## 3.5. Pooling Strategy & Layer Choice"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f1CI97kNn8dD"
      },
      "source": [
        "Below are a couple additional resources for exploring this topic."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P3D5qnRNmq5_"
      },
      "source": [
        "**BERT Authors**\n",
        "\n",
        "The BERT authors tested word-embedding strategies by feeding different vector combinations as input features to a BiLSTM used on a named entity recognition task and observing the resulting F1 scores.\n",
        "\n",
        "(Image from [Jay Allamar](http://jalammar.github.io/illustrated-bert/)'s blog)\n",
        "\n",
        "\n",
        "![alt text](http://jalammar.github.io/images/bert-feature-extraction-contextualized-embeddings.png)\n",
        "\n",
        "While concatenation of the last four layers produced the best results on this specific task, many of the other methods come in a close second and in general it is advisable to test different versions for your specific application: results may vary.\n",
        "\n",
        "This is partially demonstrated by noting that the different layers of BERT encode very different kinds of information, so the appropriate pooling strategy will change depending on the application because different layers encode different kinds of information. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m7_CVgejm5pr"
      },
      "source": [
        "**Han Xiao's BERT-as-service**\n",
        "\n",
        "Han Xiao created an open-source project named [bert-as-service](https://github.com/hanxiao/bert-as-service) on GitHub which is intended to create word embeddings for your text using BERT. Han experimented with different approaches to combining these embeddings, and shared some conclusions and rationale on the [FAQ page](https://github.com/hanxiao/bert-as-service#speech_balloon-faq) of the project. \n",
        "\n",
        "`bert-as-service`, by default, uses the outputs from the **second-to-last layer** of the model. \n",
        "\n",
        "I would summarize Han's perspective by the following:\n",
        "\n",
        "1. The embeddings start out in the first layer as having no contextual information (i.e., the meaning of the initial 'bank' embedding isn't specific to river bank or financial bank).\n",
        "2. As the embeddings move deeper into the network, they pick up more and more contextual information with each layer.\n",
        "3. As you approach the final layer, however, you start picking up information that is specific to BERT's pre-training tasks (the \"Masked Language Model\" (MLM) and \"Next Sentence Prediction\" (NSP)). \n",
        "    * What we want is embeddings that encode the word meaning well... \n",
        "    * BERT is motivated to do this, but it is also motivated to encode anything else that would help it determine what a missing word is (MLM), or whether the second sentence came after the first (NSP). \n",
        "4. The second-to-last layer is what Han settled on as a reasonable sweet-spot.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ONLJ36JfPuqf"
      },
      "source": [
        "# 4. Appendix\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jdw7cLJWMr_Y"
      },
      "source": [
        "## 4.1. Special tokens\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jyx2kQxbnHbM"
      },
      "source": [
        "\n",
        "It should be noted that although the `[CLS]` acts as an \"aggregate representation\" for classification tasks, this is not the best choice for a high quality sentence embedding vector. [According to](https://github.com/google-research/bert/issues/164) BERT author Jacob Devlin: \"*I'm not sure what these vectors are, since BERT does not generate meaningful sentence vectors. It seems that this is is doing average pooling over the word tokens to get a sentence vector, but we never suggested that this will generate meaningful sentence representations*.\"\n",
        "\n",
        "(However, the [CLS] token does become meaningful if the model has been fine-tuned, where the last hidden layer of this token is used as the \"sentence vector\" for sequence classification.)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EbS8_z6XMuTJ"
      },
      "source": [
        "\n",
        "## 4.2. Out of vocabulary words\n",
        "\n",
        "For **out of vocabulary words** that are composed of multiple sentence and character-level embeddings, there is a further issue of how best to recover this embedding. Averaging the embeddings is the most straightforward solution (one that is relied upon in similar embedding models with subword vocabularies like fasttext), but summation of subword embeddings and simply taking the last token embedding (remember that the vectors are context sensitive) are acceptable alternative strategies.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BokW7CAgMxCB"
      },
      "source": [
        "\n",
        "## 4.3. Similarity metrics\n",
        "\n",
        "It is worth noting that word-level **similarity comparisons** are not appropriate with BERT embeddings because these embeddings are contextually dependent, meaning that the word vector changes depending on the sentence it appears in. This allows wonderful things like polysemy so that e.g. your representation encodes river \"bank\" and not a financial institution \"bank\",  but makes direct word-to-word similarity comparisons less valuable. However, for sentence embeddings similarity comparison is still valid such that one can query, for example, a single sentence against a dataset of other sentences in order to find the most similar. Depending on the similarity metric used, the resulting similarity values will be less informative than the relative ranking of similarity outputs since many similarity metrics make assumptions about the vector space (equally-weighted dimensions, for example) that do not hold for our 768-dimensional vector space.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0unZ2xh4QDap"
      },
      "source": [
        "## 4.4. Implementations\n",
        "\n",
        "You can use the code in this notebook as the foundation of your own application to extract BERT features from text. However, official [tensorflow](https://github.com/google-research/bert/blob/master/extract_features.py) and well-regarded [pytorch](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/extract_features.py) implementations already exist that do this for you.  Additionally, [bert-as-a-service](https://github.com/hanxiao/bert-as-service) is an excellent tool designed specifically for running this task with high performance, and is the one I would recommend for production applications. The author has taken great care in the tool's implementation and provides excellent documentation (some of which was used to help create this tutorial) to help users understand the more nuanced details the user faces, like resource management and pooling strategy."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OhbZxbKRxMvM"
      },
      "source": [
        "## Cite\n",
        "Chris McCormick and Nick Ryan. (2019, May 14). *BERT Word Embeddings Tutorial*. Retrieved from http://www.mccormickml.com\n"
      ]
    }
  ]
}