turn_words_into_tokens.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "l09c01_nlp_turn_words_into_tokens.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3.8.5 64-bit"
    },
    "interpreter": {
      "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fFv-USWkhQKA"
      },
      "source": [
        "# Tokenizing text and creating sequences for sentences"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7hMGfCIDPnm8"
      },
      "source": [
        "This colab shows you how to tokenize text and create sequences for sentences as the first stage of preparing text for use with TensorFlow models."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4mGaRDFcSamt"
      },
      "source": [
        "## Import the Tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EN1-FZodOuPl"
      },
      "source": [
        "# Import the Tokenizer\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C5Qwn_7FSXW-"
      },
      "source": [
        "## Write some sentences\n",
        "\n",
        "Feel free to change and add sentences as you like"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RMiq8BpWVVRa"
      },
      "source": [
        "sentences = [\n",
        "    'My favorite food is ice cream',\n",
        "    'do you like ice cream too?',\n",
        "    'My dog likes ice cream!',\n",
        "    \"your favorite flavor of icecream is chocolate\",\n",
        "    \"chocolate isn't good for dogs\",\n",
        "    \"your dog, your cat, and your parrot prefer broccoli\"\n",
        "]"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wz845OtfRBCM"
      },
      "source": [
        "## Tokenize the words\n",
        "\n",
        "The first step to preparing text to be used in a machine learning model is to tokenize the text, in other words, to generate numbers for the words."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZHTK1DAlQ1zO"
      },
      "source": [
        "# Optionally set the max number of words to tokenize.\n",
        "# The out of vocabulary (OOV) token represents words that are not in the index.\n",
        "# Call fit_on_text() on the tokenizer to generate unique numbers for each word\n",
        "tokenizer = Tokenizer(num_words = 100, oov_token=\"<OOV>\")\n",
        "tokenizer.fit_on_texts(sentences)\n"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mylv-WuiRzd0"
      },
      "source": [
        "## View the word index\n",
        "After you tokenize the text, the tokenizer has a word index that contains key-value pairs for all the words and their numbers.\n",
        "\n",
        "The word is the key, and the number is the value.\n",
        "\n",
        "Notice that the OOV token is the first entry.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kX4VvsLySC7Z"
      },
      "source": [
        "# Examine the word index\n",
        "word_index = tokenizer.word_index\n",
        "print(word_index)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'<OOV>': 1, 'your': 2, 'ice': 3, 'cream': 4, 'my': 5, 'favorite': 6, 'is': 7, 'dog': 8, 'chocolate': 9, 'food': 10, 'do': 11, 'you': 12, 'like': 13, 'too': 14, 'likes': 15, 'flavor': 16, 'of': 17, 'icecream': 18, \"isn't\": 19, 'good': 20, 'for': 21, 'dogs': 22, 'cat': 23, 'and': 24, 'parrot': 25, 'prefer': 26, 'broccoli': 27}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JXKrGxsIVtLo"
      },
      "source": [
        "# Get the number for a given word\n",
        "print(word_index['favorite'])"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "6\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kcN_yM8O1oSX"
      },
      "source": [
        "# Create sequences for the sentences\n",
        "\n",
        "After you tokenize the words, the word index contains a unique number for each word. However, the numbers in the word index are not ordered. Words in a sentence have an order. So after tokenizing the words, the next step is to generate sequences for the sentences."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QlUL6Ybf1sso"
      },
      "source": [
        "sequences = tokenizer.texts_to_sequences(sentences)\n",
        "print (sequences)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[5, 6, 10, 7, 3, 4], [11, 12, 13, 3, 4, 14], [5, 8, 15, 3, 4], [2, 6, 16, 17, 18, 7, 9], [9, 19, 20, 21, 22], [2, 8, 2, 23, 24, 2, 25, 26, 27]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AswZPbuW8f-f"
      },
      "source": [
        "# Sequence sentences that contain words that are not in the word index\n",
        "\n",
        "Let's take a look at what happens if the sentence being sequenced contains words that are not in the word index.\n",
        "\n",
        "The Out of Vocabluary (OOV) token is the first entry in the word index. You will see it shows up in the sequences in place of any word that is not in the word index."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fir7qd6X8eZc"
      },
      "source": [
        "sentences2 = [\"I like hot chocolate\", \"My dogs and my hedgehog like kibble but my squirrel prefers grapes and my chickens like ice cream, preferably vanilla\"]\n",
        "\n",
        "sequences2 = tokenizer.texts_to_sequences(sentences2)\n",
        "print(sequences2)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[1, 13, 1, 9], [5, 22, 24, 5, 1, 13, 1, 1, 5, 1, 1, 1, 24, 5, 1, 13, 3, 4, 1, 1]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": []
    }
  ]
}