nlp_embeddings_and_sentiment.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "l09c04_nlp_embeddings_and_sentiment.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3.8.5 64-bit"
    },
    "interpreter": {
      "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9uBA1i1BbiJn"
      },
      "source": [
        "# Word Embeddings and Sentiment"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1iLe4E0dB7tj"
      },
      "source": [
        "In this colab, you'll work with word embeddings and train a basic neural network to predict text sentiment. At the end, you'll be able to visualize how the network sees the related sentiment of each word in the dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wqvz1jVgbwIN"
      },
      "source": [
        "## Import TensorFlow and related functions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XIG52aKPdpux"
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pazU5OmxehIA"
      },
      "source": [
        "## Get the dataset\n",
        "\n",
        "We're going to use a dataset containing Amazon and Yelp reviews, with their related sentiment (1 for positive, 0 for negative). This dataset was originally extracted from [here](https://www.kaggle.com/marklvl/sentiment-labelled-sentences-data-set)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qpwQT2E9ez5B"
      },
      "source": [
        "!wget --no-check-certificate \\\n",
        "    -O /tmp/sentiment.csv 'https://drive.google.com/uc?id=13ySLC_ue6Umt9RJYSeM2t-V0kCv-4C-P'"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2021-06-25 18:52:37--  https://drive.google.com/uc?id=13ySLC_ue6Umt9RJYSeM2t-V0kCv-4C-P\n",
            "Resolving drive.google.com (drive.google.com)... 172.217.160.206, 2404:6800:4009:80b::200e\n",
            "Connecting to drive.google.com (drive.google.com)|172.217.160.206|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Moved Temporarily\n",
            "Location: https://doc-08-ak-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/rlkihrtuad9f4joeqrehv8tt072mm6nj/1624627350000/11118900490791463723/*/13ySLC_ue6Umt9RJYSeM2t-V0kCv-4C-P [following]\n",
            "Warning: wildcards not supported in HTTP.\n",
            "--2021-06-25 18:52:38--  https://doc-08-ak-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/rlkihrtuad9f4joeqrehv8tt072mm6nj/1624627350000/11118900490791463723/*/13ySLC_ue6Umt9RJYSeM2t-V0kCv-4C-P\n",
            "Resolving doc-08-ak-docs.googleusercontent.com (doc-08-ak-docs.googleusercontent.com)... 142.250.192.97, 2404:6800:4009:82a::2001\n",
            "Connecting to doc-08-ak-docs.googleusercontent.com (doc-08-ak-docs.googleusercontent.com)|142.250.192.97|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 127831 (125K) [text/csv]\n",
            "Saving to: ‘/tmp/sentiment.csv’\n",
            "\n",
            "/tmp/sentiment.csv  100%[===================>] 124.83K   606KB/s    in 0.2s    \n",
            "\n",
            "2021-06-25 18:52:39 (606 KB/s) - ‘/tmp/sentiment.csv’ saved [127831/127831]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6Zvp9NScfMnw"
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "dataset = pd.read_csv('/tmp/sentiment.csv')\n",
        "\n",
        "sentences = dataset['text'].tolist()\n",
        "labels = dataset['sentiment'].tolist()\n",
        "\n",
        "# Separate out the sentences and labels into training and test sets\n",
        "training_size = int(len(sentences) * 0.8)\n",
        "\n",
        "training_sentences = sentences[0:training_size]\n",
        "testing_sentences = sentences[training_size:]\n",
        "training_labels = labels[0:training_size]\n",
        "testing_labels = labels[training_size:]\n",
        "\n",
        "# Make labels into numpy arrays for use with the network later\n",
        "training_labels_final = np.array(training_labels)\n",
        "testing_labels_final = np.array(testing_labels)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NHpvqaSigcST"
      },
      "source": [
        "## Tokenize the dataset\n",
        "\n",
        "Tokenize the dataset, including padding and OOV"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "78icewYRgfxh"
      },
      "source": [
        "vocab_size = 1000\n",
        "embedding_dim = 16\n",
        "max_length = 100\n",
        "trunc_type='post'\n",
        "padding_type='post'\n",
        "oov_tok = \"<OOV>\"\n",
        "\n",
        "\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "\n",
        "tokenizer = Tokenizer(num_words = vocab_size, oov_token=oov_tok)\n",
        "tokenizer.fit_on_texts(training_sentences)\n",
        "word_index = tokenizer.word_index\n",
        "sequences = tokenizer.texts_to_sequences(training_sentences)\n",
        "padded = pad_sequences(sequences,maxlen=max_length, padding=padding_type, \n",
        "                       truncating=trunc_type)\n",
        "\n",
        "testing_sequences = tokenizer.texts_to_sequences(testing_sentences)\n",
        "testing_padded = pad_sequences(testing_sequences,maxlen=max_length, \n",
        "                               padding=padding_type, truncating=trunc_type)"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q4yIEk_8kszh"
      },
      "source": [
        "## Review a Sequence\n",
        "\n",
        "Let's quickly take a look at one of the padded sequences to ensure everything above worked appropriately."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JTU3FmVGk100"
      },
      "source": [
        "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
        "\n",
        "def decode_review(text):\n",
        "    return ' '.join([reverse_word_index.get(i, '?') for i in text])\n",
        "\n",
        "print(decode_review(padded[1]))\n",
        "print(training_sentences[1])"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "good case excellent value ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?\nGood case Excellent value.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RI91liJnlA92"
      },
      "source": [
        "## Train a Basic Sentiment Model with Embeddings"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bBMgzp-_lMTp"
      },
      "source": [
        "# Build a basic sentiment network\n",
        "# Note the embedding layer is first, \n",
        "# and the output is only 1 node as it is either 0 or 1 (negative or positive)\n",
        "model = tf.keras.Sequential([\n",
        "    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),\n",
        "    tf.keras.layers.Flatten(),\n",
        "    tf.keras.layers.Dense(6, activation='relu'),\n",
        "    tf.keras.layers.Dense(1, activation='sigmoid')\n",
        "])\n",
        "model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])\n",
        "model.summary()"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n_________________________________________________________________\nLayer (type)                 Output Shape              Param #   \n=================================================================\nembedding (Embedding)        (None, 100, 16)           16000     \n_________________________________________________________________\nflatten (Flatten)            (None, 1600)              0         \n_________________________________________________________________\ndense (Dense)                (None, 6)                 9606      \n_________________________________________________________________\ndense_1 (Dense)              (None, 1)                 7         \n=================================================================\nTotal params: 25,613\nTrainable params: 25,613\nNon-trainable params: 0\n_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pfl1W-zVldpn"
      },
      "source": [
        "num_epochs = 10\n",
        "model.fit(padded, training_labels_final, epochs=num_epochs, validation_data=(testing_padded, testing_labels_final))"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "50/50 [==============================] - 1s 7ms/step - loss: 0.6925 - accuracy: 0.5116 - val_loss: 0.6981 - val_accuracy: 0.4110\n",
            "Epoch 2/10\n",
            "50/50 [==============================] - 0s 3ms/step - loss: 0.6909 - accuracy: 0.5229 - val_loss: 0.6998 - val_accuracy: 0.4110\n",
            "Epoch 3/10\n",
            "50/50 [==============================] - 0s 3ms/step - loss: 0.6855 - accuracy: 0.5229 - val_loss: 0.6982 - val_accuracy: 0.4110\n",
            "Epoch 4/10\n",
            "50/50 [==============================] - 0s 3ms/step - loss: 0.6662 - accuracy: 0.5907 - val_loss: 0.6712 - val_accuracy: 0.6015\n",
            "Epoch 5/10\n",
            "50/50 [==============================] - 0s 3ms/step - loss: 0.6001 - accuracy: 0.7439 - val_loss: 0.6032 - val_accuracy: 0.7193\n",
            "Epoch 6/10\n",
            "50/50 [==============================] - 0s 3ms/step - loss: 0.4938 - accuracy: 0.8544 - val_loss: 0.5493 - val_accuracy: 0.7569\n",
            "Epoch 7/10\n",
            "50/50 [==============================] - 0s 3ms/step - loss: 0.3845 - accuracy: 0.8895 - val_loss: 0.5060 - val_accuracy: 0.7644\n",
            "Epoch 8/10\n",
            "50/50 [==============================] - 0s 3ms/step - loss: 0.2956 - accuracy: 0.9234 - val_loss: 0.4791 - val_accuracy: 0.7744\n",
            "Epoch 9/10\n",
            "50/50 [==============================] - 0s 3ms/step - loss: 0.2305 - accuracy: 0.9391 - val_loss: 0.4557 - val_accuracy: 0.7769\n",
            "Epoch 10/10\n",
            "50/50 [==============================] - 0s 3ms/step - loss: 0.1867 - accuracy: 0.9485 - val_loss: 0.4904 - val_accuracy: 0.7469\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f7dcc15e8b0>"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GjMZ4ZFQl_48"
      },
      "source": [
        "## Get files for visualizing the network\n",
        "\n",
        "The code below will download two files for visualizing how your network \"sees\" the sentiment related to each word. Head to http://projector.tensorflow.org/ and load these files, then click the \"Sphereize\" checkbox."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S2lB46FirAVx"
      },
      "source": [
        "# First get the weights of the embedding layer\n",
        "e = model.layers[0]\n",
        "weights = e.get_weights()[0]\n",
        "print(weights.shape) # shape: (vocab_size, embedding_dim)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(1000, 16)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xcha0oGemHX2"
      },
      "source": [
        "import io\n",
        "\n",
        "# Write out the embedding vectors and metadata\n",
        "out_v = io.open('vecs.tsv', 'w', encoding='utf-8')\n",
        "out_m = io.open('meta.tsv', 'w', encoding='utf-8')\n",
        "for word_num in range(1, vocab_size):\n",
        "  word = reverse_word_index[word_num]\n",
        "  embeddings = weights[word_num]\n",
        "  out_m.write(word + \"\\n\")\n",
        "  out_v.write('\\t'.join([str(x) for x in embeddings]) + \"\\n\")\n",
        "out_v.close()\n",
        "out_m.close()"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g-Q6ALywmWVz"
      },
      "source": [
        "# Download the files\n",
        "try:\n",
        "  from google.colab import files\n",
        "except ImportError:\n",
        "  pass\n",
        "else:\n",
        "  files.download('vecs.tsv')\n",
        "  files.download('meta.tsv')"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GNoxfY-i3Ir1"
      },
      "source": [
        "## Predicting Sentiment in New Reviews\n",
        "\n",
        "Now that you've trained and visualized your network, take a look below at how we can predict sentiment in new reviews the network has never seen before."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QXtfw-OY3WoZ"
      },
      "source": [
        "# Use the model to predict a review   \n",
        "fake_reviews = ['I love this phone', 'I hate spaghetti', \n",
        "                'Everything was cold',\n",
        "                'Everything was hot exactly as I wanted', \n",
        "                'Everything was green', \n",
        "                'the host seated us immediately',\n",
        "                'they gave us free chocolate cake', \n",
        "                'not sure about the wilted flowers on the table',\n",
        "                'only works when I stand on tippy toes', \n",
        "                'does not work when I stand on my head']\n",
        "\n",
        "print(fake_reviews) \n",
        "\n",
        "# Create the sequences\n",
        "padding_type='post'\n",
        "sample_sequences = tokenizer.texts_to_sequences(fake_reviews)\n",
        "fakes_padded = pad_sequences(sample_sequences, padding=padding_type, maxlen=max_length)           \n",
        "\n",
        "print('\\nHOT OFF THE PRESS! HERE ARE SOME NEWLY MINTED, ABSOLUTELY GENUINE REVIEWS!\\n')              \n",
        "\n",
        "classes = model.predict(fakes_padded)\n",
        "\n",
        "# The closer the class is to 1, the more positive the review is deemed to be\n",
        "for x in range(len(fake_reviews)):\n",
        "  print(fake_reviews[x])\n",
        "  print(classes[x])\n",
        "  print('\\n')\n",
        "\n",
        "# Try adding reviews of your own\n",
        "# Add some negative words (such as \"not\") to the good reviews and see what happens\n",
        "# For example:\n",
        "# they gave us free chocolate cake and did not charge us"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['I love this phone', 'I hate spaghetti', 'Everything was cold', 'Everything was hot exactly as I wanted', 'Everything was green', 'the host seated us immediately', 'they gave us free chocolate cake', 'not sure about the wilted flowers on the table', 'only works when I stand on tippy toes', 'does not work when I stand on my head']\n\nHOT OFF THE PRESS! HERE ARE SOME NEWLY MINTED, ABSOLUTELY GENUINE REVIEWS!\n\nI love this phone\n[0.97886586]\n\n\nI hate spaghetti\n[0.09715581]\n\n\nEverything was cold\n[0.5105124]\n\n\nEverything was hot exactly as I wanted\n[0.79080886]\n\n\nEverything was green\n[0.6139845]\n\n\nthe host seated us immediately\n[0.7759546]\n\n\nthey gave us free chocolate cake\n[0.86759305]\n\n\nnot sure about the wilted flowers on the table\n[0.03722844]\n\n\nonly works when I stand on tippy toes\n[0.95355314]\n\n\ndoes not work when I stand on my head\n[0.01605681]\n\n\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": []
    }
  ]
}