generating_paper_titles.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "generating-paper-titles.ipynb",
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kyrcha/deep-learning-pipelines/blob/master/generating_paper_titles.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DrNrVSkahAOt",
        "colab_type": "text"
      },
      "source": [
        "# Generating plausible paper titles with Recurrent Neural Networks\n",
        "\n",
        "In this notebook we will generate fictional paper titles using recurrent neural networks and more specifically LSTM. We reused some ideas and code from:\n",
        "- https://adventuresinmachinelearning.com/keras-lstm-tutorial/ (Keras stuff)\n",
        "- https://github.com/dennybritz/rnn-tutorial-rnnlm/blob/master/RNNLM.ipynb (Preprocessing stuff)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D4QWo-Iug6Qs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import csv\n",
        "import itertools\n",
        "import operator\n",
        "import numpy as np\n",
        "import nltk\n",
        "import sys\n",
        "from datetime import datetime\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ebCy5zRFiWjV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%capture\n",
        "# Download NLTK model data (you need to do this once)\n",
        "nltk.download(\"book\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZZItj8rzg_mV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Declare tokens to be used for unknown words, start and end of titles.\n",
        "unknown_token = \"UNKNOWN_TOKEN\"\n",
        "title_start_token = \"TITLE_START\"\n",
        "title_end_token = \"TITLE_END\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uf3IHMkgh2UD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Read the file\n",
        "with open('data/ieee-tnnls-titles.txt', 'r') as f:\n",
        "    text = f.read()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dlh8RrFkj2KB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Test what I've read\n",
        "#print(text)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d5ssgd3FjcPd",
        "colab_type": "text"
      },
      "source": [
        "## Explore the data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gVlfoxavjNzl",
        "colab_type": "code",
        "outputId": "7e86b58e-155c-4c4b-de4b-1778d87607d2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "print('Dataset Stats')\n",
        "print('Roughly the number of unique words: {}'.format(len({word: None for word in text.split()})))\n",
        "titles = text.splitlines()\n",
        "print('Number of titles: {}'.format(len(titles)))\n",
        "\n",
        "word_count_sentence = [len(title.split()) for title in titles]\n",
        "print('Average number of words in each title: {}'.format(np.average(word_count_sentence)))"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Dataset Stats\n",
            "Roughly the number of unique words: 2705\n",
            "Number of titles: 1207\n",
            "Average number of words in each title: 10.198011599005799\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DbCEW3lwmuz2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Tokenize\n",
        "from nltk.tokenize import sent_tokenize, word_tokenize\n",
        "sentences = itertools.chain(*[nltk.sent_tokenize(x.lower()) for x in titles])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c_QaTFvGoo6_",
        "colab_type": "code",
        "outputId": "782b2820-721f-475f-d226-0c68ca06a8f3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Append SENTENCE_START and SENTENCE_END\n",
        "tokenized_titles = [\"%s %s %s\" % (title_start_token, x, title_end_token) for x in sentences]\n",
        "print(\"Parsed %d sentences.\" % (len(tokenized_titles)))\n"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Parsed 1207 sentences.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TFJGXdbSrmfG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Tokenize the sentences into words\n",
        "tokenized_titles = [nltk.word_tokenize(title) for title in tokenized_titles]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s-VK_FUiS_C0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "final_title = []\n",
        "for title in tokenized_titles:\n",
        "  final_title.append([token for token in title if token.isalpha() or token == title_start_token or token == title_end_token])\n",
        "  \n",
        "tokenized_titles = final_title"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "efGmyMWzXT_d",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Test the titles\n",
        "#print(tokenized_titles)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "we4wL7CqWUPB",
        "colab_type": "code",
        "outputId": "444240ad-6750-4e18-bb13-6eb8a38d0b57",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Count the word frequencies\n",
        "word_freq = nltk.FreqDist(itertools.chain(*tokenized_titles))\n",
        "print(\"Found %d unique words tokens.\" % len(word_freq.items()))"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Found 2073 unique words tokens.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CqjMCVEyr0Y6",
        "colab_type": "code",
        "outputId": "a9838adf-7329-4571-ccc5-7de1cd03e684",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# Get the most common words and build index_to_word and word_to_index vectors\n",
        "vocabulary_size = 250\n",
        "vocab = word_freq.most_common(vocabulary_size-1)\n",
        "index_to_word = [x[0] for x in vocab]\n",
        "index_to_word.append(unknown_token)\n",
        "word_to_index = dict([(w,i) for i,w in enumerate(index_to_word)])\n",
        "\n",
        "print(\"Using vocabulary size %d.\" % vocabulary_size)\n",
        "print(\"The least frequent word in our vocabulary is '%s' and appeared %d times.\" % (vocab[-1][0], vocab[-1][1]))"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using vocabulary size 250.\n",
            "The least frequent word in our vocabulary is 'stable' and appeared 7 times.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xxh8vZ76SKbT",
        "colab_type": "code",
        "outputId": "13109c9d-ebba-4578-9dfa-1ec0d88b2914",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# What does the vocabulary looks like?\n",
        "vocab"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('TITLE_START', 1207),\n",
              " ('TITLE_END', 1207),\n",
              " ('for', 553),\n",
              " ('of', 400),\n",
              " ('with', 318),\n",
              " ('and', 286),\n",
              " ('neural', 284),\n",
              " ('learning', 267),\n",
              " ('networks', 253),\n",
              " ('a', 242),\n",
              " ('control', 186),\n",
              " ('systems', 145),\n",
              " ('in', 129),\n",
              " ('adaptive', 123),\n",
              " ('on', 118),\n",
              " ('network', 112),\n",
              " ('nonlinear', 104),\n",
              " ('the', 102),\n",
              " ('using', 98),\n",
              " ('via', 87),\n",
              " ('to', 82),\n",
              " ('data', 78),\n",
              " ('analysis', 71),\n",
              " ('deep', 70),\n",
              " ('based', 68),\n",
              " ('classification', 64),\n",
              " ('approach', 57),\n",
              " ('feature', 54),\n",
              " ('by', 52),\n",
              " ('synchronization', 51),\n",
              " ('an', 51),\n",
              " ('clustering', 50),\n",
              " ('dynamic', 49),\n",
              " ('robust', 49),\n",
              " ('image', 45),\n",
              " ('method', 42),\n",
              " ('stochastic', 42),\n",
              " ('algorithm', 42),\n",
              " ('model', 42),\n",
              " ('tracking', 41),\n",
              " ('stability', 41),\n",
              " ('online', 41),\n",
              " ('multiple', 38),\n",
              " ('estimation', 38),\n",
              " ('sparse', 37),\n",
              " ('selection', 37),\n",
              " ('optimization', 36),\n",
              " ('optimal', 36),\n",
              " ('recurrent', 35),\n",
              " ('design', 34),\n",
              " ('delays', 34),\n",
              " ('regression', 33),\n",
              " ('matrix', 32),\n",
              " ('distributed', 31),\n",
              " ('reinforcement', 29),\n",
              " ('detection', 28),\n",
              " ('dynamical', 28),\n",
              " ('state', 28),\n",
              " ('uncertain', 27),\n",
              " ('delayed', 27),\n",
              " ('output', 27),\n",
              " ('machine', 27),\n",
              " ('class', 27),\n",
              " ('programming', 27),\n",
              " ('efficient', 26),\n",
              " ('linear', 26),\n",
              " ('system', 26),\n",
              " ('representation', 26),\n",
              " ('visual', 26),\n",
              " ('kernel', 26),\n",
              " ('modeling', 26),\n",
              " ('recognition', 25),\n",
              " ('hierarchical', 25),\n",
              " ('global', 25),\n",
              " ('novel', 23),\n",
              " ('vector', 23),\n",
              " ('convolutional', 23),\n",
              " ('coupled', 23),\n",
              " ('new', 22),\n",
              " ('under', 22),\n",
              " ('from', 21),\n",
              " ('boolean', 21),\n",
              " ('training', 21),\n",
              " ('framework', 20),\n",
              " ('multiagent', 20),\n",
              " ('unknown', 19),\n",
              " ('markovian', 19),\n",
              " ('application', 19),\n",
              " ('reduction', 19),\n",
              " ('exponential', 19),\n",
              " ('complex', 19),\n",
              " ('delay', 18),\n",
              " ('generalized', 18),\n",
              " ('support', 18),\n",
              " ('switching', 18),\n",
              " ('structure', 18),\n",
              " ('spiking', 18),\n",
              " ('semisupervised', 17),\n",
              " ('machines', 17),\n",
              " ('discriminative', 17),\n",
              " ('time', 17),\n",
              " ('feedback', 17),\n",
              " ('factorization', 17),\n",
              " ('manifold', 17),\n",
              " ('object', 16),\n",
              " ('local', 16),\n",
              " ('constraints', 16),\n",
              " ('function', 16),\n",
              " ('stabilization', 16),\n",
              " ('input', 16),\n",
              " ('bayesian', 15),\n",
              " ('probabilistic', 15),\n",
              " ('consensus', 15),\n",
              " ('unsupervised', 15),\n",
              " ('random', 15),\n",
              " ('noise', 14),\n",
              " ('prediction', 14),\n",
              " ('heterogeneous', 14),\n",
              " ('regularized', 14),\n",
              " ('models', 14),\n",
              " ('problems', 14),\n",
              " ('regularization', 14),\n",
              " ('nonnegative', 14),\n",
              " ('constrained', 14),\n",
              " ('through', 13),\n",
              " ('memristive', 13),\n",
              " ('multiview', 13),\n",
              " ('quantized', 13),\n",
              " ('approximation', 13),\n",
              " ('applications', 13),\n",
              " ('space', 13),\n",
              " ('transfer', 13),\n",
              " ('graph', 13),\n",
              " ('information', 13),\n",
              " ('supervised', 12),\n",
              " ('tensor', 12),\n",
              " ('communication', 12),\n",
              " ('its', 12),\n",
              " ('identification', 12),\n",
              " ('classifiers', 12),\n",
              " ('sequential', 12),\n",
              " ('switched', 12),\n",
              " ('subspace', 12),\n",
              " ('coding', 12),\n",
              " ('performance', 12),\n",
              " ('impulsive', 12),\n",
              " ('saliency', 12),\n",
              " ('computing', 11),\n",
              " ('memory', 11),\n",
              " ('cooperative', 11),\n",
              " ('metric', 11),\n",
              " ('face', 11),\n",
              " ('active', 11),\n",
              " ('extraction', 11),\n",
              " ('ensemble', 11),\n",
              " ('projection', 11),\n",
              " ('temporal', 11),\n",
              " ('mode', 11),\n",
              " ('neurodynamic', 11),\n",
              " ('improved', 11),\n",
              " ('functions', 11),\n",
              " ('decomposition', 11),\n",
              " ('policy', 11),\n",
              " ('multilayer', 11),\n",
              " ('approximate', 11),\n",
              " ('dynamics', 11),\n",
              " ('algorithms', 11),\n",
              " ('inference', 10),\n",
              " ('features', 10),\n",
              " ('unified', 10),\n",
              " ('controller', 10),\n",
              " ('joint', 10),\n",
              " ('distance', 10),\n",
              " ('boosting', 10),\n",
              " ('segmentation', 10),\n",
              " ('jump', 10),\n",
              " ('iterative', 10),\n",
              " ('domain', 10),\n",
              " ('power', 10),\n",
              " ('gradient', 10),\n",
              " ('hybrid', 10),\n",
              " ('component', 10),\n",
              " ('cluster', 9),\n",
              " ('regulation', 9),\n",
              " ('editorial', 9),\n",
              " ('latent', 9),\n",
              " ('markov', 9),\n",
              " ('networked', 9),\n",
              " ('games', 9),\n",
              " ('fast', 9),\n",
              " ('methods', 9),\n",
              " ('hashing', 9),\n",
              " ('spectral', 9),\n",
              " ('study', 9),\n",
              " ('minimization', 8),\n",
              " ('knowledge', 8),\n",
              " ('mixture', 8),\n",
              " ('gaussian', 8),\n",
              " ('action', 8),\n",
              " ('structural', 8),\n",
              " ('collaborative', 8),\n",
              " ('multimodal', 8),\n",
              " ('event', 8),\n",
              " ('over', 8),\n",
              " ('label', 8),\n",
              " ('nonparametric', 8),\n",
              " ('solution', 8),\n",
              " ('iteration', 8),\n",
              " ('solving', 8),\n",
              " ('adaptation', 8),\n",
              " ('pinning', 8),\n",
              " ('extreme', 8),\n",
              " ('generalization', 8),\n",
              " ('mimo', 8),\n",
              " ('disturbances', 8),\n",
              " ('weighted', 8),\n",
              " ('sensor', 8),\n",
              " ('least', 8),\n",
              " ('quality', 8),\n",
              " ('improving', 8),\n",
              " ('processes', 8),\n",
              " ('artificial', 8),\n",
              " ('loss', 7),\n",
              " ('quadratic', 7),\n",
              " ('as', 7),\n",
              " ('topology', 7),\n",
              " ('special', 7),\n",
              " ('distributions', 7),\n",
              " ('nonconvex', 7),\n",
              " ('transformation', 7),\n",
              " ('multitask', 7),\n",
              " ('sequences', 7),\n",
              " ('process', 7),\n",
              " ('smart', 7),\n",
              " ('boltzmann', 7),\n",
              " ('concept', 7),\n",
              " ('mining', 7),\n",
              " ('adversarial', 7),\n",
              " ('flexible', 7),\n",
              " ('sliding', 7),\n",
              " ('embedding', 7),\n",
              " ('multiobjective', 7),\n",
              " ('extended', 7),\n",
              " ('fuzzy', 7),\n",
              " ('set', 7),\n",
              " ('implementation', 7),\n",
              " ('problem', 7),\n",
              " ('manipulators', 7),\n",
              " ('stable', 7)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZH9wQ6EvsK_S",
        "colab_type": "code",
        "outputId": "7e85ce65-1792-47c9-f66b-e0ccf464ccf7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "# Replace all words not in our vocabulary with the unknown token\n",
        "for i, sent in enumerate(tokenized_titles):\n",
        "    tokenized_titles[i] = [w if w in word_to_index else unknown_token for w in sent]\n",
        "\n",
        "print(\"\\nExample sentence: '%s'\" % titles[100])\n",
        "print(\"\\nExample sentence after Pre-processing: '%s'\" % tokenized_titles[100])"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Example sentence: 'Plume Tracing via Model-Free Reinforcement Learning Method'\n",
            "\n",
            "Example sentence after Pre-processing: '['TITLE_START', 'UNKNOWN_TOKEN', 'UNKNOWN_TOKEN', 'via', 'reinforcement', 'learning', 'method', 'TITLE_END']'\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zxk0MqZjw2g6",
        "colab_type": "code",
        "outputId": "481ee545-defa-49b9-f1d4-2f2b36313868",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        }
      },
      "source": [
        "# Just an example of how to make a training dataset without a generator\n",
        "X_train = np.asarray([[word_to_index[w] for w in sent[:-1]] for sent in tokenized_titles])\n",
        "y_train = np.asarray([[word_to_index[w] for w in sent[1:]] for sent in tokenized_titles])\n",
        "print(X_train[0])\n",
        "print(y_train[0])\n",
        "# Print an training data example\n",
        "x_example, y_example = X_train[0], y_train[0]\n",
        "print(\"x:\\n%s\\n%s\" % (\" \".join([index_to_word[x] for x in x_example]), x_example))\n",
        "print(\"\\ny:\\n%s\\n%s\" % (\" \".join([index_to_word[x] for x in y_example]), y_example))"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[0, 104, 55, 4, 23, 7, 9, 249]\n",
            "[104, 55, 4, 23, 7, 9, 249, 1]\n",
            "x:\n",
            "TITLE_START object detection with deep learning a UNKNOWN_TOKEN\n",
            "[0, 104, 55, 4, 23, 7, 9, 249]\n",
            "\n",
            "y:\n",
            "object detection with deep learning a UNKNOWN_TOKEN TITLE_END\n",
            "[104, 55, 4, 23, 7, 9, 249, 1]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eJRP6yOEyVEc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class KerasBatchGenerator(object):\n",
        "\n",
        "  def __init__(self, data, num_steps, batch_size, vocabulary, skip_step=5):\n",
        "    self.data = data\n",
        "    self.num_steps = num_steps\n",
        "    self.batch_size = batch_size\n",
        "    self.vocabulary = vocabulary\n",
        "    # this will track the progress of the batches sequentially through the\n",
        "    # data set - once the data reaches the end of the data set it will reset\n",
        "    # back to zero\n",
        "    self.current_idx = 0\n",
        "    # skip_step is the number of words which will be skipped before the next\n",
        "    # batch is skimmed from the data set\n",
        "    self.skip_step = skip_step\n",
        "\n",
        "  def generate(self):\n",
        "    x = np.zeros((self.batch_size, self.num_steps))\n",
        "    y = np.zeros((self.batch_size, self.num_steps, self.vocabulary))\n",
        "    while True:\n",
        "      i = 0\n",
        "      while i < self.batch_size:\n",
        "        # I don't want to see in x a title end token to predict y \n",
        "        if self.current_idx < len(self.data) and self.data[self.current_idx] == word_to_index[title_end_token]:\n",
        "          self.current_idx += self.skip_step\n",
        "        if self.current_idx + self.num_steps >= len(self.data):\n",
        "          # reset the index back to the start of the data set\n",
        "          self.current_idx = 0\n",
        "        x[i, :] = self.data[self.current_idx:self.current_idx + self.num_steps]\n",
        "        temp_y = self.data[self.current_idx + 1:self.current_idx + self.num_steps + 1]\n",
        "        # convert all of temp_y into a one hot representation\n",
        "        y[i, :, :] = to_categorical(temp_y, num_classes=self.vocabulary)\n",
        "        self.current_idx += self.skip_step\n",
        "        i += 1\n",
        "      yield x, y\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jas8V5FSy6Ed",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "num_steps = 1\n",
        "skip_step = 1\n",
        "batch_size = 10\n",
        "\n",
        "# set seeds for reproducibility\n",
        "from numpy.random import seed\n",
        "seed(123)\n",
        "from tensorflow import set_random_seed\n",
        "set_random_seed(234)\n",
        "\n",
        "# Create the training data\n",
        "# A concatenation of all tokens as integers (indices)\n",
        "X = list(itertools.chain(*np.asarray([[word_to_index[w] for w in sent] for sent in tokenized_titles])))\n",
        "# Create 2 batch generators out of the concatenation\n",
        "train_data_generator = KerasBatchGenerator(X[:10000], num_steps, batch_size, vocabulary_size, skip_step)\n",
        "valid_data_generator = KerasBatchGenerator(X[10001:], num_steps, batch_size, vocabulary_size, skip_step)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w_IKmz1rzZyc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Activation, Embedding, Dropout, TimeDistributed\n",
        "from keras.layers import LSTM\n",
        "from keras.optimizers import Adam\n",
        "from keras.utils import to_categorical\n",
        "from keras.callbacks import ModelCheckpoint\n",
        "\n",
        "hidden_size = 250\n",
        "\n",
        "model = Sequential()\n",
        "model.add(Embedding(vocabulary_size, hidden_size, input_length=num_steps))\n",
        "model.add(LSTM(hidden_size, return_sequences=True))\n",
        "model.add(LSTM(hidden_size, return_sequences=True))\n",
        "model.add(Dropout(rate=0.5))\n",
        "model.add(TimeDistributed(Dense(vocabulary_size)))\n",
        "model.add(Activation('softmax'))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4FZ-Sa19z-QO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['categorical_accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nBeNxC6M0G4a",
        "colab_type": "code",
        "outputId": "a72f043c-8fb2-45aa-c73d-b46728236af9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        }
      },
      "source": [
        "num_epochs = 10\n",
        "\n",
        "model.fit_generator(train_data_generator.generate(), len(X[:10000])//(batch_size*num_steps), num_epochs,\n",
        "                        validation_data=valid_data_generator.generate(),\n",
        "                        validation_steps=len(X[10001:])//(batch_size*num_steps))"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "1000/1000 [==============================] - 13s 13ms/step - loss: 3.8697 - categorical_accuracy: 0.2929 - val_loss: 3.5585 - val_categorical_accuracy: 0.3143\n",
            "Epoch 2/10\n",
            "1000/1000 [==============================] - 12s 12ms/step - loss: 3.4911 - categorical_accuracy: 0.3258 - val_loss: 3.4157 - val_categorical_accuracy: 0.3362\n",
            "Epoch 3/10\n",
            "1000/1000 [==============================] - 12s 12ms/step - loss: 3.3471 - categorical_accuracy: 0.3509 - val_loss: 3.3525 - val_categorical_accuracy: 0.3526\n",
            "Epoch 4/10\n",
            "1000/1000 [==============================] - 12s 12ms/step - loss: 3.2879 - categorical_accuracy: 0.3605 - val_loss: 3.2804 - val_categorical_accuracy: 0.3578\n",
            "Epoch 5/10\n",
            "1000/1000 [==============================] - 12s 12ms/step - loss: 3.2310 - categorical_accuracy: 0.3608 - val_loss: 3.3125 - val_categorical_accuracy: 0.3500\n",
            "Epoch 6/10\n",
            "1000/1000 [==============================] - 12s 12ms/step - loss: 3.1837 - categorical_accuracy: 0.3613 - val_loss: 3.2606 - val_categorical_accuracy: 0.3622\n",
            "Epoch 7/10\n",
            "1000/1000 [==============================] - 12s 12ms/step - loss: 3.1388 - categorical_accuracy: 0.3619 - val_loss: 3.2551 - val_categorical_accuracy: 0.3622\n",
            "Epoch 8/10\n",
            "1000/1000 [==============================] - 12s 12ms/step - loss: 3.0918 - categorical_accuracy: 0.3712 - val_loss: 3.2519 - val_categorical_accuracy: 0.3617\n",
            "Epoch 9/10\n",
            "1000/1000 [==============================] - 12s 12ms/step - loss: 3.0481 - categorical_accuracy: 0.3707 - val_loss: 3.2304 - val_categorical_accuracy: 0.3680\n",
            "Epoch 10/10\n",
            "1000/1000 [==============================] - 12s 12ms/step - loss: 3.0428 - categorical_accuracy: 0.3683 - val_loss: 3.2125 - val_categorical_accuracy: 0.3701\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7fb9ab639470>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VFDlko750Rxz",
        "colab_type": "code",
        "outputId": "0a5291e1-7b7a-4092-b47c-cfa382ff2913",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 527
        }
      },
      "source": [
        "def generate_title(model):\n",
        "    # We start the sentence with the start token\n",
        "    new_title = [word_to_index[title_start_token]]\n",
        "    # Repeat until we get an end token\n",
        "    while not new_title[-1] == word_to_index[title_end_token]:\n",
        "        x = np.zeros((1,1))\n",
        "        x[0, :] = new_title[-1]\n",
        "        next_word_probs = model.predict(x)[0][0]\n",
        "        sampled_word = word_to_index[unknown_token]\n",
        "        # We don't want to sample unknown words\n",
        "        while sampled_word == word_to_index[unknown_token]:\n",
        "            samples = np.random.multinomial(1, next_word_probs)\n",
        "            sampled_word = np.argmax(samples)\n",
        "        new_title.append(sampled_word)\n",
        "    title_str = [index_to_word[x] for x in new_title[1:-1]]\n",
        "    return title_str\n",
        "\n",
        "num_sentences = 30\n",
        "senten_min_length = 7\n",
        "senten_max_length = 15\n",
        "\n",
        "for i in range(num_sentences):\n",
        "    sent = []\n",
        "    # We want long sentences, not sentences with one or two words\n",
        "    while len(sent) < senten_min_length or len(sent) > senten_max_length:\n",
        "        sent = generate_title(model)\n",
        "    print(\" \".join(sent))"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "a flexible neural networks with deep output regulation for support vector optimization\n",
            "output regulation of linear systems using approximate dynamic programming data\n",
            "dynamic regularized constrained on adaptive neural networks\n",
            "a unified embedding of multiple delays for efficient regularization using a switching of hierarchical problem\n",
            "a exponential synchronization of the the learning based on bayesian learning\n",
            "feature selection by an classification with domain selection for linear optimization for the an synchronization\n",
            "new unified and based on communication constraints for the an deep learning control\n",
            "a recurrent neural networks with input delay\n",
            "neural networks approximation data with switching methods\n",
            "a weighted linear systems through neural data\n",
            "a novel boosting based on neural networks for networked uncertain classification\n",
            "nonlinear systems on uncertain systems based on neural networks\n",
            "face classification using neural networks with uncertain systems\n",
            "multiview optimal nonlinear systems and improved convolutional neural networks\n",
            "class of multiagent systems and exponential synchronization of neural networks\n",
            "sparse factorization with time on neural networks based on a deep learning with switching\n",
            "class of online adaptive output feedback functions of systems\n",
            "adaptive neural networks by class of deep convolutional neural networks of multiple delays\n",
            "synchronization of an an a coupled neural networks with an synchronization of neural network\n",
            "nonlinear networked uncertain and efficient adaptive networks\n",
            "recurrent nonlinear multiagent systems with adaptive adaptive neural networks with time selection\n",
            "improved application to adaptive neural networks with impulsive control of delay for linear multiagent systems\n",
            "random on the a synchronization of control for gradient\n",
            "state estimation for semisupervised learning games with input clustering\n",
            "multiview boosting a unified performance of joint control of adaptation\n",
            "a class of a coupled neural networks for nonlinear stochastic switching\n",
            "improved neural networks with a class for neural networks and clustering\n",
            "regularized nonlinear systems with uncertain nonlinear systems\n",
            "stability analysis machines for face a support vector control of distributed adaptive delayed neural networks\n",
            "a stochastic nonlinear switched systems with delay and a regularized approximation\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}