Blogpost_generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ODfOC4Sm077z"
   },
   "source": [
    "# Learning and Generating text from Blog Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data, remove punctuations & tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 124
    },
    "colab_type": "code",
    "id": "RJpS--M9xtzp",
    "outputId": "3e82fb67-0489-44d3-9c8d-bb78a80082e1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "i took out reading from my list of hobbies a couple of years ago. i remember going to public library to borrow algorithms texts last year. the year before that i browsed the shelves for books on devel\n",
      "i took out reading from my list of hobbies a couple of years ago  i remember going to public library to borrow algorithms texts last year  the year before that i browsed the shelves for books on devel\n",
      "['i', 'took', 'out', 'reading', 'from', 'my', 'list', 'of', 'hobbies', 'a', 'couple', 'of', 'years', 'ago', 'i', 'remember', 'going', 'to', 'public', 'library', 'to', 'borrow', 'algorithms', 'texts', 'last', 'year', 'the', 'year', 'before', 'that', 'i', 'browsed', 'the', 'shelves', 'for', 'books', 'on', 'development', 'studies', 'and', 'sociology', 'this', 'year', 'ive', 'visited', 'only', 'on', 'behalf', 'of', 'my', 'mother', 'to', 'get', 'her', 'yoga', 'books', 'from', 'the', 'ground', 'floor', 'main', 'section', 'it', 'is', 'quite', 'probable', 'that', 'i', 'was', 'never', 'an', 'avid', 'reader', 'i', 'mostly', 'read', 'what', 'achu', 'annan', 'suggested', 'bought', 'me', 'an', 'omnibus', 'for', 'every', 'birthday', 'when', 'i', 'was', 'young', 'a', 'lot', 'of', 'paulo', 'coelho', 'from', 'our', 'school', 'library', 'those', 'standard', 'books', 'almost', 'everyone', 'reads', 'kite', 'runner', 'to', 'kill', 'a', 'mockingbird', 'jd', 'salinger', 'he', 'bought', 'me', 'cats', 'eye', 'in', 'my', 'final', 'year', 'and', 'it', 'was', 'great', 'for', 'some', 'reason', 'i', 'really', 'liked', 'reading', 'books', 'written', 'by', 'women', 'there', 'was', 'this', 'book', 'by', 'a', 'lady', 'i', 'forget', 'the', 'name', 'but', 'im', 'sure', 'i', 'have', 'it', 'written', 'down', 'in', 'one', 'of', 'my', 'diaries', 'it', 'talked', 'of', 'suitors', 'and', 'hemming', 'lines', 'dainty', 'glass', 'cups', 'taken', 'out', 'in', 'the', 'afternoons', 'with', 'designs', 'on', 'them', 'extra', 'linen', 'stored', 'away', 'in', 'cupboards', 'and', 'wardrobes', 'recipe', 'books', 'with', 'extra', 'scribblings', 'in', 'pencil', 'on', 'margins', 'and', 'double']\n",
      "Total Tokens: 32523\n",
      "Unique Tokens: 5921\n"
     ]
    }
   ],
   "source": [
    "# load doc into memory\n",
    "def load_doc(filename):\n",
    "    # open the file as read only\n",
    "    file = open(filename, 'r')\n",
    "    # read all text\n",
    "    text = file.read()\n",
    "    # close the file\n",
    "    file.close()\n",
    "    return text\n",
    "\n",
    "# load document\n",
    "in_filename = 'blogdata_nonextline.txt'\n",
    "doc = load_doc(in_filename)\n",
    "print(doc[:200])\n",
    "\n",
    "def clean_doc(doc):\n",
    "  #print(doc)\n",
    "  doc = doc.replace('.', ' ')\n",
    "  print(doc[:200])\n",
    "  tokens = doc.split(' ') \n",
    "  tokens = [word for word in tokens if word.isalpha()]\n",
    "  tokens = [word.lower() for word in tokens]\n",
    "  return tokens\n",
    "  \n",
    "tokens = clean_doc(doc)\n",
    "print(tokens[:200])\n",
    "print('Total Tokens: %d' % len(tokens))\n",
    "print('Unique Tokens: %d' % len(set(tokens)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Aand as you can see here, my vocabulary isn't extensive. \n",
    "A total of 5921 unique tokens of which at least 150-200 must be Malayalam (my mother tongue) words :D So much room to (and must) improve my writing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating sequences from the text data\n",
    "\n",
    "Sequences of (20+1) words built from the entire text data. Since all punctuations have been removed, it has become a single piece of text, and sequences are generated from it. Sample sequences are displayed further below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "h3uGmSjNR0cc",
    "outputId": "bde1dbd9-88a1-4454-c153-5295a5f124ad"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Sequences: 32502\n"
     ]
    }
   ],
   "source": [
    "length = 20 + 1\n",
    "sequences = list()\n",
    "for i in range(length, len(tokens)):\n",
    "    # select sequence of tokens\n",
    "    seq = tokens[i-length:i]\n",
    "    # convert into a line\n",
    "    line = ' '.join(seq)\n",
    "    # store\n",
    "    sequences.append(line)\n",
    "print('Total Sequences: %d' % len(sequences))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VeuRQo22VQcZ"
   },
   "outputs": [],
   "source": [
    "def save_doc(lines, filename):\n",
    "    data = '\\n'.join(lines)\n",
    "    file = open(filename, 'w')\n",
    "    file.write(data)\n",
    "    file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aNnqhv7SVRY7"
   },
   "outputs": [],
   "source": [
    "# save sequences to file\n",
    "\n",
    "out_filename = 'blogdata_nonl_sequences.txt'\n",
    "save_doc(sequences, out_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sample Sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 55
    },
    "colab_type": "code",
    "id": "4jUFvSq-VXC0",
    "outputId": "0f6fbce8-b230-47ef-d908-e865c11cce12"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['i took out reading from my list of hobbies a couple of years ago i remember going to public library to', 'took out reading from my list of hobbies a couple of years ago i remember going to public library to borrow', 'out reading from my list of hobbies a couple of years ago i remember going to public library to borrow algorithms', 'reading from my list of hobbies a couple of years ago i remember going to public library to borrow algorithms texts', 'from my list of hobbies a couple of years ago i remember going to public library to borrow algorithms texts last', 'my list of hobbies a couple of years ago i remember going to public library to borrow algorithms texts last year', 'list of hobbies a couple of years ago i remember going to public library to borrow algorithms texts last year the', 'of hobbies a couple of years ago i remember going to public library to borrow algorithms texts last year the year', 'hobbies a couple of years ago i remember going to public library to borrow algorithms texts last year the year before', 'a couple of years ago i remember going to public library to borrow algorithms texts last year the year before that']\n"
     ]
    }
   ],
   "source": [
    "print(sequences[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kyQXLGR-Vu4o"
   },
   "source": [
    "#  Building the Language Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gUQ9L1xQVxh9"
   },
   "outputs": [],
   "source": [
    "\n",
    "from numpy import array\n",
    "from pickle import dump\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.utils import to_categorical\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import LSTM\n",
    "from keras.layers import Embedding\n",
    "\n",
    "\n",
    "\n",
    "# load doc into memory\n",
    "def load_doc(filename):\n",
    "    # open the file as read only\n",
    "    file = open(filename, 'r')\n",
    "    # read all text\n",
    "    text = file.read()\n",
    "    # close the file\n",
    "    file.close()\n",
    "    return text\n",
    " \n",
    "# load\n",
    "in_filename = 'blogdata_nonl_sequences.txt'\n",
    "doc = load_doc(in_filename)\n",
    "lines = doc.split('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 7280
    },
    "colab_type": "code",
    "id": "1Vr6RyzxWI8J",
    "outputId": "8b5c0064-4ee2-417e-84e8-02759b36b74c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_3 (Embedding)      (None, 20, 50)            296100    \n",
      "_________________________________________________________________\n",
      "lstm_5 (LSTM)                (None, 20, 100)           60400     \n",
      "_________________________________________________________________\n",
      "lstm_6 (LSTM)                (None, 100)               80400     \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 100)               10100     \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 5922)              598122    \n",
      "=================================================================\n",
      "Total params: 1,045,122\n",
      "Trainable params: 1,045,122\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Epoch 1/200\n",
      "32502/32502 [==============================] - 49s 2ms/step - loss: 7.2430 - acc: 0.0434\n",
      "Epoch 2/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 6.8108 - acc: 0.0441\n",
      "Epoch 3/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 6.6993 - acc: 0.0442\n",
      "Epoch 4/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 6.5536 - acc: 0.0522\n",
      "Epoch 5/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 6.4542 - acc: 0.0552\n",
      "Epoch 6/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 6.3655 - acc: 0.0561\n",
      "Epoch 7/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 6.2838 - acc: 0.0592\n",
      "Epoch 8/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 6.1990 - acc: 0.0617\n",
      "Epoch 9/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 6.1262 - acc: 0.0649\n",
      "Epoch 10/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 6.0470 - acc: 0.0674\n",
      "Epoch 11/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 5.9709 - acc: 0.0697\n",
      "Epoch 12/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 5.8992 - acc: 0.0712\n",
      "Epoch 13/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 5.8317 - acc: 0.0749\n",
      "Epoch 14/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 5.7685 - acc: 0.0782\n",
      "Epoch 15/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 5.7074 - acc: 0.0811\n",
      "Epoch 16/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 5.6512 - acc: 0.0836\n",
      "Epoch 17/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 5.5934 - acc: 0.0869\n",
      "Epoch 18/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 5.5391 - acc: 0.0900\n",
      "Epoch 19/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 5.4825 - acc: 0.0931\n",
      "Epoch 20/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 5.4264 - acc: 0.0949\n",
      "Epoch 21/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 5.3670 - acc: 0.0984\n",
      "Epoch 22/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 5.3064 - acc: 0.0999\n",
      "Epoch 23/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 5.2450 - acc: 0.1032\n",
      "Epoch 24/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 5.1877 - acc: 0.1046\n",
      "Epoch 25/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 5.1277 - acc: 0.1068\n",
      "Epoch 26/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 5.0646 - acc: 0.1087\n",
      "Epoch 27/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 5.0012 - acc: 0.1105\n",
      "Epoch 28/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 4.9461 - acc: 0.1119\n",
      "Epoch 29/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 4.8873 - acc: 0.1138\n",
      "Epoch 30/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 4.8272 - acc: 0.1167\n",
      "Epoch 31/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 4.7688 - acc: 0.1196\n",
      "Epoch 32/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 4.7143 - acc: 0.1222\n",
      "Epoch 33/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 4.6593 - acc: 0.1252\n",
      "Epoch 34/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 4.6025 - acc: 0.1293\n",
      "Epoch 35/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 4.5546 - acc: 0.1301\n",
      "Epoch 36/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 4.4965 - acc: 0.1341\n",
      "Epoch 37/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 4.4520 - acc: 0.1390\n",
      "Epoch 38/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 4.3986 - acc: 0.1436\n",
      "Epoch 39/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 4.3558 - acc: 0.1467\n",
      "Epoch 40/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 4.3073 - acc: 0.1520\n",
      "Epoch 41/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 4.2575 - acc: 0.1578\n",
      "Epoch 42/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 4.2112 - acc: 0.1596\n",
      "Epoch 43/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 4.1649 - acc: 0.1649\n",
      "Epoch 44/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 4.1273 - acc: 0.1717\n",
      "Epoch 45/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 4.0836 - acc: 0.1761\n",
      "Epoch 46/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 4.0463 - acc: 0.1804\n",
      "Epoch 47/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 4.0075 - acc: 0.1843\n",
      "Epoch 48/200\n",
      "32502/32502 [==============================] - 49s 1ms/step - loss: 3.9695 - acc: 0.1892\n",
      "Epoch 49/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 3.9323 - acc: 0.1948\n",
      "Epoch 50/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.8925 - acc: 0.1996\n",
      "Epoch 51/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.8604 - acc: 0.2056\n",
      "Epoch 52/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.8242 - acc: 0.2098\n",
      "Epoch 53/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.7928 - acc: 0.2138\n",
      "Epoch 54/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.7556 - acc: 0.2186\n",
      "Epoch 55/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.7272 - acc: 0.2228\n",
      "Epoch 56/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.6909 - acc: 0.2299\n",
      "Epoch 57/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.6606 - acc: 0.2332\n",
      "Epoch 58/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.6321 - acc: 0.2381\n",
      "Epoch 59/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.5979 - acc: 0.2403\n",
      "Epoch 60/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.5669 - acc: 0.2476\n",
      "Epoch 61/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 3.5390 - acc: 0.2521\n",
      "Epoch 62/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.5080 - acc: 0.2558\n",
      "Epoch 63/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.4800 - acc: 0.2599\n",
      "Epoch 64/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.4553 - acc: 0.2636\n",
      "Epoch 65/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.4212 - acc: 0.2691\n",
      "Epoch 66/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.3934 - acc: 0.2769\n",
      "Epoch 67/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.3697 - acc: 0.2772\n",
      "Epoch 68/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.3473 - acc: 0.2835\n",
      "Epoch 69/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.3167 - acc: 0.2874\n",
      "Epoch 70/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.2904 - acc: 0.2906\n",
      "Epoch 71/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.2640 - acc: 0.2954\n",
      "Epoch 72/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 3.2363 - acc: 0.3024\n",
      "Epoch 73/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.2887 - acc: 0.2936\n",
      "Epoch 74/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 3.2058 - acc: 0.3045\n",
      "Epoch 75/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.1582 - acc: 0.3133\n",
      "Epoch 76/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.1357 - acc: 0.3184\n",
      "Epoch 77/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.1077 - acc: 0.3228\n",
      "Epoch 78/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.0911 - acc: 0.3267\n",
      "Epoch 79/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.0677 - acc: 0.3294\n",
      "Epoch 80/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 3.0502 - acc: 0.3290\n",
      "Epoch 81/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 3.0271 - acc: 0.3359\n",
      "Epoch 82/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 3.0060 - acc: 0.3413\n",
      "Epoch 83/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 2.9783 - acc: 0.3443\n",
      "Epoch 84/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 2.9562 - acc: 0.3486\n",
      "Epoch 85/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 2.9305 - acc: 0.3530\n",
      "Epoch 86/200\n",
      "32502/32502 [==============================] - 50s 2ms/step - loss: 2.9138 - acc: 0.3571\n",
      "Epoch 87/200\n",
      "32502/32502 [==============================] - 49s 2ms/step - loss: 2.8932 - acc: 0.3599\n",
      "Epoch 88/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.8787 - acc: 0.3601\n",
      "Epoch 89/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.8603 - acc: 0.3652\n",
      "Epoch 90/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 2.8328 - acc: 0.3711\n",
      "Epoch 91/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 2.8068 - acc: 0.3731\n",
      "Epoch 92/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 2.7889 - acc: 0.3782\n",
      "Epoch 93/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.7718 - acc: 0.3814\n",
      "Epoch 94/200\n",
      "32502/32502 [==============================] - 49s 1ms/step - loss: 2.7600 - acc: 0.3849\n",
      "Epoch 95/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 2.7299 - acc: 0.3883\n",
      "Epoch 96/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.7132 - acc: 0.3928\n",
      "Epoch 97/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.6927 - acc: 0.3945\n",
      "Epoch 98/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.6819 - acc: 0.3963\n",
      "Epoch 99/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.6570 - acc: 0.4000\n",
      "Epoch 100/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.6385 - acc: 0.4044\n",
      "Epoch 101/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.6296 - acc: 0.4096\n",
      "Epoch 102/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 2.6057 - acc: 0.4118\n",
      "Epoch 103/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 2.5872 - acc: 0.4144\n",
      "Epoch 104/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.5592 - acc: 0.4236\n",
      "Epoch 105/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 2.5414 - acc: 0.4210\n",
      "Epoch 106/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 2.5283 - acc: 0.4263\n",
      "Epoch 107/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.5094 - acc: 0.4297\n",
      "Epoch 108/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.4938 - acc: 0.4313\n",
      "Epoch 109/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.4764 - acc: 0.4332\n",
      "Epoch 110/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.4668 - acc: 0.4346\n",
      "Epoch 111/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.4370 - acc: 0.4433\n",
      "Epoch 112/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.4125 - acc: 0.4481\n",
      "Epoch 113/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.3981 - acc: 0.4511\n",
      "Epoch 114/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.3900 - acc: 0.4493\n",
      "Epoch 115/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.3809 - acc: 0.4517\n",
      "Epoch 116/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.3548 - acc: 0.4582\n",
      "Epoch 117/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.3380 - acc: 0.4608\n",
      "Epoch 118/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.3110 - acc: 0.4670\n",
      "Epoch 119/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 2.2963 - acc: 0.4686\n",
      "Epoch 120/200\n",
      "32502/32502 [==============================] - 49s 2ms/step - loss: 2.2877 - acc: 0.4689\n",
      "Epoch 121/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.2639 - acc: 0.4792\n",
      "Epoch 122/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.2534 - acc: 0.4764\n",
      "Epoch 123/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.2373 - acc: 0.4798\n",
      "Epoch 124/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.2255 - acc: 0.4826\n",
      "Epoch 125/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.2069 - acc: 0.4879\n",
      "Epoch 126/200\n",
      "32502/32502 [==============================] - 49s 2ms/step - loss: 2.1868 - acc: 0.4922\n",
      "Epoch 127/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 2.1673 - acc: 0.4944\n",
      "Epoch 128/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.1428 - acc: 0.4996\n",
      "Epoch 129/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.1216 - acc: 0.5065\n",
      "Epoch 130/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.1221 - acc: 0.5030\n",
      "Epoch 131/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.1121 - acc: 0.5066\n",
      "Epoch 132/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.0804 - acc: 0.5118\n",
      "Epoch 133/200\n",
      "32502/32502 [==============================] - 50s 2ms/step - loss: 2.0669 - acc: 0.5141\n",
      "Epoch 134/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.0641 - acc: 0.5162\n",
      "Epoch 135/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.0288 - acc: 0.5228\n",
      "Epoch 136/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 2.0130 - acc: 0.5262\n",
      "Epoch 137/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.9981 - acc: 0.5295\n",
      "Epoch 138/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.9859 - acc: 0.5303\n",
      "Epoch 139/200\n",
      "32502/32502 [==============================] - 49s 2ms/step - loss: 1.9635 - acc: 0.5363\n",
      "Epoch 140/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.9498 - acc: 0.5378\n",
      "Epoch 141/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.9268 - acc: 0.5443\n",
      "Epoch 142/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.9097 - acc: 0.5485\n",
      "Epoch 143/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.8938 - acc: 0.5529\n",
      "Epoch 144/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.8847 - acc: 0.5538\n",
      "Epoch 145/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.8797 - acc: 0.5510\n",
      "Epoch 146/200\n",
      "32502/32502 [==============================] - 49s 2ms/step - loss: 1.8660 - acc: 0.5580\n",
      "Epoch 147/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.8502 - acc: 0.5621\n",
      "Epoch 148/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.8329 - acc: 0.5623\n",
      "Epoch 149/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.8082 - acc: 0.5664\n",
      "Epoch 150/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.7841 - acc: 0.5753\n",
      "Epoch 151/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.7662 - acc: 0.5781\n",
      "Epoch 152/200\n",
      "32502/32502 [==============================] - 49s 2ms/step - loss: 1.7534 - acc: 0.5834\n",
      "Epoch 153/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.7622 - acc: 0.5760\n",
      "Epoch 154/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.7400 - acc: 0.5847\n",
      "Epoch 155/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.7413 - acc: 0.5815\n",
      "Epoch 156/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.7167 - acc: 0.5885\n",
      "Epoch 157/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.6853 - acc: 0.5972\n",
      "Epoch 158/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.6771 - acc: 0.5990\n",
      "Epoch 159/200\n",
      "32502/32502 [==============================] - 49s 2ms/step - loss: 1.6543 - acc: 0.6033\n",
      "Epoch 160/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.6477 - acc: 0.6033\n",
      "Epoch 161/200\n",
      "32502/32502 [==============================] - 54s 2ms/step - loss: 1.6393 - acc: 0.6039\n",
      "Epoch 162/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.6200 - acc: 0.6095\n",
      "Epoch 163/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.6045 - acc: 0.6157\n",
      "Epoch 164/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.5912 - acc: 0.6187\n",
      "Epoch 165/200\n",
      "32502/32502 [==============================] - 49s 2ms/step - loss: 1.5912 - acc: 0.6141\n",
      "Epoch 166/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.5746 - acc: 0.6181\n",
      "Epoch 167/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.5529 - acc: 0.6239\n",
      "Epoch 168/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.5398 - acc: 0.6316\n",
      "Epoch 169/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.5275 - acc: 0.6309\n",
      "Epoch 170/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.5137 - acc: 0.6336\n",
      "Epoch 171/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.4963 - acc: 0.6413\n",
      "Epoch 172/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.4900 - acc: 0.6399\n",
      "Epoch 173/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.4757 - acc: 0.6437\n",
      "Epoch 174/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.4573 - acc: 0.6480\n",
      "Epoch 175/200\n",
      "32502/32502 [==============================] - 46s 1ms/step - loss: 1.4561 - acc: 0.6492\n",
      "Epoch 176/200\n",
      "32502/32502 [==============================] - 53s 2ms/step - loss: 1.4320 - acc: 0.6525\n",
      "Epoch 177/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.4240 - acc: 0.6529\n",
      "Epoch 178/200\n",
      "32502/32502 [==============================] - 49s 1ms/step - loss: 1.4162 - acc: 0.6564\n",
      "Epoch 179/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.3924 - acc: 0.6623\n",
      "Epoch 180/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.3864 - acc: 0.6628\n",
      "Epoch 181/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.3867 - acc: 0.6654\n",
      "Epoch 182/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.3694 - acc: 0.6652\n",
      "Epoch 183/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.3567 - acc: 0.6686\n",
      "Epoch 184/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.3454 - acc: 0.6722\n",
      "Epoch 185/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.3190 - acc: 0.6794\n",
      "Epoch 186/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.2956 - acc: 0.6868\n",
      "Epoch 187/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.2927 - acc: 0.6872\n",
      "Epoch 188/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.2971 - acc: 0.6818\n",
      "Epoch 189/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.2851 - acc: 0.6874\n",
      "Epoch 190/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.2776 - acc: 0.6873\n",
      "Epoch 191/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.2488 - acc: 0.6943\n",
      "Epoch 192/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.2337 - acc: 0.6964\n",
      "Epoch 193/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.2298 - acc: 0.6987\n",
      "Epoch 194/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.2141 - acc: 0.7030\n",
      "Epoch 195/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.2028 - acc: 0.7077\n",
      "Epoch 196/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.1988 - acc: 0.7068\n",
      "Epoch 197/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.1894 - acc: 0.7096\n",
      "Epoch 198/200\n",
      "32502/32502 [==============================] - 48s 1ms/step - loss: 1.1871 - acc: 0.7096\n",
      "Epoch 199/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.1652 - acc: 0.7143\n",
      "Epoch 200/200\n",
      "32502/32502 [==============================] - 47s 1ms/step - loss: 1.1418 - acc: 0.7218\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f9b104fc950>"
      ]
     },
     "execution_count": 12,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# integer encode sequences of words\n",
    "tokenizer = Tokenizer()\n",
    "tokenizer.fit_on_texts(lines)\n",
    "sequences = tokenizer.texts_to_sequences(lines)\n",
    "# vocabulary size\n",
    "vocab_size = len(tokenizer.word_index) + 1\n",
    "\n",
    "# separate into input and output\n",
    "sequences = array(sequences)\n",
    "X, y = sequences[:,:-1], sequences[:,-1]\n",
    "y = to_categorical(y, num_classes=vocab_size)\n",
    "seq_length = X.shape[1]\n",
    "\n",
    "\n",
    "# define model\n",
    "model = Sequential()\n",
    "model.add(Embedding(vocab_size, 50, input_length=seq_length))\n",
    "model.add(LSTM(100, return_sequences=True))\n",
    "model.add(LSTM(100))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(vocab_size, activation='softmax'))\n",
    "print(model.summary())\n",
    "\n",
    "\n",
    "# compile model\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "# fit model\n",
    "\n",
    "model.fit(X, y, batch_size=128,epochs=200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-JK0_I_aWO8r"
   },
   "outputs": [],
   "source": [
    "# save the model to file\n",
    "model.save('./model.h5')\n",
    "# save the tokenizer\n",
    "dump(tokenizer, open('tokenizer.pkl', 'wb'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Oz0U0V0qYxHZ"
   },
   "source": [
    "# Loading Language Model to Generate Text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 89
    },
    "colab_type": "code",
    "id": "hnRoeu8mY2bM",
    "outputId": "aa31f264-f149-44d7-e988-48631dff666f",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SEED:i have always wondered why i like to observe people \n",
      "\n",
      "GENERATED:i just meant stunned full adorable theyre smug at the window directly facing my villagers from the chain library there couldnt start their burn all at the morning trying in chinese corner at my laundry enthayi and beneath their backdrop\n"
     ]
    }
   ],
   "source": [
    "from random import randint\n",
    "from pickle import load\n",
    "from keras.models import load_model\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "# load doc into memory\n",
    "def load_doc(filename):\n",
    "    # open the file as read only\n",
    "    file = open(filename, 'r')\n",
    "    # read all text\n",
    "    text = file.read()\n",
    "    # close the file\n",
    "    file.close()\n",
    "    return text\n",
    "\n",
    "# generate a sequence from a language model\n",
    "def generate_seq(model, tokenizer, seq_length, seed_text, n_words):\n",
    "    result = list()\n",
    "    in_text = seed_text\n",
    "    # generate a fixed number of words\n",
    "    for _ in range(n_words):\n",
    "        # encode the text as integer\n",
    "        encoded = tokenizer.texts_to_sequences([in_text])[0]\n",
    "        # truncate sequences to a fixed length\n",
    "        encoded = pad_sequences([encoded], maxlen=seq_length, truncating='pre')\n",
    "        # predict probabilities for each word\n",
    "        yhat = model.predict_classes(encoded, verbose=0)\n",
    "        # map predicted word index to word\n",
    "        out_word = ''\n",
    "        for word, index in tokenizer.word_index.items():\n",
    "            if index == yhat:\n",
    "                out_word = word\n",
    "                break\n",
    "        # append to input\n",
    "        in_text += ' ' + out_word\n",
    "        result.append(out_word)\n",
    "    return ' '.join(result)\n",
    "\n",
    "# load cleaned text sequences\n",
    "in_filename = 'blogdata_nonl_sequences.txt'\n",
    "doc = load_doc(in_filename)\n",
    "lines = doc.split('\\n')\n",
    "seq_length = len(lines[0].split()) - 1\n",
    "\n",
    "# load the model\n",
    "model = load_model('model.h5')\n",
    "\n",
    "# load the tokenizer\n",
    "tokenizer = load(open('tokenizer.pkl', 'rb'))\n",
    "\n",
    "# select a seed text\n",
    "# seed_text = lines[randint(0,len(lines))]\n",
    "seed_text = \"i have always wondered why i like to observe people \"\n",
    "print(\"SEED:\"+seed_text + '\\n')\n",
    "\n",
    "# generate new text\n",
    "generated = generate_seq(model, tokenizer, seq_length, seed_text, 40)\n",
    "print(\"GENERATED:\"+generated)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TADA!"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Blogpost_generation.ipynb",
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python (myenv)",
   "language": "python",
   "name": "nithin"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}