Text_Generation_Project_Gutenberg.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    },
    "colab": {
      "name": "Text Generation - Project Gutenberg.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/bosecodes/slytherin-slingshot/blob/master/Text_Generation_Project_Gutenberg.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JfuW2dckBrax",
        "colab_type": "text"
      },
      "source": [
        "# Develop a Small LSTM Recurrent Neural Network\n",
        "\n",
        "In this section we will develop a simple LSTM network to learn the sequences of characters from Alice in Wonderland. In the next section, we will use this model to generate new sequences of characters.\n",
        "\n",
        "Let's start off by importing the classes and the functions we intend to use to train our model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ojb653boBra0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "2aaf0cb0-4f59-409a-ebb1-590519b69e99"
      },
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n",
        "from tensorflow.python.keras.models import Sequential\n",
        "from tensorflow.python.keras.layers import Dense, Dropout, LSTM\n",
        "from tensorflow.python.keras.callbacks import ModelCheckpoint\n",
        "from tensorflow.python.keras.utils import np_utils"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Num GPUs Available:  1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Eva00JI3CFxI",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1-Qmj4j9Bra-",
        "colab_type": "text"
      },
      "source": [
        "Next we need to load the ASCII text for the book into memory and convert all the characters to lowercase to reduce the vocabulary that the network must learn"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SN16NQmyBrbA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# load ascii text and convert to lowercase\n",
        "location = '/content/wonderland.txt'\n",
        "raw_text = open(location, 'r', encoding = 'utf-8').read()\n",
        "raw_text = raw_text.lower()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XntG42aEBrbI",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "c564803c-d20b-404e-8acf-b061b4e4a6ff"
      },
      "source": [
        "raw_text[:100]"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"project gutenberg's alice's adventures in wonderland, by lewis carroll\\n\\nthis ebook is for the use of\""
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bU7skywrBrbR",
        "colab_type": "text"
      },
      "source": [
        "Now that the book is loaded, we must prepare the data for modelling by the neural network. We can't model the characters directly, we instead must convert the characters to integers.\n",
        "\n",
        "We can do this easily by first creating a set of all the distinct characters in the book, then creating a map of each character to a unique integer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZMgD6NUgBrbS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# create a mapping of unique chars to integers\n",
        "chars = sorted(list(set(raw_text)))\n",
        "char_to_int = dict((c,i) for i, c in enumerate(chars))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZvzXwWI-Brba",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "02ca9089-c518-48d6-c653-98cec7ed7d47"
      },
      "source": [
        "char_to_int"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'\\n': 0,\n",
              " ' ': 1,\n",
              " '!': 2,\n",
              " '\"': 3,\n",
              " '#': 4,\n",
              " '$': 5,\n",
              " '%': 6,\n",
              " \"'\": 7,\n",
              " '(': 8,\n",
              " ')': 9,\n",
              " '*': 10,\n",
              " ',': 11,\n",
              " '-': 12,\n",
              " '.': 13,\n",
              " '/': 14,\n",
              " '0': 15,\n",
              " '1': 16,\n",
              " '2': 17,\n",
              " '3': 18,\n",
              " '4': 19,\n",
              " '5': 20,\n",
              " '6': 21,\n",
              " '7': 22,\n",
              " '8': 23,\n",
              " '9': 24,\n",
              " ':': 25,\n",
              " ';': 26,\n",
              " '?': 27,\n",
              " '@': 28,\n",
              " '[': 29,\n",
              " ']': 30,\n",
              " '_': 31,\n",
              " 'a': 32,\n",
              " 'b': 33,\n",
              " 'c': 34,\n",
              " 'd': 35,\n",
              " 'e': 36,\n",
              " 'f': 37,\n",
              " 'g': 38,\n",
              " 'h': 39,\n",
              " 'i': 40,\n",
              " 'j': 41,\n",
              " 'k': 42,\n",
              " 'l': 43,\n",
              " 'm': 44,\n",
              " 'n': 45,\n",
              " 'o': 46,\n",
              " 'p': 47,\n",
              " 'q': 48,\n",
              " 'r': 49,\n",
              " 's': 50,\n",
              " 't': 51,\n",
              " 'u': 52,\n",
              " 'v': 53,\n",
              " 'w': 54,\n",
              " 'x': 55,\n",
              " 'y': 56,\n",
              " 'z': 57}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "61-RuP9mBrbh",
        "colab_type": "text"
      },
      "source": [
        "We can further remove some other characters that would further clean up the dataset that will reduce the vocabulary and improve the modelling process.\n",
        "\n",
        "Now that the book has been loaded and the mapping prepared, we can summarise the dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Uh4T5dWDBrbj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "26faeb32-e079-4cb6-a138-75f460c21be0"
      },
      "source": [
        "n_chars = len(raw_text)\n",
        "n_vocab = len(chars)\n",
        "print('Total Characters: ', n_chars)\n",
        "print('Total Vocab: ', n_vocab)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Total Characters:  163779\n",
            "Total Vocab:  58\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fAMvqvrJBrbs",
        "colab_type": "text"
      },
      "source": [
        "Thus the book has about 160,000 characters and that when converted to lowercase, there are only 58 distinct characters in the vocabulary for the network to learn. Much more than the 26 in the alphabet.\n",
        "\n",
        "We need to now define the training data for our network. There is a lot of flexibility in how you choose to break up the text and expose it to the network during training.\n",
        "\n",
        "In this tutorial we will split the book text up to subsequences of 100 characters, an arbitrary length. We could just as easily split the data by sentences and pad the shorter sequences and truncate the longer ones.\n",
        "\n",
        "Each training pattern is comprised of 100 time steps of one character (X) followed by one character output(y). When creating these sequences, we slide this window along the whole book, one character at a time, allowing each character a chance to be learned from the 100 characters that preceded it.(except the first 100 characters ofcourse).\n",
        "\n",
        "For example, if the sequence length is 5 (for simplicity, then the two training patterns would be as follows:\n",
        "\n",
        "CHAPT -> E\n",
        "\n",
        "HAPTE -> R\n",
        "\n",
        "As we split up the book into sequences, we convert the characters into integers according to the lookup table(dictionary), that we created earlier."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IKGnglCABrbt",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "7a0ee4c4-39e6-4129-b013-ab8ff3a4811b"
      },
      "source": [
        "# prepare the dataset of input to output pairs encoded as integers\n",
        "seq_length = 100\n",
        "dataX = []\n",
        "dataY = []\n",
        "\n",
        "for i in range(0, n_chars - seq_length, 1):\n",
        "    seq_in = raw_text[i: i + seq_length]\n",
        "    seq_out = raw_text[i+seq_length] # that is, the first character\n",
        "    # after the sequence is complete\n",
        "    dataX.append([char_to_int[char] for char in seq_in])\n",
        "    dataY.append(char_to_int[seq_out])\n",
        "n_patterns = len(dataX)\n",
        "print('Total Patterns: ', n_patterns)"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Total Patterns:  163679\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-NB2QMBWBrb1",
        "colab_type": "text"
      },
      "source": [
        "Thus the number of total pattterns we have 100 less patterns, caused by barring the first 100 characters. We have one training pattern to predict each of the remaining characters.\n",
        "\n",
        "Now that we have the prepared training data, we need to transform it so that it is suitable for use with Keras.\n",
        "- First we must transform the input sequences to the form [samples, time steps, features] expected by the LSTM network.\n",
        "- Next we need to rescale the integers to the range of 0-1 to make the patterns easier to learn by the LSTM network that uses a sigmoid activation function by default\n",
        "- Finally we need to convert the output patterns(single characters converted to integers) to a one hot encoding. This is so that we can configure the network to predict the probability of the 57 different characters in teh vocabulary (an easier representation) rather than trying to force it to predict precisely the next character. Each y value is converted to a Sparse Vector with a length of 57, full of zeroes except with a 1 in the column where teh letter that the pattern represents.\n",
        "\n",
        "For example, when 'n' integer value 31 is one hot encoded, it looks like:\n",
        "\n",
        "[ 0 0 0 0 0 0 0 0 0 0\n",
        "  0 0 0 1 0 0 0 0 0 0\n",
        "  0 0 0 0 0 0 ]\n",
        "\n",
        "We can implement these steps as follows:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AMRTsX_fBrb2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "outputId": "08aefc8c-a845-4f3d-a115-77ebd2408ee0"
      },
      "source": [
        "print(np.reshape(dataX, (n_patterns, seq_length), ))\n",
        "# that is how the input data is supposed to be\n",
        "# however we add a third variable in the tuple called '1'\n",
        "# this is to wrap the entire 2-D array to a \n",
        "# 3-D array"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[47 49 46 ...  1 46 37]\n",
            " [49 46 41 ... 46 37  1]\n",
            " [46 41 36 ... 37  1 32]\n",
            " ...\n",
            " [ 1 51 46 ... 33 46 46]\n",
            " [51 46  1 ... 46 46 42]\n",
            " [46  1 39 ... 46 42 50]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zS6xQnWwBrb-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "6ce03b53-a3af-4795-a3d2-b514ce800143"
      },
      "source": [
        "# reshape X to be [samples, time steps, features]\n",
        "X = np.reshape(dataX, (n_patterns, seq_length,1), )\n",
        "# Normalize\n",
        "print(X[:5])\n",
        "X = X / float(n_vocab)\n",
        "\n",
        "# One Hot Encode the output variable\n",
        "y = np_utils.to_categorical(dataY)"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[[47]\n",
            "  [49]\n",
            "  [46]\n",
            "  [41]\n",
            "  [36]\n",
            "  [34]\n",
            "  [51]\n",
            "  [ 1]\n",
            "  [38]\n",
            "  [52]\n",
            "  [51]\n",
            "  [36]\n",
            "  [45]\n",
            "  [33]\n",
            "  [36]\n",
            "  [49]\n",
            "  [38]\n",
            "  [ 7]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [32]\n",
            "  [43]\n",
            "  [40]\n",
            "  [34]\n",
            "  [36]\n",
            "  [ 7]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [32]\n",
            "  [35]\n",
            "  [53]\n",
            "  [36]\n",
            "  [45]\n",
            "  [51]\n",
            "  [52]\n",
            "  [49]\n",
            "  [36]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [40]\n",
            "  [45]\n",
            "  [ 1]\n",
            "  [54]\n",
            "  [46]\n",
            "  [45]\n",
            "  [35]\n",
            "  [36]\n",
            "  [49]\n",
            "  [43]\n",
            "  [32]\n",
            "  [45]\n",
            "  [35]\n",
            "  [11]\n",
            "  [ 1]\n",
            "  [33]\n",
            "  [56]\n",
            "  [ 1]\n",
            "  [43]\n",
            "  [36]\n",
            "  [54]\n",
            "  [40]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [34]\n",
            "  [32]\n",
            "  [49]\n",
            "  [49]\n",
            "  [46]\n",
            "  [43]\n",
            "  [43]\n",
            "  [ 0]\n",
            "  [ 0]\n",
            "  [51]\n",
            "  [39]\n",
            "  [40]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [36]\n",
            "  [33]\n",
            "  [46]\n",
            "  [46]\n",
            "  [42]\n",
            "  [ 1]\n",
            "  [40]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [37]\n",
            "  [46]\n",
            "  [49]\n",
            "  [ 1]\n",
            "  [51]\n",
            "  [39]\n",
            "  [36]\n",
            "  [ 1]\n",
            "  [52]\n",
            "  [50]\n",
            "  [36]\n",
            "  [ 1]\n",
            "  [46]\n",
            "  [37]]\n",
            "\n",
            " [[49]\n",
            "  [46]\n",
            "  [41]\n",
            "  [36]\n",
            "  [34]\n",
            "  [51]\n",
            "  [ 1]\n",
            "  [38]\n",
            "  [52]\n",
            "  [51]\n",
            "  [36]\n",
            "  [45]\n",
            "  [33]\n",
            "  [36]\n",
            "  [49]\n",
            "  [38]\n",
            "  [ 7]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [32]\n",
            "  [43]\n",
            "  [40]\n",
            "  [34]\n",
            "  [36]\n",
            "  [ 7]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [32]\n",
            "  [35]\n",
            "  [53]\n",
            "  [36]\n",
            "  [45]\n",
            "  [51]\n",
            "  [52]\n",
            "  [49]\n",
            "  [36]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [40]\n",
            "  [45]\n",
            "  [ 1]\n",
            "  [54]\n",
            "  [46]\n",
            "  [45]\n",
            "  [35]\n",
            "  [36]\n",
            "  [49]\n",
            "  [43]\n",
            "  [32]\n",
            "  [45]\n",
            "  [35]\n",
            "  [11]\n",
            "  [ 1]\n",
            "  [33]\n",
            "  [56]\n",
            "  [ 1]\n",
            "  [43]\n",
            "  [36]\n",
            "  [54]\n",
            "  [40]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [34]\n",
            "  [32]\n",
            "  [49]\n",
            "  [49]\n",
            "  [46]\n",
            "  [43]\n",
            "  [43]\n",
            "  [ 0]\n",
            "  [ 0]\n",
            "  [51]\n",
            "  [39]\n",
            "  [40]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [36]\n",
            "  [33]\n",
            "  [46]\n",
            "  [46]\n",
            "  [42]\n",
            "  [ 1]\n",
            "  [40]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [37]\n",
            "  [46]\n",
            "  [49]\n",
            "  [ 1]\n",
            "  [51]\n",
            "  [39]\n",
            "  [36]\n",
            "  [ 1]\n",
            "  [52]\n",
            "  [50]\n",
            "  [36]\n",
            "  [ 1]\n",
            "  [46]\n",
            "  [37]\n",
            "  [ 1]]\n",
            "\n",
            " [[46]\n",
            "  [41]\n",
            "  [36]\n",
            "  [34]\n",
            "  [51]\n",
            "  [ 1]\n",
            "  [38]\n",
            "  [52]\n",
            "  [51]\n",
            "  [36]\n",
            "  [45]\n",
            "  [33]\n",
            "  [36]\n",
            "  [49]\n",
            "  [38]\n",
            "  [ 7]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [32]\n",
            "  [43]\n",
            "  [40]\n",
            "  [34]\n",
            "  [36]\n",
            "  [ 7]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [32]\n",
            "  [35]\n",
            "  [53]\n",
            "  [36]\n",
            "  [45]\n",
            "  [51]\n",
            "  [52]\n",
            "  [49]\n",
            "  [36]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [40]\n",
            "  [45]\n",
            "  [ 1]\n",
            "  [54]\n",
            "  [46]\n",
            "  [45]\n",
            "  [35]\n",
            "  [36]\n",
            "  [49]\n",
            "  [43]\n",
            "  [32]\n",
            "  [45]\n",
            "  [35]\n",
            "  [11]\n",
            "  [ 1]\n",
            "  [33]\n",
            "  [56]\n",
            "  [ 1]\n",
            "  [43]\n",
            "  [36]\n",
            "  [54]\n",
            "  [40]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [34]\n",
            "  [32]\n",
            "  [49]\n",
            "  [49]\n",
            "  [46]\n",
            "  [43]\n",
            "  [43]\n",
            "  [ 0]\n",
            "  [ 0]\n",
            "  [51]\n",
            "  [39]\n",
            "  [40]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [36]\n",
            "  [33]\n",
            "  [46]\n",
            "  [46]\n",
            "  [42]\n",
            "  [ 1]\n",
            "  [40]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [37]\n",
            "  [46]\n",
            "  [49]\n",
            "  [ 1]\n",
            "  [51]\n",
            "  [39]\n",
            "  [36]\n",
            "  [ 1]\n",
            "  [52]\n",
            "  [50]\n",
            "  [36]\n",
            "  [ 1]\n",
            "  [46]\n",
            "  [37]\n",
            "  [ 1]\n",
            "  [32]]\n",
            "\n",
            " [[41]\n",
            "  [36]\n",
            "  [34]\n",
            "  [51]\n",
            "  [ 1]\n",
            "  [38]\n",
            "  [52]\n",
            "  [51]\n",
            "  [36]\n",
            "  [45]\n",
            "  [33]\n",
            "  [36]\n",
            "  [49]\n",
            "  [38]\n",
            "  [ 7]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [32]\n",
            "  [43]\n",
            "  [40]\n",
            "  [34]\n",
            "  [36]\n",
            "  [ 7]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [32]\n",
            "  [35]\n",
            "  [53]\n",
            "  [36]\n",
            "  [45]\n",
            "  [51]\n",
            "  [52]\n",
            "  [49]\n",
            "  [36]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [40]\n",
            "  [45]\n",
            "  [ 1]\n",
            "  [54]\n",
            "  [46]\n",
            "  [45]\n",
            "  [35]\n",
            "  [36]\n",
            "  [49]\n",
            "  [43]\n",
            "  [32]\n",
            "  [45]\n",
            "  [35]\n",
            "  [11]\n",
            "  [ 1]\n",
            "  [33]\n",
            "  [56]\n",
            "  [ 1]\n",
            "  [43]\n",
            "  [36]\n",
            "  [54]\n",
            "  [40]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [34]\n",
            "  [32]\n",
            "  [49]\n",
            "  [49]\n",
            "  [46]\n",
            "  [43]\n",
            "  [43]\n",
            "  [ 0]\n",
            "  [ 0]\n",
            "  [51]\n",
            "  [39]\n",
            "  [40]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [36]\n",
            "  [33]\n",
            "  [46]\n",
            "  [46]\n",
            "  [42]\n",
            "  [ 1]\n",
            "  [40]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [37]\n",
            "  [46]\n",
            "  [49]\n",
            "  [ 1]\n",
            "  [51]\n",
            "  [39]\n",
            "  [36]\n",
            "  [ 1]\n",
            "  [52]\n",
            "  [50]\n",
            "  [36]\n",
            "  [ 1]\n",
            "  [46]\n",
            "  [37]\n",
            "  [ 1]\n",
            "  [32]\n",
            "  [45]]\n",
            "\n",
            " [[36]\n",
            "  [34]\n",
            "  [51]\n",
            "  [ 1]\n",
            "  [38]\n",
            "  [52]\n",
            "  [51]\n",
            "  [36]\n",
            "  [45]\n",
            "  [33]\n",
            "  [36]\n",
            "  [49]\n",
            "  [38]\n",
            "  [ 7]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [32]\n",
            "  [43]\n",
            "  [40]\n",
            "  [34]\n",
            "  [36]\n",
            "  [ 7]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [32]\n",
            "  [35]\n",
            "  [53]\n",
            "  [36]\n",
            "  [45]\n",
            "  [51]\n",
            "  [52]\n",
            "  [49]\n",
            "  [36]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [40]\n",
            "  [45]\n",
            "  [ 1]\n",
            "  [54]\n",
            "  [46]\n",
            "  [45]\n",
            "  [35]\n",
            "  [36]\n",
            "  [49]\n",
            "  [43]\n",
            "  [32]\n",
            "  [45]\n",
            "  [35]\n",
            "  [11]\n",
            "  [ 1]\n",
            "  [33]\n",
            "  [56]\n",
            "  [ 1]\n",
            "  [43]\n",
            "  [36]\n",
            "  [54]\n",
            "  [40]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [34]\n",
            "  [32]\n",
            "  [49]\n",
            "  [49]\n",
            "  [46]\n",
            "  [43]\n",
            "  [43]\n",
            "  [ 0]\n",
            "  [ 0]\n",
            "  [51]\n",
            "  [39]\n",
            "  [40]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [36]\n",
            "  [33]\n",
            "  [46]\n",
            "  [46]\n",
            "  [42]\n",
            "  [ 1]\n",
            "  [40]\n",
            "  [50]\n",
            "  [ 1]\n",
            "  [37]\n",
            "  [46]\n",
            "  [49]\n",
            "  [ 1]\n",
            "  [51]\n",
            "  [39]\n",
            "  [36]\n",
            "  [ 1]\n",
            "  [52]\n",
            "  [50]\n",
            "  [36]\n",
            "  [ 1]\n",
            "  [46]\n",
            "  [37]\n",
            "  [ 1]\n",
            "  [32]\n",
            "  [45]\n",
            "  [56]]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qo9kI-YhBrcF",
        "colab_type": "text"
      },
      "source": [
        "We can now define our LSTM model. Here, we define a single LSTM layer with 256 memory units. The network uses dropout with a probability of 20. The output layer is a Dense (Fully connected) layer, using the softmax activation function to output the probability prediction for each of the 57 characters between 0 and 1.\n",
        "\n",
        "The problem is really \"a single classification problem with 57 classes\" and as such defined as \"optimizing the log loss (cross entropy)\", here \"using the ADAM optimization algorithm\" for speed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kM2OpAMZBrcG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# define the LSTM model\n",
        "model = Sequential()\n",
        "model.add(LSTM(256, input_shape = (X.shape[1], X.shape[2])))\n",
        "model.add(Dropout(0.2))\n",
        "model.add(Dense(y.shape[1], activation = 'softmax'))\n",
        "model.compile(loss = 'categorical_crossentropy', optimizer = 'Adam')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "npFbwU5uBrcO",
        "colab_type": "text"
      },
      "source": [
        "There is no test dataset. We are modeling the entire training dataset to learn the probability of each character in a sequence.\n",
        "\n",
        "We are not interested in the most accurate (classification accuracy) model of the training dataset. This would be a model that predicts each character in the training dataset perfectly. Instead we are interested in the generalization of the dataset that minimizes the chosen loss function. We are seeking a balance between generalization and overfitting but short of memorization.\n",
        "\n",
        "The network is slow to train, because of the slowness and because of our optimization requirements, we will use model checkpointing to record all of the network weights to file each time an improvement in the loss is observed at the end of the epoch. We will use the best sort of weights (lowest loss), to instantiate our generative model in the next section."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TmEVzveTBrcP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# define the checkpoint\n",
        "filepath = \"weights-improvement-{epoch:02d}-{loss:.4f}.hdf5\"\n",
        "checkpoint = ModelCheckpoint(filepath, monitor = 'loss', verbose = 1,\n",
        "                            save_best_only = True, mode = 'min')\n",
        "callbacks_list = [checkpoint]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZFEKpVbFBrcW",
        "colab_type": "text"
      },
      "source": [
        "We can now fit our model to the data. Here we use a modest number of 20 epochs and a large batch size of 128 patterns."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XKyFzdHWBrcX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "62275e71-86e8-40af-923b-5c524bf0822a"
      },
      "source": [
        "model.fit(X, y, epochs = 20, batch_size = 128, callbacks = callbacks_list)"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/20\n",
            "1278/1279 [============================>.] - ETA: 0s - loss: 2.9759\n",
            "Epoch 00001: loss improved from inf to 2.97587, saving model to weights-improvement-01-2.9759.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.9759\n",
            "Epoch 2/20\n",
            "1279/1279 [==============================] - ETA: 0s - loss: 2.7970\n",
            "Epoch 00002: loss improved from 2.97587 to 2.79695, saving model to weights-improvement-02-2.7970.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.7970\n",
            "Epoch 3/20\n",
            "1278/1279 [============================>.] - ETA: 0s - loss: 2.7142\n",
            "Epoch 00003: loss improved from 2.79695 to 2.71407, saving model to weights-improvement-03-2.7141.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.7141\n",
            "Epoch 4/20\n",
            "1276/1279 [============================>.] - ETA: 0s - loss: 2.6466\n",
            "Epoch 00004: loss improved from 2.71407 to 2.64663, saving model to weights-improvement-04-2.6466.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.6466\n",
            "Epoch 5/20\n",
            "1279/1279 [==============================] - ETA: 0s - loss: 2.5892\n",
            "Epoch 00005: loss improved from 2.64663 to 2.58923, saving model to weights-improvement-05-2.5892.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.5892\n",
            "Epoch 6/20\n",
            "1275/1279 [============================>.] - ETA: 0s - loss: 2.5368\n",
            "Epoch 00006: loss improved from 2.58923 to 2.53683, saving model to weights-improvement-06-2.5368.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.5368\n",
            "Epoch 7/20\n",
            "1276/1279 [============================>.] - ETA: 0s - loss: 2.4841\n",
            "Epoch 00007: loss improved from 2.53683 to 2.48403, saving model to weights-improvement-07-2.4840.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.4840\n",
            "Epoch 8/20\n",
            "1279/1279 [==============================] - ETA: 0s - loss: 2.4372\n",
            "Epoch 00008: loss improved from 2.48403 to 2.43724, saving model to weights-improvement-08-2.4372.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.4372\n",
            "Epoch 9/20\n",
            "1277/1279 [============================>.] - ETA: 0s - loss: 2.3927\n",
            "Epoch 00009: loss improved from 2.43724 to 2.39278, saving model to weights-improvement-09-2.3928.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.3928\n",
            "Epoch 10/20\n",
            "1276/1279 [============================>.] - ETA: 0s - loss: 2.3509\n",
            "Epoch 00010: loss improved from 2.39278 to 2.35117, saving model to weights-improvement-10-2.3512.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.3512\n",
            "Epoch 11/20\n",
            "1278/1279 [============================>.] - ETA: 0s - loss: 2.3126\n",
            "Epoch 00011: loss improved from 2.35117 to 2.31276, saving model to weights-improvement-11-2.3128.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.3128\n",
            "Epoch 12/20\n",
            "1278/1279 [============================>.] - ETA: 0s - loss: 2.2762\n",
            "Epoch 00012: loss improved from 2.31276 to 2.27605, saving model to weights-improvement-12-2.2761.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.2761\n",
            "Epoch 13/20\n",
            "1277/1279 [============================>.] - ETA: 0s - loss: 2.2393\n",
            "Epoch 00013: loss improved from 2.27605 to 2.23934, saving model to weights-improvement-13-2.2393.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.2393\n",
            "Epoch 14/20\n",
            "1275/1279 [============================>.] - ETA: 0s - loss: 2.2084\n",
            "Epoch 00014: loss improved from 2.23934 to 2.20824, saving model to weights-improvement-14-2.2082.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.2082\n",
            "Epoch 15/20\n",
            "1276/1279 [============================>.] - ETA: 0s - loss: 2.1760\n",
            "Epoch 00015: loss improved from 2.20824 to 2.17623, saving model to weights-improvement-15-2.1762.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.1762\n",
            "Epoch 16/20\n",
            "1278/1279 [============================>.] - ETA: 0s - loss: 2.1497\n",
            "Epoch 00016: loss improved from 2.17623 to 2.14968, saving model to weights-improvement-16-2.1497.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.1497\n",
            "Epoch 17/20\n",
            "1278/1279 [============================>.] - ETA: 0s - loss: 2.1203\n",
            "Epoch 00017: loss improved from 2.14968 to 2.12046, saving model to weights-improvement-17-2.1205.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.1205\n",
            "Epoch 18/20\n",
            "1277/1279 [============================>.] - ETA: 0s - loss: 2.0957\n",
            "Epoch 00018: loss improved from 2.12046 to 2.09562, saving model to weights-improvement-18-2.0956.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.0956\n",
            "Epoch 19/20\n",
            "1279/1279 [==============================] - ETA: 0s - loss: 2.0722\n",
            "Epoch 00019: loss improved from 2.09562 to 2.07221, saving model to weights-improvement-19-2.0722.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.0722\n",
            "Epoch 20/20\n",
            "1277/1279 [============================>.] - ETA: 0s - loss: 2.0474\n",
            "Epoch 00020: loss improved from 2.07221 to 2.04755, saving model to weights-improvement-20-2.0475.hdf5\n",
            "1279/1279 [==============================] - 16s 12ms/step - loss: 2.0475\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f327c3ee2e8>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RWh99Se2Brce",
        "colab_type": "text"
      },
      "source": [
        "After running the network, we will have a number of checkpoint files in the working directory, we can delete all of them except for the one with the minimum loss.\n",
        "\n",
        "The network loss decreased almost every epoch and it can thereby be expected that the network would benefit from training for many more epochs.\n",
        "\n",
        "In the next section, let's concentrate about how we can use this model to generate new text sequeces."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TMhOfVggBrcg",
        "colab_type": "text"
      },
      "source": [
        "# Generating Text with an LSTM Network\n",
        "\n",
        "Generating text with this network is actually pretty straightforward now. \n",
        "Firstly, we load the data and define teh network in exactly the same way, except that:\n",
        "- the network weights are loaded from a checkpoint file, and, \n",
        "- the network does not need to trained."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g4YpYbqZBrch",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# load the network weights\n",
        "filename = 'weights-improvement-20-2.0475.hdf5'\n",
        "model.load_weights(filename)\n",
        "model.compile(loss = 'categorical_crossentropy', optimizer = 'Adam')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Awz-aVOfBrco",
        "colab_type": "text"
      },
      "source": [
        "Also, when preparing the mapping of the unique characters to integers, we must also create a reverse mapping that we can use to convert the integers back to characters so that we can understand the predictions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QfifNCqIBrcp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "int_to_char = dict((i, c) for i, c in enumerate(chars))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zpLP2-E1Brcx",
        "colab_type": "text"
      },
      "source": [
        "Finally, we need to actually make predictions.\n",
        "\n",
        "The simplest way to use the Keras LSTM model to make predictions is to first start off with a seed sequence as input, generate the next character and then update the seed sequence to add the generated character on the end and trim off the first character. This process is repeated for as long as we want to predict new characters (eg: A sequence of 1000 characters in length).\n",
        "\n",
        "We can pick a random input pattern as our sequence, then print the generated chaaracters as we generate them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2S37QbLhBrcy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        },
        "outputId": "6a8594c5-85d8-4d1a-9554-d6d26148e977"
      },
      "source": [
        "# pick a random seed\n",
        "import sys\n",
        "start = np.random.randint(0, len(dataX)-1)\n",
        "pattern = dataX[start]\n",
        "\n",
        "print('Seed: ')\n",
        "print( \"\\\"\", ''.join([int_to_char[value] for value in pattern]), \"\\\"\")\n",
        "\n",
        "# generate characters\n",
        "for i in range(1000):\n",
        "    x = np.reshape(pattern, (1, len(pattern), 1))\n",
        "    x = x / float(n_vocab)\n",
        "    prediction = model.predict(x, verbose = 0)\n",
        "    index = np.argmax(prediction)\n",
        "    result = int_to_char[index]\n",
        "    seq_in = [int_to_char[value] for value in pattern]\n",
        "    sys.stdout.write(result)\n",
        "    pattern.append(index)\n",
        "    pattern = pattern[1: len(pattern)]\n",
        "print('\\nDone!')"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Seed: \n",
            "\" she went round the court and got behind him, and\n",
            "very soon found an opportunity of taking it away. s \"\n",
            "he was not in the tooee to tee thet sas the was sored ho the caree in the rabbet sore, and she whit hn was toe tirte to tee the was soenk at inrs an the was sotnd the tas of the couro, and tae toine to be a sore of the toeet an inss to the toeee tf the career, and see woine to betit all the war sot what it was toenking to the toeee tf the caree in the caree in the rabbet sore, and she whit hn was toe tirte to tee the was sor that she was sot thsh the tone, \n",
            "'the mase thing so be in toe to tee the gocster saseer ' said the mock turtle. \n",
            "'i dane tae thing toe to toen toe toieg to tee thet ' said the mock turtle.\n",
            "\n",
            "'i dane the mabt thrn saeee'' said the maccit. \n",
            "anice was tor tas toe tirtee to tee thet sas thth the mooe, and the was tointing an the woond to tee the was sor that she was sot thsh the tone, \n",
            "'the mase thing so be in toe to tee the gocster saseer ' said the mock turtle. \n",
            "'i dane tae thing toe to toen toe toieg to tee thet ' said the mock turtle.\n",
            "\n",
            "'i dane the mabt thrn saeee'' \n",
            "Done!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m-VmXTifBrc_",
        "colab_type": "text"
      },
      "source": [
        "We can now note some observations about the generated text.\n",
        "\n",
        "- It generally conforms to the line format observed in the original text of less than 80 characters before a new line.\n",
        "- The characters are seperated into word-like groups and most groups are actual English words, (e.g., 'the', 'little', 'said' and 'was'), but many do not (e.g. 'lott', 'tiie', 'taede' 'ssemme').\n",
        "\n",
        "The fact that this character based model of the book produces an output like this is pretty impressive. It gives you a sense of the learning capabilities of LSTM networks.\n",
        "\n",
        "The results are not perfect. Now, we'll look at the techniques to make some improvements by developing a much larger LSTM network.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8p9epo87BrdF",
        "colab_type": "text"
      },
      "source": [
        "# Larger LSTM Recurrent Neural Network\n",
        "\n",
        "We got results, but not excellent results in the last section. Now, we'll try to improve the quality of the generated network by using a much larger network.\n",
        "\n",
        "We will keep the number of memory units the same, i.e., 256, but, add a second additional layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PLytOxmgBrdM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = Sequential()\n",
        "model.add(LSTM(256, input_shape = (X.shape[1], X.shape[2]), return_sequences = True))\n",
        "model.add(Dropout(0.2))\n",
        "model.add(LSTM(256))\n",
        "model.add(Dropout(0.2))\n",
        "model.add(Dense(y.shape[1], activation = 'softmax'))\n",
        "model.compile(loss = 'categorical_crossentropy', optimizer = 'adam')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RpBwDA45BrdS",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "044bfd69-de22-4bd8-e16c-14965700bc54"
      },
      "source": [
        "# Define the checkpoint\n",
        "filepath = 'weights-improvement-{epoch:02d}-{loss:.4f}-bigger.hdf5'\n",
        "checkpoint = ModelCheckpoint(filepath, monitor = 'loss', verbose = 1, save_best_only = True, mode = 'min')\n",
        "callbacks_list = [checkpoint]\n",
        "\n",
        "# Fit the model\n",
        "model.fit(X, y, epochs = 50, batch_size = 64, callbacks = callbacks_list)"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 2.8195\n",
            "Epoch 00001: loss improved from inf to 2.81906, saving model to weights-improvement-01-2.8191-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 2.8191\n",
            "Epoch 2/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 2.5245\n",
            "Epoch 00002: loss improved from 2.81906 to 2.52433, saving model to weights-improvement-02-2.5243-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 2.5243\n",
            "Epoch 3/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 2.3421\n",
            "Epoch 00003: loss improved from 2.52433 to 2.34200, saving model to weights-improvement-03-2.3420-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 2.3420\n",
            "Epoch 4/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 2.2078\n",
            "Epoch 00004: loss improved from 2.34200 to 2.20771, saving model to weights-improvement-04-2.2077-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 21ms/step - loss: 2.2077\n",
            "Epoch 5/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 2.1022\n",
            "Epoch 00005: loss improved from 2.20771 to 2.10220, saving model to weights-improvement-05-2.1022-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 2.1022\n",
            "Epoch 6/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 2.0225\n",
            "Epoch 00006: loss improved from 2.10220 to 2.02232, saving model to weights-improvement-06-2.0223-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 2.0223\n",
            "Epoch 7/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 1.9512\n",
            "Epoch 00007: loss improved from 2.02232 to 1.95139, saving model to weights-improvement-07-1.9514-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.9514\n",
            "Epoch 8/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 1.8973\n",
            "Epoch 00008: loss improved from 1.95139 to 1.89716, saving model to weights-improvement-08-1.8972-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.8972\n",
            "Epoch 9/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 1.8441\n",
            "Epoch 00009: loss improved from 1.89716 to 1.84386, saving model to weights-improvement-09-1.8439-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.8439\n",
            "Epoch 10/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 1.8031\n",
            "Epoch 00010: loss improved from 1.84386 to 1.80299, saving model to weights-improvement-10-1.8030-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 21ms/step - loss: 1.8030\n",
            "Epoch 11/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 1.7641\n",
            "Epoch 00011: loss improved from 1.80299 to 1.76411, saving model to weights-improvement-11-1.7641-bigger.hdf5\n",
            "2558/2558 [==============================] - 54s 21ms/step - loss: 1.7641\n",
            "Epoch 12/50\n",
            "2558/2558 [==============================] - ETA: 0s - loss: 1.7250\n",
            "Epoch 00012: loss improved from 1.76411 to 1.72502, saving model to weights-improvement-12-1.7250-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 21ms/step - loss: 1.7250\n",
            "Epoch 13/50\n",
            "2558/2558 [==============================] - ETA: 0s - loss: 1.6912\n",
            "Epoch 00013: loss improved from 1.72502 to 1.69125, saving model to weights-improvement-13-1.6912-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.6912\n",
            "Epoch 14/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 1.6638\n",
            "Epoch 00014: loss improved from 1.69125 to 1.66382, saving model to weights-improvement-14-1.6638-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.6638\n",
            "Epoch 15/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 1.6351\n",
            "Epoch 00015: loss improved from 1.66382 to 1.63515, saving model to weights-improvement-15-1.6352-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.6352\n",
            "Epoch 16/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 1.6103\n",
            "Epoch 00016: loss improved from 1.63515 to 1.61044, saving model to weights-improvement-16-1.6104-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.6104\n",
            "Epoch 17/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 1.5890\n",
            "Epoch 00017: loss improved from 1.61044 to 1.58905, saving model to weights-improvement-17-1.5891-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.5891\n",
            "Epoch 18/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 1.5676\n",
            "Epoch 00018: loss improved from 1.58905 to 1.56762, saving model to weights-improvement-18-1.5676-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.5676\n",
            "Epoch 19/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 1.5498\n",
            "Epoch 00019: loss improved from 1.56762 to 1.54978, saving model to weights-improvement-19-1.5498-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.5498\n",
            "Epoch 20/50\n",
            "2558/2558 [==============================] - ETA: 0s - loss: 1.5249\n",
            "Epoch 00020: loss improved from 1.54978 to 1.52489, saving model to weights-improvement-20-1.5249-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.5249\n",
            "Epoch 21/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 1.5099\n",
            "Epoch 00021: loss improved from 1.52489 to 1.50980, saving model to weights-improvement-21-1.5098-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.5098\n",
            "Epoch 22/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 1.4939\n",
            "Epoch 00022: loss improved from 1.50980 to 1.49383, saving model to weights-improvement-22-1.4938-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.4938\n",
            "Epoch 23/50\n",
            "2558/2558 [==============================] - ETA: 0s - loss: 1.4802\n",
            "Epoch 00023: loss improved from 1.49383 to 1.48020, saving model to weights-improvement-23-1.4802-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.4802\n",
            "Epoch 24/50\n",
            "2558/2558 [==============================] - ETA: 0s - loss: 1.4677\n",
            "Epoch 00024: loss improved from 1.48020 to 1.46768, saving model to weights-improvement-24-1.4677-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.4677\n",
            "Epoch 25/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 1.4552\n",
            "Epoch 00025: loss improved from 1.46768 to 1.45517, saving model to weights-improvement-25-1.4552-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.4552\n",
            "Epoch 26/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 1.4456\n",
            "Epoch 00026: loss improved from 1.45517 to 1.44552, saving model to weights-improvement-26-1.4455-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.4455\n",
            "Epoch 27/50\n",
            "2558/2558 [==============================] - ETA: 0s - loss: 1.4327\n",
            "Epoch 00027: loss improved from 1.44552 to 1.43268, saving model to weights-improvement-27-1.4327-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.4327\n",
            "Epoch 28/50\n",
            "2558/2558 [==============================] - ETA: 0s - loss: 1.4272\n",
            "Epoch 00028: loss improved from 1.43268 to 1.42717, saving model to weights-improvement-28-1.4272-bigger.hdf5\n",
            "2558/2558 [==============================] - 56s 22ms/step - loss: 1.4272\n",
            "Epoch 29/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 1.4145\n",
            "Epoch 00029: loss improved from 1.42717 to 1.41431, saving model to weights-improvement-29-1.4143-bigger.hdf5\n",
            "2558/2558 [==============================] - 56s 22ms/step - loss: 1.4143\n",
            "Epoch 30/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 1.4023\n",
            "Epoch 00030: loss improved from 1.41431 to 1.40236, saving model to weights-improvement-30-1.4024-bigger.hdf5\n",
            "2558/2558 [==============================] - 56s 22ms/step - loss: 1.4024\n",
            "Epoch 31/50\n",
            "2558/2558 [==============================] - ETA: 0s - loss: 1.3980\n",
            "Epoch 00031: loss improved from 1.40236 to 1.39804, saving model to weights-improvement-31-1.3980-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.3980\n",
            "Epoch 32/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 1.3890\n",
            "Epoch 00032: loss improved from 1.39804 to 1.38924, saving model to weights-improvement-32-1.3892-bigger.hdf5\n",
            "2558/2558 [==============================] - 56s 22ms/step - loss: 1.3892\n",
            "Epoch 33/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 1.3908\n",
            "Epoch 00033: loss did not improve from 1.38924\n",
            "2558/2558 [==============================] - 56s 22ms/step - loss: 1.3909\n",
            "Epoch 34/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 1.3776\n",
            "Epoch 00034: loss improved from 1.38924 to 1.37767, saving model to weights-improvement-34-1.3777-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.3777\n",
            "Epoch 35/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 1.3738\n",
            "Epoch 00035: loss improved from 1.37767 to 1.37386, saving model to weights-improvement-35-1.3739-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.3739\n",
            "Epoch 36/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 1.4365\n",
            "Epoch 00036: loss did not improve from 1.37386\n",
            "2558/2558 [==============================] - 56s 22ms/step - loss: 1.4365\n",
            "Epoch 37/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 1.3491\n",
            "Epoch 00037: loss improved from 1.37386 to 1.34902, saving model to weights-improvement-37-1.3490-bigger.hdf5\n",
            "2558/2558 [==============================] - 56s 22ms/step - loss: 1.3490\n",
            "Epoch 38/50\n",
            "2558/2558 [==============================] - ETA: 0s - loss: 1.3572\n",
            "Epoch 00038: loss did not improve from 1.34902\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.3572\n",
            "Epoch 39/50\n",
            "2558/2558 [==============================] - ETA: 0s - loss: 1.3544\n",
            "Epoch 00039: loss did not improve from 1.34902\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.3544\n",
            "Epoch 40/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 1.3439\n",
            "Epoch 00040: loss improved from 1.34902 to 1.34404, saving model to weights-improvement-40-1.3440-bigger.hdf5\n",
            "2558/2558 [==============================] - 56s 22ms/step - loss: 1.3440\n",
            "Epoch 41/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 1.3500\n",
            "Epoch 00041: loss did not improve from 1.34404\n",
            "2558/2558 [==============================] - 56s 22ms/step - loss: 1.3500\n",
            "Epoch 42/50\n",
            "2558/2558 [==============================] - ETA: 0s - loss: 1.3349\n",
            "Epoch 00042: loss improved from 1.34404 to 1.33492, saving model to weights-improvement-42-1.3349-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.3349\n",
            "Epoch 43/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 1.3349\n",
            "Epoch 00043: loss did not improve from 1.33492\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.3350\n",
            "Epoch 44/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 1.3317\n",
            "Epoch 00044: loss improved from 1.33492 to 1.33175, saving model to weights-improvement-44-1.3317-bigger.hdf5\n",
            "2558/2558 [==============================] - 56s 22ms/step - loss: 1.3317\n",
            "Epoch 45/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 1.3285\n",
            "Epoch 00045: loss improved from 1.33175 to 1.32850, saving model to weights-improvement-45-1.3285-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.3285\n",
            "Epoch 46/50\n",
            "2558/2558 [==============================] - ETA: 0s - loss: 1.4262\n",
            "Epoch 00046: loss did not improve from 1.32850\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.4262\n",
            "Epoch 47/50\n",
            "2557/2558 [============================>.] - ETA: 0s - loss: 1.3250\n",
            "Epoch 00047: loss improved from 1.32850 to 1.32492, saving model to weights-improvement-47-1.3249-bigger.hdf5\n",
            "2558/2558 [==============================] - 56s 22ms/step - loss: 1.3249\n",
            "Epoch 48/50\n",
            "2558/2558 [==============================] - ETA: 0s - loss: 1.3225\n",
            "Epoch 00048: loss improved from 1.32492 to 1.32250, saving model to weights-improvement-48-1.3225-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.3225\n",
            "Epoch 49/50\n",
            "2558/2558 [==============================] - ETA: 0s - loss: 1.3134\n",
            "Epoch 00049: loss improved from 1.32250 to 1.31341, saving model to weights-improvement-49-1.3134-bigger.hdf5\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.3134\n",
            "Epoch 50/50\n",
            "2556/2558 [============================>.] - ETA: 0s - loss: 1.3155\n",
            "Epoch 00050: loss did not improve from 1.31341\n",
            "2558/2558 [==============================] - 55s 22ms/step - loss: 1.3155\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f3264213470>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LxRiqDsrBrdY",
        "colab_type": "text"
      },
      "source": [
        "We finally choose the weights file with the minimum loss and then use it to predict the text by using a random seed (just as in the previous section)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EqsRIiXyBrde",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        },
        "outputId": "594d963c-7413-41a8-a06c-a55bc7f884b0"
      },
      "source": [
        "# pick a random seed\n",
        "start = np.random.randint(0, len(dataX) - 1)\n",
        "pattern = dataX[start]\n",
        "print('Seed: ')\n",
        "print(\"\\\"\", ''.join([int_to_char[value] for value in pattern]), \"\\\"\")\n",
        "\n",
        "# generate characters\n",
        "for i in range(1000):\n",
        "  x = np.reshape(pattern, (1, len(pattern), 1))\n",
        "  x = x/float(n_vocab)\n",
        "  prediction = model.predict(x, verbose = 0)\n",
        "  index = np.argmax(prediction)\n",
        "  result = int_to_char[index]\n",
        "  seq_in = [int_to_char[value] for value in pattern]\n",
        "  sys.stdout.write(result)\n",
        "  pattern.append(index)\n",
        "  pattern = pattern[1: len(pattern)]\n",
        "\n",
        "print('\\nDone!')"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Seed: \n",
            "\" eaching it tricks very much, if--if i'd\n",
            "only been the right size to do it! oh dear! i'd nearly forgo \"\n",
            "ttent the sempe it to and say \"with the noce tore.\n",
            "\n",
            "'what is the dart iist,' said the mock turtle. \n",
            "'i dan't she sea, she mock turtle in the dancer!' and the mock turtle said to the thing so a shmed off,\n",
            "\n",
            "'the mouse sealln,' said the mock turtle. \n",
            "'i dan't she sea would be a mine the moose in the dartu,' said alice, ''the mors of menpoesce of the soot of the sea.' \n",
            "'i don't know all the boom,' she said to herself, 'it was a linute or two she was so the shme in the distance,\n",
            "\n",
            "'the dar'' said the ming, 'i must be a better world be a monk all the tort of the sea. aut i must be she shaner to say it is in the mind, and the lors of the shme it was oo thme to oueer the pight with one of the court, and she was not a minute or two of the sooes of the garden with one of the coor, and was going to see it would be a minute or two she was a linute or two of the sooes of the garden with one of the coor, and was going to see it would be a minute or two she was a linute or two of the sooes of the gard\n",
            "Done!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RAy8QkdBBrdk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JClUxt7UBrdq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mn9TJ0tLBrd0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UuJW6Ka6Brd7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xVTiXpNRBreA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EikWHbxxBreG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KD3RcUXUBreM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iONC2D8uBreT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cBveEtPWBreY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xQ2FRuErBref",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}