sequences.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "sequences.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyP157oCDVC8tMBpiPWCxOOe"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n-vpnSIG7_sQ"
      },
      "source": [
        "# analyzing sequence data\n",
        "\n",
        "In this jupyter notebook, we'll compare a few different approaches for analyzing sequence data.\n",
        "\n",
        "Of course, we'll need some sequence data to analyze.\n",
        "\n",
        "In tensorflow, a 'sequence' is a 1-dimensional 'list' of data, with each 'data point' being a vector. So:\n",
        "\n",
        "    [ 1.2, 2.3, 3.4, 4.5 ]\n",
        "\n",
        "is *not* a sequence in tensorflow, but:\n",
        "\n",
        "    [ [1.2], [2.3], [3.4], [4.5] ]\n",
        "\n",
        "*is* a sequence.\n",
        "\n",
        "You can think of a 'sequence' as similar to a 2-dimensional 'image' in tensorflow, exept you are stuck in 1-dimension, in the case of sequence data. Similar to the case of 'image' data, we can discuss the concept of 'channels' in sequence data.\n",
        "\n",
        "The sequence above has a single 'channel', whereas the sequence below has 3 'channels':\n",
        "\n",
        "    [ [1,2,3], [4,5,6], [7,8,9], [10,11,12] ]\n",
        "\n",
        "As in the case of 2D 'image' data, the *meaning* of the channels will depend on the *meaning* of the data, which is particular to how the data were collected and what the analysis problem is trying to accomplish.\n",
        "\n",
        "For this exercise, we'll just simulate some sequence data, so we can gain some experience analyzing such data and compare different models.\n",
        "\n",
        "In this case, we'll use the scikit-learn make_classification function to simulate sequences of 128 data points (or 'features'), each of which is one-dimensional. For classification, we'll set 32 features as 'informative' or correlated with class labels, while the remaining fetures in the sequence are 'random' with regard to class labels (ie, not informative). We'll simulate a total of 38,262 data samples and use binary class labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PIv8n6q377y2"
      },
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import sklearn.datasets\n",
        "import sklearn.model_selection\n",
        "\n",
        "\n",
        "x, y = sklearn.datasets.make_classification(n_samples=38262,\n",
        "                                            n_features=128,\n",
        "                                            n_informative=32,\n",
        "                                            random_state=8792439)\n",
        "train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(x,y, test_size=0.2, random_state=849691)\n",
        "train_x = np.expand_dims(train_x, axis=-1)\n",
        "valid_x = np.expand_dims(valid_x, axis=-1)\n",
        "train_data = tf.data.Dataset.from_tensor_slices((train_x, train_y)).batch(32)\n",
        "valid_data = tf.data.Dataset.from_tensor_slices((valid_x, valid_y)).batch(32)\n",
        "print(train_data, valid_data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UvAfbsrL7-Vw"
      },
      "source": [
        "In this case, we used the scikit-learn function train_test_split to randomly divide our simulated data into 80% training and 20% validation data.\n",
        "\n",
        "Also notice that we had to use the numpy expand_dims function to transform our simulated data into the correct 'shape' for tensorflow sequences. The scikit-learn make_classification function creates 'sequences' of the form:\n",
        "\n",
        "    [ a, b, c, d ]\n",
        "\n",
        "which is *not* a tensorflow sequence, as it lacks the 'channel dimension' (or rank). To fix this, we 'expand' the last dimension, so the data takes on the form:\n",
        "\n",
        "    [ [a], [b], [c], [d] ]\n",
        "\n",
        "which is compatible with how tensorflow sequence models work.\n",
        "\n",
        "After getting our data in the correct format for tensorflow, it's easy to create tensorflow Dataset objects using the from_tensor_slices function.\n",
        "\n",
        "You should see that your Dataset objects have the shape:\n",
        "\n",
        "    ((None, 128,1), (None,))\n",
        "\n",
        "So, the explanatory variables (ie, the sequence data) have shape:\n",
        "\n",
        "    (None, 128, 1)\n",
        "\n",
        "That is, there are 128 'features' in the sequence, and each feature has dimensionality 1. Another way of saying this is that the sequences have *length* 128, and the sequence data has a single *channel*."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0tWSdc_KG_zu"
      },
      "source": [
        "## convolution model\n",
        "\n",
        "First, we'll analyze our sequence data using a convolution neural-network model, similar to what we did with the 2-dimensional 'image' data.\n",
        "\n",
        "The only 'difference' we need to be aware of when applying convolution models to sequence data is that we'll need to use a tf.keras.layers.Conv1D object to model the 1-dimensional sequence data, rather than the tf.keras.layers.Conv2D object we used to model 2-dimensional 'image' data. \n",
        "\n",
        "For the 1D convolutions, there is only a single 'rank' in the kernel_size, so we'll need to set:\n",
        "\n",
        "    kernel_size=(3,)\n",
        "\n",
        "when we create the 1D convolution layer. Of course, we could use a larger kernel size, if we wanted to.\n",
        "\n",
        "For now, we'll just use a single Conv1D layer with ReLU activation in our model. We'll need to decide on the number of filters we want to use in our convolution layer. For now, we'll just set:\n",
        "\n",
        "    filters=64\n",
        "\n",
        "as a reasonable starting value.\n",
        "\n",
        "As with the 2D 'image' data, we'll need to 'flatten' our sequence data to remove the 'channel' dimension, before sending it to the classification layer.\n",
        "\n",
        "This is a binary classification problem, so we'll use a Dense layer with a single unit and sigmoid activation as the 'decision layer' in the network.\n",
        "\n",
        "Finally, we'll need to use binary crossentropy loss as our loss function. In this example, we'll use the Adam optimizer, and we'll make sure to track the 'accuracy' of our model as it trains, which is commonly done for classification problems, as humans tend to 'reason' better with accuracy than with loss outputs.\n",
        "\n",
        "With such a simple model, we should get a reasonalble model fit after 50 epochs of training using the Adam optimizer. You might want to make sure you're running the model fit with GPU hardware acceleration, though. Click on the downward-arrow in the upper-right corner; select 'View resources', then click on 'Change Runtime Type' and select GPU hardware acceleration before saving. Then you can run the following code cell.\n",
        "\n",
        "Training should take about 5 seconds / epoch with GPU acceleration. So, about 5 minutes or so."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ByW4eKMDJQsM"
      },
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import sklearn.datasets\n",
        "import sklearn.model_selection\n",
        "\n",
        "# simulate and package data\n",
        "x, y = sklearn.datasets.make_classification(n_samples=38262,\n",
        "                                            n_features=128,\n",
        "                                            n_informative=32,\n",
        "                                            random_state=8792439)\n",
        "train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(x,y, test_size=0.2, random_state=849691)\n",
        "train_x = np.expand_dims(train_x, axis=-1)\n",
        "valid_x = np.expand_dims(valid_x, axis=-1)\n",
        "train_data = tf.data.Dataset.from_tensor_slices((train_x, train_y)).batch(32)\n",
        "valid_data = tf.data.Dataset.from_tensor_slices((valid_x, valid_y)).batch(32)\n",
        "print(train_data, valid_data)\n",
        "\n",
        "# build model\n",
        "model = tf.keras.models.Sequential()\n",
        "model.add(tf.keras.layers.Conv1D(filters=64, kernel_size=(3,), activation=tf.keras.activations.relu, input_shape=(128,1)))\n",
        "model.add(tf.keras.layers.Flatten())\n",
        "model.add(tf.keras.layers.Dense(units=1, activation=tf.keras.activations.sigmoid))\n",
        "model.summary()\n",
        "\n",
        "# compile model\n",
        "model.compile(optimizer=tf.keras.optimizers.Adam(),\n",
        "              loss=tf.keras.losses.BinaryCrossentropy(),\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# fit model\n",
        "model.fit(train_data, epochs=50, validation_data=valid_data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lav3kwvTKRLB"
      },
      "source": [
        "This 1D convolution network has 8321 trainable parameters, the vast majority of which (8065) are in the final decision layer. \n",
        "\n",
        "There are only 256 trainable parameters in the convolution layer of the network. We defined our convolution kernel size as (3,), so each convolution neuron has 3 input weights, plus a bias term, so there are 3+1=4 trainable parameters per convolution 'filter'. We wanted 64 filters, so there are 64*4=256 trainable parameters in the convolution layer.\n",
        "\n",
        "After training for 50 epochs using Adam optimization, my convolution network was able to achieve 0.91 classification accuracy on the training data and 0.89 classification accuracy on the validation data. So, about 90% of the sequences are correctly classified by this model.\n",
        "\n",
        "Also note how long it takes to train this model. In my case, each epoch took about 4 seconds to train using GPU hardware acceleration."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eH-DwkfKLrlX"
      },
      "source": [
        "## simple Recurrent Neural Network (RNN) model\n",
        "\n",
        "Next, let's compare the convolution model to what we'd get if we analyzed our data using a simple recurrent neural network model (aka, \"RNN\").\n",
        "\n",
        "Tensorflow implements a simple RNN using a tf.keras.layers.SimpleRNN object, which we can add to our Sequential model just like any other Layer object.\n",
        "\n",
        "There are a couple options to the SimpleRNN that we'll want to set. First, we'll want to set the number of RNN \"units\" in the layer. In the Conv1D model, we used 64 'filters'. In the SimpleRNN model, we'll use *half* that number, or 32 'units' in the RNN:\n",
        "\n",
        "    units=32\n",
        "\n",
        "This will result in an RNN model with slightly *fewer* trainable parameters, compared to the Conv1D model we used previously.\n",
        "\n",
        "We'll also want to set the:\n",
        "\n",
        "    return_sequences=True\n",
        "\n",
        "option flag. In the default behaviour, the SimpleRNN object *only* returns the output from the *last* neuron in the sequence. By setting return_sequences=True, we'll get the otuputs from *all* the neurons in the sequence layer, so the decision layer will have more information from which to classify the sequences.\n",
        "\n",
        "Other than that, we'll just need to make sure we set the input_shape option, as the SimpleRNN layer will be the first layer in the Sequential model.\n",
        "\n",
        "The SimpleRNN model's complation and fitting is implemented in exactly the same way as in the case of the convolution model.\n",
        "\n",
        "By the way, you should be prepared to have something else to do for awhile once you start the model fitting process in the following code cell. Remember how we discussed the sequential nature of RNN models, and how they take longer to train... In fact, the RNN model is so slow, that we're only going to train it for 20 epochs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZD9EkGEpMLPK"
      },
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import sklearn.datasets\n",
        "import sklearn.model_selection\n",
        "\n",
        "# simulate and package data\n",
        "x, y = sklearn.datasets.make_classification(n_samples=38262,\n",
        "                                            n_features=128,\n",
        "                                            n_informative=32,\n",
        "                                            random_state=8792439)\n",
        "train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(x,y, test_size=0.2, random_state=849691)\n",
        "train_x = np.expand_dims(train_x, axis=-1)\n",
        "valid_x = np.expand_dims(valid_x, axis=-1)\n",
        "train_data = tf.data.Dataset.from_tensor_slices((train_x, train_y)).batch(32)\n",
        "valid_data = tf.data.Dataset.from_tensor_slices((valid_x, valid_y)).batch(32)\n",
        "print(train_data, valid_data)\n",
        "\n",
        "# build model\n",
        "model = tf.keras.models.Sequential()\n",
        "FIXME - add SimpleRNN layer with 32 units and return_sequences=True\n",
        "model.add(tf.keras.layers.Flatten())\n",
        "model.add(tf.keras.layers.Dense(units=1, activation=tf.keras.activations.sigmoid))\n",
        "model.summary()\n",
        "\n",
        "# compile model\n",
        "model.compile(optimizer=tf.keras.optimizers.Adam(),\n",
        "              loss=tf.keras.losses.BinaryCrossentropy(),\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# fit model\n",
        "model.fit(train_data, epochs=20, validation_data=valid_data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9ceXpw4IN0Nj"
      },
      "source": [
        "Make sure you keep track of the model's number of parameters, how long it took to train, and the final accuracy and loss values, for the associated quiz!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K6MwIHczOjYy"
      },
      "source": [
        "## Long Short-Term Memory (LSTM) model\n",
        "\n",
        "Next, let's compare a similar LSTM model to the SimpleRNN model we just implemented.\n",
        "\n",
        "Tensorflow implements LSTM neurons as the object, tf.keras.layers.LSTM, so all we need to do is to replace the code:\n",
        "\n",
        "    tf.keras.layers.SimpleRNN(...)\n",
        "\n",
        "With:\n",
        "\n",
        "    tf.keras.layers.LSTM(...)\n",
        "\n",
        "in the above code cell, and we'll have a similar model that uses LSTM neurons. We can keep all the other parameters and options the same as in the previous code cell.\n",
        "\n",
        "Implement the LSTM model below, and run the following code cell to train your LSTM network."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uDTyl8hNNzkZ"
      },
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import sklearn.datasets\n",
        "import sklearn.model_selection\n",
        "\n",
        "# simulate and package data\n",
        "x, y = sklearn.datasets.make_classification(n_samples=38262,\n",
        "                                            n_features=128,\n",
        "                                            n_informative=32,\n",
        "                                            random_state=8792439)\n",
        "train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(x,y, test_size=0.2, random_state=849691)\n",
        "train_x = np.expand_dims(train_x, axis=-1)\n",
        "valid_x = np.expand_dims(valid_x, axis=-1)\n",
        "train_data = tf.data.Dataset.from_tensor_slices((train_x, train_y)).batch(32)\n",
        "valid_data = tf.data.Dataset.from_tensor_slices((valid_x, valid_y)).batch(32)\n",
        "print(train_data, valid_data)\n",
        "\n",
        "# build model\n",
        "model = tf.keras.models.Sequential()\n",
        "FIXME - create model with single LSTM layer having 20 units\n",
        "model.summary()\n",
        "\n",
        "# compile model\n",
        "FIXME - use Adam optimizer and BinaryCrossentropy loss\n",
        "\n",
        "# fit model\n",
        "FIXME - train model for 20 epochs"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YsqfElK8hGM9"
      },
      "source": [
        "Again, keep track of the model's parameter count, training time and accuracy values for the associated quiz.\n",
        "\n",
        "You may have noticed that the LSTM training time is *much less* than the training time required for the SimpleRNN model!\n",
        "\n",
        "That's odd; I thought that LSTM neurons were *more* complicated than SimpleRNN neurons, and LSTM would therefore require *more* training time??\n",
        "\n",
        "Well, in this case there are some back-end optimizations that significantly speed up LSTM training. In particular, specific combinations of options allow LSTM training to take advantage of NVIDIA's cuDNN library for neural network training, which is very fast. So long as you are happy with the default LSTM parameters, you can run LSTM training much faster than the simpler SimpleRNN model."
      ]
    }
  ]
}