program-a-neuron.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "coskmeJSP8rt"
      },
      "source": [
        "# Introduction to Neural Networks\n",
        "\n",
        "Neural networks are a type of algorithm designed to recognize patterns and make decisions. It's called a \"neural network\" because the basic building block of the algorithm, a \"neuron,\" is inspired by how neurons in the human brain work.  \n",
        "\n",
        "Biological neurons have dendrites to receive input, a body for processing inputs (like a threshold for sodium levels), and an axon with terminal ends to pass along information.  \n",
        "![](https://nickmccullum.com/images/python-deep-learning/understanding-neurons-deep-learning/neuron-anatomy.png)  \n",
        "\n",
        "Information moves from one neuron to the next in a network by getting passed from the first neuron's axon to the second neuron's dendrites.  \n",
        "![](https://i1.wp.com/www.brains-explained.com/wp-content/uploads/2015/06/synapse.jpg)  \n",
        "\n",
        "In __artificial neural networks__, neurons can be modeled or represented as a sequence of mathematical functions. Each neuron has multiple inputs (dendrites) and an output that can connect to other neurons (axon terminals). The neuron multiplies each of its inputs by a specific weight and adds a fixed value called a bias, similar to point-slope formula for making a line in algebra. The sum of all the inputs is calculated, and then a special transformation called an \"activation function\" is used to convert the sum into a value between 0 and 1 (sort of like the signal processing that occurs in the body of a real-life neuron). The output can then be passed to the next layer of neurons in the network.  \n",
        "![](https://miro.medium.com/v2/resize:fit:1302/format:webp/1*UA30b0mJUPYoPvN8yJr2iQ.jpeg)  \n",
        "\n",
        "To build an artificial neural network, multiple neurons can be connected with the outputs of one neuron becoming the inputs of the next neuron. These connections are sort of like synapses between biological neurons in the brain.  \n",
        "![](https://nickmccullum.com/images/python-deep-learning/understanding-neurons-deep-learning/neuron-functionality.png)  \n",
        "\n",
        "\n",
        "Neurons can be connected in layers with different patterns that perform a sequence of mathematical operations on the input data. The structure of the layera and the order of the operations is what allows the network to learn how to find patterns in the data. The first layer of neurons that receives the data is the input layer. The last layer of neurons outputs the prediction of the network, so this is called the \"output layer.\" The layers in-between the input and output layers are called \"hidden layers.\" In this diagram, each circle is a neuron in the artificial neural network.  \n",
        "\n",
        "![](https://nickmccullum.com/images/python-deep-learning/what-is-deep-learning/artificial-neural-net.png)  \n",
        "\n",
        "For articial neural networks to make accurate predictions, they must be trained to find what the between combinations of weights and biases should be for each neuron in the network. The values of the weights and biases are usually found through \"forward passes\" and \"backpropagation.\"  \n",
        "\n",
        "In the _forward pass_, input data is fed into the network, and the network calculates a prediction. The accuracy of that prediction is then check using a \"loss function.\" Based on how accurate or inaccurate the prediction was, the weights and biases for each of the neurons are updated to a new value in a process called _backpropagation_. To get the most accurate predictions, the forward passes and backpropagation need to be repeated with A LOT of data.  \n",
        "\n",
        "This training process uses calculus and linear algebra, so for this class we are going to use another method to train the network and learn the best values for the weights of a neuron in a neural network.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YoKyI9R8PEnY"
      },
      "source": [
        "## Coding a Simple Neuron from Scratch\n",
        "Without using back propagation or Python classes. Instead, we will use functions and grid search to find the best combinations of weights for the dendrite of one neuron in a network.\n",
        "\n",
        "For this project, we will use two python libraries:\n",
        "1. [`NumPy`](https://numpy.org/doc/stable/index.html) for fast numerical calculations\n",
        "2. [`Itertools`](https://docs.python.org/3/library/itertools.html), which is a built-in module that we can use to generate search grids\n",
        "\n",
        "From `itertools`, we will use the `product()` function, which calculates all possible combinations of items from a series of lists. For example, if we had two lists: `list_1 = [1, 2, 3]` and `list_2 = [5,6,7]`, then the output from `itertools.product()` would be:\n",
        "\n",
        "|   | 5 | 6 | 7 |\n",
        "|---|---|---|---|\n",
        "| **1** | 1,5 | 1,6 | 1,7 |\n",
        "| **2** | 2,5 | 2,6 | 2,7 |\n",
        "| **3** | 3,5 | 3,6 | 3,7 |\n",
        "\n",
        "For the sake of simplicity and demonstration, we are going to assume the all of the neuron's _biases_ equal zero.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "4LuNKoDwLvHW"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "from itertools import product"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uc1PSaO3O_l3"
      },
      "source": [
        "### Generate Some Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AsuOrJ-AL2KO"
      },
      "outputs": [],
      "source": [
        "# generate random synthetic data for training\n",
        "np.random.seed(2024)\n",
        "X_train = np.random.rand(1000, 3)\n",
        "\n",
        "# make two classes based on the sum of each feature in the data\n",
        "# if the features for one observation sum to > 1.5, then assign the class label \"1\". Otherwise the class label is \"0\"\n",
        "y_train = (X_train[:, 0] + X_train[:, 1] + X_train[:, 2] > 1.5).astype(int) "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IbtoWXc3PO_r"
      },
      "source": [
        "### Exercises\n",
        "#### Define Helper Functions\n",
        "To emulate a neuron in an artificial neural network, we'll write Python functions for the neuron's activation function, a function for the neuron to make predictions, a function to check the neuron's accuracy, and a function to perform a grid search and find the best weights for the neuron's dendrites.\n",
        "\n",
        "#### Exercise 1: Activation Function\n",
        "Write the activation function to use in the neuron. Let's start with a **sigmoid activation function** that forces the neuron's output to be between 0 and 1. A _sigmoid_ function is any mathematical function that creates a graph with an S-shaped curve. For our case, let's use the logistic function (the picture below) as our sigmoid activation function.\n",
        "\n",
        "![A logistic curve or sigmoid function](https://upload.wikimedia.org/wikipedia/commons/thumb/8/88/Logistic-curve.svg/320px-Logistic-curve.svg.png)\n",
        "\n",
        "The equation for the logistic function is $y = \\frac{1}{1 + e^{-x}}$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FwchsCNPTLkG"
      },
      "outputs": [],
      "source": [
        "def sigmoid_activation(x):\n",
        "    # implement the logistic function. \n",
        "    # Your function should take the value x and return the value for y.\n",
        "    # You can use the exponential function from the NumPy package (np.exp)\n",
        "    pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4nX3gR0NT2a-"
      },
      "source": [
        "#### Exercise 2: Prediction Function\n",
        "Write a function that uses the neuron's weights to make a prediction given an input data point. Since we have a vector of weights for each feature, and vector of inputs, we need to take the [dot product](https://www.mathsisfun.com/algebra/vectors-dot-product.html) between the inputs and the weights. Luckily, we do not need to write any extra code to do this because the NumPy package already has the function (`np.dot`). If you want to learn more about the function and tips for how to use it in different situations, you can read the [documentation](https://numpy.org/doc/stable/reference/generated/numpy.dot.html#numpy.dot)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a3p5BHlMUHBG"
      },
      "outputs": [],
      "source": [
        "def predict(X, weights):\n",
        "  # Use the `dot` function from NumPy to take the dot product between \n",
        "  # the input data (X) and the neuron's current vector of weights\n",
        "  # If the output from np.dot is > 0.5, then assign the value of 1\n",
        "  # otherwise, make the value 0\n",
        "  # then return the result\n",
        "  pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Narbfgu8UVaK"
      },
      "source": [
        "#### Exercise 3: Calculate the Accuracy of the Neuron\n",
        "Write a function to calculate the percent accuracy of the neuron's predictions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aOZ5bJ7QUe32"
      },
      "outputs": [],
      "source": [
        "def accuracy_score(true_labels, pred_labels):\n",
        "  # Calculate the number of predicted labels that were correct\n",
        "  # Then divide by the total number of labels and multiply by 100\n",
        "  # Return the result\n",
        "  pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-clvAgnxUg5y"
      },
      "source": [
        "#### Exercise 4: Grid Search\n",
        "Write a function to perform a grid search and optimize values for the neuron's weights. There are **three** features for each observation (row) in the data set. That means we need to have three weights (one for each feature). We can use a for loop to test each combination of potential weights with the grid search, and our earlier helper functions to make predictions and check the accuracy of a prediction.\n",
        "\n",
        "Take a moment and think about what the \"pseudocode\" () for this function should look like. What steps should we take and in what order?\n",
        "\n",
        "Try to think of the pseudocode on your own at first. This is an important skill for programming because it helps organize your thinking with a logical flow. As with almost anything in programming, there are multiple ways to achieve a goal. If you want, you can click below to reveal my pseudocode to check your work or to get inspiration.\n",
        "\n",
        "<details><summary>Click to reveal pseudocode</summary>\n",
        "\n",
        "```\n",
        "create some weights, for now their value can be None\n",
        "set the accuracy to 0\n",
        "\n",
        "for each combination of weights (w1, w2, w3) in the range of possible values:  \n",
        "    use the weights to make a prediction  \n",
        "    check the accuracy of the prediction  \n",
        "    if the new accuracy is better than the current accuracy:  \n",
        "        update the current accuracy  \n",
        "        update the best vector of weights  \n",
        "return the best weights and the best accuracy  \n",
        "```\n",
        "\n",
        "</details>\n",
        "\n",
        "Tips: \n",
        "1. Check out the [`product`](https://docs.python.org/3/library/itertools.html) function from the Python `itertools` module\n",
        "2. Check out how to [return multiple objects in Python](https://pythonbasics.org/multiple-return/)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8U7O8Rf7MTIq"
      },
      "outputs": [],
      "source": [
        "def grid_search(X, y, weight_range):\n",
        "  # write your grid search for the best weights and accuracy\n",
        "  # then return the weights and accuracy (something like this: return best_weights, best_accuracy)\n",
        "  pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zSH0HHWxPSB9"
      },
      "source": [
        "#### Exercise 5: Train the Neuron!\n",
        "To train the neuron with a grid search, we first need to establish the upper and lower limits or the search space. You can use the [`linspace`](https://numpy.org/doc/stable/reference/generated/numpy.linspace.html#numpy-linspace) function from NumPy to generate a vector of certain length between an upper and a lower bound. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "udv2ut5JMTvJ",
        "outputId": "25956a7c-9292-4791-fd1a-46c49f0378cd"
      },
      "outputs": [],
      "source": [
        "# define the bounds of our grid search for each parameter\n",
        "# you can create a list or numpy array of possible values\n",
        "# or you can use the linspace function from NumPy\n",
        "\n",
        "# uncomment the line below and fill it in!\n",
        "# weight_range = \n",
        "\n",
        "# obtain the weights and accuracy of our model by using the grid search function\n",
        "best_weights, best_accuracy = grid_search(X_train, y_train, weight_range)\n",
        "\n",
        "# let's see how we did!\n",
        "print(f\"Best weights: {best_weights}\")\n",
        "print(f\"Best accuracy: {best_accuracy}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2R9_QlqIPVl7"
      },
      "source": [
        "#### Exercise 6: Test the Neuron!\n",
        "Generate some new data following the same generation procedure for the training data set. Then use the neuron to predict on the new data and check the neuron's accuracy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2BYREXoJOdK6"
      },
      "outputs": [],
      "source": [
        "# generate random synthetic data for testing\n",
        "# change the random seed so that way it is replicable but the generated numbers are different\n",
        "np.random.seed(123) \n",
        "\n",
        "# uncomment the lines below and fill them in!\n",
        "# X_test = \n",
        "# y_test = "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iY_fUSBlQuM3",
        "outputId": "80fb3e2c-e3ed-47e1-c157-292940905624"
      },
      "outputs": [],
      "source": [
        "test_predictions = predict(X_test, best_weights)\n",
        "prediction_accuracy = accuracy_score(y_test, test_predictions)\n",
        "print(f\"Accuracy on the test data: {100*np.round(prediction_accuracy, 2)}%\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "34C0k2CrSy9F"
      },
      "source": [
        "## Extra Exercises\n",
        "1. Try an expanded grid search. (Easy)\n",
        "2. Try another activation function. (Medium)\n",
        "3. Add another parameter (weight) to learn. (Hard)\n",
        "4. Put more neurons together! Try making two neurons feed into a third neuron. (Very Hard!)\n",
        "\n",
        "### Exercise 2.1\n",
        "Try to make the neuron more accurate by increasing the range of potential values each of the weights can be. What changes do you notice when you try to train the neuron with the expanded grid search?\n",
        "\n",
        "**BONUS: How long does it take to train the neuron with different grid searches?**\n",
        "Look at the documentation for Python's [time](https://docs.python.org/3/library/time.html) module and see if you can figure out how long it will take with bigger and bigger search grids."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pSb9gzzad01K",
        "outputId": "2fed783e-4950-4f70-85b5-64ce66d950e2"
      },
      "outputs": [],
      "source": [
        "# uncomment the line below and fill it in\n",
        "# new_weight_range = \n",
        "\n",
        "# obtain the weights and accuracy of our model by a grid search\n",
        "best_weights, best_accuracy = grid_search(X_train, y_train, new_weight_range)\n",
        "\n",
        "# let's see how we did!\n",
        "print(f\"Best weights: {best_weights}\")\n",
        "print(f\"Best accuracy: {best_accuracy}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u-iwKrYzdFg_"
      },
      "source": [
        "### Exercise 2.2\n",
        "Write a helper function (call it `relu_activation()`) for the [ReLU (Rectified Linear Unit)](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) activation function. Modify the prediction function (call it `predict_relu()`) from exercise 1.2 to use the new `relu_activation()` function. Then use the grid search function to train the neuron with the new activation function. Which activaiton function led to a more accurate neuron?\n",
        "\n",
        "The ReLU function looks like this when it is plotted:  \n",
        "<img src=\"https://machinelearningmastery.com/wp-content/uploads/2018/10/Line-Plot-of-Rectified-Linear-Activation-for-Negative-and-Positive-Inputs.png\" style=\"width:480px;\"/>\n",
        "\n",
        "Mathematically, ReLU is the maximum of either 0 or the input value \"x\": $ReLU(x) = max(0, x)$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "02u_QL8HVVAg"
      },
      "outputs": [],
      "source": [
        "def relu_activation(x):\n",
        "  pass\n",
        "\n",
        "# modify the prediction function you wrote earlier to use the relu_activation() function\n",
        "def predict_relu(X, weights):\n",
        "  pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-maM7BrGbe3v"
      },
      "source": [
        "#### Exercise 2.3\n",
        "Modify the data generation code to include another measurement about each data point. Then adjust the `grid_search()` function to learn the weight for the additional measurement. Re-train the neuron and see how well it predicts with more data!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "913A1yBtcKeY"
      },
      "outputs": [],
      "source": [
        "# generate 1000 random synthetic data points for training that have four features for each data point\n",
        "# follow the same procedure as before, this time make the threshold for the \"1\" label be 1.75\n",
        "\n",
        "np.random.seed(2024)\n",
        "\n",
        "# X_train2 = \n",
        "# y_train2 = \n",
        "\n",
        "# modify the grid search function to accomodate a fourth input value (or to find a fourth weight)\n",
        "def grid_search_2(X, y, weight_range):\n",
        "  pass\n",
        "\n",
        "# define the bounds of our grid search for each parameter\n",
        "# weight_range = \n",
        "\n",
        "# obtain the weights and accuracy of our model by a grid search\n",
        "best_weights, best_accuracy = grid_search(X_train2, y_train2, weight_range)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NSupyDHweF-C"
      },
      "source": [
        "### Exercise 2.4\n",
        "Let's make a neural _network_! Try training two neurons together, that feed into a third neuron to make the final prediction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CXvaJ2SqfHpt"
      },
      "outputs": [],
      "source": [
        "# this is an extra hard challenge! \n",
        "# Think about how you would take the output of one neuron and feed in as intput to another. \n",
        "# How do you need to modify the grid search function?"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}