XOR.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyMlff+jr697KwmY4c5qurC+",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/RobInLabUJI/MultilayerPerceptron/blob/main/XOR.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Learning the XOR function with a multilayer perceptron\n",
        "\n",
        "[<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/4/46/Colored_neural_network.svg/199px-Colored_neural_network.svg.png\" align=\"right\">](https://en.wikipedia.org/wiki/Multilayer_perceptron)\n",
        "\n",
        "*From Wikipedia, the free encyclopedia*\n",
        "\n",
        "A [multilayer perceptron](https://en.wikipedia.org/wiki/Multilayer_perceptron) (MLP) is a feedforward artificial neural network model that maps sets of input data onto a set of appropriate outputs. An MLP consists of multiple layers of nodes in a directed graph, with each layer fully connected to the next one. Except for the input nodes, each node is a neuron (or processing element) with a nonlinear activation function. MLP utilizes a supervised learning technique called backpropagation for training the network. MLP is a modification of the standard linear perceptron and can distinguish data that are not linearly separable.\n",
        "\n",
        "Consequently, MLP can outperform the perceptron and solve the XOR problem."
      ],
      "metadata": {
        "id": "1NWf9XAVTqFR"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yZKJDaCAToWB"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "from sklearn.neural_network import MLPClassifier\n",
        "import joblib\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def plot_decision_boundary(mlp, X, Y):\n",
        "    h = .02\n",
        "    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5\n",
        "    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5\n",
        "    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n",
        "    Z = mlp.predict(np.c_[xx.ravel(), yy.ravel()])\n",
        "    Z = Z.reshape(xx.shape)\n",
        "    plt.pcolormesh(xx, yy, Z, shading='auto', cmap=plt.cm.Paired);\n",
        "    plt.xlim(xx.min(), xx.max());\n",
        "    plt.ylim(yy.min(), yy.max());\n",
        "\n",
        "def plot_data(x,y):\n",
        "    colormap = np.array(['r', 'k'])\n",
        "    plt.scatter(x[:,0], x[:,1], c=colormap[y.astype(int)], s=50);\n",
        "\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load in the data\n",
        "\n",
        "This is the data for the truth table of the XOR function."
      ],
      "metadata": {
        "id": "dBgzuMGuUFaA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "x = np.array([[0.,0.],\\\n",
        "              [0.,1.],\\\n",
        "              [1.,0.],\\\n",
        "              [1.,1.]])\n",
        "y = np.array([0.,1.,1.,0.])"
      ],
      "metadata": {
        "id": "47MwJzwOUETY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Build the model\n",
        "\n",
        "Create a [MLP object](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html) with the following arguments:\n",
        "* [stochastic gradient descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent) solver (standard technique in [backpropagation](https://en.wikipedia.org/wiki/Backpropagation))\n",
        "* one hidden layer with 5 neurons\n",
        "* 4000 iterations maximum\n",
        "\n",
        "The rest of the arguments are set to their default values ([see documentation](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html))."
      ],
      "metadata": {
        "id": "Ko2da9nLUK2A"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "net = MLPClassifier(solver='sgd',\\\n",
        "                    hidden_layer_sizes=(5, ),\\\n",
        "                    max_iter=4000)"
      ],
      "metadata": {
        "id": "dG8_OTeKUIV2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Train\n",
        "\n",
        "The `fit` function automatically iterates until convergence or the maximum number of iterations is reached, so you only need to execute the following cell once. When the loss or score is not improving by at least *tol* (default 1e-4) for two consecutive iterations, convergence is considered to be reached and training stops."
      ],
      "metadata": {
        "id": "ia3VRFsQUQce"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "net.fit(x,y)"
      ],
      "metadata": {
        "id": "sVLUuALkUNbx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Plot the data\n",
        "\n",
        "The following figure displays the XOR data and the decision boundary of the MLP. This boundary is not linear anymore: it is represented in colors, so that the blue region in the plane corresponds to the features that are classified as 0's and consequently the brown region is classified as 1's."
      ],
      "metadata": {
        "id": "FjbVW5OmUl9m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "plot_decision_boundary(net,x,y)\n",
        "plot_data(x, y)"
      ],
      "metadata": {
        "id": "veDfY9YKUjiF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Success or failure?\n",
        "\n",
        "There is not a unique solution to this problem. The figure below depicts some possible solutions:\n",
        "\n",
        "<table border=\"0\">\n",
        "<tr><td>\n",
        "<img src=\"https://github.com/RobInLabUJI/MultilayerPerceptron/raw/main/img/xor_1.png\" width=200>\n",
        "</td><td>\n",
        "<img src=\"https://github.com/RobInLabUJI/MultilayerPerceptron/raw/main/img/xor_2.png\" width=200>\n",
        "</td><td>\n",
        "<img src=\"https://github.com/RobInLabUJI/MultilayerPerceptron/raw/main/img/xor_3.png\" width=200>\n",
        "</td></tr><tr><td>\n",
        "<img src=\"https://github.com/RobInLabUJI/MultilayerPerceptron/raw/main/img/xor_4.png\" width=200>\n",
        "</td><td>\n",
        "<img src=\"https://github.com/RobInLabUJI/MultilayerPerceptron/raw/main/img/xor_5.png\" width=200>\n",
        "</td><td>\n",
        "<img src=\"https://github.com/RobInLabUJI/MultilayerPerceptron/raw/main/img/xor_6.png\" width=200>\n",
        "</td></tr>\n",
        "</table>\n",
        "\n",
        "However, the network may **not** converge. Sometimes, it can get stuck in a local minima. The algorithm stops but the result is not correct at all. For example:\n",
        "\n",
        "<table border=\"0\">\n",
        "<tr><td>\n",
        "<img src=\"https://github.com/RobInLabUJI/MultilayerPerceptron/raw/main/img/xor_fail_1.png\" width=200>\n",
        "</td><td>\n",
        "<img src=\"https://github.com/RobInLabUJI/MultilayerPerceptron/raw/main/img/xor_fail_2.png\" width=200>\n",
        "</td><td>\n",
        "<img src=\"https://github.com/RobInLabUJI/MultilayerPerceptron/raw/main/img/xor_fail_3.png\" width=200>\n",
        "</td></tr>\n",
        "</table>\n",
        "\n",
        "Check the result of your network. You should find at least one convergent network, and one non-converging network. Then use the code below for saving each of the networks into a separate file. The networks can later be loaded and analysed without need of training."
      ],
      "metadata": {
        "id": "wYrtaU-MUtsh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Non-converging network\n",
        "\n",
        "Percentage of correct classification of the training data:"
      ],
      "metadata": {
        "id": "N0lIPq8KUvYu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "net.score(x,y) * 100"
      ],
      "metadata": {
        "id": "hwmHxrMJUtNj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Number of iterations during training:"
      ],
      "metadata": {
        "id": "Jju32vI7UyeY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "net.n_iter_"
      ],
      "metadata": {
        "id": "HVAUlmTKUo9X"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Loss curve: (currently, [MLPClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier) supports only the [Cross-Entropy loss function](https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_error_function_and_logistic_regression))"
      ],
      "metadata": {
        "id": "5I09HwWgU46Y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "plt.plot(net.loss_curve_);\n",
        "plt.xlabel('Iterations');\n",
        "plt.ylabel('Loss');"
      ],
      "metadata": {
        "id": "jNBQtj8YU35Z"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Save the network into a file."
      ],
      "metadata": {
        "id": "8ccLlm-8U-nn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "joblib.dump(net, 'xor_non_converging.pkl') "
      ],
      "metadata": {
        "id": "zYlvRI1wU9eY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Converged network\n",
        "\n",
        "Percentage of correct classification of the training data:"
      ],
      "metadata": {
        "id": "1IlgeS9sVCt5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "net.score(x,y) * 100"
      ],
      "metadata": {
        "id": "r51qeOqnVEAe"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Number of iterations during training:"
      ],
      "metadata": {
        "id": "xvyUkCqtVH-5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "net.n_iter_"
      ],
      "metadata": {
        "id": "noooFlBLVGiH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Loss curve: (currently, [MLPClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier) supports only the [Cross-Entropy loss function](https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_error_function_and_logistic_regression))"
      ],
      "metadata": {
        "id": "vFjuxroHVL5I"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "plt.plot(net.loss_curve_);\n",
        "plt.xlabel('Iterations');\n",
        "plt.ylabel('Loss');"
      ],
      "metadata": {
        "id": "F20sbYBSVLRw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Save the network into a file."
      ],
      "metadata": {
        "id": "t4PalUkbVRDz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "joblib.dump(net, 'xor_converged.pkl') "
      ],
      "metadata": {
        "id": "OtlBrpHlVPxc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Download the files (tested in Google Chrome, won't work with Firefox)."
      ],
      "metadata": {
        "id": "8MKe3xlSWPX9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import files\n",
        "files.download('xor_non_converging.pkl')\n",
        "files.download('xor_converged.pkl')"
      ],
      "metadata": {
        "id": "xB4dc2uDWSPv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You should now check the saved files in [this notebook](https://github.com/RobInLabUJI/MultilayerPerceptron/blob/main/XOR_check.ipynb)."
      ],
      "metadata": {
        "id": "wZMzxtkiVVm3"
      }
    }
  ]
}