Assignment(BOOSTING2).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ec2f1b01-6a5e-47cb-ba03-4d345908f578",
   "metadata": {},
   "source": [
    "**Q1. What is Gradient Boosting Regression?**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7926d3af-28cd-4c5e-b61b-eca2a14dbaf6",
   "metadata": {},
   "source": [
    "Gradient Boosting Regression (GBR) is a machine learning algorithm that belongs to the family of boosting algorithms. It is a supervised learning algorithm used for both regression and classification problems.\n",
    "\n",
    "In Gradient Boosting Regression (GBR), a decision tree is fitted to the data in a step-wise manner, with each new tree attempting to correct the errors made by the previous tree. The prediction of the ensemble model is obtained by summing the predictions of all the individual trees.\n",
    "\n",
    "The \"gradient\" in gradient boosting refers to the use of the gradient descent optimization algorithm to minimize the loss function (e.g., mean squared error) of the model. At each step, the gradient of the loss function with respect to the prediction is computed, and the tree is fitted to the negative gradient, which corresponds to the direction of steepest descent.\n",
    "\n",
    "GBR is a powerful algorithm that can handle a large number of features and nonlinear relationships between the features and the target variable. However, it can be prone to overfitting if the model is too complex or if the data is noisy. Therefore, it is important to tune the hyperparameters of the model (e.g., the number of trees, the learning rate, the maximum depth of the trees) to achieve the best performance on the validation set."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69b2f942-2747-42b0-98b1-aa142f1d47ae",
   "metadata": {},
   "source": [
    "**Q2. Implement a simple gradient boosting algorithm from scratch using Python and NumPy. Use a simple regression problem as an example and train the model on a small dataset. Evaluate the model's performance using metrics such as mean squared error and R-squared.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b9b4526a-3ee7-44a9-93e0-27c49908b983",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean squared error: 963.63\n",
      "R-squared: 0.84\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import make_regression\n",
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "import numpy as np\n",
    "\n",
    "class GradientBoostingRegressor:\n",
    "    \n",
    "    def __init__(self, n_estimators=100, learning_rate=0.1, max_depth=3):\n",
    "        self.n_estimators = n_estimators\n",
    "        self.learning_rate = learning_rate\n",
    "        self.max_depth = max_depth\n",
    "        self.trees = []\n",
    "        self.intercept = None\n",
    "        \n",
    "    def fit(self, X, y):\n",
    "        self.intercept = np.mean(y)\n",
    "        residual = y - self.intercept\n",
    "        \n",
    "        for i in range(self.n_estimators):\n",
    "            tree = DecisionTreeRegressor(max_depth=self.max_depth)\n",
    "            tree.fit(X, residual)\n",
    "            self.trees.append(tree)\n",
    "            pred = tree.predict(X)\n",
    "            residual -= self.learning_rate * pred\n",
    "            \n",
    "    def predict(self, X):\n",
    "        preds = np.array([tree.predict(X) for tree in self.trees])\n",
    "        return self.intercept + self.learning_rate * np.sum(preds, axis=0)\n",
    "\n",
    "# Generate a random regression problem\n",
    "X, y = make_regression(n_samples=100, n_features=5, noise=0.5)\n",
    "\n",
    "# Split data into training and testing sets\n",
    "n_train = int(0.8 * len(X))\n",
    "X_train, y_train = X[:n_train], y[:n_train]\n",
    "X_test, y_test = X[n_train:], y[n_train:]\n",
    "\n",
    "# Train a gradient boosting regressor on the training set\n",
    "gb = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1, max_depth=3)\n",
    "gb.fit(X_train, y_train)\n",
    "\n",
    "# Evaluate the model on the testing set\n",
    "y_pred = gb.predict(X_test)\n",
    "mse = mean_squared_error(y_test, y_pred)\n",
    "r2 = r2_score(y_test, y_pred)\n",
    "\n",
    "print(f\"Mean squared error: {mse:.2f}\")\n",
    "print(f\"R-squared: {r2:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "364a9720-1644-4985-bb62-1ca4dc313d52",
   "metadata": {},
   "source": [
    "**Q3. Experiment with different hyperparameters such as learning rate, number of trees, and tree depth to optimise the performance of the model. Use grid search or random search to find the best hyperparameters.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a6d42e7-0355-41e3-bfda-d46aed70b5a5",
   "metadata": {},
   "source": [
    "Here is an implementation of grid search to find the best hyperparameters for the GradientBoostingRegressor model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ab4d047-b447-4451-8514-8ec73ef5fb09",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_regression\n",
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from numpy import arange\n",
    "\n",
    "# Generate a random regression problem\n",
    "X, y = make_regression(n_samples=100, n_features=5, noise=0.5)\n",
    "\n",
    "# Split data into training and testing sets\n",
    "n_train = int(0.8 * len(X))\n",
    "X_train, y_train = X[:n_train], y[:n_train]\n",
    "X_test, y_test = X[n_train:], y[n_train:]\n",
    "\n",
    "# Define hyperparameters for grid search\n",
    "param_grid = {\n",
    "    'n_estimators': [50, 100, 150],\n",
    "    'learning_rate': [0.01, 0.1, 0.5],\n",
    "    'max_depth': [2, 3, 4],\n",
    "}\n",
    "\n",
    "# Create a GradientBoostingRegressor object\n",
    "gb = GradientBoostingRegressor()\n",
    "\n",
    "# Use GridSearchCV to search for the best hyperparameters\n",
    "grid_search = GridSearchCV(gb, param_grid=param_grid, cv=5)\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "# Print the best hyperparameters\n",
    "print(f\"Best hyperparameters: {grid_search.best_params_}\")\n",
    "\n",
    "# Evaluate the best model on the testing set\n",
    "y_pred = grid_search.predict(X_test)\n",
    "mse = mean_squared_error(y_test, y_pred)\n",
    "r2 = r2_score(y_test, y_pred)\n",
    "\n",
    "print(f\"Mean squared error: {mse:.2f}\")\n",
    "print(f\"R-squared: {r2:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "778ba6e8-cb43-4d31-a62f-7a7e241d2838",
   "metadata": {},
   "source": [
    "In this implementation, we use scikit-learn's GridSearchCV to search for the best hyperparameters. We can define a dictionary of hyperparameters to search over, and then pass this dictionary and the GradientBoostingRegressor object to GridSearchCV. We also specify a 5-fold cross-validation to evaluate the models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1b9e960-129b-44e4-b44b-bc6a327255a9",
   "metadata": {},
   "source": [
    "You can also use RandomizedSearchCV to perform a random search over the hyperparameter space. Here's an example of how to use RandomizedSearchCV:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7308e98e-9361-4b11-97ec-edef826ad3fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from scipy.stats import uniform\n",
    "\n",
    "# Define hyperparameters for random search\n",
    "param_dist = {\n",
    "    'n_estimators': range(50, 151, 10),\n",
    "    'learning_rate': uniform(0.01, 0.5),\n",
    "    'max_depth': range(2, 5),\n",
    "}\n",
    "\n",
    "# Create a GradientBoostingRegressor object\n",
    "gb = GradientBoostingRegressor()\n",
    "\n",
    "# Use RandomizedSearchCV to search for the best hyperparameters\n",
    "random_search = RandomizedSearchCV(gb, param_distributions=param_dist, cv=5, n_iter=20)\n",
    "random_search.fit(X_train, y_train)\n",
    "\n",
    "# Print the best hyperparameters\n",
    "print(f\"Best hyperparameters: {random_search.best_params_}\")\n",
    "\n",
    "# Evaluate the best model on the testing set\n",
    "y_pred = random_search.predict(X_test)\n",
    "mse = mean_squared_error(y_test, y_pred)\n",
    "r2 = r2_score(y_test, y_pred)\n",
    "\n",
    "print(f\"Mean squared error: {mse:.2f}\")\n",
    "print(f\"R-squared: {r2:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "166372de-94d8-48b5-b05f-9913aef9569e",
   "metadata": {},
   "source": [
    "**Steps to use grid search or random search to find the best hyperparameters for a machine learning model:**\n",
    "1. **Define the hyperparameters that you want to optimize**: In this case, we will optimize the learning rate, number of trees, and tree depth of a gradient boosting model, given for any dataset.\n",
    "2. **Define a range of values for each hyperparameter**: For example, the learning rate will be of 0.01 to 0.1, the number of trees will be of 100 to 1000, and the tree depth will be of 2 to 10.\n",
    "3. **Create a grid or a list of hyperparameter combinations to search over**: For example, we can create a grid of all possible combinations of hyperparameters. Also, we can randomly select a certain number of possible combinations of hyperparameters.\n",
    "4. **Train and evaluate the model for each hyperparameter combination**: For each combination of hyperparameters, train the model on the training set and evaluate its performance on the validation set using a suitable evaluation metric, such as accuracy or mean squared error.\n",
    "5. **Select the best hyperparameters**: Choose the hyperparameters that result in the best performance on the validation set."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a11c321-291b-43a5-968b-a7f87586dd13",
   "metadata": {},
   "source": [
    "**Q4. What is a weak learner in Gradient Boosting?**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66d110db-164b-4ef3-88ba-547219388e63",
   "metadata": {},
   "source": [
    "In Gradient Boosting, a weak learner is a machine learning algorithm that performs only slightly better than random guessing, but is still able to learn from the training data. The concept of a weak learner in Gradient Boosting algorithm, combines many weak learners to create a strong learner.\n",
    "\n",
    "In Gradient Boosting, each weak learner is typically a decision tree with a shallow depth, often referred to as a decision stump. During training, the algorithm fits the weak learner to the data, and then adjusts the weights of the training examples to focus on the examples that were poorly predicted by the previous weak learner. The algorithm then adds the new weak learner to the ensemble, and repeats this process many times to gradually improve the model's accuracy.\n",
    "\n",
    "The strength of the Gradient Boosting algorithm lies in its ability to combine many weak learners to create a strong learner. By adding many weak learners and focusing on the examples that are difficult to predict, the algorithm is able to learn complex relationships in the data and create a highly accurate model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "563a081a-592f-4dac-b07c-1653b72dcb3b",
   "metadata": {},
   "source": [
    "**Q5. What is the intuition behind the Gradient Boosting algorithm?**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9052c04-05a6-450b-a39e-ea4bcbde8f7e",
   "metadata": {},
   "source": [
    "The intuition behind the Gradient Boosting algorithm is to iteratively improve the accuracy of a model by combining many weak learners to create a strong learner.\n",
    "\n",
    "The algorithm starts by fitting a weak learner to the training data. This initial model may not be very accurate, but it provides a starting point for the algorithm to build on. The algorithm then evaluates the errors made by this model, and focuses on the examples that were poorly predicted. The next weak learner is then trained to focus on these difficult examples, in an attempt to correct the errors made by the previous model.\n",
    "\n",
    "This process of adding a new weak learner and adjusting the weights of the examples is repeated many times, with each new weak learner improving on the errors made by the previous model. The final model is then created by combining all of the weak learners into a single ensemble model.\n",
    "\n",
    "The key idea behind Gradient Boosting is that each weak learner focuses on the examples that are difficult to predict, and by combining many of these weak learners, the algorithm is able to learn complex relationships in the data and create a highly accurate model. The \"gradient\" in the name comes from the fact that the algorithm uses the gradient of the loss function to determine how to adjust the weights of the training examples at each step, in order to focus on the examples that are difficult to predict."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "413b0cc6-886a-46e9-9c9b-b59224592cf4",
   "metadata": {},
   "source": [
    "**Q6. How does Gradient Boosting algorithm build an ensemble of weak learners?**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d25a76f6-ac0d-4b31-ba56-6b8bafe4b1cd",
   "metadata": {},
   "source": [
    "The Gradient Boosting algorithm builds an ensemble of weak learners by iteratively adding new weak learners to the ensemble and adjusting the weights of the training examples to focus on the examples that are difficult to predict. This algorithm works as follows:\n",
    "1. **Initialize the model**: The algorithm starts by initializing the model with a single weak learner, such as a decision tree with a shallow depth.\n",
    "2. **Fit the model**: The weak learner is fit to the training data, and the model makes predictions on the training set.\n",
    "3. **Compute the residuals**: The algorithm computes the difference between the predicted values and the true values on the training set. These differences are called the residuals.\n",
    "4. **Update the weights**: The weights of the training examples are adjusted based on the residuals. Examples, or simply features, that were poorly predicted by the previous model are given more weight, while examples that were well predicted are given less weight.\n",
    "5. **Fit a new weak learner**: A new weak learner is fit to the training data, with the weights of the examples, or simply features, that were adjusted to focus on the examples that are difficult to predict.\n",
    "6. **Add the new weak learner to the ensemble**: The new weak learner is added to the ensemble of weak learners.\n",
    "7. **Repeat**: Steps 2-6 are repeated many times, with each new weak learner focusing on the examples that are difficult to predict, and the weights of the training examples adjusted to emphasize these examples.\n",
    "8. **Combine the weak learners**: The final model is created by combining all of the weak learners in the ensemble. The predictions of the weak learners are weighted according to their performance on the training set, and the weighted sum of the predictions is used to make the final prediction.\n",
    "\n",
    "By iteratively adding new weak learners and adjusting the weights of the training examples, the Gradient Boosting algorithm is able to create a highly accurate ensemble model that can learn complex relationships in the data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5410c56f-2d1e-4ef4-846d-33ecdaf1d290",
   "metadata": {},
   "source": [
    "**Q7. What are the steps involved in constructing the mathematical intuition of Gradient Boosting algorithm?**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e5abb98-374c-4ec7-9ab2-9b1db922cf17",
   "metadata": {},
   "source": [
    "The mathematical intuition behind the Gradient Boosting algorithm can be broken down into several key steps:\n",
    "1. **Define the Loss Function**: The first step is to define a loss function that measures the difference between the predicted values and the true values. Common loss functions for regression problems include the mean squared error (MSE) and the mean absolute error (MAE). For classification problems, common loss functions include the binary cross-entropy loss and the multi-class cross-entropy loss.\n",
    "2. **Fit the Initial Model**: The algorithm starts by fitting an initial model to the data, such as a decision tree with a shallow depth. This initial model may not be very accurate, but it provides a starting point for creating an algorithm.\n",
    "3. **Compute the Residuals**: The algorithm computes the difference between the predicted values and the true values on the training set. These differences are called the residuals.\n",
    "4. **Fit a New Weak Learner**: The next step is to fit a new weak learner to the residuals. The new weak learner should focus on the examples that were poorly predicted by the previous model, in an attempt to correct the errors made by the previous model. The most common choice for a weak learner in Gradient Boosting is a decision tree with a shallow depth.\n",
    "5. **Update the Model**: The predictions of the new weak learner are added to the predictions of the previous model, and the model is updated to minimize the loss function. This is typically done using a gradient descent algorithm, where the gradient of the loss function is used to determine the direction and magnitude of the update.\n",
    "6. **Repeat**: Steps 3-5 are repeated many times, with each new weak learner focusing on the examples that are difficult to predict, and the model updated to minimize the loss function.\n",
    "7. **Combine the Weak Learners**: The final model is created by combining all of the weak learners in the ensemble. The predictions of the weak learners are weighted according to their performance on the training set, and the weighted sum of the predictions is used to make the final prediction.\n",
    "\n",
    "By iteratively adding new weak learners and adjusting the model to minimize the loss function, the Gradient Boosting algorithm is able to create a highly accurate ensemble model that can learn complex relationships in the data. The key insight is that each new weak learner focuses on the examples that are difficult to predict, and by combining many of these weak learners, the algorithm is able to improve the accuracy of the model over time."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}