GAI_Mod1_4_3_DataProcessing_MissingValues.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyMUeArr9aaMazgPO3Nt+ENW",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/sumanthkrishna/genAIPath/blob/main/GAI_Mod1_4_3_DataProcessing_MissingValues.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Data Cleaning: Missing Values in Exploratory Data Analysis (EDA)**\n",
        "\n",
        "Missing values are a common challenge in real-world datasets, and addressing them is a crucial step in the exploratory data analysis (EDA) phase of data science. Missing values can arise due to various reasons, including data collection errors, system failures, or simply because certain information is not available. In this detailed exploration, we will delve into identifying missing values across different variable types, approaches to handle them, and demonstrate Python code snippets using the well-known Iris dataset.\n",
        "\n",
        "### **1. Identifying Missing Values:**\n",
        "\n",
        "#### **1.1 Across Different Variable Types:**\n",
        "\n",
        "**1.1.1 Numerical Variables:**\n",
        "For numerical variables, we can use the Pandas library to easily identify missing values. The `isnull()` function can be applied to the dataframe to check for missing values, and `sum()` can then be used to get a count for each column.\n",
        "\n",
        "```python\n",
        "import pandas as pd\n",
        "\n",
        "# Assuming df is your dataframe\n",
        "missing_numerical = df.select_dtypes(include=['float64', 'int64']).isnull().sum()\n",
        "```\n",
        "\n",
        "**1.1.2 Categorical Variables:**\n",
        "For categorical variables, the same approach can be used. You need to include object types in the `select_dtypes` parameter.\n",
        "\n",
        "```python\n",
        "missing_categorical = df.select_dtypes(include=['object']).isnull().sum()\n",
        "```\n",
        "\n",
        "#### **1.2 Approaches to Handle Missing Values:**\n",
        "\n",
        "### **2. Approaches to Handle Missing Values:**\n",
        "\n",
        "#### **2.1 Removal:**\n",
        "If the missing values are relatively small in number, removing rows or columns might be an option.\n",
        "\n",
        "```python\n",
        "# Removing rows with any missing values\n",
        "df_cleaned_rows = df.dropna()\n",
        "\n",
        "# Removing columns with any missing values\n",
        "df_cleaned_columns = df.dropna(axis=1)\n",
        "```\n",
        "\n",
        "#### **2.2 Imputation:**\n",
        "For numerical variables, imputation is often done by replacing missing values with the mean, median, or a custom value.\n",
        "\n",
        "```python\n",
        "# Impute missing values with mean\n",
        "df['numerical_column'].fillna(df['numerical_column'].mean(), inplace=True)\n",
        "\n",
        "# Impute missing values with median\n",
        "df['numerical_column'].fillna(df['numerical_column'].median(), inplace=True)\n",
        "```\n",
        "\n",
        "For categorical variables, common imputation strategies include replacing missing values with the mode (most frequent category) or a custom value.\n",
        "\n",
        "```python\n",
        "# Impute missing values with mode\n",
        "df['categorical_column'].fillna(df['categorical_column'].mode()[0], inplace=True)\n",
        "\n",
        "# Impute missing values with a custom value\n",
        "df['categorical_column'].fillna('Not Available', inplace=True)\n",
        "```\n",
        "\n",
        "#### **2.3 Advanced Imputation:**\n",
        "Machine learning models can be used for more advanced imputation. The `SimpleImputer` from scikit-learn is a useful tool for this purpose.\n",
        "\n",
        "```python\n",
        "from sklearn.impute import SimpleImputer\n",
        "\n",
        "imputer = SimpleImputer(strategy='mean')\n",
        "df['numerical_column'] = imputer.fit_transform(df[['numerical_column']])\n",
        "```\n",
        "\n",
        "### **3. Applying Concepts to Iris Dataset:**\n",
        "\n",
        "Now, let's apply these concepts to the Iris dataset:\n",
        "\n",
        "```python\n",
        "# Load the Iris dataset\n",
        "from sklearn.datasets import load_iris\n",
        "import pandas as pd\n",
        "\n",
        "iris = load_iris()\n",
        "df_iris = pd.DataFrame(data=iris.data, columns=iris.feature_names)\n",
        "df_iris['target'] = iris.target\n",
        "\n",
        "# Introduce missing values\n",
        "import numpy as np\n",
        "\n",
        "# Randomly set some values to NaN\n",
        "df_iris.iloc[3:5, 1] = np.nan\n",
        "df_iris.iloc[2:4, 2] = np.nan\n",
        "```\n",
        "\n",
        "With this introduction of missing values, you can then follow the steps outlined earlier to identify and handle missing values based on the variable types in the Iris dataset.\n",
        "\n",
        "In conclusion, handling missing values is a critical aspect of data cleaning in the EDA phase of data science. The approach chosen depends on the nature of the data and the specific requirements of the analysis or modeling task. The examples provided offer a practical guide on how to tackle missing values using Python and Pandas, with a real-world application to the Iris dataset."
      ],
      "metadata": {
        "id": "5gYQQ7h60LEm"
      }
    }
  ]
}