Tutorials/Python_Data_Engineering/4. Python Advance/Pandas Basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8625f6ad",
   "metadata": {},
   "source": [
    "# Introduction to\n",
    "***\n",
    "![Pandas_logo](images/pandas.png)\n",
    "Image source:https://en.wikipedia.org/wiki/Pandas_(software)\n",
    "***\n",
    "The most widely used Python library for data science"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9510287e",
   "metadata": {},
   "source": [
    "It has nothing to do with cute bears. Instead it stands for **Pan**el **da**ta - **Pandas**\n",
    "***\n",
    "![cute_pandas](images/cute_pandas.jpg)\n",
    "Image source:https://wallpaper-house.com/wallpaper-id-399850.php"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "165a0622",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Why Pandas ?\n",
    "***\n",
    "<img src=\"images/why-pandas.jpg\" width=\"70%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4cbfb74",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Features of Pandas\n",
    "***\n",
    "<img src=\"images/pandas-features.jpg\" width=\"70%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3712337c",
   "metadata": {},
   "source": [
    "\n",
    "# Pandas Data Structures\n",
    "***\n",
    "<img src=\"images/pandas-datastructures.jpg\" width=\"70%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07251010",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "# Pandas Series\n",
    "***\n",
    "* Very similar to a NumPy array.\n",
    "\n",
    "* What differentiates the NumPy array from a Series, is that a Series can have axis labels, meaning it can be indexed by a label, instead of just a number location."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6548d8ef",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## How to create a Series?\n",
    "***\n",
    "You can convert a list, numpy array, or dictionary to a Series. To create a series you call the function pd.Series(). Let's see an example of the same.\n",
    "\n",
    "We will create a series from \n",
    "- list\n",
    "- numpy array \n",
    "- dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b5305527",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "14f8fdc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_list = [10, 20, 30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9b8a00fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "series = pd.Series(my_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ba60ced1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([10, 20, 30])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "series.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0f2ac6ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    10\n",
      "1    20\n",
      "2    30\n",
      "dtype: int64\n",
      "RangeIndex(start=0, stop=3, step=1)\n",
      "[10 20 30]\n"
     ]
    }
   ],
   "source": [
    "#creating a pandas series from a list\n",
    "import pandas as pd\n",
    "\n",
    "my_list = [10, 20, 30]\n",
    "series = pd.Series(my_list)\n",
    "\n",
    "print(series)\n",
    "print(series.index)\n",
    "print(series.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3d1a06dd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    10\n",
       "b    20\n",
       "c    30\n",
       "dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# creating a series from numPy Array\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "index = ['a','b','c']\n",
    "arr = np.array([10,20,30])\n",
    "\n",
    "pd.Series(data=arr,index=index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "90cad633",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    10\n",
       "b    20\n",
       "c    30\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# creating a series from dictionary\n",
    "import pandas as pd\n",
    "\n",
    "d = {'a':10, 'b':20, 'c':30}\n",
    "pd.Series(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ab23c51",
   "metadata": {},
   "source": [
    "\n",
    "## Using Index in a Series\n",
    "***\n",
    "* The key to using a Series is understanding its index.\n",
    "\n",
    "* Pandas makes use of these index names or numbers by allowing for **fast lookups** of information (works like a hash table or dictionary).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d51f8368",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "# Custom index\n",
    "import pandas as pd\n",
    "ser1 = pd.Series([1,2,3,4], index=['USA', 'Germany','USSR', 'Japan']) \n",
    "ser2 = pd.Series([1,2,5,4], index=['USA', 'Germany','Italy', 'Japan'])   \n",
    "\n",
    "# get the value of 'USA'\n",
    "print(ser1['USA'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9b2c8099",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "USA        1\n",
       "Germany    2\n",
       "Italy      5\n",
       "Japan      4\n",
       "dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "9339516c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Germany    4.0\n",
      "Italy      NaN\n",
      "Japan      8.0\n",
      "USA        2.0\n",
      "USSR       NaN\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "print(ser1 + ser2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b33b9da",
   "metadata": {},
   "source": [
    "# DataFrame Basics\n",
    "\n",
    "***\n",
    "\n",
    "We'll talk about\n",
    "- How to create a DataFrame, the primary data structure in pandas \n",
    "- How to find the shape and rank of the created or existing DataFrames\n",
    "- How to read DataFrames from a file\n",
    "- What are indexes, and how do they work in the domain of Pandas DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "566a9e43",
   "metadata": {},
   "source": [
    "## What are DataFrames?\n",
    "\n",
    "***\n",
    "\n",
    "DataFrames are a way to store data in rectangular grids that can easily be overviewed. Each row of these grids corresponds to measurements or values of an instance, while each column is a vector containing data for a specific variable. This means that a data frame’s rows do not need to contain, but can contain, the same type of values: they can be numeric, character, logical, etc.\n",
    "<br> </br>\n",
    "\n",
    "Data frames in Python come within the Pandas library, and they are defined as a two-dimensional labeled data structures with columns of potentially different types.\n",
    "<br> </br>\n",
    "\n",
    "In general, you could say that the Pandas data frame consists of three main components: the data, the index, and the columns.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54d87221",
   "metadata": {},
   "source": [
    "**Creating DataFrames manually**\n",
    "\n",
    "The function that you will use is the Pandas Dataframe() function: it requires you to pass the data that you want to put in, the indices and the columns.\n",
    "\n",
    "Remember that the data that is contained within the data frame doesn’t have to be homogenous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "676e99a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0  1  2\n",
       "0  1  2  3\n",
       "1  3  4  5\n",
       "2  5  6  7\n",
       "3  7  8  9"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "df = pd.DataFrame([[1, 2, 3],\n",
    "                   [3, 4, 5],\n",
    "                   [5, 6, 7],\n",
    "                   [7, 8, 9]])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "52e2ee83",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (4, 3)\n",
      "Index: RangeIndex(start=0, stop=4, step=1)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0  1  2\n",
       "0  1  2  3\n",
       "1  3  4  5\n",
       "2  5  6  7\n",
       "3  7  8  9"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame([[1, 2, 3], [3, 4, 5], [5, 6, 7], [7, 8, 9]])\n",
    "\n",
    "print(\"Shape:\", df.shape)\n",
    "print(\"Index:\", df.index)\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46a3fd23",
   "metadata": {},
   "source": [
    "**Understanding the Index**\n",
    "\n",
    "Before you start with adding, deleting and renaming the components of your DataFrame, you first need to know how you can select these elements.\n",
    "\n",
    "This is where Indexes come into play, just the way you can use an idex page in a book to locate your chapters, you can use the loc() or iloc() function in pandas to access data in particular columns of your DataFrame.\n",
    "\n",
    "We will learn about how these functions work and their subtle differences in the next sections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "f6d3db63",
   "metadata": {},
   "outputs": [],
   "source": [
    "df2 = pd.DataFrame([[1, 2, 3], [3, 4, 5], [5, 6, 7], [7, 8, 9]],\n",
    "                  index= ['a','b','c','d'], columns=['x','y','z'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d7ed61d6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   x  y  z\n",
       "a  1  2  3\n",
       "b  3  4  5\n",
       "c  5  6  7\n",
       "d  7  8  9"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "142b0d80",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (4, 3)\n",
      "Index: RangeIndex(start=0, stop=4, step=1)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   x  y  z\n",
       "a  1  2  3\n",
       "b  3  4  5\n",
       "c  5  6  7\n",
       "d  7  8  9"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = pd.DataFrame([[1, 2, 3], [3, 4, 5], [5, 6, 7], [7, 8, 9]],\n",
    "                   index=['a', 'b', 'c', 'd'], columns=['x', 'y', 'z'])\n",
    "\n",
    "print(\"Shape:\", df.shape)\n",
    "print(\"Index:\", df.index)\n",
    "\n",
    "df2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b67e13d2",
   "metadata": {},
   "source": [
    "# The Weather Dataset: Reading DataFrames from Files\n",
    "***\n",
    "The Weather Dataset is a time-series data set with per-hour information about the weather conditions at a particular location. It records Temperature, Dew Point Temperature, Relative Humidity, Wind Speed, Visibility, Pressure, and Conditions.\n",
    "\n",
    "<img src=\"images/weather.jpg\" alt=\"Weather\" style=\"width: 200px;\"/>\n",
    "\n",
    "This data is available as a CSV file. We are going to use Pandas DataFrames and analyse this dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "id": "25031878",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (8784, 8)\n",
      "Index: RangeIndex(start=0, stop=8784, step=1)\n"
     ]
    }
   ],
   "source": [
    "# Read the data into a data frame\n",
    "\n",
    "weather_df = pd.read_csv(\"data/weather_2012.csv\") # \n",
    "\n",
    "print(\"Shape:\", weather_df.shape)\n",
    "print(\"Index:\", weather_df.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "id": "4083f8b9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date/Time</th>\n",
       "      <th>Temp (C)</th>\n",
       "      <th>Dew Point Temp (C)</th>\n",
       "      <th>Rel Hum (%)</th>\n",
       "      <th>Wind Spd (km/h)</th>\n",
       "      <th>Visibility (km)</th>\n",
       "      <th>Stn Press (kPa)</th>\n",
       "      <th>Weather</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-01 00:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.9</td>\n",
       "      <td>86</td>\n",
       "      <td>4</td>\n",
       "      <td>8.0</td>\n",
       "      <td>101.24</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-01-01 01:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.7</td>\n",
       "      <td>87</td>\n",
       "      <td>4</td>\n",
       "      <td>8.0</td>\n",
       "      <td>101.24</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-01-01 02:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.4</td>\n",
       "      <td>89</td>\n",
       "      <td>7</td>\n",
       "      <td>4.0</td>\n",
       "      <td>101.26</td>\n",
       "      <td>Freezing Drizzle,Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2012-01-01 03:00:00</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>-3.2</td>\n",
       "      <td>88</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>101.27</td>\n",
       "      <td>Freezing Drizzle,Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-01-01 04:00:00</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>-3.3</td>\n",
       "      <td>88</td>\n",
       "      <td>7</td>\n",
       "      <td>4.8</td>\n",
       "      <td>101.23</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Date/Time  Temp (C)  Dew Point Temp (C)  Rel Hum (%)  \\\n",
       "0  2012-01-01 00:00:00      -1.8                -3.9           86   \n",
       "1  2012-01-01 01:00:00      -1.8                -3.7           87   \n",
       "2  2012-01-01 02:00:00      -1.8                -3.4           89   \n",
       "3  2012-01-01 03:00:00      -1.5                -3.2           88   \n",
       "4  2012-01-01 04:00:00      -1.5                -3.3           88   \n",
       "\n",
       "   Wind Spd (km/h)  Visibility (km)  Stn Press (kPa)               Weather  \n",
       "0                4              8.0           101.24                   Fog  \n",
       "1                4              8.0           101.24                   Fog  \n",
       "2                7              4.0           101.26  Freezing Drizzle,Fog  \n",
       "3                6              4.0           101.27  Freezing Drizzle,Fog  \n",
       "4                7              4.8           101.23                   Fog  "
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "id": "a0c7ac1c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2012-01-01 00:00:00\n",
       "1    2012-01-01 01:00:00\n",
       "2    2012-01-01 02:00:00\n",
       "3    2012-01-01 03:00:00\n",
       "4    2012-01-01 04:00:00\n",
       "Name: Date/Time, dtype: object"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df['Date/Time'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4ec2280",
   "metadata": {},
   "source": [
    "**Let's convert the `Date/Time` column datatype from `object` to `timestamp` so that we can access the month directly using the attribute `dt.month`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "id": "b262f280",
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_df['Date/Time'] = pd.to_datetime(weather_df['Date/Time'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "id": "de39c0b4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0   2012-01-01 00:00:00\n",
       "1   2012-01-01 01:00:00\n",
       "2   2012-01-01 02:00:00\n",
       "3   2012-01-01 03:00:00\n",
       "4   2012-01-01 04:00:00\n",
       "Name: Date/Time, dtype: datetime64[ns]"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df['Date/Time'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbbfde20",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "# How to Analyze DataFrames?\n",
    "***\n",
    "The following functions help you understand and explore summaries of your data without having to view the whole DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b03bfa44",
   "metadata": {},
   "source": [
    "## `.info()`\n",
    "***\n",
    "Provides a summary of a DataFrame: rows, columns, data types of columns (if automatically detected) and the memory usage.\n",
    "\n",
    "For detailed summaries of the the DataFrame, you can pass optional arguments verbose=True and null_counts=True to the .info() method to output information for all of the columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "7459dd26",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 8784 entries, 0 to 8783\n",
      "Data columns (total 8 columns):\n",
      " #   Column              Non-Null Count  Dtype         \n",
      "---  ------              --------------  -----         \n",
      " 0   Date/Time           8784 non-null   datetime64[ns]\n",
      " 1   Temp (C)            8784 non-null   float64       \n",
      " 2   Dew Point Temp (C)  8784 non-null   float64       \n",
      " 3   Rel Hum (%)         8784 non-null   int64         \n",
      " 4   Wind Spd (km/h)     8784 non-null   int64         \n",
      " 5   Visibility (km)     8784 non-null   float64       \n",
      " 6   Stn Press (kPa)     8784 non-null   float64       \n",
      " 7   Weather             8784 non-null   object        \n",
      "dtypes: datetime64[ns](1), float64(4), int64(2), object(1)\n",
      "memory usage: 549.1+ KB\n"
     ]
    }
   ],
   "source": [
    "weather_df.info() # Bring the cursor inside the brackets of info() and hit shift+tab & see what you get.\n",
    "                  # This will work for any function in Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2212091c",
   "metadata": {},
   "source": [
    "## `.head()`\n",
    "***\n",
    "It is used to preview a part of a large DataFrame, similar to the linux `head` command. This reduces time and resources required if  whole DataFrame was to be fetched instead. Shows the first N rows in the data (by default, N=5)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "3e574e4b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date/Time</th>\n",
       "      <th>Temp (C)</th>\n",
       "      <th>Dew Point Temp (C)</th>\n",
       "      <th>Rel Hum (%)</th>\n",
       "      <th>Wind Spd (km/h)</th>\n",
       "      <th>Visibility (km)</th>\n",
       "      <th>Stn Press (kPa)</th>\n",
       "      <th>Weather</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-01 00:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.9</td>\n",
       "      <td>86</td>\n",
       "      <td>4</td>\n",
       "      <td>8.0</td>\n",
       "      <td>101.24</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-01-01 01:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.7</td>\n",
       "      <td>87</td>\n",
       "      <td>4</td>\n",
       "      <td>8.0</td>\n",
       "      <td>101.24</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-01-01 02:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.4</td>\n",
       "      <td>89</td>\n",
       "      <td>7</td>\n",
       "      <td>4.0</td>\n",
       "      <td>101.26</td>\n",
       "      <td>Freezing Drizzle,Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2012-01-01 03:00:00</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>-3.2</td>\n",
       "      <td>88</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>101.27</td>\n",
       "      <td>Freezing Drizzle,Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-01-01 04:00:00</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>-3.3</td>\n",
       "      <td>88</td>\n",
       "      <td>7</td>\n",
       "      <td>4.8</td>\n",
       "      <td>101.23</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Date/Time  Temp (C)  Dew Point Temp (C)  Rel Hum (%)  \\\n",
       "0 2012-01-01 00:00:00      -1.8                -3.9           86   \n",
       "1 2012-01-01 01:00:00      -1.8                -3.7           87   \n",
       "2 2012-01-01 02:00:00      -1.8                -3.4           89   \n",
       "3 2012-01-01 03:00:00      -1.5                -3.2           88   \n",
       "4 2012-01-01 04:00:00      -1.5                -3.3           88   \n",
       "\n",
       "   Wind Spd (km/h)  Visibility (km)  Stn Press (kPa)               Weather  \n",
       "0                4              8.0           101.24                   Fog  \n",
       "1                4              8.0           101.24                   Fog  \n",
       "2                7              4.0           101.26  Freezing Drizzle,Fog  \n",
       "3                6              4.0           101.27  Freezing Drizzle,Fog  \n",
       "4                7              4.8           101.23                   Fog  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c81995a",
   "metadata": {},
   "source": [
    "## `.index`\n",
    "***\n",
    "This attribute provides the `index` of the dataframe.\n",
    "\n",
    "Indexing identifies data using known indicators that allows intuitive getting and setting of subsets of the data set.\n",
    "\n",
    "A major advantage of Pandas over NumPy is that each of the columns and rows has a label. Working with column positions is possible, but it can be hard to keep track of which number corresponds to which column.\n",
    "\n",
    "We can work with labels using the **pandas.DataFrame.loc** method, which allows us to index using labels instead of positions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "15d21e37",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=8784, step=1)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df.index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be7947d7",
   "metadata": {},
   "source": [
    "## `.unique()`\n",
    "***\n",
    "This method, which belongs to the `Series` object, can be useful when trying to identify unique values in a column.\n",
    "- Uniques are returned in order of appearance. \n",
    "- It is significantly faster than numpy.unique and includes N/A values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "8d4a2d56",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                        Fog\n",
       "1                        Fog\n",
       "2       Freezing Drizzle,Fog\n",
       "3       Freezing Drizzle,Fog\n",
       "4                        Fog\n",
       "                ...         \n",
       "8779                    Snow\n",
       "8780                    Snow\n",
       "8781                    Snow\n",
       "8782                    Snow\n",
       "8783                    Snow\n",
       "Name: Weather, Length: 8784, dtype: object"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df['Weather']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "a0d5c1e6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Fog', 'Freezing Drizzle,Fog', 'Mostly Cloudy', 'Cloudy', 'Rain',\n",
       "       'Rain Showers', 'Mainly Clear', 'Snow Showers', 'Snow', 'Clear',\n",
       "       'Freezing Rain,Fog', 'Freezing Rain', 'Freezing Drizzle',\n",
       "       'Rain,Snow', 'Moderate Snow', 'Freezing Drizzle,Snow',\n",
       "       'Freezing Rain,Snow Grains', 'Snow,Blowing Snow', 'Freezing Fog',\n",
       "       'Haze', 'Rain,Fog', 'Drizzle,Fog', 'Drizzle',\n",
       "       'Freezing Drizzle,Haze', 'Freezing Rain,Haze', 'Snow,Haze',\n",
       "       'Snow,Fog', 'Snow,Ice Pellets', 'Rain,Haze', 'Thunderstorms,Rain',\n",
       "       'Thunderstorms,Rain Showers', 'Thunderstorms,Heavy Rain Showers',\n",
       "       'Thunderstorms,Rain Showers,Fog', 'Thunderstorms',\n",
       "       'Thunderstorms,Rain,Fog',\n",
       "       'Thunderstorms,Moderate Rain Showers,Fog', 'Rain Showers,Fog',\n",
       "       'Rain Showers,Snow Showers', 'Snow Pellets', 'Rain,Snow,Fog',\n",
       "       'Moderate Rain,Fog', 'Freezing Rain,Ice Pellets,Fog',\n",
       "       'Drizzle,Ice Pellets,Fog', 'Drizzle,Snow', 'Rain,Ice Pellets',\n",
       "       'Drizzle,Snow,Fog', 'Rain,Snow Grains', 'Rain,Snow,Ice Pellets',\n",
       "       'Snow Showers,Fog', 'Moderate Snow,Blowing Snow'], dtype=object)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df['Weather'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8fa7a76",
   "metadata": {},
   "source": [
    "\n",
    "## Mini - Challenge - 1\n",
    "***\n",
    "Find all the unique Wind Speed values recorded in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a05cf7b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "1c09bb26",
   "metadata": {},
   "source": [
    "## `.nunique()`\n",
    "***\n",
    "This method belongs to the `Series` object and can be useful when trying to identify the number of unique values in a column. \n",
    "- Excludes NA values by default\n",
    "- Always returns an integer value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "84d7b715",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df['Weather'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b4da5b7",
   "metadata": {},
   "source": [
    "## `.value_counts()`\n",
    "***\n",
    "This method, which belongs to the `Series` object, can be useful when trying to identify unique values and their counts in a column\n",
    "- The resulting object will be in descending order so that the first element is the most frequently-occurring element. \n",
    "- Excludes NA values by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "da5bcded",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Mainly Clear                               2106\n",
       "Mostly Cloudy                              2069\n",
       "Cloudy                                     1728\n",
       "Clear                                      1326\n",
       "Snow                                        390\n",
       "Rain                                        306\n",
       "Rain Showers                                188\n",
       "Fog                                         150\n",
       "Rain,Fog                                    116\n",
       "Drizzle,Fog                                  80\n",
       "Snow Showers                                 60\n",
       "Drizzle                                      41\n",
       "Snow,Fog                                     37\n",
       "Snow,Blowing Snow                            19\n",
       "Rain,Snow                                    18\n",
       "Thunderstorms,Rain Showers                   16\n",
       "Haze                                         16\n",
       "Drizzle,Snow,Fog                             15\n",
       "Freezing Rain                                14\n",
       "Freezing Drizzle,Snow                        11\n",
       "Freezing Drizzle                              7\n",
       "Snow,Ice Pellets                              6\n",
       "Freezing Drizzle,Fog                          6\n",
       "Snow,Haze                                     5\n",
       "Freezing Fog                                  4\n",
       "Snow Showers,Fog                              4\n",
       "Moderate Snow                                 4\n",
       "Rain,Snow,Ice Pellets                         4\n",
       "Freezing Rain,Fog                             4\n",
       "Freezing Drizzle,Haze                         3\n",
       "Rain,Haze                                     3\n",
       "Thunderstorms,Rain                            3\n",
       "Thunderstorms,Rain Showers,Fog                3\n",
       "Freezing Rain,Haze                            2\n",
       "Drizzle,Snow                                  2\n",
       "Rain Showers,Snow Showers                     2\n",
       "Thunderstorms                                 2\n",
       "Moderate Snow,Blowing Snow                    2\n",
       "Rain Showers,Fog                              1\n",
       "Thunderstorms,Moderate Rain Showers,Fog       1\n",
       "Snow Pellets                                  1\n",
       "Rain,Snow,Fog                                 1\n",
       "Moderate Rain,Fog                             1\n",
       "Freezing Rain,Ice Pellets,Fog                 1\n",
       "Drizzle,Ice Pellets,Fog                       1\n",
       "Thunderstorms,Rain,Fog                        1\n",
       "Rain,Ice Pellets                              1\n",
       "Rain,Snow Grains                              1\n",
       "Thunderstorms,Heavy Rain Showers              1\n",
       "Freezing Rain,Snow Grains                     1\n",
       "Name: Weather, dtype: int64"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df['Weather'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d783bc0b",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Mini-Challenge - 2\n",
    "***\n",
    "\n",
    "Find the number of times when the weather was exactly 'Clear'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28986064",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "bdb8577c",
   "metadata": {},
   "source": [
    "\n",
    "### How can we get the count for Clear ?\n",
    "***\n",
    "\n",
    "In order to get only the number of times when the weather was exactly 'Clear' , we need to used Square brackets after `.value_counts` \n",
    "eg. `.value_counts()[3]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f9e5831",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ff6375b6",
   "metadata": {},
   "source": [
    "## Mini-Challenge - 3\n",
    "***\n",
    "**Try one more time..!!**\n",
    "\n",
    "Find the number of times when the wind speed was exactly 4 km/h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95e44737",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "85244521",
   "metadata": {},
   "source": [
    "\n",
    "# Data Manipulation : Gets you desired results\n",
    "***\n",
    "The true power of the Pandas DataFrame is the ease and flexibility of manipulating data to get your desired results.\n",
    "\n",
    "Pandas is best at handling tabular data sets comprising different variable types (integer, float, double, etc.). \n",
    "\n",
    "In addition, the pandas library can also be used to perform even the most naive of tasks such as loading data or doing feature engineering on time series data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db9eec72",
   "metadata": {},
   "source": [
    "## Selection (Part 1)\n",
    "***\n",
    "How do you select particular rows/columns from the DataFrame ?\n",
    "\n",
    "The DataFrame object supports indexing operations just like the Python `list` class and the Pandas Series object, but is much faster and more powerful.\n",
    "\n",
    "Note that when you extract a single row or column, you get a one-dimensional object as output. That is called a pandas Series. The values on the left are just labels taken from the dataframe index. \n",
    "\n",
    "On the other hand, when we extract portions of a pandas dataframe, we get a two-dimensional DataFrame type of object. Something to keep in mind for later."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0409bd6a",
   "metadata": {},
   "source": [
    "### How to get the Weather column from the \"weather_df\" dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "2c1c4da7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.series.Series'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0                     Fog\n",
       "1                     Fog\n",
       "2    Freezing Drizzle,Fog\n",
       "3    Freezing Drizzle,Fog\n",
       "4                     Fog\n",
       "Name: Weather, dtype: object"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col = weather_df['Weather']\n",
    "\n",
    "print(type(col))\n",
    "col.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4dade05",
   "metadata": {},
   "source": [
    "### How to get the Weather and Temperature columns from the \"weather_df\" dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "d2b231fb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Weather</th>\n",
       "      <th>Temp (C)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Fog</td>\n",
       "      <td>-1.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Fog</td>\n",
       "      <td>-1.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Freezing Drizzle,Fog</td>\n",
       "      <td>-1.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Freezing Drizzle,Fog</td>\n",
       "      <td>-1.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fog</td>\n",
       "      <td>-1.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Weather  Temp (C)\n",
       "0                   Fog      -1.8\n",
       "1                   Fog      -1.8\n",
       "2  Freezing Drizzle,Fog      -1.8\n",
       "3  Freezing Drizzle,Fog      -1.5\n",
       "4                   Fog      -1.5"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "two_cols = weather_df[['Weather', 'Temp (C)']] # Take a good look at those brackets. There are two sets of them\n",
    "                                               # to access more than one columns. \n",
    "print(type(two_cols))\n",
    "two_cols.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "542e8c2b",
   "metadata": {},
   "source": [
    "**Keep in mind Whenever you need to take more than two columns need to put double sqaure  like above example**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5abf9159",
   "metadata": {},
   "source": [
    "## **Get the first 25 rows from the \"weather_df\" dataframe**\n",
    "***\n",
    "**Important**: This slicing would work even if the row index had non-numeric labels, because slicing works here the same way as a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "c65b59e3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date/Time</th>\n",
       "      <th>Temp (C)</th>\n",
       "      <th>Dew Point Temp (C)</th>\n",
       "      <th>Rel Hum (%)</th>\n",
       "      <th>Wind Spd (km/h)</th>\n",
       "      <th>Visibility (km)</th>\n",
       "      <th>Stn Press (kPa)</th>\n",
       "      <th>Weather</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-01 00:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.9</td>\n",
       "      <td>86</td>\n",
       "      <td>4</td>\n",
       "      <td>8.0</td>\n",
       "      <td>101.24</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-01-01 01:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.7</td>\n",
       "      <td>87</td>\n",
       "      <td>4</td>\n",
       "      <td>8.0</td>\n",
       "      <td>101.24</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-01-01 02:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.4</td>\n",
       "      <td>89</td>\n",
       "      <td>7</td>\n",
       "      <td>4.0</td>\n",
       "      <td>101.26</td>\n",
       "      <td>Freezing Drizzle,Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2012-01-01 03:00:00</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>-3.2</td>\n",
       "      <td>88</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>101.27</td>\n",
       "      <td>Freezing Drizzle,Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-01-01 04:00:00</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>-3.3</td>\n",
       "      <td>88</td>\n",
       "      <td>7</td>\n",
       "      <td>4.8</td>\n",
       "      <td>101.23</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2012-01-01 05:00:00</td>\n",
       "      <td>-1.4</td>\n",
       "      <td>-3.3</td>\n",
       "      <td>87</td>\n",
       "      <td>9</td>\n",
       "      <td>6.4</td>\n",
       "      <td>101.27</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2012-01-01 06:00:00</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>-3.1</td>\n",
       "      <td>89</td>\n",
       "      <td>7</td>\n",
       "      <td>6.4</td>\n",
       "      <td>101.29</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2012-01-01 07:00:00</td>\n",
       "      <td>-1.4</td>\n",
       "      <td>-3.6</td>\n",
       "      <td>85</td>\n",
       "      <td>7</td>\n",
       "      <td>8.0</td>\n",
       "      <td>101.26</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2012-01-01 08:00:00</td>\n",
       "      <td>-1.4</td>\n",
       "      <td>-3.6</td>\n",
       "      <td>85</td>\n",
       "      <td>9</td>\n",
       "      <td>8.0</td>\n",
       "      <td>101.23</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2012-01-01 09:00:00</td>\n",
       "      <td>-1.3</td>\n",
       "      <td>-3.1</td>\n",
       "      <td>88</td>\n",
       "      <td>15</td>\n",
       "      <td>4.0</td>\n",
       "      <td>101.20</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2012-01-01 10:00:00</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>-2.3</td>\n",
       "      <td>91</td>\n",
       "      <td>9</td>\n",
       "      <td>1.2</td>\n",
       "      <td>101.15</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2012-01-01 11:00:00</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>-2.1</td>\n",
       "      <td>89</td>\n",
       "      <td>7</td>\n",
       "      <td>4.0</td>\n",
       "      <td>100.98</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2012-01-01 12:00:00</td>\n",
       "      <td>-0.2</td>\n",
       "      <td>-2.0</td>\n",
       "      <td>88</td>\n",
       "      <td>9</td>\n",
       "      <td>4.8</td>\n",
       "      <td>100.79</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2012-01-01 13:00:00</td>\n",
       "      <td>0.2</td>\n",
       "      <td>-1.7</td>\n",
       "      <td>87</td>\n",
       "      <td>13</td>\n",
       "      <td>4.8</td>\n",
       "      <td>100.58</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2012-01-01 14:00:00</td>\n",
       "      <td>0.8</td>\n",
       "      <td>-1.1</td>\n",
       "      <td>87</td>\n",
       "      <td>20</td>\n",
       "      <td>4.8</td>\n",
       "      <td>100.31</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>2012-01-01 15:00:00</td>\n",
       "      <td>1.8</td>\n",
       "      <td>-0.4</td>\n",
       "      <td>85</td>\n",
       "      <td>22</td>\n",
       "      <td>6.4</td>\n",
       "      <td>100.07</td>\n",
       "      <td>Fog</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>2012-01-01 16:00:00</td>\n",
       "      <td>2.6</td>\n",
       "      <td>-0.2</td>\n",
       "      <td>82</td>\n",
       "      <td>13</td>\n",
       "      <td>12.9</td>\n",
       "      <td>99.93</td>\n",
       "      <td>Mostly Cloudy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>2012-01-01 17:00:00</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>81</td>\n",
       "      <td>13</td>\n",
       "      <td>16.1</td>\n",
       "      <td>99.81</td>\n",
       "      <td>Cloudy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>2012-01-01 18:00:00</td>\n",
       "      <td>3.8</td>\n",
       "      <td>1.0</td>\n",
       "      <td>82</td>\n",
       "      <td>15</td>\n",
       "      <td>12.9</td>\n",
       "      <td>99.74</td>\n",
       "      <td>Rain</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>2012-01-01 19:00:00</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.3</td>\n",
       "      <td>88</td>\n",
       "      <td>15</td>\n",
       "      <td>12.9</td>\n",
       "      <td>99.68</td>\n",
       "      <td>Rain</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>2012-01-01 20:00:00</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>87</td>\n",
       "      <td>19</td>\n",
       "      <td>25.0</td>\n",
       "      <td>99.50</td>\n",
       "      <td>Cloudy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>2012-01-01 21:00:00</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.7</td>\n",
       "      <td>85</td>\n",
       "      <td>20</td>\n",
       "      <td>25.0</td>\n",
       "      <td>99.39</td>\n",
       "      <td>Cloudy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>2012-01-01 22:00:00</td>\n",
       "      <td>4.4</td>\n",
       "      <td>1.9</td>\n",
       "      <td>84</td>\n",
       "      <td>24</td>\n",
       "      <td>19.3</td>\n",
       "      <td>99.32</td>\n",
       "      <td>Rain Showers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>2012-01-01 23:00:00</td>\n",
       "      <td>5.3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>79</td>\n",
       "      <td>30</td>\n",
       "      <td>25.0</td>\n",
       "      <td>99.31</td>\n",
       "      <td>Cloudy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>2012-01-02 00:00:00</td>\n",
       "      <td>5.2</td>\n",
       "      <td>1.5</td>\n",
       "      <td>77</td>\n",
       "      <td>35</td>\n",
       "      <td>25.0</td>\n",
       "      <td>99.26</td>\n",
       "      <td>Rain Showers</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Date/Time  Temp (C)  Dew Point Temp (C)  Rel Hum (%)  \\\n",
       "0  2012-01-01 00:00:00      -1.8                -3.9           86   \n",
       "1  2012-01-01 01:00:00      -1.8                -3.7           87   \n",
       "2  2012-01-01 02:00:00      -1.8                -3.4           89   \n",
       "3  2012-01-01 03:00:00      -1.5                -3.2           88   \n",
       "4  2012-01-01 04:00:00      -1.5                -3.3           88   \n",
       "5  2012-01-01 05:00:00      -1.4                -3.3           87   \n",
       "6  2012-01-01 06:00:00      -1.5                -3.1           89   \n",
       "7  2012-01-01 07:00:00      -1.4                -3.6           85   \n",
       "8  2012-01-01 08:00:00      -1.4                -3.6           85   \n",
       "9  2012-01-01 09:00:00      -1.3                -3.1           88   \n",
       "10 2012-01-01 10:00:00      -1.0                -2.3           91   \n",
       "11 2012-01-01 11:00:00      -0.5                -2.1           89   \n",
       "12 2012-01-01 12:00:00      -0.2                -2.0           88   \n",
       "13 2012-01-01 13:00:00       0.2                -1.7           87   \n",
       "14 2012-01-01 14:00:00       0.8                -1.1           87   \n",
       "15 2012-01-01 15:00:00       1.8                -0.4           85   \n",
       "16 2012-01-01 16:00:00       2.6                -0.2           82   \n",
       "17 2012-01-01 17:00:00       3.0                 0.0           81   \n",
       "18 2012-01-01 18:00:00       3.8                 1.0           82   \n",
       "19 2012-01-01 19:00:00       3.1                 1.3           88   \n",
       "20 2012-01-01 20:00:00       3.2                 1.3           87   \n",
       "21 2012-01-01 21:00:00       4.0                 1.7           85   \n",
       "22 2012-01-01 22:00:00       4.4                 1.9           84   \n",
       "23 2012-01-01 23:00:00       5.3                 2.0           79   \n",
       "24 2012-01-02 00:00:00       5.2                 1.5           77   \n",
       "\n",
       "    Wind Spd (km/h)  Visibility (km)  Stn Press (kPa)               Weather  \n",
       "0                 4              8.0           101.24                   Fog  \n",
       "1                 4              8.0           101.24                   Fog  \n",
       "2                 7              4.0           101.26  Freezing Drizzle,Fog  \n",
       "3                 6              4.0           101.27  Freezing Drizzle,Fog  \n",
       "4                 7              4.8           101.23                   Fog  \n",
       "5                 9              6.4           101.27                   Fog  \n",
       "6                 7              6.4           101.29                   Fog  \n",
       "7                 7              8.0           101.26                   Fog  \n",
       "8                 9              8.0           101.23                   Fog  \n",
       "9                15              4.0           101.20                   Fog  \n",
       "10                9              1.2           101.15                   Fog  \n",
       "11                7              4.0           100.98                   Fog  \n",
       "12                9              4.8           100.79                   Fog  \n",
       "13               13              4.8           100.58                   Fog  \n",
       "14               20              4.8           100.31                   Fog  \n",
       "15               22              6.4           100.07                   Fog  \n",
       "16               13             12.9            99.93         Mostly Cloudy  \n",
       "17               13             16.1            99.81                Cloudy  \n",
       "18               15             12.9            99.74                  Rain  \n",
       "19               15             12.9            99.68                  Rain  \n",
       "20               19             25.0            99.50                Cloudy  \n",
       "21               20             25.0            99.39                Cloudy  \n",
       "22               24             19.3            99.32          Rain Showers  \n",
       "23               30             25.0            99.31                Cloudy  \n",
       "24               35             25.0            99.26          Rain Showers  "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df[:25]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11191254",
   "metadata": {},
   "source": [
    "\n",
    "**How to get the first 3 alternating rows from the \"weather_df\" dataframe, but only the Visibility and Relative Humidity columns**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "7f85863b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date/Time</th>\n",
       "      <th>Visibility (km)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-01 00:00:00</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-01-01 02:00:00</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-01-01 04:00:00</td>\n",
       "      <td>4.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2012-01-01 06:00:00</td>\n",
       "      <td>6.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2012-01-01 08:00:00</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Date/Time  Visibility (km)\n",
       "0 2012-01-01 00:00:00              8.0\n",
       "2 2012-01-01 02:00:00              4.0\n",
       "4 2012-01-01 04:00:00              4.8\n",
       "6 2012-01-01 06:00:00              6.4\n",
       "8 2012-01-01 08:00:00              8.0"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df[['Date/Time','Visibility (km)']][0:10:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "2f192d40",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Rel Hum (%)</th>\n",
       "      <th>Visibility (km)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Rel Hum (%)  Visibility (km)\n",
       "0         True             True\n",
       "2         True             True\n",
       "4         True             True"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# solution 1\n",
    "result1 = weather_df[:6:2][['Rel Hum (%)', 'Visibility (km)']]\n",
    "\n",
    "# solution 2\n",
    "result2 = weather_df[['Rel Hum (%)', 'Visibility (km)']][:6:2]\n",
    "\n",
    "# are they the same?\n",
    "result1 == result2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33b6fb74",
   "metadata": {},
   "source": [
    "#### So which of the two solutions should you use?\n",
    "***\n",
    "**Answer**: Neither. Because we're indexing more than once (Chained Indexing).\n",
    "When you use chained indexing, the order and type of the indexing operation partially determine whether the result is a slice into the original object, or a copy of the slice.\n",
    "\n",
    "Let's analyse (break down) one of the above solutions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "f1cfdeaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# first indexing\n",
    "df1 = weather_df[:6:2]\n",
    "\n",
    "# second indexing\n",
    "df2 = df1[['Rel Hum (%)', 'Visibility (km)']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c006874",
   "metadata": {},
   "source": [
    "While both results are correct in this **read-only** case, chained indexing may give unpredictable behaviours when **writing** to a dataframe.\n",
    "\n",
    "This is because indexing could either return a \"view\" (of slices of the dataframe), or a copy of the dataframe."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87c37c97",
   "metadata": {},
   "source": [
    "## Selection (Part 2)\n",
    "***\n",
    "Pandas provides a powerful way to work with both rows and columns together, optionally using their label indices or numeric indices.\n",
    "\n",
    "- **`.loc :`**<br/>\n",
    "Purely label-location based indexer for selection by label (but may also be used with a boolean array).<br/>\n",
    "**Important: If you use slicing in loc, it will return the end index as well**\n",
    "<br/><br/>\n",
    "\n",
    "- **`.iloc:`**<br/>\n",
    "Purely integer-location based indexing for selection by position (but may also be used with a boolean array).\n",
    "\n",
    "Allowed inputs are:\n",
    "- A single label, e.g. 5 or 'a'\n",
    "- A list or array of labels, e.g. ['a', 'b', 'c']\n",
    "- A slice object with labels, e.g. 'a':'f'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "24df3544",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Visibility (km)</th>\n",
       "      <th>Rel Hum (%)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8.0</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8.0</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.0</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.0</td>\n",
       "      <td>88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.8</td>\n",
       "      <td>88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6.4</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Visibility (km)  Rel Hum (%)\n",
       "0              8.0           86\n",
       "1              8.0           87\n",
       "2              4.0           89\n",
       "3              4.0           88\n",
       "4              4.8           88\n",
       "5              6.4           87"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df.loc[0:5, ['Visibility (km)','Rel Hum (%)']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "3707e956",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Visibility (km)</th>\n",
       "      <th>Rel Hum (%)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8.0</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8.0</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.0</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.0</td>\n",
       "      <td>88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.8</td>\n",
       "      <td>88</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Visibility (km)  Rel Hum (%)\n",
       "0              8.0           86\n",
       "1              8.0           87\n",
       "2              4.0           89\n",
       "3              4.0           88\n",
       "4              4.8           88"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df[['Visibility (km)','Rel Hum (%)']].iloc[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3572455d",
   "metadata": {},
   "source": [
    "\n",
    "## Mini-Challenge - 4\n",
    "***\n",
    "\n",
    "Get the first 3 alternating rows from the `weather_df` dataframe, but only the Visibility and Relative Humidity columns\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "9d8a5716",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using loc or iloc\n",
    "# NOTE: slice is `:5:2`, not 6 !\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21d2bd82",
   "metadata": {},
   "source": [
    "\n",
    "## Filtering\n",
    "***\n",
    "\n",
    "![Filter](images/filters1.jpg)\n",
    "Image Source:https://pixabay.com/en/yashica-filter-camera-vintage-711794/\n",
    "<br/>\n",
    "\n",
    "Anything that takes in data, processes it, and provides an output\n",
    "\n",
    "Input Data ⟶ Filter ⟶ Output Data\n",
    "\n",
    "Filtering rows of a DataFrame is an almost mandatory task for Data Analysis with Python. Given a Data Frame, we may not be interested in the entire dataset but only in specific rows."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a63c913",
   "metadata": {},
   "source": [
    "### Find all instances when snow was recorded\n",
    "***\n",
    "Whether or not it snowed can be found out using the Weather column.\n",
    "\n",
    "Note that this routine does not filter a dataframe on its contents. The filter is applied to the labels of the index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "2c9a161a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Fog', 'Freezing Drizzle,Fog', 'Mostly Cloudy', 'Cloudy', 'Rain',\n",
       "       'Rain Showers', 'Mainly Clear', 'Snow Showers', 'Snow', 'Clear',\n",
       "       'Freezing Rain,Fog', 'Freezing Rain', 'Freezing Drizzle',\n",
       "       'Rain,Snow', 'Moderate Snow', 'Freezing Drizzle,Snow',\n",
       "       'Freezing Rain,Snow Grains', 'Snow,Blowing Snow', 'Freezing Fog',\n",
       "       'Haze', 'Rain,Fog', 'Drizzle,Fog', 'Drizzle',\n",
       "       'Freezing Drizzle,Haze', 'Freezing Rain,Haze', 'Snow,Haze',\n",
       "       'Snow,Fog', 'Snow,Ice Pellets', 'Rain,Haze', 'Thunderstorms,Rain',\n",
       "       'Thunderstorms,Rain Showers', 'Thunderstorms,Heavy Rain Showers',\n",
       "       'Thunderstorms,Rain Showers,Fog', 'Thunderstorms',\n",
       "       'Thunderstorms,Rain,Fog',\n",
       "       'Thunderstorms,Moderate Rain Showers,Fog', 'Rain Showers,Fog',\n",
       "       'Rain Showers,Snow Showers', 'Snow Pellets', 'Rain,Snow,Fog',\n",
       "       'Moderate Rain,Fog', 'Freezing Rain,Ice Pellets,Fog',\n",
       "       'Drizzle,Ice Pellets,Fog', 'Drizzle,Snow', 'Rain,Ice Pellets',\n",
       "       'Drizzle,Snow,Fog', 'Rain,Snow Grains', 'Rain,Snow,Ice Pellets',\n",
       "       'Snow Showers,Fog', 'Moderate Snow,Blowing Snow'], dtype=object)"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df['Weather'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "26e97317",
   "metadata": {},
   "outputs": [],
   "source": [
    "snowed_filter = weather_df['Weather'].str.lower().str.contains('snow')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "70a00220",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       False\n",
       "1       False\n",
       "2       False\n",
       "3       False\n",
       "4       False\n",
       "        ...  \n",
       "8779     True\n",
       "8780     True\n",
       "8781     True\n",
       "8782     True\n",
       "8783     True\n",
       "Name: Weather, Length: 8784, dtype: bool"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "snowed_filter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "cc289646",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date/Time</th>\n",
       "      <th>Temp (C)</th>\n",
       "      <th>Dew Point Temp (C)</th>\n",
       "      <th>Rel Hum (%)</th>\n",
       "      <th>Wind Spd (km/h)</th>\n",
       "      <th>Visibility (km)</th>\n",
       "      <th>Stn Press (kPa)</th>\n",
       "      <th>Weather</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>2012-01-02 17:00:00</td>\n",
       "      <td>-2.1</td>\n",
       "      <td>-9.5</td>\n",
       "      <td>57</td>\n",
       "      <td>22</td>\n",
       "      <td>25.0</td>\n",
       "      <td>99.66</td>\n",
       "      <td>Snow Showers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>2012-01-02 20:00:00</td>\n",
       "      <td>-5.6</td>\n",
       "      <td>-13.4</td>\n",
       "      <td>54</td>\n",
       "      <td>24</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.07</td>\n",
       "      <td>Snow Showers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>2012-01-02 21:00:00</td>\n",
       "      <td>-5.8</td>\n",
       "      <td>-12.8</td>\n",
       "      <td>58</td>\n",
       "      <td>26</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.15</td>\n",
       "      <td>Snow Showers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>2012-01-02 23:00:00</td>\n",
       "      <td>-7.4</td>\n",
       "      <td>-14.1</td>\n",
       "      <td>59</td>\n",
       "      <td>17</td>\n",
       "      <td>19.3</td>\n",
       "      <td>100.27</td>\n",
       "      <td>Snow Showers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>2012-01-03 00:00:00</td>\n",
       "      <td>-9.0</td>\n",
       "      <td>-16.0</td>\n",
       "      <td>57</td>\n",
       "      <td>28</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.35</td>\n",
       "      <td>Snow Showers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8779</th>\n",
       "      <td>2012-12-31 19:00:00</td>\n",
       "      <td>0.1</td>\n",
       "      <td>-2.7</td>\n",
       "      <td>81</td>\n",
       "      <td>30</td>\n",
       "      <td>9.7</td>\n",
       "      <td>100.13</td>\n",
       "      <td>Snow</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8780</th>\n",
       "      <td>2012-12-31 20:00:00</td>\n",
       "      <td>0.2</td>\n",
       "      <td>-2.4</td>\n",
       "      <td>83</td>\n",
       "      <td>24</td>\n",
       "      <td>9.7</td>\n",
       "      <td>100.03</td>\n",
       "      <td>Snow</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8781</th>\n",
       "      <td>2012-12-31 21:00:00</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>93</td>\n",
       "      <td>28</td>\n",
       "      <td>4.8</td>\n",
       "      <td>99.95</td>\n",
       "      <td>Snow</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8782</th>\n",
       "      <td>2012-12-31 22:00:00</td>\n",
       "      <td>-0.2</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>89</td>\n",
       "      <td>28</td>\n",
       "      <td>9.7</td>\n",
       "      <td>99.91</td>\n",
       "      <td>Snow</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8783</th>\n",
       "      <td>2012-12-31 23:00:00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-2.1</td>\n",
       "      <td>86</td>\n",
       "      <td>30</td>\n",
       "      <td>11.3</td>\n",
       "      <td>99.89</td>\n",
       "      <td>Snow</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>583 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                Date/Time  Temp (C)  Dew Point Temp (C)  Rel Hum (%)  \\\n",
       "41    2012-01-02 17:00:00      -2.1                -9.5           57   \n",
       "44    2012-01-02 20:00:00      -5.6               -13.4           54   \n",
       "45    2012-01-02 21:00:00      -5.8               -12.8           58   \n",
       "47    2012-01-02 23:00:00      -7.4               -14.1           59   \n",
       "48    2012-01-03 00:00:00      -9.0               -16.0           57   \n",
       "...                   ...       ...                 ...          ...   \n",
       "8779  2012-12-31 19:00:00       0.1                -2.7           81   \n",
       "8780  2012-12-31 20:00:00       0.2                -2.4           83   \n",
       "8781  2012-12-31 21:00:00      -0.5                -1.5           93   \n",
       "8782  2012-12-31 22:00:00      -0.2                -1.8           89   \n",
       "8783  2012-12-31 23:00:00       0.0                -2.1           86   \n",
       "\n",
       "      Wind Spd (km/h)  Visibility (km)  Stn Press (kPa)       Weather  \n",
       "41                 22             25.0            99.66  Snow Showers  \n",
       "44                 24             25.0           100.07  Snow Showers  \n",
       "45                 26             25.0           100.15  Snow Showers  \n",
       "47                 17             19.3           100.27  Snow Showers  \n",
       "48                 28             25.0           100.35  Snow Showers  \n",
       "...               ...              ...              ...           ...  \n",
       "8779               30              9.7           100.13          Snow  \n",
       "8780               24              9.7           100.03          Snow  \n",
       "8781               28              4.8            99.95          Snow  \n",
       "8782               28              9.7            99.91          Snow  \n",
       "8783               30             11.3            99.89          Snow  \n",
       "\n",
       "[583 rows x 8 columns]"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df[snowed_filter]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "a7d018ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date/Time</th>\n",
       "      <th>Temp (C)</th>\n",
       "      <th>Dew Point Temp (C)</th>\n",
       "      <th>Rel Hum (%)</th>\n",
       "      <th>Wind Spd (km/h)</th>\n",
       "      <th>Visibility (km)</th>\n",
       "      <th>Stn Press (kPa)</th>\n",
       "      <th>Weather</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>2012-01-02 17:00:00</td>\n",
       "      <td>-2.1</td>\n",
       "      <td>-9.5</td>\n",
       "      <td>57</td>\n",
       "      <td>22</td>\n",
       "      <td>25.0</td>\n",
       "      <td>99.66</td>\n",
       "      <td>Snow Showers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>2012-01-02 20:00:00</td>\n",
       "      <td>-5.6</td>\n",
       "      <td>-13.4</td>\n",
       "      <td>54</td>\n",
       "      <td>24</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.07</td>\n",
       "      <td>Snow Showers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>2012-01-02 21:00:00</td>\n",
       "      <td>-5.8</td>\n",
       "      <td>-12.8</td>\n",
       "      <td>58</td>\n",
       "      <td>26</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.15</td>\n",
       "      <td>Snow Showers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>2012-01-02 23:00:00</td>\n",
       "      <td>-7.4</td>\n",
       "      <td>-14.1</td>\n",
       "      <td>59</td>\n",
       "      <td>17</td>\n",
       "      <td>19.3</td>\n",
       "      <td>100.27</td>\n",
       "      <td>Snow Showers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>2012-01-03 00:00:00</td>\n",
       "      <td>-9.0</td>\n",
       "      <td>-16.0</td>\n",
       "      <td>57</td>\n",
       "      <td>28</td>\n",
       "      <td>25.0</td>\n",
       "      <td>100.35</td>\n",
       "      <td>Snow Showers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8779</th>\n",
       "      <td>2012-12-31 19:00:00</td>\n",
       "      <td>0.1</td>\n",
       "      <td>-2.7</td>\n",
       "      <td>81</td>\n",
       "      <td>30</td>\n",
       "      <td>9.7</td>\n",
       "      <td>100.13</td>\n",
       "      <td>Snow</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8780</th>\n",
       "      <td>2012-12-31 20:00:00</td>\n",
       "      <td>0.2</td>\n",
       "      <td>-2.4</td>\n",
       "      <td>83</td>\n",
       "      <td>24</td>\n",
       "      <td>9.7</td>\n",
       "      <td>100.03</td>\n",
       "      <td>Snow</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8781</th>\n",
       "      <td>2012-12-31 21:00:00</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>93</td>\n",
       "      <td>28</td>\n",
       "      <td>4.8</td>\n",
       "      <td>99.95</td>\n",
       "      <td>Snow</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8782</th>\n",
       "      <td>2012-12-31 22:00:00</td>\n",
       "      <td>-0.2</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>89</td>\n",
       "      <td>28</td>\n",
       "      <td>9.7</td>\n",
       "      <td>99.91</td>\n",
       "      <td>Snow</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8783</th>\n",
       "      <td>2012-12-31 23:00:00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-2.1</td>\n",
       "      <td>86</td>\n",
       "      <td>30</td>\n",
       "      <td>11.3</td>\n",
       "      <td>99.89</td>\n",
       "      <td>Snow</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>583 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                Date/Time  Temp (C)  Dew Point Temp (C)  Rel Hum (%)  \\\n",
       "41    2012-01-02 17:00:00      -2.1                -9.5           57   \n",
       "44    2012-01-02 20:00:00      -5.6               -13.4           54   \n",
       "45    2012-01-02 21:00:00      -5.8               -12.8           58   \n",
       "47    2012-01-02 23:00:00      -7.4               -14.1           59   \n",
       "48    2012-01-03 00:00:00      -9.0               -16.0           57   \n",
       "...                   ...       ...                 ...          ...   \n",
       "8779  2012-12-31 19:00:00       0.1                -2.7           81   \n",
       "8780  2012-12-31 20:00:00       0.2                -2.4           83   \n",
       "8781  2012-12-31 21:00:00      -0.5                -1.5           93   \n",
       "8782  2012-12-31 22:00:00      -0.2                -1.8           89   \n",
       "8783  2012-12-31 23:00:00       0.0                -2.1           86   \n",
       "\n",
       "      Wind Spd (km/h)  Visibility (km)  Stn Press (kPa)       Weather  \n",
       "41                 22             25.0            99.66  Snow Showers  \n",
       "44                 24             25.0           100.07  Snow Showers  \n",
       "45                 26             25.0           100.15  Snow Showers  \n",
       "47                 17             19.3           100.27  Snow Showers  \n",
       "48                 28             25.0           100.35  Snow Showers  \n",
       "...               ...              ...              ...           ...  \n",
       "8779               30              9.7           100.13          Snow  \n",
       "8780               24              9.7           100.03          Snow  \n",
       "8781               28              4.8            99.95          Snow  \n",
       "8782               28              9.7            99.91          Snow  \n",
       "8783               30             11.3            99.89          Snow  \n",
       "\n",
       "[583 rows x 8 columns]"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Basically, we want a way to \"filter out\" records that have the word \"snow\" (case insensitive) in the last column\n",
    "\n",
    "snowed_filter = weather_df['Weather'].str.lower().str.contains('snow')\n",
    "weather_df[snowed_filter]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c71c34ac",
   "metadata": {},
   "source": [
    "**Find all instances when wind speed was above 24 and visibility was 25**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "229d5815",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date/Time</th>\n",
       "      <th>Temp (C)</th>\n",
       "      <th>Dew Point Temp (C)</th>\n",
       "      <th>Rel Hum (%)</th>\n",
       "      <th>Wind Spd (km/h)</th>\n",
       "      <th>Visibility (km)</th>\n",
       "      <th>Stn Press (kPa)</th>\n",
       "      <th>Weather</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>2012-01-01 23:00:00</td>\n",
       "      <td>5.3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>79</td>\n",
       "      <td>30</td>\n",
       "      <td>25.0</td>\n",
       "      <td>99.31</td>\n",
       "      <td>Cloudy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>2012-01-02 00:00:00</td>\n",
       "      <td>5.2</td>\n",
       "      <td>1.5</td>\n",
       "      <td>77</td>\n",
       "      <td>35</td>\n",
       "      <td>25.0</td>\n",
       "      <td>99.26</td>\n",
       "      <td>Rain Showers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>2012-01-02 01:00:00</td>\n",
       "      <td>4.6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>72</td>\n",
       "      <td>39</td>\n",
       "      <td>25.0</td>\n",
       "      <td>99.26</td>\n",
       "      <td>Cloudy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>2012-01-02 02:00:00</td>\n",
       "      <td>3.9</td>\n",
       "      <td>-0.9</td>\n",
       "      <td>71</td>\n",
       "      <td>32</td>\n",
       "      <td>25.0</td>\n",
       "      <td>99.26</td>\n",
       "      <td>Mostly Cloudy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>2012-01-02 03:00:00</td>\n",
       "      <td>3.7</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>69</td>\n",
       "      <td>33</td>\n",
       "      <td>25.0</td>\n",
       "      <td>99.30</td>\n",
       "      <td>Mostly Cloudy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Date/Time  Temp (C)  Dew Point Temp (C)  Rel Hum (%)  \\\n",
       "23  2012-01-01 23:00:00       5.3                 2.0           79   \n",
       "24  2012-01-02 00:00:00       5.2                 1.5           77   \n",
       "25  2012-01-02 01:00:00       4.6                 0.0           72   \n",
       "26  2012-01-02 02:00:00       3.9                -0.9           71   \n",
       "27  2012-01-02 03:00:00       3.7                -1.5           69   \n",
       "\n",
       "    Wind Spd (km/h)  Visibility (km)  Stn Press (kPa)        Weather  \n",
       "23               30             25.0            99.31         Cloudy  \n",
       "24               35             25.0            99.26   Rain Showers  \n",
       "25               39             25.0            99.26         Cloudy  \n",
       "26               32             25.0            99.26  Mostly Cloudy  \n",
       "27               33             25.0            99.30  Mostly Cloudy  "
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = weather_df[(weather_df['Wind Spd (km/h)'] > 24) & (weather_df['Visibility (km)']== 25)]\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cb274fd",
   "metadata": {},
   "source": [
    "## Summary / Cheatsheet: Selection/Indexing/Filtering\n",
    "***\n",
    "This is a handy reminder for what syntax will get what result.\n",
    "\n",
    "Syntax | Function | Remarks\n",
    ":--- | :--- | :---\n",
    "**`df['some_label']`** |  Get the (single) Column referenced by name `some_label` | A **str** is provided\n",
    "**`df[['label1', 'label2']]`** | Get multiple columns referenced by given names | A **list** is provided \n",
    "**`df[start:end:step]`** | Get corresponding rows (same as list slicing) | A **slicing operator**<br/> is provided\n",
    "**`df[boolean array/df]`** | Get corresponding rows (same as list slicing) | A **filter object** is provided\n",
    "**`df.loc [row_sel, col_sel]`** | Select specified rows and columns (by labels) | \n",
    "**`df.iloc[row_sel, col_sel]`** | Select specified rows and columns (by index) | \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "778279bd",
   "metadata": {},
   "source": [
    "\n",
    "## Working with Columns\n",
    "***\n",
    "- We will learn how to carry out Series operations on DataFrame Columns\n",
    "- How to add or update columns within a DataFrame\n",
    "- How to rename specific columns\n",
    "- How to delete or drop a column that is no longer required for analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2d19a9b",
   "metadata": {},
   "source": [
    "### Series Operations\n",
    "***\n",
    "A series is a one-dimensional ndarray with axis labels (including time series).\n",
    "\n",
    "Labels need not be unique but must be a hashable type. The object supports both integer- and label-based indexing and provides a host of methods for performing operations involving the index. Statistical methods from ndarray have been overridden to automatically exclude missing data (currently represented as NaN).\n",
    "\n",
    "Operations between Series (+, -, /, , *) align values based on their associated index values– they need not be the same length. The result index after operation will be the sorted union of the two indexes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5337789",
   "metadata": {},
   "source": [
    "Add 10 to the values in the column \"Wind Spd (km/h)\" using the \"+\" operator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "0db14d87",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2.0\n",
       "1    2.0\n",
       "2    3.5\n",
       "3    3.0\n",
       "4    3.5\n",
       "Name: Wind Spd (km/h), dtype: float64"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df[\"Wind Spd (km/h)\"].head() * 2 / 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "6179ab6b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    14\n",
       "1    14\n",
       "2    17\n",
       "3    16\n",
       "4    17\n",
       "Name: Wind Spd (km/h), dtype: int64"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "add_10 = weather_df[\"Wind Spd (km/h)\"] + 10\n",
    "add_10.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c47b133",
   "metadata": {},
   "source": [
    "Multiply the values in the 'Visibility (km)' column by 2 using the asterisk (*) operator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "edb980af",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    16.0\n",
       "1    16.0\n",
       "2     8.0\n",
       "3     8.0\n",
       "4     9.6\n",
       "Name: Visibility (km), dtype: float64"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mult_2 = weather_df['Visibility (km)'] * 2\n",
    "mult_2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e373fca5",
   "metadata": {},
   "source": [
    "Add the \"Temp (C)\" and \"Dew Point Temp (C)\" columns as series \"temperature\"\n",
    "\n",
    "THis can be done by simply passing the column names and using the \"+\" operator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "9057193e",
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_df['new_temp_col'] = weather_df[\"Temp (C)\"] + weather_df[\"Dew Point Temp (C)\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "c8fbd51a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date/Time</th>\n",
       "      <th>Temp (C)</th>\n",
       "      <th>Dew Point Temp (C)</th>\n",
       "      <th>Rel Hum (%)</th>\n",
       "      <th>Wind Spd (km/h)</th>\n",
       "      <th>Visibility (km)</th>\n",
       "      <th>Stn Press (kPa)</th>\n",
       "      <th>Weather</th>\n",
       "      <th>new_temp_col</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-01 00:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.9</td>\n",
       "      <td>86</td>\n",
       "      <td>4</td>\n",
       "      <td>8.0</td>\n",
       "      <td>101.24</td>\n",
       "      <td>Fog</td>\n",
       "      <td>-5.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-01-01 01:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.7</td>\n",
       "      <td>87</td>\n",
       "      <td>4</td>\n",
       "      <td>8.0</td>\n",
       "      <td>101.24</td>\n",
       "      <td>Fog</td>\n",
       "      <td>-5.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-01-01 02:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.4</td>\n",
       "      <td>89</td>\n",
       "      <td>7</td>\n",
       "      <td>4.0</td>\n",
       "      <td>101.26</td>\n",
       "      <td>Freezing Drizzle,Fog</td>\n",
       "      <td>-5.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2012-01-01 03:00:00</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>-3.2</td>\n",
       "      <td>88</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>101.27</td>\n",
       "      <td>Freezing Drizzle,Fog</td>\n",
       "      <td>-4.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-01-01 04:00:00</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>-3.3</td>\n",
       "      <td>88</td>\n",
       "      <td>7</td>\n",
       "      <td>4.8</td>\n",
       "      <td>101.23</td>\n",
       "      <td>Fog</td>\n",
       "      <td>-4.8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Date/Time  Temp (C)  Dew Point Temp (C)  Rel Hum (%)  \\\n",
       "0  2012-01-01 00:00:00      -1.8                -3.9           86   \n",
       "1  2012-01-01 01:00:00      -1.8                -3.7           87   \n",
       "2  2012-01-01 02:00:00      -1.8                -3.4           89   \n",
       "3  2012-01-01 03:00:00      -1.5                -3.2           88   \n",
       "4  2012-01-01 04:00:00      -1.5                -3.3           88   \n",
       "\n",
       "   Wind Spd (km/h)  Visibility (km)  Stn Press (kPa)               Weather  \\\n",
       "0                4              8.0           101.24                   Fog   \n",
       "1                4              8.0           101.24                   Fog   \n",
       "2                7              4.0           101.26  Freezing Drizzle,Fog   \n",
       "3                6              4.0           101.27  Freezing Drizzle,Fog   \n",
       "4                7              4.8           101.23                   Fog   \n",
       "\n",
       "   new_temp_col  \n",
       "0          -5.7  \n",
       "1          -5.5  \n",
       "2          -5.2  \n",
       "3          -4.7  \n",
       "4          -4.8  "
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "8bc2ccaf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0   -5.7\n",
       "1   -5.5\n",
       "2   -5.2\n",
       "3   -4.7\n",
       "4   -4.8\n",
       "dtype: float64"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "temperature = weather_df[\"Temp (C)\"] + weather_df[\"Dew Point Temp (C)\"]\n",
    "temperature.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37d83840",
   "metadata": {},
   "source": [
    "## Apply / Call Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f65567e6",
   "metadata": {},
   "source": [
    "## `.apply()`\n",
    "***\n",
    "\n",
    "You can pass any number of arguments to the function that apply is calling through either unnamed arguments, passed as a tuple to the args parameter, or through other keyword arguments internally captured as a dictionary by the kwds parameter.\n",
    "\n",
    "Invoke function on values of Series. Can be a NumPy function that applies to the entire Series or a Python function that only works on single values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "cace021f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Applying custom functions\n",
    "\n",
    "def times2(value):\n",
    "    return value * 2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "839dc3c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    8.0\n",
       "1    8.0\n",
       "2    4.0\n",
       "3    4.0\n",
       "4    4.8\n",
       "Name: Visibility (km), dtype: float64"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df['Visibility (km)'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "de42c7ce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    16.0\n",
       "1    16.0\n",
       "2     8.0\n",
       "3     8.0\n",
       "4     9.6\n",
       "Name: Visibility (km), dtype: float64"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df['Visibility (km)'].apply(times2).head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca8e9f51",
   "metadata": {},
   "source": [
    "## `.describe()`\n",
    "***\n",
    "\n",
    "The above function is used to summarize the  central tendency, dispersion and shape of a dataset’s distribution, excluding NaN values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "dfa794af",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    8784.000000\n",
       "mean       27.664447\n",
       "std        12.622688\n",
       "min         0.200000\n",
       "25%        24.100000\n",
       "50%        25.000000\n",
       "75%        25.000000\n",
       "max        48.300000\n",
       "Name: Visibility (km), dtype: float64"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df['Visibility (km)'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0776b0f2",
   "metadata": {},
   "source": [
    "### Adding/Updating Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "e9f53d34",
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_df[\"Visibility (m)\"] = weather_df[\"Visibility (km)\"] * 1000  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "867937d4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date/Time</th>\n",
       "      <th>Temp (C)</th>\n",
       "      <th>Dew Point Temp (C)</th>\n",
       "      <th>Rel Hum (%)</th>\n",
       "      <th>Wind Spd (km/h)</th>\n",
       "      <th>Visibility (km)</th>\n",
       "      <th>Stn Press (kPa)</th>\n",
       "      <th>Weather</th>\n",
       "      <th>Visibility (m)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-01 00:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.9</td>\n",
       "      <td>86</td>\n",
       "      <td>4</td>\n",
       "      <td>8.0</td>\n",
       "      <td>101.24</td>\n",
       "      <td>Fog</td>\n",
       "      <td>8000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-01-01 01:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.7</td>\n",
       "      <td>87</td>\n",
       "      <td>4</td>\n",
       "      <td>8.0</td>\n",
       "      <td>101.24</td>\n",
       "      <td>Fog</td>\n",
       "      <td>8000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-01-01 02:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.4</td>\n",
       "      <td>89</td>\n",
       "      <td>7</td>\n",
       "      <td>4.0</td>\n",
       "      <td>101.26</td>\n",
       "      <td>Freezing Drizzle,Fog</td>\n",
       "      <td>4000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2012-01-01 03:00:00</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>-3.2</td>\n",
       "      <td>88</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>101.27</td>\n",
       "      <td>Freezing Drizzle,Fog</td>\n",
       "      <td>4000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-01-01 04:00:00</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>-3.3</td>\n",
       "      <td>88</td>\n",
       "      <td>7</td>\n",
       "      <td>4.8</td>\n",
       "      <td>101.23</td>\n",
       "      <td>Fog</td>\n",
       "      <td>4800.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Date/Time  Temp (C)  Dew Point Temp (C)  Rel Hum (%)  \\\n",
       "0 2012-01-01 00:00:00      -1.8                -3.9           86   \n",
       "1 2012-01-01 01:00:00      -1.8                -3.7           87   \n",
       "2 2012-01-01 02:00:00      -1.8                -3.4           89   \n",
       "3 2012-01-01 03:00:00      -1.5                -3.2           88   \n",
       "4 2012-01-01 04:00:00      -1.5                -3.3           88   \n",
       "\n",
       "   Wind Spd (km/h)  Visibility (km)  Stn Press (kPa)               Weather  \\\n",
       "0                4              8.0           101.24                   Fog   \n",
       "1                4              8.0           101.24                   Fog   \n",
       "2                7              4.0           101.26  Freezing Drizzle,Fog   \n",
       "3                6              4.0           101.27  Freezing Drizzle,Fog   \n",
       "4                7              4.8           101.23                   Fog   \n",
       "\n",
       "   Visibility (m)  \n",
       "0          8000.0  \n",
       "1          8000.0  \n",
       "2          4000.0  \n",
       "3          4000.0  \n",
       "4          4800.0  "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "visibility_in_meter = weather_df[\"Visibility (km)\"] * 1000  \n",
    "weather_df[\"Visibility (m)\"] = visibility_in_meter\n",
    "\n",
    "weather_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fea27c10",
   "metadata": {},
   "source": [
    "### Renaming Columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c5a8be6",
   "metadata": {},
   "source": [
    "## `.rename()`\n",
    "***\n",
    "\n",
    "Alter Series index labels or name. It will replace the existing names with the names you provide, in the order you provide.\n",
    "\n",
    "You can also assign the names by index.\n",
    "\n",
    "The rename() method allows you to relabel an axis based on some mapping (a dict or Series) or an arbitrary function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "41c81d6a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date/Time</th>\n",
       "      <th>Temp (C)</th>\n",
       "      <th>Dew Point Temp (C)</th>\n",
       "      <th>Rel Hum (%)</th>\n",
       "      <th>Wind Spd (km/h)</th>\n",
       "      <th>Visibility (km)</th>\n",
       "      <th>Stn Press (kPa)</th>\n",
       "      <th>Weather</th>\n",
       "      <th>new_temp_col</th>\n",
       "      <th>Visibility (meters)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-01 00:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.9</td>\n",
       "      <td>86</td>\n",
       "      <td>4</td>\n",
       "      <td>8.0</td>\n",
       "      <td>101.24</td>\n",
       "      <td>Fog</td>\n",
       "      <td>-5.7</td>\n",
       "      <td>8000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-01-01 01:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.7</td>\n",
       "      <td>87</td>\n",
       "      <td>4</td>\n",
       "      <td>8.0</td>\n",
       "      <td>101.24</td>\n",
       "      <td>Fog</td>\n",
       "      <td>-5.5</td>\n",
       "      <td>8000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-01-01 02:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.4</td>\n",
       "      <td>89</td>\n",
       "      <td>7</td>\n",
       "      <td>4.0</td>\n",
       "      <td>101.26</td>\n",
       "      <td>Freezing Drizzle,Fog</td>\n",
       "      <td>-5.2</td>\n",
       "      <td>4000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2012-01-01 03:00:00</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>-3.2</td>\n",
       "      <td>88</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>101.27</td>\n",
       "      <td>Freezing Drizzle,Fog</td>\n",
       "      <td>-4.7</td>\n",
       "      <td>4000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-01-01 04:00:00</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>-3.3</td>\n",
       "      <td>88</td>\n",
       "      <td>7</td>\n",
       "      <td>4.8</td>\n",
       "      <td>101.23</td>\n",
       "      <td>Fog</td>\n",
       "      <td>-4.8</td>\n",
       "      <td>4800.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Date/Time  Temp (C)  Dew Point Temp (C)  Rel Hum (%)  \\\n",
       "0  2012-01-01 00:00:00      -1.8                -3.9           86   \n",
       "1  2012-01-01 01:00:00      -1.8                -3.7           87   \n",
       "2  2012-01-01 02:00:00      -1.8                -3.4           89   \n",
       "3  2012-01-01 03:00:00      -1.5                -3.2           88   \n",
       "4  2012-01-01 04:00:00      -1.5                -3.3           88   \n",
       "\n",
       "   Wind Spd (km/h)  Visibility (km)  Stn Press (kPa)               Weather  \\\n",
       "0                4              8.0           101.24                   Fog   \n",
       "1                4              8.0           101.24                   Fog   \n",
       "2                7              4.0           101.26  Freezing Drizzle,Fog   \n",
       "3                6              4.0           101.27  Freezing Drizzle,Fog   \n",
       "4                7              4.8           101.23                   Fog   \n",
       "\n",
       "   new_temp_col  Visibility (meters)  \n",
       "0          -5.7               8000.0  \n",
       "1          -5.5               8000.0  \n",
       "2          -5.2               4000.0  \n",
       "3          -4.7               4000.0  \n",
       "4          -4.8               4800.0  "
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Notice the \"inplace=True\" parameter. This means the renaming has been assigned in the old DataFrame itself\n",
    "\n",
    "weather_df.rename(columns={'Visibility (m)': 'Visibility (meters)'}, inplace=True)\n",
    "weather_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "787cbd61",
   "metadata": {},
   "source": [
    "### Deleting Columns\n",
    "\n",
    "## `.drop()`\n",
    "***\n",
    "\n",
    "Return new object with labels in requested axis removed.\n",
    "\n",
    "Note that Pandas uses zero based numbering, so 0 is the first row, 1 is the second row, etc.\n",
    "You can select ranges relative to the top or drop relative to the bottom of the DataFrame as well.\n",
    "\n",
    "- Note: Specifying both labels and index or columns will raise a ValueError."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "56e7c070",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date/Time</th>\n",
       "      <th>Temp (C)</th>\n",
       "      <th>Dew Point Temp (C)</th>\n",
       "      <th>Rel Hum (%)</th>\n",
       "      <th>Wind Spd (km/h)</th>\n",
       "      <th>Visibility (km)</th>\n",
       "      <th>Stn Press (kPa)</th>\n",
       "      <th>Weather</th>\n",
       "      <th>new_temp_col</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-01 00:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.9</td>\n",
       "      <td>86</td>\n",
       "      <td>4</td>\n",
       "      <td>8.0</td>\n",
       "      <td>101.24</td>\n",
       "      <td>Fog</td>\n",
       "      <td>-5.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-01-01 01:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.7</td>\n",
       "      <td>87</td>\n",
       "      <td>4</td>\n",
       "      <td>8.0</td>\n",
       "      <td>101.24</td>\n",
       "      <td>Fog</td>\n",
       "      <td>-5.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-01-01 02:00:00</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>-3.4</td>\n",
       "      <td>89</td>\n",
       "      <td>7</td>\n",
       "      <td>4.0</td>\n",
       "      <td>101.26</td>\n",
       "      <td>Freezing Drizzle,Fog</td>\n",
       "      <td>-5.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2012-01-01 03:00:00</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>-3.2</td>\n",
       "      <td>88</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>101.27</td>\n",
       "      <td>Freezing Drizzle,Fog</td>\n",
       "      <td>-4.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-01-01 04:00:00</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>-3.3</td>\n",
       "      <td>88</td>\n",
       "      <td>7</td>\n",
       "      <td>4.8</td>\n",
       "      <td>101.23</td>\n",
       "      <td>Fog</td>\n",
       "      <td>-4.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8779</th>\n",
       "      <td>2012-12-31 19:00:00</td>\n",
       "      <td>0.1</td>\n",
       "      <td>-2.7</td>\n",
       "      <td>81</td>\n",
       "      <td>30</td>\n",
       "      <td>9.7</td>\n",
       "      <td>100.13</td>\n",
       "      <td>Snow</td>\n",
       "      <td>-2.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8780</th>\n",
       "      <td>2012-12-31 20:00:00</td>\n",
       "      <td>0.2</td>\n",
       "      <td>-2.4</td>\n",
       "      <td>83</td>\n",
       "      <td>24</td>\n",
       "      <td>9.7</td>\n",
       "      <td>100.03</td>\n",
       "      <td>Snow</td>\n",
       "      <td>-2.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8781</th>\n",
       "      <td>2012-12-31 21:00:00</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>93</td>\n",
       "      <td>28</td>\n",
       "      <td>4.8</td>\n",
       "      <td>99.95</td>\n",
       "      <td>Snow</td>\n",
       "      <td>-2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8782</th>\n",
       "      <td>2012-12-31 22:00:00</td>\n",
       "      <td>-0.2</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>89</td>\n",
       "      <td>28</td>\n",
       "      <td>9.7</td>\n",
       "      <td>99.91</td>\n",
       "      <td>Snow</td>\n",
       "      <td>-2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8783</th>\n",
       "      <td>2012-12-31 23:00:00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-2.1</td>\n",
       "      <td>86</td>\n",
       "      <td>30</td>\n",
       "      <td>11.3</td>\n",
       "      <td>99.89</td>\n",
       "      <td>Snow</td>\n",
       "      <td>-2.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8784 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                Date/Time  Temp (C)  Dew Point Temp (C)  Rel Hum (%)  \\\n",
       "0     2012-01-01 00:00:00      -1.8                -3.9           86   \n",
       "1     2012-01-01 01:00:00      -1.8                -3.7           87   \n",
       "2     2012-01-01 02:00:00      -1.8                -3.4           89   \n",
       "3     2012-01-01 03:00:00      -1.5                -3.2           88   \n",
       "4     2012-01-01 04:00:00      -1.5                -3.3           88   \n",
       "...                   ...       ...                 ...          ...   \n",
       "8779  2012-12-31 19:00:00       0.1                -2.7           81   \n",
       "8780  2012-12-31 20:00:00       0.2                -2.4           83   \n",
       "8781  2012-12-31 21:00:00      -0.5                -1.5           93   \n",
       "8782  2012-12-31 22:00:00      -0.2                -1.8           89   \n",
       "8783  2012-12-31 23:00:00       0.0                -2.1           86   \n",
       "\n",
       "      Wind Spd (km/h)  Visibility (km)  Stn Press (kPa)               Weather  \\\n",
       "0                   4              8.0           101.24                   Fog   \n",
       "1                   4              8.0           101.24                   Fog   \n",
       "2                   7              4.0           101.26  Freezing Drizzle,Fog   \n",
       "3                   6              4.0           101.27  Freezing Drizzle,Fog   \n",
       "4                   7              4.8           101.23                   Fog   \n",
       "...               ...              ...              ...                   ...   \n",
       "8779               30              9.7           100.13                  Snow   \n",
       "8780               24              9.7           100.03                  Snow   \n",
       "8781               28              4.8            99.95                  Snow   \n",
       "8782               28              9.7            99.91                  Snow   \n",
       "8783               30             11.3            99.89                  Snow   \n",
       "\n",
       "      new_temp_col  \n",
       "0             -5.7  \n",
       "1             -5.5  \n",
       "2             -5.2  \n",
       "3             -4.7  \n",
       "4             -4.8  \n",
       "...            ...  \n",
       "8779          -2.6  \n",
       "8780          -2.2  \n",
       "8781          -2.0  \n",
       "8782          -2.0  \n",
       "8783          -2.1  \n",
       "\n",
       "[8784 rows x 9 columns]"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Since we have not mentioned inplace=True, it returns a new dataframe.\n",
    "weather_df.drop(labels=['Visibility (meters)'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74f7a0ab",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Sorting\n",
    "***\n",
    "\n",
    "## `.sort_values()`\n",
    "***\n",
    "\n",
    "Sort by the values along either axis, in a user specified order. The order can be specified by selecting true or false for the \"ascending\" parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "01d1e220",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date/Time</th>\n",
       "      <th>Temp (C)</th>\n",
       "      <th>Dew Point Temp (C)</th>\n",
       "      <th>Rel Hum (%)</th>\n",
       "      <th>Wind Spd (km/h)</th>\n",
       "      <th>Visibility (km)</th>\n",
       "      <th>Stn Press (kPa)</th>\n",
       "      <th>Weather</th>\n",
       "      <th>Visibility (meters)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4695</th>\n",
       "      <td>2012-07-14 15:00:00</td>\n",
       "      <td>33.0</td>\n",
       "      <td>16.8</td>\n",
       "      <td>38</td>\n",
       "      <td>22</td>\n",
       "      <td>48.3</td>\n",
       "      <td>101.31</td>\n",
       "      <td>Mainly Clear</td>\n",
       "      <td>48300.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4143</th>\n",
       "      <td>2012-06-21 15:00:00</td>\n",
       "      <td>33.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>44</td>\n",
       "      <td>24</td>\n",
       "      <td>24.1</td>\n",
       "      <td>100.20</td>\n",
       "      <td>Mainly Clear</td>\n",
       "      <td>24100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4696</th>\n",
       "      <td>2012-07-14 16:00:00</td>\n",
       "      <td>32.9</td>\n",
       "      <td>15.3</td>\n",
       "      <td>35</td>\n",
       "      <td>24</td>\n",
       "      <td>48.3</td>\n",
       "      <td>101.26</td>\n",
       "      <td>Mainly Clear</td>\n",
       "      <td>48300.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5199</th>\n",
       "      <td>2012-08-04 15:00:00</td>\n",
       "      <td>32.8</td>\n",
       "      <td>18.8</td>\n",
       "      <td>44</td>\n",
       "      <td>17</td>\n",
       "      <td>24.1</td>\n",
       "      <td>101.39</td>\n",
       "      <td>Clear</td>\n",
       "      <td>24100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4694</th>\n",
       "      <td>2012-07-14 14:00:00</td>\n",
       "      <td>32.7</td>\n",
       "      <td>15.3</td>\n",
       "      <td>35</td>\n",
       "      <td>28</td>\n",
       "      <td>48.3</td>\n",
       "      <td>101.35</td>\n",
       "      <td>Mainly Clear</td>\n",
       "      <td>48300.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Date/Time  Temp (C)  Dew Point Temp (C)  Rel Hum (%)  \\\n",
       "4695 2012-07-14 15:00:00      33.0                16.8           38   \n",
       "4143 2012-06-21 15:00:00      33.0                19.0           44   \n",
       "4696 2012-07-14 16:00:00      32.9                15.3           35   \n",
       "5199 2012-08-04 15:00:00      32.8                18.8           44   \n",
       "4694 2012-07-14 14:00:00      32.7                15.3           35   \n",
       "\n",
       "      Wind Spd (km/h)  Visibility (km)  Stn Press (kPa)       Weather  \\\n",
       "4695               22             48.3           101.31  Mainly Clear   \n",
       "4143               24             24.1           100.20  Mainly Clear   \n",
       "4696               24             48.3           101.26  Mainly Clear   \n",
       "5199               17             24.1           101.39         Clear   \n",
       "4694               28             48.3           101.35  Mainly Clear   \n",
       "\n",
       "      Visibility (meters)  \n",
       "4695              48300.0  \n",
       "4143              24100.0  \n",
       "4696              48300.0  \n",
       "5199              24100.0  \n",
       "4694              48300.0  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted_by_temp = weather_df.sort_values('Temp (C)', ascending=False)  # can be inplace as well\n",
    "sorted_by_temp.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "968bcb72",
   "metadata": {},
   "source": [
    "### Which were the top 10 hottest values and their counts?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "6f42580e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16.6    65\n",
       "1.1     58\n",
       "0.8     47\n",
       "1.5     45\n",
       "19.3    44\n",
       "21.1    43\n",
       "2.6     43\n",
       "0.4     41\n",
       "1.3     40\n",
       "14.6    39\n",
       "Name: Temp (C), dtype: int64"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted_value_counts = weather_df['Temp (C)'].value_counts().sort_values(ascending=False)\n",
    "sorted_value_counts.iloc[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "88f51a30",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1     -7.371505\n",
       "2     -4.225000\n",
       "3      3.121237\n",
       "4      7.009306\n",
       "5     16.237769\n",
       "6     20.134028\n",
       "7     22.790054\n",
       "8     22.279301\n",
       "9     16.484444\n",
       "10    10.954973\n",
       "11     0.931389\n",
       "12    -3.306317\n",
       "dtype: float64"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# One Solution\n",
    "mean_temperatures = {}\n",
    "\n",
    "for month in range(1, 13):\n",
    "    mean_temperatures[month] = weather_df.loc[weather_df['Date/Time'].dt.month == month, 'Temp (C)'].mean()\n",
    "\n",
    "pd.Series(mean_temperatures)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdbf58e0",
   "metadata": {},
   "source": [
    "\n",
    "# Pivot Tables : Excellent way to Summarize your Data!\n",
    "***\n",
    "- A pivot table is a tool that allows you to reorganize and summarize selected columns and rows of data in a dataframe <br/><br/>\n",
    "\n",
    "- Pivot tables provide an easy way to subset by one column and then apply a calculation like a sum or a mean <br/><br/>\n",
    "\n",
    "- Pivot tables first groups and only then applies a calculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "cd343b08",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>foo</td>\n",
       "      <td>one</td>\n",
       "      <td>x</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>foo</td>\n",
       "      <td>one</td>\n",
       "      <td>y</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>foo</td>\n",
       "      <td>two</td>\n",
       "      <td>x</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>bar</td>\n",
       "      <td>two</td>\n",
       "      <td>y</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>bar</td>\n",
       "      <td>one</td>\n",
       "      <td>x</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>bar</td>\n",
       "      <td>one</td>\n",
       "      <td>y</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B  C  D\n",
       "0  foo  one  x  1\n",
       "1  foo  one  y  3\n",
       "2  foo  two  x  2\n",
       "3  bar  two  y  5\n",
       "4  bar  one  x  4\n",
       "5  bar  one  y  1"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = {\n",
    "    'A': ['foo','foo','foo','bar','bar','bar'],\n",
    "    'B': ['one','one','two','two','one','one'],\n",
    "    'C': ['x','y','x','y','x','y'],\n",
    "    'D': [1, 3, 2, 5, 4, 1]\n",
    "}\n",
    "\n",
    "df = pd.DataFrame(data)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "48ed6493",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>C</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bar</th>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>foo</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "C    x  y\n",
       "A        \n",
       "bar  4  6\n",
       "foo  3  3"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pivot_df = df.pivot_table(\n",
    "                values='D',      # We want to aggregate the values of which column?\n",
    "                index='A',       # We want to use which column as the new index?\n",
    "                columns=['C'],   # We want to use the values of which column as the new columns? (optional)\n",
    "                aggfunc=np.sum)  # What aggregation function to use ?\n",
    "\n",
    "\n",
    "pivot_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "7a902fe0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>C</th>\n",
       "      <th>A</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bar</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>foo</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "C    A  x  y\n",
       "0  bar  4  6\n",
       "1  foo  3  3"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert it back to a simple index\n",
    "\n",
    "pivot_df.reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceaed435",
   "metadata": {},
   "source": [
    "### What is the mean temperature recorded by month?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "980117de",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Temp (C)</th>\n",
       "      <th>Date/Time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-1.8</td>\n",
       "      <td>2012-01-01 00:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.8</td>\n",
       "      <td>2012-01-01 01:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1.8</td>\n",
       "      <td>2012-01-01 02:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1.5</td>\n",
       "      <td>2012-01-01 03:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1.5</td>\n",
       "      <td>2012-01-01 04:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8779</th>\n",
       "      <td>0.1</td>\n",
       "      <td>2012-12-31 19:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8780</th>\n",
       "      <td>0.2</td>\n",
       "      <td>2012-12-31 20:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8781</th>\n",
       "      <td>-0.5</td>\n",
       "      <td>2012-12-31 21:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8782</th>\n",
       "      <td>-0.2</td>\n",
       "      <td>2012-12-31 22:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8783</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2012-12-31 23:00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8784 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Temp (C)            Date/Time\n",
       "0         -1.8  2012-01-01 00:00:00\n",
       "1         -1.8  2012-01-01 01:00:00\n",
       "2         -1.8  2012-01-01 02:00:00\n",
       "3         -1.5  2012-01-01 03:00:00\n",
       "4         -1.5  2012-01-01 04:00:00\n",
       "...        ...                  ...\n",
       "8779       0.1  2012-12-31 19:00:00\n",
       "8780       0.2  2012-12-31 20:00:00\n",
       "8781      -0.5  2012-12-31 21:00:00\n",
       "8782      -0.2  2012-12-31 22:00:00\n",
       "8783       0.0  2012-12-31 23:00:00\n",
       "\n",
       "[8784 rows x 2 columns]"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df[['Temp (C)','Date/Time']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "id": "776adcc7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        1\n",
       "1        1\n",
       "2        1\n",
       "3        1\n",
       "4        1\n",
       "        ..\n",
       "8779    12\n",
       "8780    12\n",
       "8781    12\n",
       "8782    12\n",
       "8783    12\n",
       "Name: Date/Time, Length: 8784, dtype: int64"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df['Date/Time'].dt.month"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "id": "b252e335",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Temp (C)</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date/Time</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-7.371505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-4.225000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.121237</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7.009306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>16.237769</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>20.134028</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>22.790054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>22.279301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>16.484444</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>10.954973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.931389</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>-3.306317</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Temp (C)\n",
       "Date/Time           \n",
       "1          -7.371505\n",
       "2          -4.225000\n",
       "3           3.121237\n",
       "4           7.009306\n",
       "5          16.237769\n",
       "6          20.134028\n",
       "7          22.790054\n",
       "8          22.279301\n",
       "9          16.484444\n",
       "10         10.954973\n",
       "11          0.931389\n",
       "12         -3.306317"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_temperature_df = weather_df.pivot_table(values='Temp (C)', index=weather_df['Date/Time'].dt.month, aggfunc=np.mean)\n",
    "mean_temperature_df # the numbers 1 to 12 denote the respective months from January to December."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b038761",
   "metadata": {},
   "source": [
    "\n",
    "# Group By\n",
    "***\n",
    "The groupby method allows you to group rows of data together and call aggregate functions that applies to the whole group.\n",
    "\n",
    "Any groupby operation involves one of the following operations on the original object. They are −\n",
    "- Splitting the Object\n",
    "- Applying a function\n",
    "- Combining the results\n",
    "\n",
    "In many situations, we split the data into sets and we apply some functionality on each subset. In the apply functionality, we can perform the following operations −\n",
    "\n",
    "- Aggregation − computing a summary statistic\n",
    "- Transformation − perform some group-specific operation\n",
    "- Filtration − discarding the data with some condition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "id": "3feaddd6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date/Time</th>\n",
       "      <th>Temp (C)</th>\n",
       "      <th>Dew Point Temp (C)</th>\n",
       "      <th>Rel Hum (%)</th>\n",
       "      <th>Wind Spd (km/h)</th>\n",
       "      <th>Visibility (km)</th>\n",
       "      <th>Stn Press (kPa)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>-7.371505</td>\n",
       "      <td>-12.294758</td>\n",
       "      <td>68.383065</td>\n",
       "      <td>18.108871</td>\n",
       "      <td>22.100269</td>\n",
       "      <td>101.005349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>-4.225000</td>\n",
       "      <td>-9.221695</td>\n",
       "      <td>68.956897</td>\n",
       "      <td>14.837644</td>\n",
       "      <td>25.182184</td>\n",
       "      <td>101.142414</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>3.121237</td>\n",
       "      <td>-3.488575</td>\n",
       "      <td>64.862903</td>\n",
       "      <td>14.514785</td>\n",
       "      <td>26.177957</td>\n",
       "      <td>101.335255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>7.009306</td>\n",
       "      <td>-1.934583</td>\n",
       "      <td>56.150000</td>\n",
       "      <td>17.369444</td>\n",
       "      <td>31.777083</td>\n",
       "      <td>100.716833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>16.237769</td>\n",
       "      <td>8.080780</td>\n",
       "      <td>61.760753</td>\n",
       "      <td>12.846774</td>\n",
       "      <td>29.418548</td>\n",
       "      <td>101.057164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>20.134028</td>\n",
       "      <td>11.738056</td>\n",
       "      <td>60.643056</td>\n",
       "      <td>14.681944</td>\n",
       "      <td>32.104167</td>\n",
       "      <td>100.784222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>22.790054</td>\n",
       "      <td>14.595430</td>\n",
       "      <td>62.017473</td>\n",
       "      <td>11.887097</td>\n",
       "      <td>33.655108</td>\n",
       "      <td>100.828333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>22.279301</td>\n",
       "      <td>15.644758</td>\n",
       "      <td>67.943548</td>\n",
       "      <td>13.931452</td>\n",
       "      <td>30.192608</td>\n",
       "      <td>100.927097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>16.484444</td>\n",
       "      <td>10.757917</td>\n",
       "      <td>71.165278</td>\n",
       "      <td>14.108333</td>\n",
       "      <td>30.603472</td>\n",
       "      <td>101.087903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>10.954973</td>\n",
       "      <td>6.533468</td>\n",
       "      <td>75.731183</td>\n",
       "      <td>15.475806</td>\n",
       "      <td>25.111022</td>\n",
       "      <td>100.909368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>0.931389</td>\n",
       "      <td>-4.179306</td>\n",
       "      <td>69.984722</td>\n",
       "      <td>13.973611</td>\n",
       "      <td>26.077083</td>\n",
       "      <td>101.658306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>-3.306317</td>\n",
       "      <td>-6.129032</td>\n",
       "      <td>81.299731</td>\n",
       "      <td>17.614247</td>\n",
       "      <td>19.733199</td>\n",
       "      <td>101.174409</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Date/Time   Temp (C)  Dew Point Temp (C)  Rel Hum (%)  Wind Spd (km/h)  \\\n",
       "0           1  -7.371505          -12.294758    68.383065        18.108871   \n",
       "1           2  -4.225000           -9.221695    68.956897        14.837644   \n",
       "2           3   3.121237           -3.488575    64.862903        14.514785   \n",
       "3           4   7.009306           -1.934583    56.150000        17.369444   \n",
       "4           5  16.237769            8.080780    61.760753        12.846774   \n",
       "5           6  20.134028           11.738056    60.643056        14.681944   \n",
       "6           7  22.790054           14.595430    62.017473        11.887097   \n",
       "7           8  22.279301           15.644758    67.943548        13.931452   \n",
       "8           9  16.484444           10.757917    71.165278        14.108333   \n",
       "9          10  10.954973            6.533468    75.731183        15.475806   \n",
       "10         11   0.931389           -4.179306    69.984722        13.973611   \n",
       "11         12  -3.306317           -6.129032    81.299731        17.614247   \n",
       "\n",
       "    Visibility (km)  Stn Press (kPa)  \n",
       "0         22.100269       101.005349  \n",
       "1         25.182184       101.142414  \n",
       "2         26.177957       101.335255  \n",
       "3         31.777083       100.716833  \n",
       "4         29.418548       101.057164  \n",
       "5         32.104167       100.784222  \n",
       "6         33.655108       100.828333  \n",
       "7         30.192608       100.927097  \n",
       "8         30.603472       101.087903  \n",
       "9         25.111022       100.909368  \n",
       "10        26.077083       101.658306  \n",
       "11        19.733199       101.174409  "
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather_df.groupby(weather_df['Date/Time'].dt.month).agg(np.mean).reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "e12ff8db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date/Time</th>\n",
       "      <th>Temp (C)</th>\n",
       "      <th>Dew Point Temp (C)</th>\n",
       "      <th>Rel Hum (%)</th>\n",
       "      <th>Wind Spd (km/h)</th>\n",
       "      <th>Visibility (km)</th>\n",
       "      <th>Stn Press (kPa)</th>\n",
       "      <th>Visibility (meters)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>-7.371505</td>\n",
       "      <td>-12.294758</td>\n",
       "      <td>68.383065</td>\n",
       "      <td>18.108871</td>\n",
       "      <td>22.100269</td>\n",
       "      <td>101.005349</td>\n",
       "      <td>22100.268817</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>-4.225000</td>\n",
       "      <td>-9.221695</td>\n",
       "      <td>68.956897</td>\n",
       "      <td>14.837644</td>\n",
       "      <td>25.182184</td>\n",
       "      <td>101.142414</td>\n",
       "      <td>25182.183908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>3.121237</td>\n",
       "      <td>-3.488575</td>\n",
       "      <td>64.862903</td>\n",
       "      <td>14.514785</td>\n",
       "      <td>26.177957</td>\n",
       "      <td>101.335255</td>\n",
       "      <td>26177.956989</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>7.009306</td>\n",
       "      <td>-1.934583</td>\n",
       "      <td>56.150000</td>\n",
       "      <td>17.369444</td>\n",
       "      <td>31.777083</td>\n",
       "      <td>100.716833</td>\n",
       "      <td>31777.083333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>16.237769</td>\n",
       "      <td>8.080780</td>\n",
       "      <td>61.760753</td>\n",
       "      <td>12.846774</td>\n",
       "      <td>29.418548</td>\n",
       "      <td>101.057164</td>\n",
       "      <td>29418.548387</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>20.134028</td>\n",
       "      <td>11.738056</td>\n",
       "      <td>60.643056</td>\n",
       "      <td>14.681944</td>\n",
       "      <td>32.104167</td>\n",
       "      <td>100.784222</td>\n",
       "      <td>32104.166667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>22.790054</td>\n",
       "      <td>14.595430</td>\n",
       "      <td>62.017473</td>\n",
       "      <td>11.887097</td>\n",
       "      <td>33.655108</td>\n",
       "      <td>100.828333</td>\n",
       "      <td>33655.107527</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>22.279301</td>\n",
       "      <td>15.644758</td>\n",
       "      <td>67.943548</td>\n",
       "      <td>13.931452</td>\n",
       "      <td>30.192608</td>\n",
       "      <td>100.927097</td>\n",
       "      <td>30192.607527</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>16.484444</td>\n",
       "      <td>10.757917</td>\n",
       "      <td>71.165278</td>\n",
       "      <td>14.108333</td>\n",
       "      <td>30.603472</td>\n",
       "      <td>101.087903</td>\n",
       "      <td>30603.472222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>10.954973</td>\n",
       "      <td>6.533468</td>\n",
       "      <td>75.731183</td>\n",
       "      <td>15.475806</td>\n",
       "      <td>25.111022</td>\n",
       "      <td>100.909368</td>\n",
       "      <td>25111.021505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>0.931389</td>\n",
       "      <td>-4.179306</td>\n",
       "      <td>69.984722</td>\n",
       "      <td>13.973611</td>\n",
       "      <td>26.077083</td>\n",
       "      <td>101.658306</td>\n",
       "      <td>26077.083333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>-3.306317</td>\n",
       "      <td>-6.129032</td>\n",
       "      <td>81.299731</td>\n",
       "      <td>17.614247</td>\n",
       "      <td>19.733199</td>\n",
       "      <td>101.174409</td>\n",
       "      <td>19733.198925</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Date/Time   Temp (C)  Dew Point Temp (C)  Rel Hum (%)  Wind Spd (km/h)  \\\n",
       "0           1  -7.371505          -12.294758    68.383065        18.108871   \n",
       "1           2  -4.225000           -9.221695    68.956897        14.837644   \n",
       "2           3   3.121237           -3.488575    64.862903        14.514785   \n",
       "3           4   7.009306           -1.934583    56.150000        17.369444   \n",
       "4           5  16.237769            8.080780    61.760753        12.846774   \n",
       "5           6  20.134028           11.738056    60.643056        14.681944   \n",
       "6           7  22.790054           14.595430    62.017473        11.887097   \n",
       "7           8  22.279301           15.644758    67.943548        13.931452   \n",
       "8           9  16.484444           10.757917    71.165278        14.108333   \n",
       "9          10  10.954973            6.533468    75.731183        15.475806   \n",
       "10         11   0.931389           -4.179306    69.984722        13.973611   \n",
       "11         12  -3.306317           -6.129032    81.299731        17.614247   \n",
       "\n",
       "    Visibility (km)  Stn Press (kPa)  Visibility (meters)  \n",
       "0         22.100269       101.005349         22100.268817  \n",
       "1         25.182184       101.142414         25182.183908  \n",
       "2         26.177957       101.335255         26177.956989  \n",
       "3         31.777083       100.716833         31777.083333  \n",
       "4         29.418548       101.057164         29418.548387  \n",
       "5         32.104167       100.784222         32104.166667  \n",
       "6         33.655108       100.828333         33655.107527  \n",
       "7         30.192608       100.927097         30192.607527  \n",
       "8         30.603472       101.087903         30603.472222  \n",
       "9         25.111022       100.909368         25111.021505  \n",
       "10        26.077083       101.658306         26077.083333  \n",
       "11        19.733199       101.174409         19733.198925  "
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_temperature_df2 = weather_df.groupby(weather_df['Date/Time'].dt.month).agg(np.mean).reset_index()\n",
    "mean_temperature_df2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5947bf66",
   "metadata": {},
   "source": [
    "\n",
    "# Concat, Merge and Join \n",
    "<br/>\n",
    "\n",
    "***\n",
    "There are 3 key ways of combining DataFrames together:\n",
    "\n",
    "- **Concatenation**: Concatenation glues together DataFrames. Keep in mind that dimensions should match along the axis you are concatenating on <br/><br/>\n",
    "- **Merging**: The merge function allows you to merge DataFrames together using a similar logic as merging SQL Tables together<br/><br/>\n",
    "- **Join**: Join is a convenient method for combining the columns of two potentially differently-indexed DataFrames into a single result DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "id": "b12a84a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "df1 = pd.DataFrame({\n",
    "    'A': ['A0', 'A1', 'A2', 'A3'],\n",
    "    'B': ['B0', 'B1', 'B2', 'B3'],\n",
    "    'C': ['C0', 'C1', 'C2', 'C3'],\n",
    "    'D': ['D0', 'D1', 'D2', 'D3']\n",
    "}, index=[0, 1, 2, 3])\n",
    "\n",
    "df2 = pd.DataFrame({\n",
    "    'A': ['A4', 'A5', 'A6', 'A7'],\n",
    "    'B': ['B4', 'B5', 'B6', 'B7'],\n",
    "    'C': ['C4', 'C5', 'C6', 'C7'],\n",
    "    'D': ['D4', 'D5', 'D6', 'D7']\n",
    "}, index=[4, 5, 6, 7])\n",
    "\n",
    "df3 = pd.DataFrame({\n",
    "    'A': ['A8', 'A9', 'A10', 'A11'],\n",
    "    'B': ['B8', 'B9', 'B10', 'B11'],\n",
    "    'C': ['C8', 'C9', 'C10', 'C11'],\n",
    "    'E': ['D8', 'D9', 'D10', 'D11']\n",
    "}, index=[8, 9, 10, 11])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "id": "ac18001d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A0</td>\n",
       "      <td>B0</td>\n",
       "      <td>C0</td>\n",
       "      <td>D0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A1</td>\n",
       "      <td>B1</td>\n",
       "      <td>C1</td>\n",
       "      <td>D1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A2</td>\n",
       "      <td>B2</td>\n",
       "      <td>C2</td>\n",
       "      <td>D2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A3</td>\n",
       "      <td>B3</td>\n",
       "      <td>C3</td>\n",
       "      <td>D3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    A   B   C   D\n",
       "0  A0  B0  C0  D0\n",
       "1  A1  B1  C1  D1\n",
       "2  A2  B2  C2  D2\n",
       "3  A3  B3  C3  D3"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "id": "3cea4334",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A4</td>\n",
       "      <td>B4</td>\n",
       "      <td>C4</td>\n",
       "      <td>D4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>A5</td>\n",
       "      <td>B5</td>\n",
       "      <td>C5</td>\n",
       "      <td>D5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>A6</td>\n",
       "      <td>B6</td>\n",
       "      <td>C6</td>\n",
       "      <td>D6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>A7</td>\n",
       "      <td>B7</td>\n",
       "      <td>C7</td>\n",
       "      <td>D7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    A   B   C   D\n",
       "4  A4  B4  C4  D4\n",
       "5  A5  B5  C5  D5\n",
       "6  A6  B6  C6  D6\n",
       "7  A7  B7  C7  D7"
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "id": "083c5362",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>E</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>A8</td>\n",
       "      <td>B8</td>\n",
       "      <td>C8</td>\n",
       "      <td>D8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>A9</td>\n",
       "      <td>B9</td>\n",
       "      <td>C9</td>\n",
       "      <td>D9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>A10</td>\n",
       "      <td>B10</td>\n",
       "      <td>C10</td>\n",
       "      <td>D10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>A11</td>\n",
       "      <td>B11</td>\n",
       "      <td>C11</td>\n",
       "      <td>D11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      A    B    C    E\n",
       "8    A8   B8   C8   D8\n",
       "9    A9   B9   C9   D9\n",
       "10  A10  B10  C10  D10\n",
       "11  A11  B11  C11  D11"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "ee1b163f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "      <th>E</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A0</td>\n",
       "      <td>B0</td>\n",
       "      <td>C0</td>\n",
       "      <td>D0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A1</td>\n",
       "      <td>B1</td>\n",
       "      <td>C1</td>\n",
       "      <td>D1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A2</td>\n",
       "      <td>B2</td>\n",
       "      <td>C2</td>\n",
       "      <td>D2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A3</td>\n",
       "      <td>B3</td>\n",
       "      <td>C3</td>\n",
       "      <td>D3</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A4</td>\n",
       "      <td>B4</td>\n",
       "      <td>C4</td>\n",
       "      <td>D4</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>A5</td>\n",
       "      <td>B5</td>\n",
       "      <td>C5</td>\n",
       "      <td>D5</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>A6</td>\n",
       "      <td>B6</td>\n",
       "      <td>C6</td>\n",
       "      <td>D6</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>A7</td>\n",
       "      <td>B7</td>\n",
       "      <td>C7</td>\n",
       "      <td>D7</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>A8</td>\n",
       "      <td>B8</td>\n",
       "      <td>C8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>D8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>A9</td>\n",
       "      <td>B9</td>\n",
       "      <td>C9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>D9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>A10</td>\n",
       "      <td>B10</td>\n",
       "      <td>C10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>D10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>A11</td>\n",
       "      <td>B11</td>\n",
       "      <td>C11</td>\n",
       "      <td>NaN</td>\n",
       "      <td>D11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      A    B    C    D    E\n",
       "0    A0   B0   C0   D0  NaN\n",
       "1    A1   B1   C1   D1  NaN\n",
       "2    A2   B2   C2   D2  NaN\n",
       "3    A3   B3   C3   D3  NaN\n",
       "4    A4   B4   C4   D4  NaN\n",
       "5    A5   B5   C5   D5  NaN\n",
       "6    A6   B6   C6   D6  NaN\n",
       "7    A7   B7   C7   D7  NaN\n",
       "8    A8   B8   C8  NaN   D8\n",
       "9    A9   B9   C9  NaN   D9\n",
       "10  A10  B10  C10  NaN  D10\n",
       "11  A11  B11  C11  NaN  D11"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# if you don't specify an axis, it defaults to axis=0, which means it appends to rows\n",
    "pd.concat([df1, df2, df3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "e317e77c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>E</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A0</td>\n",
       "      <td>B0</td>\n",
       "      <td>C0</td>\n",
       "      <td>D0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A1</td>\n",
       "      <td>B1</td>\n",
       "      <td>C1</td>\n",
       "      <td>D1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A2</td>\n",
       "      <td>B2</td>\n",
       "      <td>C2</td>\n",
       "      <td>D2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A3</td>\n",
       "      <td>B3</td>\n",
       "      <td>C3</td>\n",
       "      <td>D3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A4</td>\n",
       "      <td>B4</td>\n",
       "      <td>C4</td>\n",
       "      <td>D4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A5</td>\n",
       "      <td>B5</td>\n",
       "      <td>C5</td>\n",
       "      <td>D5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A6</td>\n",
       "      <td>B6</td>\n",
       "      <td>C6</td>\n",
       "      <td>D6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A7</td>\n",
       "      <td>B7</td>\n",
       "      <td>C7</td>\n",
       "      <td>D7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A8</td>\n",
       "      <td>B8</td>\n",
       "      <td>C8</td>\n",
       "      <td>D8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A9</td>\n",
       "      <td>B9</td>\n",
       "      <td>C9</td>\n",
       "      <td>D9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A10</td>\n",
       "      <td>B10</td>\n",
       "      <td>C10</td>\n",
       "      <td>D10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>A11</td>\n",
       "      <td>B11</td>\n",
       "      <td>C11</td>\n",
       "      <td>D11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      A    B    C    D    A    B    C    D    A    B    C    E\n",
       "0    A0   B0   C0   D0  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN\n",
       "1    A1   B1   C1   D1  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN\n",
       "2    A2   B2   C2   D2  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN\n",
       "3    A3   B3   C3   D3  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN\n",
       "4   NaN  NaN  NaN  NaN   A4   B4   C4   D4  NaN  NaN  NaN  NaN\n",
       "5   NaN  NaN  NaN  NaN   A5   B5   C5   D5  NaN  NaN  NaN  NaN\n",
       "6   NaN  NaN  NaN  NaN   A6   B6   C6   D6  NaN  NaN  NaN  NaN\n",
       "7   NaN  NaN  NaN  NaN   A7   B7   C7   D7  NaN  NaN  NaN  NaN\n",
       "8   NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN   A8   B8   C8   D8\n",
       "9   NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN   A9   B9   C9   D9\n",
       "10  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  A10  B10  C10  D10\n",
       "11  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  A11  B11  C11  D11"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# axis=1 means concat along columns\n",
    "\n",
    "pd.concat([df1, df2, df3], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab66d7e9",
   "metadata": {},
   "source": [
    "## Join\n",
    "***\n",
    "Simply join two DFs having potentially different row indices\n",
    "\n",
    "You can do both inner as well as outer joins using the join function in pandas\n",
    "- Parameters {‘inner’, ‘outer’}, default ‘outer’. Outer for union and inner for intersection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "id": "eb1e89e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Join\n",
    "left_df = pd.DataFrame({\n",
    "    'A': ['A0', 'A1', 'A2'],\n",
    "    'B': ['B0', 'B1', 'B2']\n",
    "}, index=['K0', 'K1', 'K2']) \n",
    "\n",
    "right_df = pd.DataFrame({\n",
    "    'C': ['C0', 'C2', 'C3'],\n",
    "    'D': ['D0', 'D2', 'D3']\n",
    "}, index=['K0', 'K2', 'K3'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "id": "b806acee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>K0</th>\n",
       "      <td>A0</td>\n",
       "      <td>B0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>K1</th>\n",
       "      <td>A1</td>\n",
       "      <td>B1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>K2</th>\n",
       "      <td>A2</td>\n",
       "      <td>B2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A   B\n",
       "K0  A0  B0\n",
       "K1  A1  B1\n",
       "K2  A2  B2"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "left_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "id": "8a6aefe4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>K0</th>\n",
       "      <td>C0</td>\n",
       "      <td>D0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>K2</th>\n",
       "      <td>C2</td>\n",
       "      <td>D2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>K3</th>\n",
       "      <td>C3</td>\n",
       "      <td>D3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     C   D\n",
       "K0  C0  D0\n",
       "K2  C2  D2\n",
       "K3  C3  D3"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "right_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "id": "56904954",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>K0</th>\n",
       "      <td>A0</td>\n",
       "      <td>B0</td>\n",
       "      <td>C0</td>\n",
       "      <td>D0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>K1</th>\n",
       "      <td>A1</td>\n",
       "      <td>B1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>K2</th>\n",
       "      <td>A2</td>\n",
       "      <td>B2</td>\n",
       "      <td>C2</td>\n",
       "      <td>D2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>K3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>C3</td>\n",
       "      <td>D3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      A    B    C    D\n",
       "K0   A0   B0   C0   D0\n",
       "K1   A1   B1  NaN  NaN\n",
       "K2   A2   B2   C2   D2\n",
       "K3  NaN  NaN   C3   D3"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "left_df.join(right_df, how='outer')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "id": "f30467a5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>K0</th>\n",
       "      <td>A0</td>\n",
       "      <td>B0</td>\n",
       "      <td>C0</td>\n",
       "      <td>D0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>K2</th>\n",
       "      <td>A2</td>\n",
       "      <td>B2</td>\n",
       "      <td>C2</td>\n",
       "      <td>D2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A   B   C   D\n",
       "K0  A0  B0  C0  D0\n",
       "K2  A2  B2  C2  D2"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "left_df.join(right_df, how='inner')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a17ccad",
   "metadata": {},
   "source": [
    "## Merge\n",
    "\n",
    "Many a times you will be working with multiple dataframes all at once.\n",
    "\n",
    "The merge function allows them to be combined into a single data frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "id": "06a6449d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merging on multiple keys\n",
    "left = pd.DataFrame({\n",
    "    'key1': ['K0', 'K0', 'K1', 'K2'],\n",
    "    'key2': ['K0', 'K1', 'K0', 'K1'],\n",
    "    'A': ['A0', 'A1', 'A2', 'A3'],\n",
    "    'B': ['B0', 'B1', 'B2', 'B3']\n",
    "})\n",
    "    \n",
    "right = pd.DataFrame({\n",
    "    'key1': ['K0', 'K1', 'K1', 'K2'],\n",
    "    'key2': ['K0', 'K0', 'K0', 'K0'],\n",
    "    'C': ['C0', 'C1', 'C2', 'C3'],\n",
    "    'D': ['D0', 'D1', 'D2', 'D3']\n",
    "})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "id": "d309cc2c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key1</th>\n",
       "      <th>key2</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>K0</td>\n",
       "      <td>K0</td>\n",
       "      <td>A0</td>\n",
       "      <td>B0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>K0</td>\n",
       "      <td>K1</td>\n",
       "      <td>A1</td>\n",
       "      <td>B1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>K1</td>\n",
       "      <td>K0</td>\n",
       "      <td>A2</td>\n",
       "      <td>B2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>K2</td>\n",
       "      <td>K1</td>\n",
       "      <td>A3</td>\n",
       "      <td>B3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  key1 key2   A   B\n",
       "0   K0   K0  A0  B0\n",
       "1   K0   K1  A1  B1\n",
       "2   K1   K0  A2  B2\n",
       "3   K2   K1  A3  B3"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "left"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "id": "bc53d840",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key1</th>\n",
       "      <th>key2</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>K0</td>\n",
       "      <td>K0</td>\n",
       "      <td>C0</td>\n",
       "      <td>D0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>K1</td>\n",
       "      <td>K0</td>\n",
       "      <td>C1</td>\n",
       "      <td>D1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>K1</td>\n",
       "      <td>K0</td>\n",
       "      <td>C2</td>\n",
       "      <td>D2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>K2</td>\n",
       "      <td>K0</td>\n",
       "      <td>C3</td>\n",
       "      <td>D3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  key1 key2   C   D\n",
       "0   K0   K0  C0  D0\n",
       "1   K1   K0  C1  D1\n",
       "2   K1   K0  C2  D2\n",
       "3   K2   K0  C3  D3"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "right"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "28b4955e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key1</th>\n",
       "      <th>key2</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>K0</td>\n",
       "      <td>K0</td>\n",
       "      <td>A0</td>\n",
       "      <td>B0</td>\n",
       "      <td>C0</td>\n",
       "      <td>D0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>K0</td>\n",
       "      <td>K1</td>\n",
       "      <td>A1</td>\n",
       "      <td>B1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>K1</td>\n",
       "      <td>K0</td>\n",
       "      <td>A2</td>\n",
       "      <td>B2</td>\n",
       "      <td>C1</td>\n",
       "      <td>D1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>K1</td>\n",
       "      <td>K0</td>\n",
       "      <td>A2</td>\n",
       "      <td>B2</td>\n",
       "      <td>C2</td>\n",
       "      <td>D2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>K2</td>\n",
       "      <td>K1</td>\n",
       "      <td>A3</td>\n",
       "      <td>B3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  key1 key2   A   B    C    D\n",
       "0   K0   K0  A0  B0   C0   D0\n",
       "1   K0   K1  A1  B1  NaN  NaN\n",
       "2   K1   K0  A2  B2   C1   D1\n",
       "3   K1   K0  A2  B2   C2   D2\n",
       "4   K2   K1  A3  B3  NaN  NaN"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# other options are 'inner', 'left', 'right'\n",
    "pd.merge(left, right, how='left', on=['key1', 'key2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff097c0f",
   "metadata": {},
   "source": [
    "# Further Reading\n",
    "***\n",
    "- Pandas documentation: http://pandas.pydata.org/\n",
    "- 10 minutes to pandas: https://pandas.pydata.org/pandas-docs/stable/10min.html\n",
    "- Cookbook- Useful Pandas Recipes: https://pandas.pydata.org/pandas-docs/stable/cookbook.html\n",
    "- Pandas and Python Top 10: http://manishamde.github.io/blog/2013/03/07/pandas-and-python-top-10/\n",
    "- Intro to Pandas Data Structures: http://www.gregreda.com/2013/10/26/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac0bae8a",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}