03_pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Wrangling\n",
    "\n",
    "- `pandas`\n",
    "- Where to find data?\n",
    "   - Web Scraping & APIs\n",
    "   \n",
    "   \n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/COGS108/Lectures-Fa24/blob/main/03_pandas.ipynb)\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "<img src=\"https://raw.githubusercontent.com/COGS108/Lectures-Wi22/main/02_python/img/pandas.png\" alt=\"pandas\" width=\"600px\">\n",
    "</center>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas is Python library for managing heterogenous data.\n",
    "\n",
    "At its core, Pandas is used for the **DataFrame** object, which is:\n",
    "- a data structure for labeled rows and columns of data\n",
    "- associated methods and utilities for working with data.\n",
    "- each column contains a `pandas` **Series**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# After this\n",
    "\n",
    "When you are done with this exercise/lecture... the real learning can begin!\n",
    "\n",
    "One of your best tools is https://pandastutor.com\n",
    "\n",
    "Another is the documentation for pandas including https://pandas.pydata.org/docs/user_guide/10min.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import standard libraries\n",
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reminder about tab completion and contextual help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total elements: 1200 \n",
      "shape of table (200, 6)\n"
     ]
    }
   ],
   "source": [
    "# Load a csv file of data\n",
    "df = pd.read_csv('data/my_data.csv')\n",
    "print('total elements:',df.size,'\\nshape of table',df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>first_name</th>\n",
       "      <th>last_name</th>\n",
       "      <th>age</th>\n",
       "      <th>score</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>295</td>\n",
       "      <td>Andrea</td>\n",
       "      <td>Clark</td>\n",
       "      <td>46</td>\n",
       "      <td>-1</td>\n",
       "      <td>24547.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>620</td>\n",
       "      <td>Bill</td>\n",
       "      <td>Woods</td>\n",
       "      <td>46</td>\n",
       "      <td>492</td>\n",
       "      <td>46713.90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>891</td>\n",
       "      <td>Alexander</td>\n",
       "      <td>Jacobson</td>\n",
       "      <td>48</td>\n",
       "      <td>489</td>\n",
       "      <td>32071.74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>914</td>\n",
       "      <td>Derrick</td>\n",
       "      <td>Bradley</td>\n",
       "      <td>52</td>\n",
       "      <td>-1</td>\n",
       "      <td>30650.48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1736</td>\n",
       "      <td>Allison</td>\n",
       "      <td>Thomas</td>\n",
       "      <td>44</td>\n",
       "      <td>-1</td>\n",
       "      <td>9553.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>97441</td>\n",
       "      <td>Krista</td>\n",
       "      <td>Ortiz</td>\n",
       "      <td>34</td>\n",
       "      <td>-1</td>\n",
       "      <td>24074.79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>97728</td>\n",
       "      <td>Anna</td>\n",
       "      <td>Chambers</td>\n",
       "      <td>37</td>\n",
       "      <td>598</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>98115</td>\n",
       "      <td>Jennifer</td>\n",
       "      <td>Pitts</td>\n",
       "      <td>29</td>\n",
       "      <td>606</td>\n",
       "      <td>6876.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>98284</td>\n",
       "      <td>Brittany</td>\n",
       "      <td>Jenkins</td>\n",
       "      <td>34</td>\n",
       "      <td>665</td>\n",
       "      <td>43525.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>98366</td>\n",
       "      <td>Katelyn</td>\n",
       "      <td>Brown</td>\n",
       "      <td>45</td>\n",
       "      <td>501</td>\n",
       "      <td>29668.38</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>200 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id first_name last_name  age  score     value\n",
       "0      295     Andrea     Clark   46     -1  24547.87\n",
       "1      620       Bill     Woods   46    492  46713.90\n",
       "2      891  Alexander  Jacobson   48    489  32071.74\n",
       "3      914    Derrick   Bradley   52     -1  30650.48\n",
       "4     1736    Allison    Thomas   44     -1   9553.12\n",
       "..     ...        ...       ...  ...    ...       ...\n",
       "195  97441     Krista     Ortiz   34     -1  24074.79\n",
       "196  97728       Anna  Chambers   37    598      0.00\n",
       "197  98115   Jennifer     Pitts   29    606   6876.75\n",
       "198  98284   Brittany   Jenkins   34    665  43525.88\n",
       "199  98366    Katelyn     Brown   45    501  29668.38\n",
       "\n",
       "[200 rows x 6 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check out a few rows or last few rows of the dataframe using head() or tail()\n",
    "df\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas DataFrame:\n",
    "- Index for each row\n",
    "- Column name for each column (Series)\n",
    "- Stores heterogenous types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Slicing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      46\n",
       "1      46\n",
       "2      48\n",
       "3      52\n",
       "4      44\n",
       "       ..\n",
       "195    34\n",
       "196    37\n",
       "197    29\n",
       "198    34\n",
       "199    45\n",
       "Name: age, Length: 200, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Slicing (Indexing): select a Series (column) using its name\n",
    "\n",
    "df['age']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>last_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>52</td>\n",
       "      <td>Bradley</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>44</td>\n",
       "      <td>Thomas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>57</td>\n",
       "      <td>Williams</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>46</td>\n",
       "      <td>Wood</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>50</td>\n",
       "      <td>Garcia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>68</td>\n",
       "      <td>Coleman</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age last_name\n",
       "3   52   Bradley\n",
       "4   44    Thomas\n",
       "5   57  Williams\n",
       "6   46      Wood\n",
       "7   50    Garcia\n",
       "8   68   Coleman"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[3:8, ['age', 'last_name'] ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>first_name</th>\n",
       "      <th>last_name</th>\n",
       "      <th>age</th>\n",
       "      <th>score</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1736</td>\n",
       "      <td>Allison</td>\n",
       "      <td>Thomas</td>\n",
       "      <td>44</td>\n",
       "      <td>-1</td>\n",
       "      <td>9553.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2049</td>\n",
       "      <td>Stephen</td>\n",
       "      <td>Williams</td>\n",
       "      <td>57</td>\n",
       "      <td>333</td>\n",
       "      <td>138936.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2241</td>\n",
       "      <td>Malik</td>\n",
       "      <td>Wood</td>\n",
       "      <td>46</td>\n",
       "      <td>-1</td>\n",
       "      <td>10804.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2607</td>\n",
       "      <td>Amber</td>\n",
       "      <td>Garcia</td>\n",
       "      <td>50</td>\n",
       "      <td>536</td>\n",
       "      <td>9367.27</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     id first_name last_name  age  score      value\n",
       "4  1736    Allison    Thomas   44     -1    9553.12\n",
       "5  2049    Stephen  Williams   57    333  138936.92\n",
       "6  2241      Malik      Wood   46     -1   10804.47\n",
       "7  2607      Amber    Garcia   50    536    9367.27"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[4:8]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Slicing: select a row & column with 'loc'\n",
    "df.loc[4, 'age']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "name_df = df.set_index('last_name')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "name_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>first_name</th>\n",
       "      <th>age</th>\n",
       "      <th>score</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>last_name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Clark</th>\n",
       "      <td>295</td>\n",
       "      <td>Andrea</td>\n",
       "      <td>46</td>\n",
       "      <td>-1</td>\n",
       "      <td>24547.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Woods</th>\n",
       "      <td>620</td>\n",
       "      <td>Bill</td>\n",
       "      <td>46</td>\n",
       "      <td>492</td>\n",
       "      <td>46713.90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jacobson</th>\n",
       "      <td>891</td>\n",
       "      <td>Alexander</td>\n",
       "      <td>48</td>\n",
       "      <td>489</td>\n",
       "      <td>32071.74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bradley</th>\n",
       "      <td>914</td>\n",
       "      <td>Derrick</td>\n",
       "      <td>52</td>\n",
       "      <td>-1</td>\n",
       "      <td>30650.48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Thomas</th>\n",
       "      <td>1736</td>\n",
       "      <td>Allison</td>\n",
       "      <td>44</td>\n",
       "      <td>-1</td>\n",
       "      <td>9553.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ortiz</th>\n",
       "      <td>97441</td>\n",
       "      <td>Krista</td>\n",
       "      <td>34</td>\n",
       "      <td>-1</td>\n",
       "      <td>24074.79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Chambers</th>\n",
       "      <td>97728</td>\n",
       "      <td>Anna</td>\n",
       "      <td>37</td>\n",
       "      <td>598</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pitts</th>\n",
       "      <td>98115</td>\n",
       "      <td>Jennifer</td>\n",
       "      <td>29</td>\n",
       "      <td>606</td>\n",
       "      <td>6876.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jenkins</th>\n",
       "      <td>98284</td>\n",
       "      <td>Brittany</td>\n",
       "      <td>34</td>\n",
       "      <td>665</td>\n",
       "      <td>43525.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Brown</th>\n",
       "      <td>98366</td>\n",
       "      <td>Katelyn</td>\n",
       "      <td>45</td>\n",
       "      <td>501</td>\n",
       "      <td>29668.38</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>200 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              id first_name  age  score     value\n",
       "last_name                                        \n",
       "Clark        295     Andrea   46     -1  24547.87\n",
       "Woods        620       Bill   46    492  46713.90\n",
       "Jacobson     891  Alexander   48    489  32071.74\n",
       "Bradley      914    Derrick   52     -1  30650.48\n",
       "Thomas      1736    Allison   44     -1   9553.12\n",
       "...          ...        ...  ...    ...       ...\n",
       "Ortiz      97441     Krista   34     -1  24074.79\n",
       "Chambers   97728       Anna   37    598      0.00\n",
       "Pitts      98115   Jennifer   29    606   6876.75\n",
       "Jenkins    98284   Brittany   34    665  43525.88\n",
       "Brown      98366    Katelyn   45    501  29668.38\n",
       "\n",
       "[200 rows x 5 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "name_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m\n",
       "\u001b[0mdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msort_values\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mby\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'IndexLabel'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m*\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0maxis\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Axis'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mascending\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool | list[bool] | tuple[bool, ...]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0minplace\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mkind\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'SortKind'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'quicksort'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mna_position\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'last'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mignore_index\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mkey\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'ValueKeyFunc | None'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m'DataFrame | None'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Sort by the values along either axis.\n",
       "\n",
       "Parameters\n",
       "----------\n",
       "by : str or list of str\n",
       "    Name or list of names to sort by.\n",
       "\n",
       "    - if `axis` is 0 or `'index'` then `by` may contain index\n",
       "      levels and/or column labels.\n",
       "    - if `axis` is 1 or `'columns'` then `by` may contain column\n",
       "      levels and/or index labels.\n",
       "axis : \"{0 or 'index', 1 or 'columns'}\", default 0\n",
       "     Axis to be sorted.\n",
       "ascending : bool or list of bool, default True\n",
       "     Sort ascending vs. descending. Specify list for multiple sort\n",
       "     orders.  If this is a list of bools, must match the length of\n",
       "     the by.\n",
       "inplace : bool, default False\n",
       "     If True, perform operation in-place.\n",
       "kind : {'quicksort', 'mergesort', 'heapsort', 'stable'}, default 'quicksort'\n",
       "     Choice of sorting algorithm. See also :func:`numpy.sort` for more\n",
       "     information. `mergesort` and `stable` are the only stable algorithms. For\n",
       "     DataFrames, this option is only applied when sorting on a single\n",
       "     column or label.\n",
       "na_position : {'first', 'last'}, default 'last'\n",
       "     Puts NaNs at the beginning if `first`; `last` puts NaNs at the\n",
       "     end.\n",
       "ignore_index : bool, default False\n",
       "     If True, the resulting axis will be labeled 0, 1, …, n - 1.\n",
       "key : callable, optional\n",
       "    Apply the key function to the values\n",
       "    before sorting. This is similar to the `key` argument in the\n",
       "    builtin :meth:`sorted` function, with the notable difference that\n",
       "    this `key` function should be *vectorized*. It should expect a\n",
       "    ``Series`` and return a Series with the same shape as the input.\n",
       "    It will be applied to each column in `by` independently.\n",
       "\n",
       "Returns\n",
       "-------\n",
       "DataFrame or None\n",
       "    DataFrame with sorted values or None if ``inplace=True``.\n",
       "\n",
       "See Also\n",
       "--------\n",
       "DataFrame.sort_index : Sort a DataFrame by the index.\n",
       "Series.sort_values : Similar method for a Series.\n",
       "\n",
       "Examples\n",
       "--------\n",
       ">>> df = pd.DataFrame({\n",
       "...     'col1': ['A', 'A', 'B', np.nan, 'D', 'C'],\n",
       "...     'col2': [2, 1, 9, 8, 7, 4],\n",
       "...     'col3': [0, 1, 9, 4, 2, 3],\n",
       "...     'col4': ['a', 'B', 'c', 'D', 'e', 'F']\n",
       "... })\n",
       ">>> df\n",
       "  col1  col2  col3 col4\n",
       "0    A     2     0    a\n",
       "1    A     1     1    B\n",
       "2    B     9     9    c\n",
       "3  NaN     8     4    D\n",
       "4    D     7     2    e\n",
       "5    C     4     3    F\n",
       "\n",
       "Sort by col1\n",
       "\n",
       ">>> df.sort_values(by=['col1'])\n",
       "  col1  col2  col3 col4\n",
       "0    A     2     0    a\n",
       "1    A     1     1    B\n",
       "2    B     9     9    c\n",
       "5    C     4     3    F\n",
       "4    D     7     2    e\n",
       "3  NaN     8     4    D\n",
       "\n",
       "Sort by multiple columns\n",
       "\n",
       ">>> df.sort_values(by=['col1', 'col2'])\n",
       "  col1  col2  col3 col4\n",
       "1    A     1     1    B\n",
       "0    A     2     0    a\n",
       "2    B     9     9    c\n",
       "5    C     4     3    F\n",
       "4    D     7     2    e\n",
       "3  NaN     8     4    D\n",
       "\n",
       "Sort Descending\n",
       "\n",
       ">>> df.sort_values(by='col1', ascending=False)\n",
       "  col1  col2  col3 col4\n",
       "4    D     7     2    e\n",
       "5    C     4     3    F\n",
       "2    B     9     9    c\n",
       "0    A     2     0    a\n",
       "1    A     1     1    B\n",
       "3  NaN     8     4    D\n",
       "\n",
       "Putting NAs first\n",
       "\n",
       ">>> df.sort_values(by='col1', ascending=False, na_position='first')\n",
       "  col1  col2  col3 col4\n",
       "3  NaN     8     4    D\n",
       "4    D     7     2    e\n",
       "5    C     4     3    F\n",
       "2    B     9     9    c\n",
       "0    A     2     0    a\n",
       "1    A     1     1    B\n",
       "\n",
       "Sorting with a key function\n",
       "\n",
       ">>> df.sort_values(by='col4', key=lambda col: col.str.lower())\n",
       "   col1  col2  col3 col4\n",
       "0    A     2     0    a\n",
       "1    A     1     1    B\n",
       "2    B     9     9    c\n",
       "3  NaN     8     4    D\n",
       "4    D     7     2    e\n",
       "5    C     4     3    F\n",
       "\n",
       "Natural sort with the key argument,\n",
       "using the `natsort <https://github.com/SethMMorton/natsort>` package.\n",
       "\n",
       ">>> df = pd.DataFrame({\n",
       "...    \"time\": ['0hr', '128hr', '72hr', '48hr', '96hr'],\n",
       "...    \"value\": [10, 20, 30, 40, 50]\n",
       "... })\n",
       ">>> df\n",
       "    time  value\n",
       "0    0hr     10\n",
       "1  128hr     20\n",
       "2   72hr     30\n",
       "3   48hr     40\n",
       "4   96hr     50\n",
       ">>> from natsort import index_natsorted\n",
       ">>> df.sort_values(\n",
       "...     by=\"time\",\n",
       "...     key=lambda x: np.argsort(index_natsorted(df[\"time\"]))\n",
       "... )\n",
       "    time  value\n",
       "0    0hr     10\n",
       "3   48hr     40\n",
       "2   72hr     30\n",
       "4   96hr     50\n",
       "1  128hr     20\n",
       "\u001b[0;31mFile:\u001b[0m      /opt/conda/lib/python3.11/site-packages/pandas/core/frame.py\n",
       "\u001b[0;31mType:\u001b[0m      method"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.sort_values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      False\n",
       "1      False\n",
       "2      False\n",
       "3       True\n",
       "4      False\n",
       "       ...  \n",
       "195    False\n",
       "196    False\n",
       "197    False\n",
       "198    False\n",
       "199    False\n",
       "Name: age, Length: 200, dtype: bool"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['age'] > 50\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question #1\n",
    "\n",
    "What would be the output of `df['age'] > 10`?\n",
    "\n",
    "- A) subset of `df` including only rows of individuals older than 10\n",
    "- B) a Boolean with `True` for rows where age is greater than 10 and `False` otherwise\n",
    "- C) `id`s of rows where observations are greater than 10 \n",
    "- D) an error\n",
    "- E) I'm super lost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first_name</th>\n",
       "      <th>last_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>Kenneth</td>\n",
       "      <td>Johnson</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>158</th>\n",
       "      <td>Brenda</td>\n",
       "      <td>Cox</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    first_name last_name\n",
       "39     Kenneth   Johnson\n",
       "158     Brenda       Cox"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to get subset\n",
    "df.loc[ (df['age'] > 60) & (df['score']==-1) , ['first_name','last_name'] ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>first_name</th>\n",
       "      <th>last_name</th>\n",
       "      <th>age</th>\n",
       "      <th>score</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>295</td>\n",
       "      <td>Andrea</td>\n",
       "      <td>Clark</td>\n",
       "      <td>46</td>\n",
       "      <td>-1</td>\n",
       "      <td>24547.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>15137</td>\n",
       "      <td>Andrea</td>\n",
       "      <td>Simpson</td>\n",
       "      <td>55</td>\n",
       "      <td>-1</td>\n",
       "      <td>12155.47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       id first_name last_name  age  score     value\n",
       "0     295     Andrea     Clark   46     -1  24547.87\n",
       "27  15137     Andrea   Simpson   55     -1  12155.47"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.query('first_name == \"Andrea\" ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOTE: .loc uses the index, whatever it is\n",
    "# sometimes the index is NOT numeric!\n",
    "df.set_index('age').loc[52]\n",
    "\n",
    "#set_index('first_name').loc['Andrea']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'Andr'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "File \u001b[0;32m/opt/conda/lib/python3.11/site-packages/pandas/core/indexes/base.py:3805\u001b[0m, in \u001b[0;36mIndex.get_loc\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m   3804\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m-> 3805\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_engine\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_loc\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcasted_key\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   3806\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mKeyError\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m err:\n",
      "File \u001b[0;32mindex.pyx:167\u001b[0m, in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32mindex.pyx:191\u001b[0m, in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32mindex.pyx:234\u001b[0m, in \u001b[0;36mpandas._libs.index.IndexEngine._get_loc_duplicates\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32mindex.pyx:242\u001b[0m, in \u001b[0;36mpandas._libs.index.IndexEngine._maybe_get_bool_indexer\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32mindex.pyx:134\u001b[0m, in \u001b[0;36mpandas._libs.index._unpack_bool_indexer\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 'Andr'",
      "\nThe above exception was the direct cause of the following exception:\n",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[34], line 2\u001b[0m\n\u001b[1;32m      1\u001b[0m (\n\u001b[0;32m----> 2\u001b[0m     \u001b[43mdf\u001b[49m\n\u001b[1;32m      3\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mset_index\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mfirst_name\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m      4\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mloc\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mAndr\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m]\u001b[49m\n\u001b[1;32m      5\u001b[0m )\n",
      "File \u001b[0;32m/opt/conda/lib/python3.11/site-packages/pandas/core/indexing.py:1191\u001b[0m, in \u001b[0;36m_LocationIndexer.__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m   1189\u001b[0m maybe_callable \u001b[38;5;241m=\u001b[39m com\u001b[38;5;241m.\u001b[39mapply_if_callable(key, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mobj)\n\u001b[1;32m   1190\u001b[0m maybe_callable \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_check_deprecated_callable_usage(key, maybe_callable)\n\u001b[0;32m-> 1191\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_getitem_axis\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmaybe_callable\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43maxis\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43maxis\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/lib/python3.11/site-packages/pandas/core/indexing.py:1431\u001b[0m, in \u001b[0;36m_LocIndexer._getitem_axis\u001b[0;34m(self, key, axis)\u001b[0m\n\u001b[1;32m   1429\u001b[0m \u001b[38;5;66;03m# fall thru to straight lookup\u001b[39;00m\n\u001b[1;32m   1430\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_validate_key(key, axis)\n\u001b[0;32m-> 1431\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_get_label\u001b[49m\u001b[43m(\u001b[49m\u001b[43mkey\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43maxis\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43maxis\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/lib/python3.11/site-packages/pandas/core/indexing.py:1381\u001b[0m, in \u001b[0;36m_LocIndexer._get_label\u001b[0;34m(self, label, axis)\u001b[0m\n\u001b[1;32m   1379\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m_get_label\u001b[39m(\u001b[38;5;28mself\u001b[39m, label, axis: AxisInt):\n\u001b[1;32m   1380\u001b[0m     \u001b[38;5;66;03m# GH#5567 this will fail if the label is not present in the axis.\u001b[39;00m\n\u001b[0;32m-> 1381\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mobj\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mxs\u001b[49m\u001b[43m(\u001b[49m\u001b[43mlabel\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43maxis\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43maxis\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/lib/python3.11/site-packages/pandas/core/generic.py:4301\u001b[0m, in \u001b[0;36mNDFrame.xs\u001b[0;34m(self, key, axis, level, drop_level)\u001b[0m\n\u001b[1;32m   4299\u001b[0m             new_index \u001b[38;5;241m=\u001b[39m index[loc]\n\u001b[1;32m   4300\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m-> 4301\u001b[0m     loc \u001b[38;5;241m=\u001b[39m \u001b[43mindex\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_loc\u001b[49m\u001b[43m(\u001b[49m\u001b[43mkey\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   4303\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(loc, np\u001b[38;5;241m.\u001b[39mndarray):\n\u001b[1;32m   4304\u001b[0m         \u001b[38;5;28;01mif\u001b[39;00m loc\u001b[38;5;241m.\u001b[39mdtype \u001b[38;5;241m==\u001b[39m np\u001b[38;5;241m.\u001b[39mbool_:\n",
      "File \u001b[0;32m/opt/conda/lib/python3.11/site-packages/pandas/core/indexes/base.py:3812\u001b[0m, in \u001b[0;36mIndex.get_loc\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m   3807\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(casted_key, \u001b[38;5;28mslice\u001b[39m) \u001b[38;5;129;01mor\u001b[39;00m (\n\u001b[1;32m   3808\u001b[0m         \u001b[38;5;28misinstance\u001b[39m(casted_key, abc\u001b[38;5;241m.\u001b[39mIterable)\n\u001b[1;32m   3809\u001b[0m         \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;28many\u001b[39m(\u001b[38;5;28misinstance\u001b[39m(x, \u001b[38;5;28mslice\u001b[39m) \u001b[38;5;28;01mfor\u001b[39;00m x \u001b[38;5;129;01min\u001b[39;00m casted_key)\n\u001b[1;32m   3810\u001b[0m     ):\n\u001b[1;32m   3811\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m InvalidIndexError(key)\n\u001b[0;32m-> 3812\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mKeyError\u001b[39;00m(key) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01merr\u001b[39;00m\n\u001b[1;32m   3813\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mTypeError\u001b[39;00m:\n\u001b[1;32m   3814\u001b[0m     \u001b[38;5;66;03m# If we have a listlike key, _check_indexing_error will raise\u001b[39;00m\n\u001b[1;32m   3815\u001b[0m     \u001b[38;5;66;03m#  InvalidIndexError. Otherwise we fall through and re-raise\u001b[39;00m\n\u001b[1;32m   3816\u001b[0m     \u001b[38;5;66;03m#  the TypeError.\u001b[39;00m\n\u001b[1;32m   3817\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_check_indexing_error(key)\n",
      "\u001b[0;31mKeyError\u001b[0m: 'Andr'"
     ]
    }
   ],
   "source": [
    "(\n",
    "    df\n",
    "    .set_index('first_name')\n",
    "    .loc['Andr']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checking out the DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check how large our dataframe is\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['id', 'first_name', 'last_name', 'age', 'score', 'value'], dtype='object')"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check what columns we have in our DataFrame\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.set_index('first_name').index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id              int64\n",
       "first_name     object\n",
       "last_name      object\n",
       "age             int64\n",
       "score           int64\n",
       "value         float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the datatypes of our variables\n",
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the data\n",
    "\n",
    "- quantitative (numbers)\n",
    "- qualitative (categorical)\n",
    "- basic descriptive statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "first_name\n",
       "David        6\n",
       "Michael      5\n",
       "Eric         4\n",
       "Charles      4\n",
       "James        4\n",
       "Elizabeth    3\n",
       "Ashley       3\n",
       "Jonathan     3\n",
       "Sarah        3\n",
       "Brian        3\n",
       "Jennifer     3\n",
       "Jason        3\n",
       "John         3\n",
       "Aaron        2\n",
       "Sabrina      2\n",
       "Robert       2\n",
       "Lisa         2\n",
       "Mary         2\n",
       "Cynthia      2\n",
       "Amanda       2\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking categorical data\n",
    "df['first_name'].value_counts().iloc[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id       54643.50\n",
       "age         46.00\n",
       "value    17976.51\n",
       "dtype: float64"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check a particular descriptive statistic, like mean(), median(), mode(), std(), var(), etc\n",
    "df[['id','age', 'value']].median()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count       189.000000\n",
       "mean      28730.336296\n",
       "std       32493.945741\n",
       "min           0.000000\n",
       "25%        9593.030000\n",
       "50%       17976.510000\n",
       "75%       33163.310000\n",
       "max      204999.960000\n",
       "Name: value, dtype: float64"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Describe a particular column\n",
    "df['value'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>age</th>\n",
       "      <th>score</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>200.000000</td>\n",
       "      <td>200.000000</td>\n",
       "      <td>200.000000</td>\n",
       "      <td>189.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>52929.150000</td>\n",
       "      <td>46.020000</td>\n",
       "      <td>416.595000</td>\n",
       "      <td>28730.336296</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>29414.298899</td>\n",
       "      <td>10.028582</td>\n",
       "      <td>237.176674</td>\n",
       "      <td>32493.945741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>295.000000</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>26709.500000</td>\n",
       "      <td>39.000000</td>\n",
       "      <td>288.750000</td>\n",
       "      <td>9593.030000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>54643.500000</td>\n",
       "      <td>46.000000</td>\n",
       "      <td>463.500000</td>\n",
       "      <td>17976.510000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>80840.750000</td>\n",
       "      <td>53.000000</td>\n",
       "      <td>596.500000</td>\n",
       "      <td>33163.310000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>98366.000000</td>\n",
       "      <td>69.000000</td>\n",
       "      <td>942.000000</td>\n",
       "      <td>204999.960000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 id         age       score          value\n",
       "count    200.000000  200.000000  200.000000     189.000000\n",
       "mean   52929.150000   46.020000  416.595000   28730.336296\n",
       "std    29414.298899   10.028582  237.176674   32493.945741\n",
       "min      295.000000   14.000000   -1.000000       0.000000\n",
       "25%    26709.500000   39.000000  288.750000    9593.030000\n",
       "50%    54643.500000   46.000000  463.500000   17976.510000\n",
       "75%    80840.750000   53.000000  596.500000   33163.310000\n",
       "max    98366.000000   69.000000  942.000000  204999.960000"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get descriptive statistics of all numerical columns\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question #2\n",
    "\n",
    "What's the average (mean) age of the individuals in this dataset?\n",
    "\n",
    "- A) 14\n",
    "- B) 46\n",
    "- C) 28730\n",
    "- D) NA\n",
    "- E) I'm super lost/unsure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# there are two commands that could give you the correct answer...\n",
    "df['age'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `pandas`: Common Manipulations\n",
    "\n",
    "You'll want to be *very* familiar with a few common data manipulations when wrangling data, each of which is described below:\n",
    "\n",
    "Manipulation | Description\n",
    "-------|------------\n",
    "**select** | select which columns to include in dataset\n",
    "**filter** | filter dataset to only include specified rows\n",
    "**mutate** | add a new column based on values in other columns\n",
    "**groupby** | group values to apply a function within the specified groups\n",
    "**summarize** | calculate specified summary metric of a specified variable\n",
    "**arrange** | sort rows ascending or descending order of a specified column\n",
    "**merge** | join separate datasets into a single dataset based on a common column\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import this"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting & Dropping Columns\n",
    "\n",
    "- include subset of columns of larger data frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify which columns to include\n",
    "select_df = df[['id', 'age', 'score', 'value']]\n",
    "select_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop rows we don't want\n",
    "new_df = df.drop(labels=[0,2,6], axis='rows')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check out the DataFrame after dropping, what went wrong and how do we fix it?\n",
    "new_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filtering Data (slicing)\n",
    "\n",
    "- include a subset (slice) of rows from larger data frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check if we have any data from people below the age of 18\n",
    "sum(df['age'] < 18)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[(df['age'] < 18)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select only participants who are 18 or older  AND who have a score of -1\n",
    "df_new = df[ (df['age'] >= 18)] # & (df['score']==-1)]\n",
    "df_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Missing Data (NaNs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check for missing values\n",
    "df['value'].hasn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.isnull?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# note in class\n",
    "# can operate on entire dataframe\n",
    "df.isnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check for null values by row\n",
    "df.isnull().sum(axis='rows')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Have a look at the missing values\n",
    "df[df['value'].isnull()]\n",
    "# can also use .isna() or isnan()!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with Missing Data - NaNs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dealing with null values: Drop rows with missing data\n",
    "print('before dropping null rows',df.shape)\n",
    "no_na_df = df.dropna()\n",
    "print('new shape', no_na_df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# or you can fill in a value for those missing values!\n",
    "df.fillna?\n",
    "# df.fillna(100) or df.fillna( df['value'].mean() ) or just df.fillna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding Missing Data - Bad Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check for the properties of specific columns\n",
    "df['score'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df.score<0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the plot of the data for score to see the distribution\n",
    "df['score'].plot(kind='hist', bins=25);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.hist();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with Missing Data - Bad Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look for how many values have a -1 value in 'score'\n",
    "sum(df['score'] == -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop any row with -1 value in 'score'\n",
    "df = df[df['score'] != -1]\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating new columns (mutating)\n",
    "\n",
    "- `assign` can be very helpful in adding a new column\n",
    "- lambda functions can be used to carry out calculations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert age in years to age in (approximate) days\n",
    "df = df.assign(age_days = df['age'] * 365)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['age_months'] = df['age'] * 12\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grouping & summarizing\n",
    "\n",
    "- group by a particular variable\n",
    "- calculate summary statistics/metrics within group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.first_name.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# caclculate average within each age\n",
    "df.groupby('first_name').age.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sorting Rows (arrange)\n",
    "\n",
    "- specify order in which to display rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sort by values in age\n",
    "df = df.sort_values(by = ['age'],ascending=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Combining datasets\n",
    "![image of join operations](https://raw.githubusercontent.com/COGS108/Lectures-Fa24/main/img/join.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Create two DataFrames\n",
    "left = pd.DataFrame({'key': ['A', 'B', 'C', 'D'], 'value': np.random.randn(4)})    \n",
    "right = pd.DataFrame({'key': ['B', 'D', 'E', 'F'], 'value': np.random.randn(4)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "left"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "right"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "left.merge?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "left.merge(right, on='key', )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inner merge by default\n",
    "pd.merge(left, right, on='key')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# same as above\n",
    "pd.merge(left, right, on='key', how='inner')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# right merge\n",
    "pd.merge(left, right, on='key', how='right')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# left merge\n",
    "pd.merge(left, right, on='key', how='left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# outer join\n",
    "pd.merge(left, right, on='key', how='outer')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.merge(left, right, on='key', how='outer').mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do this as a check in! \n",
    "## https://forms.gle/EpNS7BYhfAxvBM4X7\n",
    "\n",
    "#### Question #3\n",
    "\n",
    "If table A had 5 rows and table B had 5 rows and 3 of those rows in each table were from the same observations present in the *other* table, how many rows would be present if an **inner merge** were carried out?\n",
    "\n",
    "- A) 3\n",
    "- B) 5\n",
    "- C) 10\n",
    "- D) 13\n",
    "- E) Totally unsure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question #4\n",
    "\n",
    "If table A had 5 rows and table B had 5 rows and 3 of those rows in each table were from the same observations present in the *other* table, how many rows would be present if a **left merge** were carried out?\n",
    "\n",
    "- A) 3\n",
    "- B) 5\n",
    "- C) 10\n",
    "- D) 13\n",
    "- E) Totally unsure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing Data\n",
    "\n",
    "- We'll have a whole lecture (or two) on visualization\n",
    "- For now, we'll just look at one uniquely-pandas approach"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot all numerical columns, and their interactions\n",
    "pd.plotting.scatter_matrix(df[['age', 'score', 'value']], figsize=[12, 12], marker=12);  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Application Program Interface (APIs)\n",
    "\n",
    "- APIs are basically a way for software to talk to software \n",
    "    - They are an interface into an application / website / database designed for computers / software."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notes on APIs:\n",
    "- Follow API guidelines! \n",
    "- These guidelines typically specify the number / rate / size of requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Github API\n",
    "\n",
    "You can access the github api with the following API. Just added specifiers for what you are looking for. \n",
    "\n",
    "https://api.github.com/\n",
    "\n",
    "For example, the following URL will search for the user 'ShanEllis':\n",
    "\n",
    "https://api.github.com/users/shanellis\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requesting Web Pages from Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The requests module allows you to send URL requests from python\n",
    "import requests  \n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Request data from the Github API on a particular user\n",
    "page = requests.get('https://api.github.com/users/jasongfleischer')  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The content we get back is a messily organized json file\n",
    "page.content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question #5\n",
    "\n",
    "What type/format of output is this?\n",
    "\n",
    "- A) CSV\n",
    "- B) XML\n",
    "- C) JSON\n",
    "- D) API\n",
    "- E) I'm super lost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can read in the json data with pandas\n",
    "git_data = pd.read_json(page.content, typ='series')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check out the pandas series object full of data\n",
    "git_data  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authorized Access - OAuth\n",
    "\n",
    "Open Authorization is a protocol to authorize access (of a user / application) to an API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OAuth provides a secure way to 'log-in' without using account names and passwords. \n",
    "\n",
    "It is effectively a set of keys, and passwords you can use to access APIs. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Web Scraping vs. APIs\n",
    "\n",
    "Web scraping and APIs are different approaches:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- APIs are an interface to interact with an application, designed for programmatic use\n",
    "    - They allow systematic, controlled access to (for example) and applications database\n",
    "    - They typically return structured (friendly) data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Web scraping (typically) involves navigating through the internet, programmatically following an architecture built for humans\n",
    "    - This can be hard to systematize, being dependent on the idiosyncracies of a web page, at the time you request it\n",
    "    - This typically returns relatively unstructured data\n",
    "    - This entails much more wrangling of the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Where to Find Data?\n",
    "\n",
    "* [Awesome Public Datasets](https://github.com/awesomedata/awesome-public-datasets/blob/master/README.rst)\n",
    "* [Data.gov](https://catalog.data.gov/dataset)\n",
    "* [Data Is Plural](https://docs.google.com/spreadsheets/d/1wZhPLMCHKJvwOkP4juclhjFgqIY8fQFMemwKL2c64vk/edit#gid=0)\n",
    "* [UCSD Datasets](https://ucsd.libguides.com/data-statistics/home)\n",
    "* [Datasets | Deep Learning](http://deeplearning.net/datasets/)\n",
    "* [Stanford | Social Science Data Collection](https://data.stanford.edu/)\n",
    "* [Eviction Lab (email required)](https://evictionlab.org/get-the-data/)\n",
    "* [San Diego Data](https://data.sandiego.gov/)\n",
    "* [US Census](https://www.census.gov/)\n",
    "* [Open Climate Data](http://openclimatedata.net/)\n",
    "* [Data and Story Library](https://dasl.datadescription.com/datafiles/)\n",
    "* [UCSD behavioral mobile data](http://extrasensory.ucsd.edu/)\n",
    "* [Kaggle](https://www.kaggle.com/)\n",
    "* [FiveThirtyEight](https://data.fivethirtyeight.com/)\n",
    "* [data.world](https://data.world/)\n",
    "* [Free Datasets - R and Data Mining ](http://www.rdatamining.com/resources/data)\n",
    "* [Data Sources for Cool Data Science Projects](https://blog.thedataincubator.com/2014/10/data-sources-for-cool-data-science-projects-part-1/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes on Working with Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Science is Ad-Hoc\n",
    "\n",
    "- It is part of the job description to put things together that were not designed to go together.\n",
    "- We do not have universal solutions, but haphazard, idiosyncratic systems, for data collection, storage and analysis.\n",
    "- Data is everywhere. But relatively little of it was collected *as data*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Collection, Curation, and Storage are Difficult\n",
    "\n",
    "- It can be difficult to choose broadly useful standards\n",
    "- Take time to think about your data, and how you will load, store, organize and save it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data is Inherently Noisy\n",
    "\n",
    "- We live in a messy, noisy, world, with messy, noisy, people, using messy, noisy instruments.\n",
    "- There is no perfect data. \n",
    "    - There is better / or worse data, given the context."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Different Objectives\n",
    "\n",
    "- Humans and computers are different.\n",
    "- We interact with '*data*' in different ways.\n",
    "- This underlies many aspects of data wrangling\n",
    "    - The 'friendliness' of data types / files\n",
    "    - The difference between web scraping and APIs\n",
    "    - A disconnect between data in the real world, and data we want to use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## So... What to do?\n",
    "\n",
    "- Think about how your data are stored & its structure?\n",
    "- Look at your data before you anayze it\n",
    "    - are there missing values? \n",
    "    - outlier values? \n",
    "- Are your data trustworthy? \n",
    "    - source?\n",
    "    - how was it generated?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Specific Recommendations\n",
    "\n",
    "- Prioritize using well structured, common, open file types\n",
    "    - Take advantage of existing tools to deal with these files (numpy, pandas, etc.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Look into, and then follow, common conventions\n",
    "    - Minimize custom objects, workflows and data files \n",
    "- Look for APIs. Ask if they are available.\n",
    "    - Acknowledge that web scraping and/or wrangling unstructured data are complex / long tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Think about data flow from the beginning. Organize your data pipeline, consider the 'wrangling' aspects throughout\n",
    "    - Set yourself up with well organized, labelled approach to your data\n",
    "    - Think about when and how you might want/need to save out intermediate results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}