salary_prediction_project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Salary Prediction\n",
    "\n",
    "Corresponding  with this notebook is a slide deck where you will need to update all the portions in red.  Completing the notebook will provide all the results needed for the slides.  **Correctly completing the slides is a required part of the project.**\n",
    "\n",
    "\n",
    "## Table of Contents\n",
    "- [Introduction](#intro)\n",
    "- [Part I - Descriptive Statistics](#descriptive)\n",
    "- [Part II - Regression](#regression)\n",
    "- [Part III - Interpret Results](#interpret)\n",
    "\n",
    "\n",
    "<a id='intro'></a>\n",
    "### Introduction\n",
    "\n",
    "Linear Regression is very commonly performed by data analysts and data scientists.  For this project, you will be working to understand the results of a Linear Regression model associated with salaries.  Your goal is to work through this notebook to understand what variables are related to salary, and how exactly they are related.\n",
    "\n",
    "As a final check, assure you meet all the criteria on the rubric.\n",
    "\n",
    "<a id='descriptive'></a>\n",
    "#### Part I - Descriptive Statistics\n",
    "\n",
    "To get started, let's import our libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "random.seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each of the parts of question `1` notice links to [pandas documentation](https://pandas.pydata.org/) is provided to assist with answering the questions.  Though there are other ways you could solve the questions, the documentation is provided to assist you with one fast way to find the answer to each question.\n",
    "\n",
    "\n",
    "`1.a)` Now, read in the `salary_data.csv` data. Store it in `df`. Read in the dataset and take a look at the top few rows here. **This question is completed for you**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Education Level</th>\n",
       "      <th>Job Title</th>\n",
       "      <th>Years of Experience</th>\n",
       "      <th>Salary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>32.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>Bachelor's</td>\n",
       "      <td>Software Engineer</td>\n",
       "      <td>5.0</td>\n",
       "      <td>90000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>28.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>Master's</td>\n",
       "      <td>Data Analyst</td>\n",
       "      <td>3.0</td>\n",
       "      <td>65000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>45.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>PhD</td>\n",
       "      <td>Senior Manager</td>\n",
       "      <td>15.0</td>\n",
       "      <td>150000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>36.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>Bachelor's</td>\n",
       "      <td>Sales Associate</td>\n",
       "      <td>7.0</td>\n",
       "      <td>60000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>52.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>Master's</td>\n",
       "      <td>Director</td>\n",
       "      <td>20.0</td>\n",
       "      <td>200000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Age  Gender Education Level          Job Title  Years of Experience  \\\n",
       "0  32.0    Male      Bachelor's  Software Engineer                  5.0   \n",
       "1  28.0  Female        Master's       Data Analyst                  3.0   \n",
       "2  45.0    Male             PhD     Senior Manager                 15.0   \n",
       "3  36.0  Female      Bachelor's    Sales Associate                  7.0   \n",
       "4  52.0    Male        Master's           Director                 20.0   \n",
       "\n",
       "     Salary  \n",
       "0   90000.0  \n",
       "1   65000.0  \n",
       "2  150000.0  \n",
       "3   60000.0  \n",
       "4  200000.0  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('salary_data.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`b)` Use the below cell to find the number of rows in the dataset. [Helpful pandas link - `Dataframe.shape`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.shape.html#pandas.DataFrame.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`c)` Do any of the rows have missing values? [Helpful Pandas Link One](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.isnull.html) and [Helpful Pandas Link Two](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.sum.html)  \n",
    "\n",
    "If there are missing values, determine a method for dealing with them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`d)` How many employees are in each `Education Level`? Build a bar chart to show the count of employees in each level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# bar chart of results - this part is done for you\n",
    "df['Education Level'].value_counts().plot(kind='bar');\n",
    "plt.title('Number of Visits From Each Education Level');\n",
    "plt.ylabel('Count of Visits');\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`e)` What are the possible values for `Salary`?  What does the distribution of `Salary` look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='regression'></a>\n",
    "#### Part II - Regression\n",
    "\n",
    "`1.` Now that you have had a chance to learn more about the dataset, let's look more at how different factors are related to `Salary`.\n",
    "\n",
    "`a)` Consider average salary by gender, is there evidence that salaries are higher for one gender over the other?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`b)` Consider average salary by education level, is there evidence that salaries are higher for increased education?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`c)` Consider average salary by years of experience, is there evidence that salaries are associated with increased years of experience?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`d)`  To make use of Job Title column lets create bool flag based on word existiance\n",
    "\n",
    "\n",
    "List of words:\n",
    "\n",
    "* Director\n",
    "* Junior\n",
    "* Senior\n",
    "* Manager\n",
    "* Analyst\n",
    "* Engineer\n",
    "\n",
    "**This question is completed for you**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flag_words = ['director', 'junior', 'senior', 'manager', 'analyst', 'engineer']\n",
    "df['Job Title'] = df['Job Title'].str.lower()\n",
    "\n",
    "for word in flag_words:\n",
    "    df['is_' + word] = df['Job Title'].str.contains(word)\n",
    "    \n",
    "df = df.drop('Job Title', axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`e)` Create a flag for gender where 1 is if a person is male and 0 if the person is not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`f)` Use statsmodels to fit a linear model to predict salary using each of the features from `a-e`.  These include:\n",
    "* Gender\n",
    "* Job TItle\n",
    "* Years of Experience\n",
    "* Education"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='interpretation'></a>\n",
    "### Part III - Interpret Results\n",
    "\n",
    "`1.` Consider you are tasked with finding which features in your linear model are significantly related to salary.  Were there any features that were not significantly related to salary in your first model?  If not, remove those features and fit a new model.  Only keep the features that were significant from the original model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# explain your answer and fit a new model with the reduced set of features here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`a)` With each additional year of experience, what is the expected change in salary?  What is the 95% confidence interval of the change?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Your Answer Here**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`b)` What is the expected difference in salary between someone with a senior title and someone with none of the other title indications?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Your Answer Here**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`c)` What is the expected difference in salary between someone with a PhD and an individual with no PhD nor master's degree?  What is the 95% confidence interval of the change?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Your Answer Here**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`d)` If a male employee has 5 years of experience as a senior engineer with a bachelor's degree, what is the expected salary of the employee?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Your Answer Here**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`e)` Imagine that the employee in question `d)` actually has a salary of $110,000, what would the residual be for this employee?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**Your Answer Here**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`f)` How well do you think your model fits?  What metrics or plots would you consider to understand if this model does a good job of predicting salary?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### Your work here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your Answer Here**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='finalcheck'></a>\n",
    "## Final Check!\n",
    "\n",
    "Congratulations!  You have reached the end of the salary prediction project!  You should be very proud of all you have accomplished!\n",
    "\n",
    "> **Tip**: Once you are satisfied with your work here, check over your notebook to make sure that it satisfies all the specifications mentioned in the rubric. You should also probably remove all of the \"Hints\" and \"Tips\" like this one so that the presentation is as polished as possible.\n",
    "\n",
    "\n",
    "<a id='submission'></a>\n",
    "## Submission\n",
    "\n",
    "Please follow the directions in the classroom to submit this notebook, as well as your completed slides."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}