01-TitanicIntroduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to open source Machine Learning utilising the Titanic dataset\n",
    "\n",
    "This notebook series provides an introduction to machine learning. For that it utilises the Titanic dataset from the Kaggle website https://www.kaggle.com/c/titanic.\n",
    "\n",
    "Let's start by having a look at the data. For that, we will utilise Pandas, an open source data analysis package for Python, using to main data structures: Series and DataFrame. \n",
    "\n",
    "#### Series:  \n",
    "Data structure for one-dimensional data (e.g. time series, ...)\n",
    "\n",
    "#### DataFrame: \n",
    "Multi-dimensional data structure (time series with multiple parameters, e.g. data formerly contained in Excel sheets, ...).\n",
    "####      \n",
    "\n",
    "You can find a documentation of Pandas here: http://pandas.pydata.org .\n",
    "\n",
    "To import data into Pandas, we first have to load the Pandas module into Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can read the data using a standard pandas function `read_csv`. If you have other file formats you can find all the functions needed in the [pandas doc](https://pandas.pydata.org/docs/user_guide/io.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('train.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let us start with a small quiz\n",
    "\n",
    "What kind of learning problem do we have here:\n",
    "- [ ] Supervised Learning\n",
    "- [ ] Unsupervised Learning\n",
    "- [ ] Reinforcement Learning\n",
    "\n",
    "What is our target here:\n",
    "- [ ] Regression\n",
    "- [ ] Classification\n",
    "- [ ] Clustering\n",
    "\n",
    "## Data Overview\n",
    "\n",
    "#### We can see, that we have 12 data columns.\n",
    "\n",
    "We can divide these columns already in two groups, numerical data and string data. NOTE: Strings have to be converted to numerical values before they can be used in a machine laerning application. This originates in the fact, that machine learning algorthims are all based on mathematical operations like multiplications, additions and non-linear functions and these only work with numbers.\n",
    "\n",
    "Let's start by having a closer look at the numerical data.\n",
    "\n",
    "#### The numerical data:\n",
    "The most important column is the `Survived` column. It provides the targets for training the algorithm.\n",
    "\n",
    "Another important one is the unique `PassengerId`, which serves throughout the whole process as a unique identification for data belonging to one person. This could also be done by the `Name` property, but similar names are possible, while similar ids aren't.\n",
    "\n",
    "#### The other columns are:\n",
    "- `Pcclass`, giving the Class in which the passanger traveled\n",
    "- `SibSp`, giving the number of siblings / spouses aboard\n",
    "- `Parch`, giving the number of parents / children aboard\n",
    "- `Age`, giving the age of the passenger\n",
    "- `Fare`, giving the price of the passenger paid for his/her ticket\n",
    "\n",
    "To get an overview of the data, we can utilise one of the multiple functions of Pandas:\n",
    "- `.columns` property shows the different columns in the DataFrame\n",
    "- `.head(n)` returns the first n rows of the DataFrame\n",
    "- `.describe()` returns a general summary statistics for all columns in the DataFrame\n",
    "- `df[col]` returns the data in a certain col of the DataFrame as a Series\n",
    "- `df[col].unique()` returns the unique values of the column of a DataFrame\n",
    "- `df[col].value_counts()` returns the number of occurences of each unique value in a DataFrame as a Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that only the numerical data appear, as these stats make only senes for those.\n",
    "\n",
    "#### Lets go throught the different rows:\n",
    "- `count` gives the number valid values (excluded missing `NaN` vales)\n",
    "- `mean` provides the mean of all the values in the column.\n",
    "- `std` provides the standard deviation from the mean in the column.\n",
    "- `min` provides the minimum value in the column.\n",
    "- `25%` provides the lower quartile value of the column.\n",
    "- `50%` provides the median value of the column.\n",
    "- `75%` provides the upper quartile value of the column.\n",
    "- `max` provides the maximum value of the column.\n",
    "\n",
    "#### We already gain some conclusions from these summary statistics:\n",
    "- It was more probable to die, than to survive on the titanic, since the mean of the `Survived` column is only 0.38.\n",
    "- The `count` value is the same for all columns except the `Age` colum, where some values are missing.\n",
    "- Most people travelled alone as the `25%` and `50%` values for both `SibSp` and `Parch` are 0, while the former has a `75%` value of 1 and a maximum value of 8 and the latter a `75%` value of 0 and a maximum value of 6.\n",
    "- The fares are rising strongly towards the more expensive tickets.\n",
    "- Most of the tickets were third class tickets.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## Pandas Indexing\n",
    "Before we continue, let us get a bit more comfortable with the way we can select data in pandas.\n",
    "\n",
    "To select data in pandas we will use `.loc` and `__getitem__`. There are even more options which can be found in the [indexing and selecting data section of the pandas documentation](https://pandas.pydata.org/docs/user_guide/indexing.html).\n",
    "\n",
    "In pandas the `__getitem__` function returns the related column, so if we do `df[\"Survived\"]` we will retrieve the column indicating if a person survived or didn't.\n",
    "\n",
    "`.loc` on the other hand allows us to select rows or rows and columns at the same time. E.g. `df[df[\"Survived\"] == True]` would return all data related to survivours, while `df[df[\"Survived\"] == True, \"Age\"]` would return the age of the survivours. Let's actually try this:\n",
    "- Select the data related to everyone older than 30\n",
    "- Return the parchment class (`\"Pclass\"`) for everyone paying more the 50 Pounds (`\"Fare\"`) for the journey"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us investigate these data closer using [visualization](./02-ExplorationNumeric.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}