Analyze_ab_test_results_notebook_jc.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyze A/B Test Results\n",
    "\n",
    "\n",
    "## Table of Contents\n",
    "- [Introduction](#intro)\n",
    "- [Part I - Probability](#probability)\n",
    "- [Part II - A/B Test](#ab_test)\n",
    "- [Part III - Regression](#regression)\n",
    "- [Conclusion](#con)\n",
    "\n",
    "\n",
    "<a id='intro'></a>\n",
    "### Introduction\n",
    "\n",
    "A/B tests are very commonly performed by data analysts and data scientists. \n",
    "\n",
    "For this project, I will be working to understand the results of an A/B test run by an e-commerce website.  My goal is to work through this notebook to help the company understand if they should implement the new page, keep the old page, or perhaps run the experiment longer to make their decision.\n",
    "\n",
    "<a id='probability'></a>\n",
    "#### Part I - Probability\n",
    "\n",
    "To get started, let's import our libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "random.seed(42)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`1.` Now, read in the `ab_data.csv` data. Store it in `df`.  \n",
    "\n",
    "a. Read in the dataset and take a look at the top few rows here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0\n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0\n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('ab_data.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. Use the below cell to find the number of rows in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(294478, 5)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 294478 observations in this dataset\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. The number of unique users in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['user_id'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "d. The proportion of users converted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12104245244060237"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.query('converted == 1')['user_id'].nunique()/df['user_id'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "e. The number of times the `new_page` and `treatment` don't line up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3893"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The row is in new page but is not in treatment group.\n",
    "# OR the row is not in new page but is in treatment group.\n",
    "df.query('landing_page == \"new_page\" & group != \"treatment\"').count()[0] + df.query('landing_page != \"new_page\" & group == \"treatment\"').count()[0]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "f. Do any of the rows have missing values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 294478 entries, 0 to 294477\n",
      "Data columns (total 5 columns):\n",
      "user_id         294478 non-null int64\n",
      "timestamp       294478 non-null object\n",
      "group           294478 non-null object\n",
      "landing_page    294478 non-null object\n",
      "converted       294478 non-null int64\n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 11.2+ MB\n"
     ]
    }
   ],
   "source": [
    "# There is no missing values in this dataset.\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` For the rows where **treatment** is not aligned with **new_page** or **control** is not aligned with **old_page**, we cannot be sure if this row truly received the new or old page.  \n",
    "\n",
    "a. Now use the answer to the quiz to create a new dataset that meets the specifications from the quiz.  Store your new dataframe in **df2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2 = df.query('landing_page == \"new_page\" & group == \"treatment\" | landing_page == \"old_page\" & group == \"control\"')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Double Check all of the correct rows were removed - this should be 0\n",
    "df2[((df2['group'] == 'treatment') == (df2['landing_page'] == 'new_page')) == False].shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`3.` "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. How many unique **user_id**s are in **df2**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2['user_id'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "b. There is one **user_id** repeated in **df2**.  What is it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jemchang/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>is_duplicate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2893</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted  \\\n",
       "2893   773192  2017-01-14 02:55:59.590927  treatment     new_page          0   \n",
       "\n",
       "      is_duplicate  \n",
       "2893          True  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# user_id = 773192\n",
    "df2[\"is_duplicate\"]= df2['user_id'].duplicated()\n",
    "df2.query('is_duplicate == True')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. What is the row information for the repeat **user_id**? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>is_duplicate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1899</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-09 05:37:58.781806</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2893</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted  \\\n",
       "1899   773192  2017-01-09 05:37:58.781806  treatment     new_page          0   \n",
       "2893   773192  2017-01-14 02:55:59.590927  treatment     new_page          0   \n",
       "\n",
       "      is_duplicate  \n",
       "1899         False  \n",
       "2893          True  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.query('user_id == \"773192\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. Remove **one** of the rows with a duplicate **user_id**, but keep your dataframe as **df2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep the first record and delete the second one\n",
    "df2 = df2.drop_duplicates(subset='user_id', keep=\"first\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check duplicates\n",
    "sum(df2['is_duplicate'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>is_duplicate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1899</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-09 05:37:58.781806</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted  \\\n",
       "1899   773192  2017-01-09 05:37:58.781806  treatment     new_page          0   \n",
       "\n",
       "      is_duplicate  \n",
       "1899         False  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the duplicate user_id\n",
    "df2.query('user_id == \"773192\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`4.` \n",
    "\n",
    "a. What is the probability of an individual converting regardless of the page they receive?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(df2.query('converted == 1').converted)/len(df2['user_id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# save convert rate in this dataset as con_obs\n",
    "con_obs = df2['converted'].mean()\n",
    "con_obs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. Given that an individual was in the `control` group, what is the probability they converted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1203863045004612"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate and save convert rate in the control group in this dataset as con_c\n",
    "con_c = len(df2.query('converted == 1 & group == \"control\"')['user_id'])/len(df2.query('group == \"control\"')['user_id'])\n",
    "con_c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. Given that an individual was in the `treatment` group, what is the probability they converted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11880806551510564"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate and save convert rate in the treatment group in this dataset as con_t\n",
    "con_t = len(df2.query('converted == 1 & group == \"treatment\"')['user_id'])/len(df2.query('group == \"treatment\"')['user_id'])\n",
    "con_t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0015782389853555567"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The difference of convert rates between the control group and treatment group\n",
    "obs_diffs = con_t - con_c\n",
    "obs_diffs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. What is the probability that an individual received the new page?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5000619442226688"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df2.query('landing_page == \"new_page\"')['user_id'])/len(df2['user_id'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e. Use the results in the previous two portions of this question to suggest if you think there is evidence that one page leads to more conversions?  Write your response below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This dataset includes two groups, old page (control) and new page (treatment), equally. The convert rate in this dataset is 11.96% regardless of pages. The convert rate in the old page is 12.04% than the convert rate in the new page, 11.88%. The differeence is only 0.15%, so it is very hard to tell which page leads to more conversions than another. We need to implement an experiment to see if the difference is statistically significant. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='ab_test'></a>\n",
    "### Part II - A/B Test\n",
    "\n",
    "Notice that because of the time stamp associated with each event, you could technically run a hypothesis test continuously as each observation was observed.  \n",
    "\n",
    "However, then the hard question is do you stop as soon as one page is considered significantly better than another or does it need to happen consistently for a certain amount of time?  How long do you run to render a decision that neither page is better than another?  \n",
    "\n",
    "These questions are the difficult parts associated with A/B tests in general.  \n",
    "\n",
    "\n",
    "`1.` For now, consider you need to make the decision just based on all the data provided.  If you want to assume that the old page is better unless the new page proves to be definitely better at a Type I error rate of 5%, what should your null and alternative hypotheses be?  You can state your hypothesis in terms of words or in terms of **$p_{old}$** and **$p_{new}$**, which are the converted rates for the old and new pages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "H0: **$p_{old}$** >= **$p_{new}$**   \n",
    "Ha: **$p_{old}$** < **$p_{new}$**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` Assume under the null hypothesis, $p_{new}$ and $p_{old}$ both have \"true\" success rates equal to the **converted** success rate regardless of page - that is $p_{new}$ and $p_{old}$ are equal. Furthermore, assume they are equal to the **converted** rate in **ab_data.csv** regardless of the page. <br><br>\n",
    "\n",
    "Use a sample size for each page equal to the ones in **ab_data.csv**.  <br><br>\n",
    "\n",
    "Perform the sampling distribution for the difference in **converted** between the two pages over 10,000 iterations of calculating an estimate from the null.  <br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. What is the **convert rate** for $p_{new}$ under the null? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pnew=con_obs\n",
    "pnew"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. What is the **convert rate** for $p_{old}$ under the null? <br><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pold=con_obs\n",
    "pold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. What is $n_{new}$?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "treatment    145310\n",
       "control      145274\n",
       "Name: group, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2['group'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "145310"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nnew = len(df2.query('group == \"treatment\"')['group'])\n",
    "nnew"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> $n_{new}$ = 145310"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. What is $n_{old}$?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "145274"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nold = len(df2.query('group == \"control\"')['group'])\n",
    "nold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    ">$n_{old}$ = 145274"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e. Simulate $n_{new}$ transactions with a convert rate of $p_{new}$ under the null.  Store these $n_{new}$ 1's and 0's in **new_page_converted**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, ..., 0, 0, 0])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_page_converted = np.random.binomial(1, pnew, nnew)\n",
    "new_page_converted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "f. Simulate $n_{old}$ transactions with a convert rate of $p_{old}$ under the null.  Store these $n_{old}$ 1's and 0's in **old_page_converted**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, ..., 0, 0, 0])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "old_page_converted = np.random.binomial(1, pold, nold)\n",
    "old_page_converted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "g. Find $p_{new}$ - $p_{old}$ for your simulated values from part (e) and (f)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0002043636209243388"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_page_converted.mean() - old_page_converted.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "h. Simulate 10,000 $p_{new}$ - $p_{old}$ values using this same process similarly to the one you calculated in parts **a. through g.** above.  Store all 10,000 values in **p_diffs**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_diffs = []\n",
    "\n",
    "for _ in range(10000):\n",
    "    new_page_conv = np.random.binomial(1, pnew, nnew).mean()\n",
    "    old_page_conv = np.random.binomial(1, pold, nold).mean()\n",
    "    p_diffs.append(new_page_conv - old_page_conv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_diffs = np.array(p_diffs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "i. Plot a histogram of the **p_diffs**.  Does this plot look like what you expected?\n",
    "\n",
    "> It looks like bell-shaped normal distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiYAAAEnCAYAAACdeoh+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xn8XOPd//HXW8RWO9GmRBNL7U2QElua2BJLKW6NVhGq2h/VulUrVGurG6Wb0tZ6h0rFVoSiDZLgtiUhBCkSQiKphBBiieDz++O6JjmZzHdLvt/vTDLv5+Mxj5m5znWuc13nnDnzmetc54wiAjMzM7NasFy1K2BmZmZW4sDEzMzMaoYDEzMzM6sZDkzMzMysZjgwMTMzs5rhwMTMzMxqRpOBiaSBkqLweF/SZEm3SfqmpOXK8nfN+QaWpZ8u6TVJn0gal9O+IGmYpFl5npNatXVLibw+B1e7HrUu71tnSdpoaV5e3t7Xt2J5J0k6uLXKW9rkbdTkfQ8k9cnHmT7tUK0mSRopaWS162FNK3wPdi2ktei4vTjHk0rfp5IGS5ra3DKWpF7V+m5avgV5DwWmAisCGwL7ATcAx0n6ekR8mPNNB3YCJpVmlLQDcB5wEXA78F6e9Evga8DAPN/kxWzH0u4g4N1qV2Ip0BU4E3gYeHkZXN7iOolUx79XuyJmdaSlx+2utPx4ssj3aRvoSsP1qsp3U0sCk3ERMbHw/q+SbgZuBn4NnAgQEXOBx8rm3SI//yUiXi5LfzoibmtZtSuTtGJe/lIlIp6qdh1qmSQBHVuprKVyHzGz2tKWx+3SMa+B79N2U7Xvpoho9EHqzQhgkwam3wZ8BKyS33fN+Qfm9yPz++JjcIW0ALrmeboBQ4CZwFxgHHBQ2XLPyvNsDfwTmAPcUZh+MGmDfgC8QwqgNiwrYzJwPXAYMAF4HxgD7FqhnV8DhgOzc76nge+W5fleTv8IeBO4Gli7Get4MjC4wjrvldfDu8A04BJgpWaUtzxwKvB8rstM4F5g80KezfK2ewf4MK+r/g2s402Bf+R1/Cqpp2u5nKcz8AlwYoV6nArMAzot5nY5Bvh3LuOgBvaZPo2sh5GkXwFfB57K+9J/52k/BB4FZuV6PAbsV5i3T1PLW8LtfX2ef2Ke/0mgbwP73f2kXsb3Sfv61mVlVfp89cyvdy3kPTGn/aqQtmlO27eQ1uTnL+frDgwD3s770P8Bu5XlGUzqad0WeChv95eAHzS1nlq6n5aldQL+RvrsvANcB3yjqX0mz/tV4JZc7w+BF4D/AVZuYP/aM2+/D4BngW9UKPMw0r48F3iOtD+PBEY2UZeuuc7fB84h/YJ+B7gT2KBC/kb3SeAu4L7CexW28yqF9CHAE03UrdT+A3O75+Y2frMs3ybAX4FX8vp8GfgzsFaFMn9M2qc/Ap4Adqbs+NjcfRT4ct53ZuTyXiMdb5Zvol0bkY53H+Ty/5DX//zvqMJnr3jc/gJwLelYPTdvq7uA9WjieELDx7zS9h9Y4TO1MzA6t20yZcdgKnwuCvNPbs5xroF1vwNwH+n74H3S8WmHVv3cN+PAMJDGA5Pv5em9yz5IA/P7LUkf6sgruhfwpfz8NOkD3Ss/VgS65B3pWeA7QD/gGuAz4IDylU7q4jod2L2wMn+Qp10D7AsMIAUerwCrle1Yr+aN+1/A/qQvsHeANQv5DiR9+Y4iHWD2JH2AzinkuSDvTL8B9gaOBl4HHgc6NLGOF9r4hXX+EulgtCfwC+BT4OxmbLNbcn0vBvqTDsi/JX/xAV8kfeBezuv466TA5VNgnwrr+FngJ7kef8hpRxfy3UuFgxgwHriz8L4l2+X1vNxvAXuQvgSPz/OfWNhnVm/iwDkjl38M6UP4lTztYuC7uex+wKW57H3y9NUbW14rbO8pue0D8vZ5lHSA2ayQb7+8He8g7YMHAo+QAoEuOc+2pAPgvYU6bkwaP/Y28MtCebeRDhCPFNK+n5exWn7f3M/fdqSD0sOkz86+pCBlLrB92QHq3dzW7wN7kQKGoEIgVraeWrSfls37UF7uDwttmELzApNDgDNIx4Ov5f3gP8DQCvvXdFKg8R3SZ214Xp+bFPLtmdffnXmbDiR9SU6n+YHJ5Lze9gGOIgUdo8ryNrlPAifnfWDF/L57rttHwN6FsqYBFzZRt5F5vbyal7Uf6Yv4s+K2BXoD55P23965/S8Cj5aVd2xu61V5mx2f2/0OCx8fm7uPvkgKbg7J2/HbpC//FRpp0wqk75RphTYNK+w7XRs5bg/Pyzw8t/NQ4C95GzZ1PJnMose8jWk4MHk31+mHpP1ucIV8Z9F0YNKcehXb+BVScDmW9Lk/hPT9+SHQvTU+9xGtE5j0y9MHlH2QiiuotMN1LZv3Yco+mKQIfyawTln6cNLppPIvzR+X5VuV1KtxTYUP+MfASWU71tsUIncW/NL8dn6vnG8MuZeggYPHpxS+BHL6LrmsRX5BleUr3/ildX52Wb67gBebKGv3PO+PGslzMYsePDuQfhk+WWEdH102/3jgX4X3h+d8xS/VHjntm4u5XT4AvlCWt08uc8+m9tucfyTpYNWjiXzLkXqZ/sXCvW4Vl9dK2/tjCj1FwGqk3pu/FtImAveXzbs66Uvp92XlXV9hOXcAIwptnEX60poHrJrThwKPLcbn737SQWeFQlqHnHZ7IW0wZQcj0g+QN4ErmlhPLdpPC+/3yss8rKy8e2hGYFI2j/K+8Z28L61TmDYyr8tNC2nr5X3j9ELa/5F6L5crpO2Y6zKyieV3zfnKg5BTcvoXW7JPkgLZAL6W358EPJO37/k5bfOcp38TdRuZ8/Uq2z7/Bh5qZL7lgV3zvNsW9s8pwN1leQ/O+YrHxyb3UWDdPN8BjbWhQt2+V6FNy5GCz6YCkzk0ftztQwPHLxo+5pW2/8AKn6ny/Xs4KUhUpc9F2fyTW1CvYhtvYdEf7quTji1/r1DHFn/uI6JVLhdWfo5WKAtS9Hc3MFvS8qUHqQu7u6TVy/KXj0/ZibSihpTNP5X0geldlv/RiHi78H58ft4wP29G6uG5KiI+a6DOe5F23vJlPk6KGsuX2Vz/KHs/vlCvhuxN2hZXNpKnN+nLaP6YoYj4lDSYuUeFdVxej2fL6nEb6UN5RCHtCFIgMiy/b+l2eSwi/tNIG5prckSMK0+UtL2kuyS9Qfrym0fajps1o8zW2N6PRcRrpTcR8R5pPe+U67cp6RdT+TI+IPWuNGcZI4CdJK1EChTXJI0HmwvslvP0AR4ozNPk50/SyqRfoDcDnxXyiNTFW163DyJiRKGtc0m9gU3tyy3dT0t2In1J31qWPrSJ5QGQ23ihpEmkdTWPdCpCpFNfRS9FxEuF+s0g/ZrfMJfVgXxqqHj8iIjHadlg/0rHAliwDpu7Tz5N+hLZPb/fnbT9HyhLm0fqdWrKlIiYPwYib5+bgR1KV2xKWiFflflvSR+WlV36vG2QHzeXlX8H6fNZ1JzviLdIPW0XSPpe/jw1x04V2vQZcFMz5h0N/FTSjyVtk8eJtERLjnkN7d8bAuu3cLkt0Ru4KyLeKSVExLuk4/zXyvIu7ue+VQKTLvl5eiuUBekXx5Gknbf4uChPX6csf/ly18vP91UoY5sK888qvokFAyNXKlteY5dnlZY5scIyV6+wzOaaVfZ+LinqbMw6wKxYcJVUJWtTeXv9h3TwXasZ9SitHyLiA9KH5HAlHUjdkTdHxEc5W0u3S2vtT4uUI6kL6Rf/2qTuy51JXx73UmhXI1pje7/RQFrpoFJaxtUVlrF/M5fxAGl/2RnoSxpo/gapp7KvpK2Az5MCmGLbmvr8rU36ZfyLCvl+CKxVdhuBYuBfstA+1ICW7qclnYG3I2JeWXqldV7J/5JOO15C+sL/KnBCnlZe5/LPBizctnVJA7cb2t7NVekzWKxPs/bJ/CU7irT9O5C+aEbkx/b5S70vMDoi3m9GvRpq1wqkcT6QTuOcRTqNsh9pjELp8vZS/Tvn5xnFgnKg82ZZ+U3uo5F+ou9F6uk+H3hR0suS/l8T7encSJuaMoD0Bf0zUi/U65J+WfZZaExLjnmN7d9tGZg09pks/zwu7ue+RVflNGQ/0vnJsa1QFqRI9yHgwgamTyt7X95T81Z+Hkjqfiv3XoW0xpQ+FI1t7NIy96byxnirQlpbeRNYW9LKjQQns0gDtcp9gbQ+Kx1sm/JX0rnvXYGVSR/wvxamt3S7tFYPXKVy+gNrkE4zzQ84Ja3SzDJbY3t/voG018vKOI0UzJX7uBnLGE/aH3YndeGXekYeAL5J6jr/mHSqoaQ5n7/lSac1LiMNKl1EI72LLbG4++l0UnDUsezgXWmdLyT3Lh0InBURfyikb9PsWi/sTdKXZkPb+9XFLLdcS/bJEaTTZLuSTiGOYsHg6q+RetEub+ZyG2rXx6TTLZDG5V0XEb8qZZC0atk8pS+79YqJOXhatyxvs74jIl0BemTuuehOCpr/JGlyRNzTwLzTga0aaFOjcm/ZCcAJkjYjHQ/PJq2HPzc1Py075jW2f5eOIR9B6rGKiOLxYnF/KEPjn8nF+d6oaIkCk3xTpwOAP+Rfza3hXlJ32nNN/OpvyCOkD9kmEXFtK9TnRVKX67GSrsiReLnhpAP1hhExvBWWuST+BQwijev5YwN5RgEnSeoaEZNh/gFgAPBUPq3QUiNIvUpHkAKTySzcFdwa26X0K3HlxZy/pBSAzP9QS/oy6Xx8sWesoeW1xvbuJalLREzJy1+NFOSXuuxfIK3DrSLigibKmluhjkRESBpF+uW4BfCnPOkB0q/Id4HHyz67zfn8zZX0EOlg/2QrBSGVLO5++iipR+cQFj59c1gzlrlinrf81+jA5ld7gYj4VNJo4L8knVVaV5J2JI0faK3ApCX75AhSj8YvSNvvnVynh0iD+tdl4dN7jekiqVfp1EfePoeSBsOX9otVWHR9Hl32fmp+HErqsSr5Bot+T7XoOyIfs8dJOpk04H1r0nijSh4Fji5r03KkQL7ZIuIF4HRJP8jLg9Y7fkHD+/drLAhMSvvW1qSLTJC0JqkHtfjZaUm9RgH7SVqt9PnLx66vk8YctYqWBCY9JK1L2qE3JHUnH0r6QJzWWhUiXYr6BPCgpEtJB+e1SCt3o4g4prGZI+JdST8FLpPUibQDzib1eHyNNNjsb82tTD64n0S6edUDkv5CioC3ANaLiDMjYpKkC4FLc6Q8ihStdiF9KVxVPNfWliJihKRbgd/mUxYPkLqSewP/iIiRwO9IB9rhks4kfUEdT7q8br/FXO5nkoaQRmB3BH5XDOJaabu8SDrffIykWaQP1AuLEUjdl8u5TtJvSL07Z5M+1MVu14aW1xrb+w3gX5LOyuWeCnwOOBfm73cnAHdIWoF0jvtN0q+inYHXIuK3uazngd0k7U/qUn2z9EVO2v6Xkc5JlwLFJ0nbvC/pqq+i5n7+TgYeBP4p6WrSL811SVfrdIiIQU20vzkWaz+NiOGSHgYuz8esl0jBzNYNzVOYd7akx4CfSJpOWufHsGTd42eSfjDcLuly0imOs0nbqlW0ZJ+MiGclzSBd+XFRoZhST8pc0hd0c7wB3Ji3z0zg/5G2T/GUyb3AUZLGk041HUzah4v1/0zS2cCVkq4ijTXZiPQjazYp6Cppch+V9BXSFYQ35mV2IO1Ln9B40HVtXubfJZ1OOrX0A9LpsAZJWoN0XBnCgst9D8z1+lfO1lrHL0iBxa8L+/e3SFd/DSwcd0vH2Cvz9lmRdJppTllZLanXuaTv/vvz/hakY9cqLHosWXxNjY5lwRUipceHpEjsNlJgorL8XVmCq3Jy+gakS8ZeJ3UJTicFQN8p5Dkrl1nxmnTS5YsjSAezD0k75zXAloU8k6l8NUOQunKLabvn8ubkx9MserXKEaT7LLyf80wgXYa6yP0GyuabTOWrcjYpy3cWFUZZVyhveeDnpB2u1KV6NwtfNbMZ6S68s0kHsMbuY7J8WfpgCqO6C+lbFfaTzRqo22Jvlzzt+6RBbZ/QxBUW5PssNDDtm6QDyEekU0uHVWpXY8tbwu19PelzURpg+RSwe4W8O5GuxnqbBfcrGArsVMizOQvuFVB+BcMWOe2xsnLvaGj90YzPX6HsoaSD91zSL95hLHxPlMHA1Aa2zSKf/Qr5mr2flqV1Ig2SfY8F9zE5sKl9Js/blXRQfy+37VJSIFS+/SvuX1S+98O3SL1gi3sfk2PL0vtUaktz90nSF3YU1yULrthpcrsU20/qNS/dx+QF8hWahXzr5v3k7fwYQhq3s9D3RM57Eun75SPyPaXyPL9ryT5KOiV0LekY+AHpNMMooF8z2rUR6XjZ7PuYkL70L8/bdg7p+DaafHVnU8cTGv4uKm3/gYW0wSx6H5NXqXBFUF5/o3NbXiRdXTaYZh7nqLwv70gz72OyuJ/70mVFZmZmzab0Pz/LR8SubbiMr5J6R46MiL82ld+WDa0x+NXMzGyJSOpGGjxaujneFqSbZ77CopfG2jLMgYmZmdWCD0njRI4kjc14m3TKYFC03sUVthTwqRwzMzOrGa1xgzUzMzOzVuHAxMzMzGqGAxMzMzOrGQ5MzMzMrGY4MDEzM7Oa4cDEzMzMaoYDEzMzM6sZDkzMzMysZjgwMTMzs5rhwMTMzMxqhgMTMzMzqxkOTMzMzKxm+N+FlwHrrrtudO3atdrVsIa88EJ63myz6tbDzBYyduzYNyOiU7XrYQtzYLIM6Nq1K2PGjKl2Nawhffqk55Ejq1kLMysj6dVq18EW5VM5ZmZmVjMcmJiZmVnNcGBiZmZmNcNjTMys1c2bN4+pU6fy0UcfVbsqZqy00kpssMEGdOzYsdpVsWZwYGJmrW7q1KmsttpqdO3aFUnVro7VsYjgrbfeYurUqXTr1q3a1bFm8KkcM2t1H330Eeuss46DEqs6SayzzjruvVuKODAxszbhoMRqhffFpYsDEzMzM6sZDkzMzMysZnjwq1kd6DroH+26vCsP6My8qe8A8JUN1mzXZVdy1llnseqqq3LKKafwy1/+kt69e7Pnnnvy0EMP8YMf/ICOHTvy6KOP8stf/pK7776bfffdl4suuqja1W6WadOm8aMf/Yhbbrllics69thjOfnkk9lyyy2XqJzBgwczZswYLr300hbPW9xWDbn99tv58pe/vMT1tNrkwMTM6so555wz//WQIUM45ZRTOProowG4/PLLmTlzJiuuuGK1qtegTz75hOWXX/SQ/cUvfrFVghKAq666qlXKaWu33347+++/vwOTZZRP5ZhZ2zrppPR/Qa35OOmkJhd73nnnsdlmm7HnnnvyQumPFIGBAwdyyy23cNVVV3HTTTdxzjnncPjhh3PAAQfw/vvvs+OOO3LjjTcuVNacOXM4+uij2WabbfjKV77CrbfeCsANN9zANttsw9Zbb82pp546P/+qq67Kz3/+c7p3706vXr144403mD17Nl27duWzzz4D4IMPPqBLly7MmzePSZMm0b9/f7bffnt22203/v3vf8+v68knn0zfvn059dRTGTVqFD169KBHjx5su+22vPfee0yePJmtt94aSFdDleq57bbbMmLECCD1YBx88MH079+fTTfdlJ/97GcV11mfPn3m/+9WpTaUe+KJJ9h5553Zdttt2XnnnRdaz1OmTKF///5sttlmnH322Yu1ra688kq++tWv0r17dw455BA++OADHnnkEYYNG8ZPf/pTevTowaRJkyrms6WXA5NWImklSU9IelrSc5LOzundJD0u6SVJN0paIaevmN9PzNO7Fso6Lae/IKlfdVpktvQaO3YsQ4cO5amnnuLvf/87o0ePXiTPscceywEHHMBFF13EkCFDGDZsGCuvvDLjxo1jwIABC+U999xzWWONNRg/fjzPPPMMu+++O9OmTePUU0/lgQceYNy4cYwePZrbb78dgPfff59evXrx9NNP07t3b6688krWWGMNunfvzqhRowC488476devHx07duS4447jj3/8I2PHjuXiiy/m+OOPn7/sF198kfvuu4/f/OY3XHzxxVx22WWMGzeOhx56iJVXXnmhel522WUAjB8/nhtuuIGjjjpq/mWy48aN48Ybb2T8+PHceOONTJkypdF1WKkN5TbffHMefPBBnnrqKc455xxOP/30+dOeeOIJhgwZwrhx47j55psb/KPRxrbVwQcfzOjRo3n66afZYostuPrqq9l5553nb7dx48ax8cYbV8xnSy+fymk9c4HdI2KOpI7Aw5LuAU4GfhcRQyX9Bfgu8Of8/HZEbCLpMOBCYICkLYHDgK2ALwL3SfpyRHxajUaZLbHf/77dF/nQQw9x0EEHscoqqwBwwAEHLFF59913H0OHDp3/fq211uLBBx+kT58+dOrUCYDDDz+cBx98kG984xussMIK7L///gBsv/32DB8+HIABAwZw44030rdvX4YOHcrxxx/PnDlzeOSRRzj00EPnlz937tz5rw899FA6dOgAwC677MLJJ5/M4YcfzsEHH8wGG2ywUD0ffvhhTjzxRCAFDV/60pd48cUXAdhjjz1YY401ANhyyy159dVX6dKlS4NtbqgNRbNnz+aoo47ipZdeQhLz5s2bP22vvfZinXXWAVKA8fDDD9OzZ89FymhsWz377LOcccYZvPPOO8yZM4d+/Sr/TmtuPls6uMeklUQyJ7/tmB8B7A6UTgBfC3wjvz4wvydP30PpYvsDgaERMTciXgEmAju0QxPMlimtee+KiFikvIhoMH/Hjh3n5+/QoQOffPIJkL5077nnHmbNmsXYsWPZfffd+eyzz1hzzTUZN27c/MeECRPml/W5z31u/utBgwZx1VVX8eGHH9KrV6/5p3yaU6fiuJlinVrahqJf/OIX9O3bl2effZY777xzoZuYla+vxrZHQ9MGDhzIpZdeyvjx4znzzDMbvElac/PZ0sGBSSuS1EHSOGAGMByYBLwTEaVP9FRg/fx6fWAKQJ4+G1inmF5hnuKyjpM0RtKYmTNntkVzzJZavXv35rbbbuPDDz/kvffe484771yi8vbee++FrjB5++232XHHHRk1ahRvvvkmn376KTfccANf+9rXGi1n1VVXZYcdduDHP/4x+++/Px06dGD11VenW7du3HzzzUAKLp5++umK80+aNIltttmGU089lZ49ey4SmPTu3ZshQ4YA6RTQa6+9xmabbbYkTW/U7NmzWX/9dHgaPHjwQtOGDx/OrFmz+PDDD7n99tvZZZddKpbR2LZ677336Ny5M/PmzZvfLoDVVluN9957r8l8tnRyYNKKIuLTiOgBbEDq5diiUrb8XOknQjSSXr6sKyKiZ0T0LHUlm1my3XbbMWDAAHr06MEhhxzCbrvttkTlnXHGGbz99ttsvfXWdO/enREjRtC5c2fOP/98+vbtS/fu3dluu+048MADmyxrwIABXH/99QuNYxkyZAhXX3013bt3Z6uttuKOO+6oOO/vf//7+XVYeeWV2WeffRaafvzxx/Ppp5+yzTbbMGDAAAYPHtymVxj97Gc/47TTTmOXXXbh008XPtu86667csQRR8zfBpVO40Dj2+rcc89lxx13ZK+99mLzzTefn37YYYdx0UUXse222zJp0qQG89nSSY11/dnik3Qm8AFwKvCFiPhE0k7AWRHRT9I/8+tHJS0P/AfoBAwCiIjzcznz8zW0rJ49e0ZDA8usBvTpk55HjqxaFapxH5PPb7gRUBv3MTGbMGECW2yx8G9FSWMjonLEZFXjHpNWIqmTpDXz65WBPYEJwAjgv3K2o4DST6Fh+T15+gORosRhwGH5qp1uwKbAE+3TCjMzs+ryVTmtpzNwraQOpIDvpoi4S9LzwFBJvwKeAkrXsV0N/FXSRGAW6UocIuI5STcBzwOfACf4ihwzW9q99dZb7LHHHouk33///fOv3jEDByatJiKeAbatkP4yFa6qiYiPgEPL0/O084DzWruOZu0liIpXslj9WmeddRg3blxVlu0hC0sXn8oxs1b36jvz+OSDd/2FYFUXEbz11lustNJK1a6KNZN7TMys1f3x8bc5EfjSmm8yYc4q1a6O1bmVVlppkZvRWe1yYGJmre7duZ9x3oNvATD5gv2qXBszW5r4VI6ZmZnVDAcmgKSvSdqx8H6gpIclXS5p1WrWzczMrJ44MEl+D3wBQNJmwOXAM8BOwEVVrJeZmVldcWCSbAyMz68PAYZHxPHA94CvV61WZmZmdcaBSRJAh/x6D+De/Po/pD/WMzMzs3bgwCQZDfxC0hHAbsA9Ob0rKTgxMzOzduDAJDkJ6AFcCpwXEZNy+qHAI1WrlZmZWZ3xfUyAiHgW+EqFSacA/p8aMzOzduLApBH5/2zMzMysnTgwASStTfrTvD2A9Sg7xRURq1ejXmZmZvXGgUlyNemfga8AppGu0jEzM7N25sAk2QPYKyIer3ZFzMzM6pmvyklmAHOqXQkzM7N658Ak+Tlwjv8Xx8zMrLp8Kic5g3QztRmSXgXmFSdGRKVLic3MzKyVOTBJbql2BczMzMyBCQARcXa162BmZmYOTBYiaXdgS9Llws9FxMjq1sjMzKy+ODABJK0P3AZsT7qPCcAXJY0BDoqIaQ3ObGZmZq3GV+Ukl5D+E2eTiOgSEV2ATXPaJVWtmZmZWR1xj0myF9AnIl4pJUTEy5J+BNxfvWqZmZnVF/eYNO6z5maU1EXSCEkTJD0n6cc5/SxJr0salx/7FuY5TdJESS9I6ldI75/TJkoa1LpNMjMzq13uMUnuBy6R9K2ImAIgaUPgDzS/x+QT4CcR8aSk1YCxkobnab+LiIuLmSVtCRwGbAV8EbhP0pfz5MtIvThTgdGShkXE80vQPjMzs6WCe0ySHwGrAC9LelXSZGBSTvtRcwqIiOkR8WR+/R4wAVi/kVkOBIZGxNx8CmkisEN+TIyIlyPiY2BozmtmZrbMc48JkHtJtpO0F7A5IOD5iLhvccqT1JX0b8WPA7sAP5R0JDCG1KvyNiloeaww21QWBDJTytJ3XJx6mJmZLW3cY1IQEcMj4o8RcckSBCWrArcCJ0XEu8CfgY2BHsB04DelrJWq0Eh6+XKOkzQSu5wlAAAdoklEQVRG0piZM2cuTlXNzMxqTt32mEg6GfhTRHyUXzcoIn7bzDI7koKSIRHx9zzvG4XpVwJ35bdTgS6F2TdgwT1UGkov1ukK4AqAnj17LhK4mJmZLY3qNjABTgSuBT7KrxsSQJOBiSQBVwMTioGMpM4RMT2/PQh4Nr8eBvxN0m9Jg183BZ4g9ZhsKqkb8DppgOy3W9AuMzOzpVbdBiYR0a3S6yWwC3AEMF7SuJx2OvAtST1IAc5k4Pt5mc9Jugl4nnRFzwkR8SmApB8C/wQ6ANdExHOtUD8zM7OaV7eBSVEemHpjRMwtS18BOCwirmuqjIh4mMrjQ+5uZJ7zgPMqpN/d2HxmZmbLKgcmyf8C9wIzytJXy9OaDEzMrLKug/5RleVOvmC/qizXzJaMr8pJRIUrX4ANgdntXBczM7O6Vdc9JpLGkwKSAEZJ+qQwuQPwJXxKxczMrN3UdWAC3JKftwb+AcwpTPuYNFj11nauk5mZWd2q68AkIs4GyLegvzEiPqpujczMzOpbXQcmJRFxbbXrYGZmZh78CqTLgiWdLelFSR9J+rT4qHb9zMzM6oUDk+Rc4CjS/9h8BvwUuAx4Czi+ivUyMzOrKw5Mkm8CP4iIy4FPgTsi4kfAmcBeVa2ZmZlZHXFgknyedGt4SFfmrJlf3wvsXZUamZmZ1SEHJslrpD/SA5gI9MuvdwI+rEqNzMzM6pADk+Q2YI/8+g/A2ZJeAQYDV1WrUmZmZvXGlwsDEXFa4fUtkqaQ/i34xYi4q3o1MzMzqy8OTABJ60bEm6X3EfE48HgVq2RmZlaXfConmSbpTknflLRStStjZmZWr9xjkuwPfBu4ErhS0t+BvwIjIqLSvw6bLZaug/5R7SqYmdU095gAEfGviBhIumz4OGBt4B5giqRfV7NuZmZm9cSBSUFEfBQRN0bEgUAPYCbwkypXy8zMrG44MCmQ9DlJ35F0D/A0sBrwqypXy8zMrG54jAkgaT/gcOAA0g3Vbgb6RMQjVa2YmZlZnXFgktwM3EkaAHt3RHxS5fqYmZnVpboPTCQtDwwCbo2I16tdHzMzs3pW92NMcu/I+ThIMzMzq7q6D0yyx4Dtq10JMzOzeudeguRK4GJJGwJjgfeLEyPiyarUyszMrM64xyT5G9AV+C0wChhTeIxuTgGSukgaIWmCpOck/Tinry1puKSX8vNaOV2SLpE0UdIzkrYrlHVUzv+SpKNat6lmZma1yz0mSbdWKOMT4CcR8aSk1YCxkoYDA4H7I+ICSYNIA21PBfYBNs2PHYE/AztKWhs4E+gJRC5nWES83Qp1NDMzq2kOTICIeLUVypgOTM+v35M0AVgfOBDok7NdC4wkBSYHAtfl/+J5TNKakjrnvMMjYhZADm76AzcsaR3NzMxqnU/lZJL2kXSXpOcldclpx0raYzHK6gpsCzwOfD4HLaXgZb2cbX1gSmG2qTmtofTyZRwnaYykMTNnzmxpFc3MzGqSAxNA0uHATcBLpNM6HfOkDsDPWljWqsCtwEkR8W5jWSukRSPpCydEXBERPSOiZ6dOnVpSRTMzs5rlwCT5GfC9iPhv0liRksdIf+bXLJI6koKSIRHx95z8Rj5FQ36ekdOnAl0Ks28ATGsk3czMbJnnwCTZFHi0QvocYPXmFCBJwNXAhIj4bWHSMKB0Zc1RwB2F9CPz1Tm9gNn5VM8/gb0lrZWv4Nk7p5mZmS3zPPg1mQZ8GSgfBNsbmNTMMnYBjgDGSxqX004HLgBukvRd4DXg0DztbmBfYCLwAXA0QETMknQuCy5TPqc0ENbMzGxZ58AkuQK4RNKx+X0XSbsBvwbOak4BEfEwlceHACwygDZfjXNCA2VdA1zTnOWamZktSxyYABHxa0lrAMOBlYARwFzg4oi4rKqVMzMzqyMOTLKI+Lmk84AtSWNvno+IOVWulpmZWV3x4NeCiPggIsYA44Fekr5U7TqZmZnVEwcmgKTBko7Pr1cg3RjtX8ALkvapauXMzMzqiAOTpB/pniUABwBrAF8gDXw9qzpVMjMzqz8OTJK1WHDjs/7ALRExAxhKGnNiZmZm7cCBSfIfYGtJHUi9J/fl9FWBeVWrlZmZWZ3xVTnJNcCNpButfQrcn9N3BP5drUqZmZnVGwcmQEScI+k5YEPg5oj4OE/6BLiwejUzMzOrLw5Msoi4tULatdWoi5mZWb3yGJNM0naSrpM0Jj/+Kmm7atfLzMysnjgwASQdTvrTvM6kP9e7G/g88ISk71SzbmZmZvXEp3KS84BfRMT/FBMlnQb8Cri+KrUyMzOrM+4xSToBN1VIvxlYr53rYmZmVrccmCQjgD4V0vsAo9q1JmZmZnWsbk/lSDq48PYe4HxJPVlwa/pewMH4lvRmZmbtpm4DE+CWCmnH5UfRH4E/tX11zMzMrG4Dk4jwaSwzM7Ma4y9nMzMzqxkOTDJJ+0l6UNKbkmZKGiVp32rXy8zMrJ44MAEkHQvcBkwCTgUGAa8At0k6ppp1MzMzqyd1O8akzKnAyRFxaSHtakljSUHKNdWplpmZWX1xj0myIXBvhfR7gC+1c13MzMzqlgOT5DVgrwrpewOvtnNdzMzM6pYDk+Ri4A+SrpR0tKSBkq4CfpenNUnSNZJmSHq2kHaWpNcljcuPfQvTTpM0UdILkvoV0vvntImSBrViG83MzGqex5gAEXG5pBnAT0h3ewWYAHwzIu5oZjGDgUuB68rSfxcRCwU3krYEDgO2Ar4I3Cfpy3nyZaTem6nAaEnDIuL5FjbJzMxsqeTAJIuI20hX5izu/A9K6trM7AcCQyNiLvCKpInADnnaxIh4GUDS0JzXgYmZmdUFn8ppez+U9Ew+1bNWTlsfmFLIMzWnNZS+CEnHSRojaczMmTPbot5mZmbtzoFJ2/ozsDHQA5gO/Canq0LeaCR90cSIKyKiZ0T07NSpU2vU1czMrOp8KqcNRcQbpdeSrgTuym+nAl0KWTcApuXXDaWbmZkt89xj0oYkdS68PQgoXbEzDDhM0oqSugGbAk8Ao4FNJXWTtAJpgOyw9qyzmZlZNdVtj4mkT4HOETFD0jXAjyPivSUo7wagD7CupKnAmUAfST1Ip2MmA98HiIjnJN1EGtT6CXBCRHyay/kh8E+gA3BNRDy3uHUyMzNb2tRtYAJ8CKwKzACOIt2WfrEDk4j4VoXkqxvJfx5wXoX0u4G7F7ceZmZmS7N6DkweAW7P/4cj4BJJH1bKGBH+Iz8zM7N2UM+ByRHAKcAmpFMt6wBzq1ojMzOzOle3gUm+YuanAJJeAb4VEW9Vt1ZmZmb1rW4Dk6KI6FbtOpiZmZkvF55P0n6SHpT0pqSZkkYV/3TPzMzM2p4DE0DSsaT/yZlEujpnEPAKcJskD3w1MzNrJz6Vk5wKnBwRlxbSrs5X7AwCrqlOtczMzOqLe0ySDYF7K6TfA3ypnetiZmZWtxyYJK8Be1VI3xt4tZ3rYmZmVrd8Kie5GPijpO1IN14LYFfSvU5OrGbFzMzM6okDEyAiLpc0A/gJcHBOngB8MyLuqF7NzMzM6osDkywibiNdmWNmZmZV4jEmZmZmVjMcmJiZmVnNcGBiZmZmNcOBiZmZmdWMug9MJHWU9LikzapdFzMzs3pX94FJRMwDupHuXWJmZmZVVPeBSXYt8L1qV8LMzKze+T4myeeAwyXtBYwF3i9OjIgfVaVWZmZmdcaBSbIF8GR+vVHZNJ/iMTMzaycOTICI6FvtOpiZmZnHmCxE0rqSdpS0YrXrYmZmVo8cmACSVpN0MzCD9O/C6+f0v0g6q5p1MzMzqycOTJILgS8C2wEfFtLvAg6qSo3MzMzqkAOT5ADgpIgYx8KDXSew6GDYiiRdI2mGpGcLaWtLGi7ppfy8Vk6XpEskTZT0jKTtCvMclfO/JOmo1mmemZnZ0sGBSbIW8FaF9NWAT5tZxmCgf1naIOD+iNgUuD+/B9gH2DQ/jgP+DCmQAc4EdgR2AM4sBTNmZmb1wIFJMprUa1JS6jX5PmnMSZMi4kFgVlnygaSbt5Gfv1FIvy6Sx4A1JXUG+gHDI2JWRLwNDGfRYMfMzGyZ5cuFk9OBf0rairROTs6vdwB6L0G5n4+I6QARMV3Sejl9fWBKId/UnNZQ+iIkHUfqbWHDDTdcgiqaLZu6DvpH1ZY9+YL9qrZss6Wde0yAiHgE2BlYAZgE7AFMA3aKiCcbm3cxqVI1GklfNDHiiojoGRE9O3Xq1KqVMzMzqxb3mGQRMR5o7cGmb0jqnHtLOpMuR4bUE9KlkG8DUiA0FehTlj6yletkZmZWs9xjkklaSdIxki7Oj2MkrbyExQ5jQbBzFHBHIf3IfHVOL2B2PuXzT2BvSWvlQa975zQzM7O64B4TIF+uexewMjA+Jx8DnCdpv+aczpF0A6m3Y11JU0lX11wA3CTpu8BrwKE5+93AvsBE4APgaICImCXpXNJgXIBzIqJ8QK2Zmdkyy4FJcgXwMHB0RLwPIOlzwDV5Ws+mCoiIbzUwaY8KeQM4oYFyrsnLNTMzqzsOTJKtgCNLQQlARLwv6RxgTPWqZWZmVl88xiT5N+mW9OU6Ay+2c13MzMzqVt32mOS7rJacAVySe0gey2m9cvqg8nnNzMysbdRtYAK8ycL3CBHwt0Ja6Z4idwAd2rFeZmZmdaueA5O+1a6AmZmZLaxuA5OIGFXtOlh1tPetyoe+nP4f8rAq3iLdzGxpUbeBSTlJKwBbA+tRNig4Iu6uSqXMzMzqjAMTQNJewF9JQUm5wGNMzMzM2oUvF04uI935tRuwCukOsKXHKlWsl5mZWV1xj0nSGfifiHi12hUxMzOrZ+4xSe4Cdq52JczMzOqde0ySHwBDJG0PPAvMK06MiOuqUiszM7M648Ak6Uf6s719Sf/2W7zxWgAOTMzMzNqBT+UkFwOXAqtFxKoRsVrhsXq1K2dmZlYvHJgkawJ/Kf67sJmZmbU/BybJrcCe1a6EmZlZvfMYk+Rl4DxJvYFnWHTw62+rUiszM7M648AkOQZ4j3TJcPllwwE4MDEzM2sHDkyAiOhW7TqYmZmZx5iYmZlZDXGPCSDpksamR8SP2qsuZmZm9cyBSbJN2fuOwOak9fNk+1fHzMysPjkwASKib3mapJWAq4GH2r9GZmZm9cljTBoQER8B5wE/r3ZdzMzM6oUDk8Z1AlZd0kIkTZY0XtI4SWNy2tqShkt6KT+vldMl6RJJEyU9I2m7JV2+mZnZ0sKncgBJJ5cnAZ2Bw4G7W2kxfSPizcL7QcD9EXGBpEH5/anAPsCm+bEj8Of8bGZmtsxzYJKcWPb+M2Am8L/A+W20zAOBPvn1tcBIUmByIHBdRATwmKQ1JXWOiOltVA8zM7Oa4cCEdrnBWgD/khTA5RFxBfD5UrAREdMlrZfzrg9MKcw7NactFJhIOg44DmDDDTds4+qbmZm1Dwcm7WOXiJiWg4/hkv7dSF5VSItFElJwcwVAz549F5luZma2NHJgkkkaAOwBrEfZoOCIOGBJyo6Iafl5hqTbgB2AN0qnaCR1Bmbk7FOBLoXZNwCmLcnyzczMlha+KgeQdBFwPdAVeAd4q+yxJGV/TtJqpdfA3sCzwDDgqJztKOCO/HoYcGS+OqcXMNvjS8zMrF64xyQ5EvhWRNzSBmV/HrhNEqT1/beIuFfSaOAmSd8FXgMOzfnvBvYFJgIfAEe3QZ3MzMxqkgOTZDlgXFsUHBEvA90rpL9FOnVUnh7ACW1RFzMzs1rnUznJFcB3ql0JMzOzeucek2RN4NuS9gKeAeYVJ/rfhc3MzNqHA5NkSxacytm8bJovxTUzM2snDkyo/O/CZmZm1v48xsTMzMxqhgMTMzMzqxkOTMzMzKxmODAxMzOzmuHAxMzMzGqGAxMzMzOrGQ5MzMzMrGY4MDEzM7Oa4RusmZm1sq6D/lGV5U6+YL+qLNesNbnHxMzMzGqGAxMzMzOrGQ5MzMzMrGZ4jIlVTbXOw5uZWe1yj4mZmZnVDAcmZmZmVjMcmJiZmVnNcGBiZmZmNcOBiZmZmdUMByZmZmZWMxyYmJmZWc1wYFKjJPWX9IKkiZIGVbs+ZmZm7cE3WKtBkjoAlwF7AVOB0ZKGRcTz1a2ZmdUy/3mgLQvcY1KbdgAmRsTLEfExMBQ4sMp1MjMza3PuMalN6wNTCu+nAju2xYJ8W3gzM6slDkxqkyqkxUIZpOOA4/LbOZJeaPNaLbAu8GY7Lq/WtKj9O5VeXLh/m1SmnXnb12/7G2y7LmznmrSeL1W7ArYoBya1aSrQpfB+A2BaMUNEXAFc0Z6VKpE0JiJ6VmPZtaCe21/PbYf6bn89t93al8eY1KbRwKaSuklaATgMGFblOpmZmbU595jUoIj4RNIPgX8CHYBrIuK5KlfLzMyszTkwqVERcTdwd7Xr0YCqnEKqIfXc/npuO9R3++u57daOFBFN5zIzMzNrBx5jYmZmZjXDgYnNJ2ltScMlvZSf12og31E5z0uSjiqkby9pfL6N/iWSVDbfKZJC0rpt3ZaWaqu2S7pI0r8lPSPpNklrtlebmqOpvz6QtKKkG/P0xyV1LUw7Lae/IKlfc8usFa3ddkldJI2QNEHSc5J+3H6taZm22O55WgdJT0m6q+1bYcusiPDDDyIC4NfAoPx6EHBhhTxrAy/n57Xy67XytCdIt+0QcA+wT2G+LqTBvK8C61a7re3VdmBvYPn8+sJK5VaxzR2AScBGwArA08CWZXmOB/6SXx8G3Jhfb5nzrwh0y+V0aE6ZtfBoo7Z3BrbLeVYDXqyXthfmOxn4G3BXtdvpx9L7cI+JFR0IXJtfXwt8o0KefsDwiJgVEW8Dw4H+kjoDq0fEoxERwHVl8/8O+BllN4qrIW3S9oj4V0R8kud/jHRPmlrRnL8+KK6XW4A9cm/QgcDQiJgbEa8AE3N5S8vfKbR62yNiekQ8CRAR7wETSHdxrjVtsd2RtAGwH3BVO7TBlmEOTKzo8xExHSA/r1chT6Xb5a+fH1MrpCPpAOD1iHi6LSrdStqk7WWOIfWm1IqG2lMxTw6wZgPrNDJvc8qsBW3R9vnyqY9tgcdbsc6tpa3a/nvSj4/PWr/KVk98uXCdkXQf8IUKk37e3CIqpEVD6ZJWyWXv3czy20x7t71s2T8HPgGGNHNZ7aHJejeSp6H0Sj92arGXrC3anmaSVgVuBU6KiHcXu4Ztp9XbLml/YEZEjJXUZwnrZ3XOgUmdiYg9G5om6Q1JnSNiej49MaNCtqlAn8L7DYCROX2DsvRpwMakc9FP5/GgGwBPStohIv6zBE1psSq0vVT2UcD+wB75VE+taPKvDwp5pkpaHlgDmNXEvE2VWQvapO2SOpKCkiER8fe2qfoSa4u2HwAcIGlfYCVgdUnXR8R32qYJtkyr9iAXP2rnAVzEwgNAf10hz9rAK6TBn2vl12vnaaOBXiwYALpvhfknU5uDX9uk7UB/4HmgU7XbWKE9y5MG8HZjwSDIrcrynMDCgyBvyq+3YuFBkC+TBlU2WWYtPNqo7SKNL/p9tdvX3m0vm7cPHvzqxxI8ql4BP2rnQTqHfD/wUn4ufen2BK4q5DuGNOhtInB0Ib0n8CxppP6l5Bv4lS2jVgOTNml7zjcFGJcff6l2W8vavS/p6pFJwM9z2jnAAfn1SsDNuR1PABsV5v15nu8FFr4Ca5Eya/HR2m0HdiWd7nimsL0XCc5r4dEW270w3YGJH0v08J1fzczMrGb4qhwzMzOrGQ5MzMzMrGY4MDEzM7Oa4cDEzMzMaoYDEzMzM6sZDkzMyki6S9LgwvuRki4tvF9F0i2SZud/S+5aKa0KVW9TuZ0hqWe162Jmyy4HJmZNOxg4rfD+GKA36b4VnUn3KamUtqyZQmrbuGpXpFok9cnB2bptuIzJkk5pq/LNap1vSW/WhIiYVZa0CTAhIsaXEiQtkrY4JHWMiHlLUkZbiYhPgXb9G4FaImmFJZy/ZretWS1xj4nVtXwKZrCkOfn/ck6vkGf+qRxJI4EfA73zL+eRldJy3hUkXShpqqT3JY2W1K9QbunX976SnpD0MdAvT/u6pLGSPpL0iqTzil+M+Vf1GZIul/RuXsZPy+q9uqQ/S5qey5kgaUBh+s6SRkn6QNLrOe/qjayrhU7lFOq/h6THczljJG3XxDpfQdL/SHpV0lxJL0v6UWF671zeR3mb/K6s7SMl/SmX8aakGZIulrRcnn6+pLEVlvuIpD8U3h8t6fm8nBcl/XepjDw9JJ0g6e+S3gf+BozIk2fm6YMbaGPFbStpY0l3SPpP3ieeVPoDvPltA74EXJTnL/45YIu2l9lSq9q3nvXDj2o+gD8Br5MCgq1Jt+F+FxhcyDMSuDS/Xhu4BniE9E/Fa1dKy3mHAI+RTvFsBPwQ+Bjonqf3Id3CfDzp35c3AjrlurwLHE36E8S+pNt/X1yo02TgrVzmJsCJuayd8nQB/0f6n57+uex9gIPy9G2AOcBPgE2BHYFHgVsaWVdd8zJ6ltX/iVzHzYF/AhOo8HcEhXJuIP0Z3CG5Xn2BI/O09YH3gb8AW5D+/PA/wG/Ktsds0i3Uvwx8k/TPzd/K07fM9dq8ME+3nPbV/P57wHTgv/K0r+fl/LAwT5D+zPHYXM+NSaf1Ii/jC8AaDbSxoW3bHfhBXv+bkG7v/nGprqR9aQpwdi7/C4u7vfzwY2l9VL0CfvhRrQewKjAXOLws7R0aCEzy+0uBkWVlLZSWv8Q+AzYsy3c78Kf8uvTldUhZngeBX5SlfSN/MZX+RmIycENZnpeAM/LrvfLyt2ig7dcBV5el9cj1Wa+BebpSOTDpV8izS07boIEyNs3T+zcw/TzS/7MsV0gbmLfTKoXt8WjZfMNZ+D+NngLOLbw/A3ih8P414IiyMk4Cni+8D+CPZXlKbW70/54a2rYN5H2stN0K2/aUJd1efvixtD48xsTq2cakf1d9tJQQEXMkLdE4kWw7Uq/F85KK6SsCD5TlHVP2fntgB0mnFtKWA1Ym/YqentOeKZtvGrBefr0tMD0iJjRQv+2BTYqndnJ9Ia2XGQ3MV0mxHtPy83qkXpFy25ICphEVpkHqJXk0Ij4rpD1M2k6bFJbVWNsBrgeOB36R3x+e05DUCegCXC7pz4V5lmfBOigp3zYttdD8kj4HnEnqCeoMdCT9YV55e8q15vYyq2kOTKyelX8JtablyKcOgPIBjx+WvX+/wrxnk04rlZtZeF1ebrBg3FhTbVsOuAr4XYVprzcxb7liPUpjIhoav9ZUvVQoo1wxvbG2QxoP8mtJO5F6WzYnnVor1u0HpNNvjSnfNi1VPv/FpFNrp5B6uD4g9YY0NbC2NbeXWU1zYGL1bCLpC64X8DLM/0W7Nelv3ZfEU6Qv2S9EREO9Aw15kjTmYOISLP9JoLOkLRroNXkS2GoJl7G49VqONK7k3grTnwe+KWm5Qq/JrqRxGM3eJhExXdIDpJ6SucAjEfFynvaGpNeBjSPiuhbW/+P83KGF85XsClwXEbcCSFqJ1OPxYtkyysuv1vYya3e+KsfqVkTMAa4GLpS0l6StSINYF/dLp1j2i6Rf6IMl/ZekjST1lHSKpIObmP0c4NuSzpG0taTNcxm/bkEV7gceB26V1E9St9zGb+TpF5JOF/1F0raSNpG0v6TLW9zYFoiIl4CbgKskHZLrtZukI3KWPwFfBP4kaQtJ+wEXkMb4fNDCxV0PDAAOy6+LzgJ+lq/E2Syv5yMlnVZeSJlXSb0z+0nqJGnVFtbpReAgSdtJ2ibXa6WyPJOB3SStrwX3S6nK9jKrBgcmVu9OIY13uC0/P0safNoajgb+F/g18G/gLtIVOq82NlNE/BPYj9Sr8ER+DCIN2GyW3NuwD+nKnOtJV8r8gXzKICKeyXXpCowCngbOB95o7jKWwJGkUy2XkNbLYGCNXK/Xc723Jd3I7RrSVTyLXMbdDLcCq5CuhrmpOCEiriLdFO8IUtsfAo4DXmmswFy/M0mDdN8gDXpuiZNJ40EeAu4hDXx9qCzPL0ljYCaRT91VeXuZtavSCH8zMzOzqnOPiZmZmdUMByZmZmZWMxyYmJmZWc1wYGJmZmY1w4GJmZmZ1QwHJmZmZlYzHJiYmZlZzXBgYmZmZjXDgYmZmZnVjP8P+iSGECjcAd4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "plt.hist(p_diffs)\n",
    "plt.axvline(obs_diffs, c='red', label='diff conversion in ab_data')\n",
    "plt.legend()\n",
    "fig.suptitle('Difference in convert rate between old and new pages distribution', fontsize=16)\n",
    "plt.xlabel('difference in convert rate', fontsize=14)\n",
    "plt.ylabel('number of observations', fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "j. What proportion of the **p_diffs** are greater than the actual difference observed in **ab_data.csv**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9061"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(p_diffs > obs_diffs).mean() # pnew - pold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "k. In words, explain what you just computed in part **j.**.  What is this value called in scientific studies?  What does this value mean in terms of whether or not there is a difference between the new and old pages?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> From j, the value, 0.9061, called p-value. At a Type I error rate of 5%, we cannot reject the null hypothesis that is the old page better than the new page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "l. We could also use a built-in to achieve similar results.  Though using the built-in might be easier to code, the above portions are a walkthrough of the ideas that are critical to correctly thinking about statistical significance. Fill in the below to calculate the number of conversions for each page, as well as the number of individuals who received each page. Let `n_old` and `n_new` refer the the number of rows associated with the old page and new pages, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(17489, 17264, 145274, 145310)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import statsmodels.api as sm\n",
    "\n",
    "convert_old = len(df2.query('converted == 1 & group == \"control\"')['user_id'])\n",
    "convert_new = len(df2.query('converted == 1 & group == \"treatment\"')['user_id'])\n",
    "n_old = nold\n",
    "n_new = nnew\n",
    "convert_old, convert_new, n_old, n_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "m. Now use `stats.proportions_ztest` to compute your test statistic and p-value.  [Here](http://knowledgetack.com/python/statsmodels/proportions_ztest/) is a helpful link on using the built in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "zscore, pvalue = sm.stats.proportions_ztest([convert_old, convert_new], [n_old, n_new], alternative = 'smaller', prop_var = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.3109241984234394, 0.9050583127590245)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zscore, pvalue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.6448536269514722"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.stats import norm\n",
    "norm.ppf(1-(0.05))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "n. What do the z-score and p-value you computed in the previous question mean for the conversion rates of the old and new pages?  Do they agree with the findings in parts **j.** and **k.**?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The z-score does not exceed 1.645 and the p-value, 0.905, is larger than the Type I error rate of 5%, so we cannot reject the null hypothesis. The p-value from j and k, 0.9061 gives us the same conclusion from this experiment. That is, the old page is better than the new page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='regression'></a>\n",
    "### Part III - A regression approach\n",
    "\n",
    "`1.` In this final part, you will see that the result you acheived in the previous A/B test can also be acheived by performing regression.<br><br>\n",
    "\n",
    "a. Since each row is either a conversion or no conversion, what type of regression should you be performing in this case?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. The goal is to use **statsmodels** to fit the regression model you specified in part **a.** to see if there is a significant difference in conversion based on which page a customer receives.  However, you first need to create a column for the intercept, and create a dummy variable column for which page each user received.  Add an **intercept** column, as well as an **ab_page** column, which is 1 when an individual receives the **treatment** and 0 if **control**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. add intercept\n",
    "df2['intercept'] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>is_duplicate</th>\n",
       "      <th>intercept</th>\n",
       "      <th>control</th>\n",
       "      <th>treatment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   is_duplicate  intercept  control  treatment  \n",
       "0         False          1        1          0  \n",
       "1         False          1        1          0  \n",
       "2         False          1        0          1  \n",
       "3         False          1        0          1  \n",
       "4         False          1        1          0  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 2. add a dummy variable for control and treatment groups\n",
    "ab_page = pd.get_dummies(df2['group'])\n",
    "df2 = df2.join(ab_page)\n",
    "df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2 = df2.rename(columns={'treatment': 'ab_page'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>is_duplicate</th>\n",
       "      <th>intercept</th>\n",
       "      <th>control</th>\n",
       "      <th>ab_page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   is_duplicate  intercept  control  ab_page  \n",
       "0         False          1        1        0  \n",
       "1         False          1        1        0  \n",
       "2         False          1        0        1  \n",
       "3         False          1        0        1  \n",
       "4         False          1        1        0  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. Use **statsmodels** to import your regression model.  Instantiate the model, and fit the model using the two columns you created in part **b.** to predict whether or not an individual converts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366118\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "model = sm.Logit(df2['converted'],df2[['intercept','ab_page']]).fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. Provide the summary of your model below, and use it as necessary to answer the following questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>     <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>               <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290582</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>               <td>MLE</td>       <th>  Df Model:          </th>   <td>     1</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>          <td>Sat, 19 Jan 2019</td> <th>  Pseudo R-squ.:     </th>  <td>8.077e-06</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>              <td>12:18:39</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>           <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th> </th>                      <td> </td>        <th>  LLR p-value:       </th>   <td>0.1899</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   -1.9888</td> <td>    0.008</td> <td> -246.669</td> <td> 0.000</td> <td>   -2.005</td> <td>   -1.973</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>   -0.0150</td> <td>    0.011</td> <td>   -1.311</td> <td> 0.190</td> <td>   -0.037</td> <td>    0.007</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290582\n",
       "Method:                           MLE   Df Model:                            1\n",
       "Date:                Sat, 19 Jan 2019   Pseudo R-squ.:               8.077e-06\n",
       "Time:                        12:18:39   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "                                        LLR p-value:                    0.1899\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -1.9888      0.008   -246.669      0.000      -2.005      -1.973\n",
       "ab_page       -0.0150      0.011     -1.311      0.190      -0.037       0.007\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e. What is the p-value associated with **ab_page**? Why does it differ from the value you found in the **Part II**?<br><br> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> For the hypothesis test in logistic regression, we calculate odds ratio (log(p/1-p)) and test that there is any relationship between convert rates and pages of groups. Here, p-value is 0.19 which means there is no relationship between convert rates and pages of groups at a Type I error of 5%. For the hypothesis tests in ab test and two sample proportion, we calculate the proportions in two groups and test that there is any difference of convert rates between new pages and old pages. Here, p-value is 0.9 means that we cannot reject the null hypothesis, the convert rate in the old page is better than in the new page at a Type I error of 5%. Because of different calculations, our p-values are different but we get the same conclusion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "f. Now, you are considering other things that might influence whether or not an individual converts.  Discuss why it is a good idea to consider other factors to add into your regression model.  Are there any disadvantages to adding additional terms into your regression model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> After implemeting the logistic regression, we cannot prove that new page has the better convert rate from the current dataset, so it is a good idea to find if there are other factors that influence the convert rate.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "g. Now along with testing if the conversion rate changes for different pages, also add an effect based on which country a user lives. You will need to read in the **countries.csv** dataset and merge together your datasets on the approporiate rows.  [Here](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.join.html) are the docs for joining tables. \n",
    "\n",
    "Does it appear that country had an impact on conversion? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>834778</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>928468</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>822059</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>711597</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>710616</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id country\n",
       "0   834778      UK\n",
       "1   928468      US\n",
       "2   822059      UK\n",
       "3   711597      UK\n",
       "4   710616      UK"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfcountries = pd.read_csv('countries.csv')\n",
    "dfcountries.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>is_duplicate</th>\n",
       "      <th>intercept</th>\n",
       "      <th>control</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   is_duplicate  intercept  control  ab_page country  \n",
       "0         False          1        1        0      US  \n",
       "1         False          1        1        0      US  \n",
       "2         False          1        0        1      US  \n",
       "3         False          1        0        1      US  \n",
       "4         False          1        1        0      US  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3 = df2.join(dfcountries.set_index('user_id'), on='user_id')\n",
    "df3.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "US    203619\n",
       "UK     72466\n",
       "CA     14499\n",
       "Name: country, dtype: int64"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3['country'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>is_duplicate</th>\n",
       "      <th>intercept</th>\n",
       "      <th>control</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>country</th>\n",
       "      <th>CA</th>\n",
       "      <th>UK</th>\n",
       "      <th>US</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   is_duplicate  intercept  control  ab_page country  CA  UK  US  \n",
       "0         False          1        1        0      US   0   0   1  \n",
       "1         False          1        1        0      US   0   0   1  \n",
       "2         False          1        0        1      US   0   0   1  \n",
       "3         False          1        0        1      US   0   0   1  \n",
       "4         False          1        1        0      US   0   0   1  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3[['CA','UK','US']] = pd.get_dummies(df3['country'])\n",
    "df3.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366113\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "model2 = sm.Logit(df3['converted'],df3[['intercept','ab_page','UK','CA']]).fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>     <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>               <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290580</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>               <td>MLE</td>       <th>  Df Model:          </th>   <td>     3</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>          <td>Sat, 19 Jan 2019</td> <th>  Pseudo R-squ.:     </th>  <td>2.323e-05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>              <td>13:24:40</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>           <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th> </th>                      <td> </td>        <th>  LLR p-value:       </th>   <td>0.1760</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   -1.9893</td> <td>    0.009</td> <td> -223.763</td> <td> 0.000</td> <td>   -2.007</td> <td>   -1.972</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>   -0.0149</td> <td>    0.011</td> <td>   -1.307</td> <td> 0.191</td> <td>   -0.037</td> <td>    0.007</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>        <td>    0.0099</td> <td>    0.013</td> <td>    0.743</td> <td> 0.457</td> <td>   -0.016</td> <td>    0.036</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CA</th>        <td>   -0.0408</td> <td>    0.027</td> <td>   -1.516</td> <td> 0.130</td> <td>   -0.093</td> <td>    0.012</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290580\n",
       "Method:                           MLE   Df Model:                            3\n",
       "Date:                Sat, 19 Jan 2019   Pseudo R-squ.:               2.323e-05\n",
       "Time:                        13:24:40   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "                                        LLR p-value:                    0.1760\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -1.9893      0.009   -223.763      0.000      -2.007      -1.972\n",
       "ab_page       -0.0149      0.011     -1.307      0.191      -0.037       0.007\n",
       "UK             0.0099      0.013      0.743      0.457      -0.016       0.036\n",
       "CA            -0.0408      0.027     -1.516      0.130      -0.093       0.012\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> After adding the country variable, p-values in UK and CA are 0.457 and 0.13 respectively, which are larger than type I error, 0.05,  which means country does not have an significant impact on convert rate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "h. Though you have now looked at the individual factors of country and page on conversion, we would now like to look at an interaction between page and country to see if there significant effects on conversion.  Create the necessary additional columns, and fit the new model.  \n",
    "\n",
    "Provide the summary results, and your conclusions based on the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>is_duplicate</th>\n",
       "      <th>intercept</th>\n",
       "      <th>control</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>country</th>\n",
       "      <th>CA</th>\n",
       "      <th>UK</th>\n",
       "      <th>US</th>\n",
       "      <th>page_CA</th>\n",
       "      <th>page_UK</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   is_duplicate  intercept  control  ab_page country  CA  UK  US  page_CA  \\\n",
       "0         False          1        1        0      US   0   0   1        0   \n",
       "1         False          1        1        0      US   0   0   1        0   \n",
       "2         False          1        0        1      US   0   0   1        0   \n",
       "3         False          1        0        1      US   0   0   1        0   \n",
       "4         False          1        1        0      US   0   0   1        0   \n",
       "\n",
       "   page_UK  \n",
       "0        0  \n",
       "1        0  \n",
       "2        0  \n",
       "3        0  \n",
       "4        0  "
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3['page_CA'] = df3['ab_page'] * df3['CA']\n",
    "df3['page_UK'] = df3['ab_page'] * df3['UK']\n",
    "df3.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366109\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "model3 = sm.Logit(df3['converted'],df3[['intercept','ab_page','UK','CA','page_CA','page_UK']]).fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>     <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>               <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290578</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>               <td>MLE</td>       <th>  Df Model:          </th>   <td>     5</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>          <td>Sat, 19 Jan 2019</td> <th>  Pseudo R-squ.:     </th>  <td>3.482e-05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>              <td>13:36:59</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>           <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th> </th>                      <td> </td>        <th>  LLR p-value:       </th>   <td>0.1920</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   -1.9865</td> <td>    0.010</td> <td> -206.344</td> <td> 0.000</td> <td>   -2.005</td> <td>   -1.968</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>   -0.0206</td> <td>    0.014</td> <td>   -1.505</td> <td> 0.132</td> <td>   -0.047</td> <td>    0.006</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>        <td>   -0.0057</td> <td>    0.019</td> <td>   -0.306</td> <td> 0.760</td> <td>   -0.043</td> <td>    0.031</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CA</th>        <td>   -0.0175</td> <td>    0.038</td> <td>   -0.465</td> <td> 0.642</td> <td>   -0.091</td> <td>    0.056</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>page_CA</th>   <td>   -0.0469</td> <td>    0.054</td> <td>   -0.872</td> <td> 0.383</td> <td>   -0.152</td> <td>    0.059</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>page_UK</th>   <td>    0.0314</td> <td>    0.027</td> <td>    1.181</td> <td> 0.238</td> <td>   -0.021</td> <td>    0.084</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290578\n",
       "Method:                           MLE   Df Model:                            5\n",
       "Date:                Sat, 19 Jan 2019   Pseudo R-squ.:               3.482e-05\n",
       "Time:                        13:36:59   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "                                        LLR p-value:                    0.1920\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -1.9865      0.010   -206.344      0.000      -2.005      -1.968\n",
       "ab_page       -0.0206      0.014     -1.505      0.132      -0.047       0.006\n",
       "UK            -0.0057      0.019     -0.306      0.760      -0.043       0.031\n",
       "CA            -0.0175      0.038     -0.465      0.642      -0.091       0.056\n",
       "page_CA       -0.0469      0.054     -0.872      0.383      -0.152       0.059\n",
       "page_UK        0.0314      0.027      1.181      0.238      -0.021       0.084\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model3.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> After adding the interaction variables between pages and countries, the p-values in page_CA and page_UK are 0.383 and 0.238 which are larger than type I error, 0.05. That is, the interaction terms do not have an significant impacts on convert rate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "i. Find if time associates with the convert rates. Here, I will convert timestamp from string to datetime and then categorize it into weekdays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 290584 entries, 0 to 294477\n",
      "Data columns (total 15 columns):\n",
      "user_id         290584 non-null int64\n",
      "timestamp       290584 non-null datetime64[ns]\n",
      "group           290584 non-null object\n",
      "landing_page    290584 non-null object\n",
      "converted       290584 non-null int64\n",
      "is_duplicate    290584 non-null bool\n",
      "intercept       290584 non-null int64\n",
      "control         290584 non-null uint8\n",
      "ab_page         290584 non-null uint8\n",
      "country         290584 non-null object\n",
      "CA              290584 non-null uint8\n",
      "UK              290584 non-null uint8\n",
      "US              290584 non-null uint8\n",
      "page_CA         290584 non-null uint8\n",
      "page_UK         290584 non-null uint8\n",
      "dtypes: bool(1), datetime64[ns](1), int64(3), object(3), uint8(7)\n",
      "memory usage: 20.0+ MB\n"
     ]
    }
   ],
   "source": [
    "# Time\n",
    "df3['timestamp'] = pd.to_datetime(df3['timestamp'])\n",
    "df3.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "df3['Day of Week'] = df3['timestamp'].apply(lambda time: time.dayofweek)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    47148\n",
       "0    45440\n",
       "6    39915\n",
       "5    39669\n",
       "4    39626\n",
       "2    39565\n",
       "3    39221\n",
       "Name: Day of Week, dtype: int64"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3['Day of Week'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>is_duplicate</th>\n",
       "      <th>intercept</th>\n",
       "      <th>control</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>country</th>\n",
       "      <th>...</th>\n",
       "      <th>page_CA</th>\n",
       "      <th>page_UK</th>\n",
       "      <th>Day of Week</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                  timestamp      group landing_page  converted  \\\n",
       "0   851104 2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228 2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590 2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541 2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975 2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   is_duplicate  intercept  control  ab_page country ...  page_CA  page_UK  \\\n",
       "0         False          1        1        0      US ...        0        0   \n",
       "1         False          1        1        0      US ...        0        0   \n",
       "2         False          1        0        1      US ...        0        0   \n",
       "3         False          1        0        1      US ...        0        0   \n",
       "4         False          1        1        0      US ...        0        0   \n",
       "\n",
       "   Day of Week  0  1  2  3  4  5  6  \n",
       "0            5  0  0  0  0  0  1  0  \n",
       "1            3  0  0  0  1  0  0  0  \n",
       "2            2  0  0  1  0  0  0  0  \n",
       "3            6  0  0  0  0  0  0  1  \n",
       "4            5  0  0  0  0  0  1  0  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dayofweek = pd.get_dummies(df3['Day of Week'])\n",
    "df3 = df3.join(dayofweek)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>is_duplicate</th>\n",
       "      <th>intercept</th>\n",
       "      <th>control</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>country</th>\n",
       "      <th>...</th>\n",
       "      <th>page_CA</th>\n",
       "      <th>page_UK</th>\n",
       "      <th>Day of Week</th>\n",
       "      <th>Mon</th>\n",
       "      <th>Tue</th>\n",
       "      <th>Wed</th>\n",
       "      <th>Thu</th>\n",
       "      <th>Fri</th>\n",
       "      <th>Sat</th>\n",
       "      <th>Sun</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                  timestamp      group landing_page  converted  \\\n",
       "0   851104 2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228 2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590 2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541 2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975 2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   is_duplicate  intercept  control  ab_page country ...   page_CA  page_UK  \\\n",
       "0         False          1        1        0      US ...         0        0   \n",
       "1         False          1        1        0      US ...         0        0   \n",
       "2         False          1        0        1      US ...         0        0   \n",
       "3         False          1        0        1      US ...         0        0   \n",
       "4         False          1        1        0      US ...         0        0   \n",
       "\n",
       "   Day of Week  Mon  Tue  Wed  Thu  Fri  Sat  Sun  \n",
       "0            5    0    0    0    0    0    1    0  \n",
       "1            3    0    0    0    1    0    0    0  \n",
       "2            2    0    0    1    0    0    0    0  \n",
       "3            6    0    0    0    0    0    0    1  \n",
       "4            5    0    0    0    0    0    1    0  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3 = df3.rename(columns={0:'Mon', 1:'Tue', 2:'Wed', 3:'Thu', 4:'Fri', 5:'Sat', 6:'Sun'})\n",
    "df3.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366109\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "model4 = sm.Logit(df3['converted'],df3[['intercept','ab_page', 'Tue','Wed','Thu','Fri','Sat','Sun']]).fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>     <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>               <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290576</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>               <td>MLE</td>       <th>  Df Model:          </th>   <td>     7</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>          <td>Sun, 20 Jan 2019</td> <th>  Pseudo R-squ.:     </th>  <td>3.402e-05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>              <td>19:29:26</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>           <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th> </th>                      <td> </td>        <th>  LLR p-value:       </th>   <td>0.4044</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   -1.9742</td> <td>    0.015</td> <td> -127.752</td> <td> 0.000</td> <td>   -2.004</td> <td>   -1.944</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>   -0.0149</td> <td>    0.011</td> <td>   -1.306</td> <td> 0.192</td> <td>   -0.037</td> <td>    0.007</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Tue</th>       <td>   -0.0152</td> <td>    0.020</td> <td>   -0.752</td> <td> 0.452</td> <td>   -0.055</td> <td>    0.024</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Wed</th>       <td>   -0.0076</td> <td>    0.021</td> <td>   -0.362</td> <td> 0.717</td> <td>   -0.049</td> <td>    0.034</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Thu</th>       <td>   -0.0111</td> <td>    0.021</td> <td>   -0.526</td> <td> 0.599</td> <td>   -0.053</td> <td>    0.030</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Fri</th>       <td>   -0.0425</td> <td>    0.021</td> <td>   -1.999</td> <td> 0.046</td> <td>   -0.084</td> <td>   -0.001</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Sat</th>       <td>   -0.0030</td> <td>    0.021</td> <td>   -0.145</td> <td> 0.885</td> <td>   -0.044</td> <td>    0.038</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Sun</th>       <td>   -0.0253</td> <td>    0.021</td> <td>   -1.199</td> <td> 0.231</td> <td>   -0.067</td> <td>    0.016</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290576\n",
       "Method:                           MLE   Df Model:                            7\n",
       "Date:                Sun, 20 Jan 2019   Pseudo R-squ.:               3.402e-05\n",
       "Time:                        19:29:26   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "                                        LLR p-value:                    0.4044\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -1.9742      0.015   -127.752      0.000      -2.004      -1.944\n",
       "ab_page       -0.0149      0.011     -1.306      0.192      -0.037       0.007\n",
       "Tue           -0.0152      0.020     -0.752      0.452      -0.055       0.024\n",
       "Wed           -0.0076      0.021     -0.362      0.717      -0.049       0.034\n",
       "Thu           -0.0111      0.021     -0.526      0.599      -0.053       0.030\n",
       "Fri           -0.0425      0.021     -1.999      0.046      -0.084      -0.001\n",
       "Sat           -0.0030      0.021     -0.145      0.885      -0.044       0.038\n",
       "Sun           -0.0253      0.021     -1.199      0.231      -0.067       0.016\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model4.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> After building another logistic regression for groups of pages and weekdays, we can see that the p-value of Friday is 0.046 smaller than the Type I error of 5%, which means Friday has an significant impact on the convert rates.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='con'></a>\n",
    "### Conclusion  \n",
    "After performing the ab test, I conclude that the difference in the convert rates between the old page and the new page is very small and the p-value cannot reject the null hypothesis, the convert rate in the old page is better than in the new page. The result from the logistic regression also indicates that there is no statistically significant between the convert rate and groups of page. I add the country variable but the conclusion still remains the same. However, after categorizing the timestamp variable as weekdays and adding it to the model, the result shows that Friday has a significant impact on the convert rate. For the next step, we can investigate more on time, add other features and run the experiment longer to see if we can get more insights."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}