Author_Gender_Prediction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extracting author names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predicting author genders from OSP \"text\" table using the gender.py library, which is a python extension of genderize.io (https://genderize.io/)\n",
    "\n",
    "Update: ranked by top authors by top text (there are repeats of authors) because there is no separate author table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from v1_db import session, Text, Citation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sqlalchemy.sql import functions as func\n",
    "from collections import Counter\n",
    "import pandas as pd\n",
    "import requests\n",
    "from gender import getGenders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMPORT RPY2 PACKAGES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import rpy2.robjects as robjects\n",
    "import rpy2.robjects.packages as rpackages\n",
    "from rpy2.robjects.packages import importr\n",
    "base = importr('base')\n",
    "utils = importr('utils')\n",
    "utils = rpackages.importr('utils')\n",
    "utils.chooseCRANmirror(ind=1)\n",
    "wiki = importr('WikidataR')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "R Wikidata lookup function:\n",
    "- approach: lookup keyword id's in search result to identify id's and necessary properties (author and gender)\n",
    "- function uses Wikidata id's (P1006 = National Thesaurus for Author Names ID, Q6581097 = male, Q6581072 = female)\n",
    "- tryCatch to prevent program crash when running an invalid name\n",
    "- function is made global so it can be accessed later on in the program\n",
    "- function f performs lookup, function g (which will be used later in the program) calls f and handles exceptions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "R object with classes: ('function',) mapped to:\n",
       "<SignatureTranslatedFunction - Python:0x10f357088 / R:0x7fdbfa9eb0b0>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "robjects.r('''\n",
    "    f <- function(name){\n",
    "        all_results <- find_item(name)\n",
    "        \n",
    "        for(i in 1:length(all_results)){\n",
    "            temp <- get_item(all_results[[i]]$id)\n",
    "            if(regexpr('P1006', toString(temp)) != -1){\n",
    "                break\n",
    "            }\n",
    "            \n",
    "        }\n",
    "        str = toString(temp)\n",
    "  \n",
    "        x = regexpr('Q6581097', str)\n",
    "  \n",
    "        if (regexpr('Q6581097', str) != -1){\n",
    "            return(\"male\")          \n",
    "        }\n",
    "        else if(regexpr('Q6581072', str) != -1){\n",
    "            return(\"female\")\n",
    "        }\n",
    "        else{\n",
    "            return(\"other\")\n",
    "        }\n",
    "    }\n",
    "    \n",
    "    g <- function(n){\n",
    "        tryCatch(f(n), error=function(e) return(\"unknown\"))\n",
    "    }\n",
    "''')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#make r functions global\n",
    "r_f = robjects.globalenv['f']\n",
    "getGender = robjects.globalenv['g']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'female'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#getGender(\"Mark Twain\")\n",
    "getGender(\"JK Rowling\")[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Query V1 data for authors of top titles "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "count = func.count(Citation.text_id)\n",
    "\n",
    "authors = (session\n",
    "          .query(Text.authors)\n",
    "          .join(Citation)\n",
    "          .filter(Text.valid==True)\n",
    "          .filter(Text.display==True)\n",
    "          .group_by(Text.id)\n",
    "          .order_by(count.desc())\n",
    "          .limit(300))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#authors.all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create dataFrames to store author/gender data into, this can later be converted to csv file if necessary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "header = ['first', 'last', 'gender', 'proportion']\n",
    "df = pd.DataFrame(columns=header)\n",
    "\n",
    "df_lt90 = pd.DataFrame(columns=header)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function that further formats the names with odd formats (names in parenthases, multiple names in first name string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def format_name(name):\n",
    "    \n",
    "    #if alternate name in parenthases, take that name\n",
    "    if \"(\" in name:\n",
    "        temp = name.replace(\")\", \"\").split(\"(\")\n",
    "        name = temp[1]\n",
    "    \n",
    "    if \" \" in name:\n",
    "        temp = name.split(\" \")\n",
    "        name = temp[0]\n",
    "    \n",
    "    \n",
    "    return name\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Format name, find gender, add to table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = 0\n",
    "for a in authors.all():\n",
    "    \n",
    "    #format entry \n",
    "    entry = str(a)\n",
    "    split_entry = entry.replace(\".'],)\", \"\").replace(\"(['\", \"\").replace(\"'],)\", \"\").split(\", \")\n",
    "    \n",
    "    #separate first and last name\n",
    "    if len(split_entry)== 1:\n",
    "        first = split_entry[0]\n",
    "        test_name = first.replace(\"'\", \"\")\n",
    "        last = \"\"\n",
    "    else:\n",
    "        first = split_entry[1]\n",
    "        last = split_entry[0]\n",
    "        test_name = format_name(first).replace(\"'\", \"\")\n",
    "           \n",
    "    \n",
    "    #get gender \n",
    "    g = getGenders(test_name)\n",
    "    gender = g[0][0]\n",
    "    p = g[0][1]\n",
    "    \n",
    "    #input data into temporary dataframe \n",
    "    df1 = pd.DataFrame({'first': first, 'last': last, 'gender': gender, 'proportion': p}, index = [x])\n",
    "    \n",
    "    #add to necessary tables \n",
    "    #frames = [df, df1]\n",
    "    #df = pd.concat(frames)\n",
    "    \n",
    "    #add to correct proportion table\n",
    "    if float(p) >= .90:\n",
    "        frames = [df, df1]\n",
    "        df = pd.concat(frames)\n",
    "    else:\n",
    "        name = test_name + \" \" + last\n",
    "        gender = getGender(name)[0]\n",
    "        if gender == \"unknown\":\n",
    "            name = first + \" \" + last\n",
    "            gender = getGender(name)[0]\n",
    "        df1 = pd.DataFrame({'first': first, 'last': last, 'gender': gender, 'proportion': p}, index = [x])\n",
    "        frames2 = [df_lt90, df1]\n",
    "        df_lt90 = pd.concat(frames2)\n",
    "        \n",
    "    \n",
    "    x += 1\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>gender</th>\n",
       "      <th>proportion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>William</td>\n",
       "      <td>Strunk</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Plato</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>0.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Karl</td>\n",
       "      <td>Marx</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Neil A.</td>\n",
       "      <td>Campbell</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Mary Wollstonecraft</td>\n",
       "      <td>Shelley</td>\n",
       "      <td>female</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Aristotle</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Thomas</td>\n",
       "      <td>Hobbes</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Niccolò</td>\n",
       "      <td>Machiavelli</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>William</td>\n",
       "      <td>Shakespeare</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Homer</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Edward W</td>\n",
       "      <td>Said</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Kate L</td>\n",
       "      <td>Turabian</td>\n",
       "      <td>female</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Homer</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Joseph</td>\n",
       "      <td>Conrad</td>\n",
       "      <td>male</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Geoffrey</td>\n",
       "      <td>Chaucer</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Martin Luther</td>\n",
       "      <td>King</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>John Stuart</td>\n",
       "      <td>Mill</td>\n",
       "      <td>male</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Thomas S</td>\n",
       "      <td>Kuhn</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>John</td>\n",
       "      <td>Milton</td>\n",
       "      <td>male</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>John Stuart</td>\n",
       "      <td>Mill</td>\n",
       "      <td>male</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Samuel P</td>\n",
       "      <td>Huntington</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Chinua</td>\n",
       "      <td>Achebe</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Plato</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>0.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Michel</td>\n",
       "      <td>Foucault</td>\n",
       "      <td>male</td>\n",
       "      <td>0.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Charles</td>\n",
       "      <td>Darwin</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Aristotle</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Kate L</td>\n",
       "      <td>Turabian</td>\n",
       "      <td>female</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Henry David</td>\n",
       "      <td>Thoreau</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Harvey M.</td>\n",
       "      <td>Deitel</td>\n",
       "      <td>male</td>\n",
       "      <td>0.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Joseph</td>\n",
       "      <td>Gibaldi</td>\n",
       "      <td>male</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>265</th>\n",
       "      <td>Barbara</td>\n",
       "      <td>Ehrenreich</td>\n",
       "      <td>female</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>266</th>\n",
       "      <td>John D</td>\n",
       "      <td>Cutnell</td>\n",
       "      <td>male</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>267</th>\n",
       "      <td>Susan</td>\n",
       "      <td>Glaspell</td>\n",
       "      <td>female</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>269</th>\n",
       "      <td>Edward W</td>\n",
       "      <td>Said</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>270</th>\n",
       "      <td>George</td>\n",
       "      <td>Orwell</td>\n",
       "      <td>male</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>271</th>\n",
       "      <td>Steven C</td>\n",
       "      <td>Levi</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>272</th>\n",
       "      <td>Tennessee</td>\n",
       "      <td>Williams</td>\n",
       "      <td>female</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>273</th>\n",
       "      <td>Sonia</td>\n",
       "      <td>Gandhi</td>\n",
       "      <td>female</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>274</th>\n",
       "      <td>Albert</td>\n",
       "      <td>Camus</td>\n",
       "      <td>male</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>275</th>\n",
       "      <td>David'</td>\n",
       "      <td>Held</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>276</th>\n",
       "      <td>Kenneth H</td>\n",
       "      <td>Rosen</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>277</th>\n",
       "      <td>Gerard J</td>\n",
       "      <td>Tortora</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>278</th>\n",
       "      <td>Brian W</td>\n",
       "      <td>Kernighan</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>279</th>\n",
       "      <td>Jeremy</td>\n",
       "      <td>Black</td>\n",
       "      <td>male</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>280</th>\n",
       "      <td>Richard</td>\n",
       "      <td>Wright</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>281</th>\n",
       "      <td>Thomas</td>\n",
       "      <td>Pynchon</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>282</th>\n",
       "      <td>Arthur</td>\n",
       "      <td>Miller</td>\n",
       "      <td>male</td>\n",
       "      <td>0.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>283</th>\n",
       "      <td>Olivier (Olivier J.)</td>\n",
       "      <td>Blanchard</td>\n",
       "      <td>male</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>284</th>\n",
       "      <td>William</td>\n",
       "      <td>Blake</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>285</th>\n",
       "      <td>Ernest</td>\n",
       "      <td>Hemingway</td>\n",
       "      <td>male</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>286</th>\n",
       "      <td>John</td>\n",
       "      <td>McMurry</td>\n",
       "      <td>male</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>287</th>\n",
       "      <td>Paul G</td>\n",
       "      <td>Hewitt</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>288</th>\n",
       "      <td>Edmund</td>\n",
       "      <td>Spenser</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>289</th>\n",
       "      <td>Johann Sebastian</td>\n",
       "      <td>Bach</td>\n",
       "      <td>male</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>291</th>\n",
       "      <td>Roger</td>\n",
       "      <td>Kamien</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>292</th>\n",
       "      <td>Matthew</td>\n",
       "      <td>Arnold</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>293</th>\n",
       "      <td>Alexander</td>\n",
       "      <td>Hamilton</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294</th>\n",
       "      <td>John P.</td>\n",
       "      <td>Kotter</td>\n",
       "      <td>male</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>295</th>\n",
       "      <td>Robert M</td>\n",
       "      <td>Axelrod</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>298</th>\n",
       "      <td>Steven</td>\n",
       "      <td>Pinker</td>\n",
       "      <td>male</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>257 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                    first         last  gender proportion\n",
       "0                 William       Strunk    male          1\n",
       "1                   Plato                 male       0.93\n",
       "2                    Karl         Marx    male          1\n",
       "3                 Neil A.     Campbell    male          1\n",
       "4     Mary Wollstonecraft      Shelley  female          1\n",
       "5               Aristotle                 male          1\n",
       "6                  Thomas       Hobbes    male          1\n",
       "7                 Niccolò  Machiavelli    male          1\n",
       "9                 William  Shakespeare    male          1\n",
       "10                  Homer                 male          1\n",
       "11               Edward W         Said    male          1\n",
       "12                 Kate L     Turabian  female       0.99\n",
       "13                  Homer                 male          1\n",
       "14                 Joseph       Conrad    male       0.99\n",
       "15               Geoffrey      Chaucer    male          1\n",
       "17          Martin Luther         King    male          1\n",
       "18            John Stuart         Mill    male       0.99\n",
       "19               Thomas S         Kuhn    male          1\n",
       "20                   John       Milton    male       0.99\n",
       "21            John Stuart         Mill    male       0.99\n",
       "22               Samuel P   Huntington    male          1\n",
       "23                 Chinua       Achebe    male          1\n",
       "24                  Plato                 male       0.93\n",
       "25                 Michel     Foucault    male       0.96\n",
       "26                Charles       Darwin    male          1\n",
       "28              Aristotle                 male          1\n",
       "29                 Kate L     Turabian  female       0.99\n",
       "30            Henry David      Thoreau    male          1\n",
       "32              Harvey M.       Deitel    male       0.98\n",
       "33                 Joseph      Gibaldi    male       0.99\n",
       "..                    ...          ...     ...        ...\n",
       "265               Barbara   Ehrenreich  female          1\n",
       "266                John D      Cutnell    male       0.99\n",
       "267                 Susan     Glaspell  female          1\n",
       "269              Edward W         Said    male          1\n",
       "270                George       Orwell    male       0.99\n",
       "271              Steven C         Levi    male          1\n",
       "272             Tennessee     Williams  female          1\n",
       "273                 Sonia       Gandhi  female          1\n",
       "274                Albert        Camus    male       0.99\n",
       "275                David'         Held    male          1\n",
       "276             Kenneth H        Rosen    male          1\n",
       "277              Gerard J      Tortora    male          1\n",
       "278               Brian W    Kernighan    male          1\n",
       "279                Jeremy        Black    male       0.99\n",
       "280               Richard       Wright    male          1\n",
       "281                Thomas      Pynchon    male          1\n",
       "282                Arthur       Miller    male       0.98\n",
       "283  Olivier (Olivier J.)    Blanchard    male       0.99\n",
       "284               William        Blake    male          1\n",
       "285                Ernest    Hemingway    male       0.99\n",
       "286                  John      McMurry    male       0.99\n",
       "287                Paul G       Hewitt    male          1\n",
       "288                Edmund      Spenser    male          1\n",
       "289      Johann Sebastian         Bach    male       0.99\n",
       "291                 Roger       Kamien    male          1\n",
       "292               Matthew       Arnold    male          1\n",
       "293             Alexander     Hamilton    male          1\n",
       "294               John P.       Kotter    male       0.99\n",
       "295              Robert M      Axelrod    male          1\n",
       "298                Steven       Pinker    male          1\n",
       "\n",
       "[257 rows x 4 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "df_lt90: dataframe of names with a proportion of less than .9\n",
    "\n",
    "These names were run through wikidata to get gender identification\n",
    "\n",
    "Initial genderize.io proportion is still listed for reference, but gender assignment is from wikidata\n",
    "\n",
    "NOTES: \n",
    "- Tennessee Williams is incorrectly assgined (with female proportion of 1)\n",
    "- Leslie Silko unidentified (listed in Wikidata with full name, \"Leslie Silko\" does not produce a search result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>gender</th>\n",
       "      <th>proportion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Sophocles</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Sophocles</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Saint</td>\n",
       "      <td>Augustine</td>\n",
       "      <td>male</td>\n",
       "      <td>0.57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Alexis de</td>\n",
       "      <td>Tocqueville</td>\n",
       "      <td>male</td>\n",
       "      <td>0.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>F. Scott (Francis Scott)</td>\n",
       "      <td>Fitzgerald</td>\n",
       "      <td>male</td>\n",
       "      <td>0.89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Toni</td>\n",
       "      <td>Morrison</td>\n",
       "      <td>female</td>\n",
       "      <td>0.66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>1694-1778</td>\n",
       "      <td>Voltaire</td>\n",
       "      <td>unknown</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>Thucydides</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>Alexis'</td>\n",
       "      <td>de Tocqueville</td>\n",
       "      <td>male</td>\n",
       "      <td>0.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>Sophocles</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>Jean Paul</td>\n",
       "      <td>Sartre</td>\n",
       "      <td>male</td>\n",
       "      <td>0.53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>Jean M</td>\n",
       "      <td>Due</td>\n",
       "      <td>male</td>\n",
       "      <td>0.53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>43 B.C.-17 A.D. or 18 A.D</td>\n",
       "      <td>Ovid</td>\n",
       "      <td>unknown</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>Confucius</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>N. Gregory</td>\n",
       "      <td>Mankiw</td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>Aristophanes</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>125</th>\n",
       "      <td>Aeschylus</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>Simone de</td>\n",
       "      <td>Beauvoir</td>\n",
       "      <td>female</td>\n",
       "      <td>0.55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>Sophocles</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>133</th>\n",
       "      <td>Sigmund</td>\n",
       "      <td>Freud</td>\n",
       "      <td>male</td>\n",
       "      <td>0.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>139</th>\n",
       "      <td>Toni</td>\n",
       "      <td>Morrison</td>\n",
       "      <td>female</td>\n",
       "      <td>0.66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>150</th>\n",
       "      <td>Francis</td>\n",
       "      <td>Fukuyama</td>\n",
       "      <td>male</td>\n",
       "      <td>0.89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>164</th>\n",
       "      <td>F. T</td>\n",
       "      <td>Paul</td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>174</th>\n",
       "      <td>Leslie</td>\n",
       "      <td>Silko</td>\n",
       "      <td>unknown</td>\n",
       "      <td>0.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>188</th>\n",
       "      <td>Aeschylus</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>Gustave</td>\n",
       "      <td>Flaubert</td>\n",
       "      <td>male</td>\n",
       "      <td>0.67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>Hesiod</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>203</th>\n",
       "      <td>Sigmund</td>\n",
       "      <td>Freud</td>\n",
       "      <td>male</td>\n",
       "      <td>0.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>206</th>\n",
       "      <td>T. W.'</td>\n",
       "      <td>ADORNO</td>\n",
       "      <td>unknown</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>224</th>\n",
       "      <td>Aphra</td>\n",
       "      <td>Behn</td>\n",
       "      <td>female</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>226</th>\n",
       "      <td>Herodotus</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>230</th>\n",
       "      <td>Herodotus</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>235</th>\n",
       "      <td>inc. Library</td>\n",
       "      <td>Time</td>\n",
       "      <td>unknown</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>237</th>\n",
       "      <td>Sun'</td>\n",
       "      <td>Tzu</td>\n",
       "      <td>male</td>\n",
       "      <td>0.51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>240</th>\n",
       "      <td>Thucydides</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>247</th>\n",
       "      <td>Jean</td>\n",
       "      <td>Rhys</td>\n",
       "      <td>female</td>\n",
       "      <td>0.53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>254</th>\n",
       "      <td>Harper</td>\n",
       "      <td>Lee</td>\n",
       "      <td>female</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>259</th>\n",
       "      <td>Aristophanes</td>\n",
       "      <td></td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>268</th>\n",
       "      <td>de France</td>\n",
       "      <td>Marie</td>\n",
       "      <td>other</td>\n",
       "      <td>0.68</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290</th>\n",
       "      <td>Aquinas</td>\n",
       "      <td>Thomas</td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>296</th>\n",
       "      <td>Émile</td>\n",
       "      <td>Durkheim</td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>297</th>\n",
       "      <td>Sigmund</td>\n",
       "      <td>Freud</td>\n",
       "      <td>male</td>\n",
       "      <td>0.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299</th>\n",
       "      <td>Jearl</td>\n",
       "      <td>Walker</td>\n",
       "      <td>male</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         first            last   gender proportion\n",
       "8                    Sophocles                     male        0.0\n",
       "16                   Sophocles                     male        0.0\n",
       "27                       Saint       Augustine     male       0.57\n",
       "31                   Alexis de     Tocqueville     male       0.52\n",
       "35    F. Scott (Francis Scott)      Fitzgerald     male       0.89\n",
       "42                        Toni        Morrison   female       0.66\n",
       "56                   1694-1778        Voltaire  unknown        0.0\n",
       "57                  Thucydides                     male        0.0\n",
       "58                     Alexis'  de Tocqueville     male       0.52\n",
       "61                   Sophocles                     male        0.0\n",
       "62                   Jean Paul          Sartre     male       0.53\n",
       "63                      Jean M             Due     male       0.53\n",
       "64   43 B.C.-17 A.D. or 18 A.D            Ovid  unknown        0.0\n",
       "79                   Confucius                     male        0.0\n",
       "103                 N. Gregory          Mankiw     male        0.0\n",
       "108               Aristophanes                     male        0.0\n",
       "125                  Aeschylus                     male        0.0\n",
       "127                  Simone de        Beauvoir   female       0.55\n",
       "128                  Sophocles                     male        0.0\n",
       "133                    Sigmund           Freud     male        0.8\n",
       "139                       Toni        Morrison   female       0.66\n",
       "150                    Francis        Fukuyama     male       0.89\n",
       "164                       F. T            Paul     male        0.0\n",
       "174                     Leslie           Silko  unknown       0.88\n",
       "188                  Aeschylus                     male        0.0\n",
       "198                    Gustave        Flaubert     male       0.67\n",
       "200                     Hesiod                     male        0.0\n",
       "203                    Sigmund           Freud     male        0.8\n",
       "206                     T. W.'          ADORNO  unknown        0.5\n",
       "224                      Aphra            Behn   female        0.0\n",
       "226                  Herodotus                     male        0.0\n",
       "230                  Herodotus                     male        0.0\n",
       "235               inc. Library            Time  unknown        0.0\n",
       "237                       Sun'             Tzu     male       0.51\n",
       "240                 Thucydides                     male        0.0\n",
       "247                       Jean            Rhys   female       0.53\n",
       "254                     Harper             Lee   female        0.5\n",
       "259               Aristophanes                     male        0.0\n",
       "268                  de France           Marie    other       0.68\n",
       "290                    Aquinas          Thomas     male        0.0\n",
       "296                      Émile        Durkheim     male        0.0\n",
       "297                    Sigmund           Freud     male        0.8\n",
       "299                      Jearl          Walker     male        0.0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_lt90"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv('top300_above90.csv')\n",
    "df_lt90.to_csv('top300_below90.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}