Hacker News.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Finding Success on Hacker News"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What this project is about:** In this project, I'll compare two different types of posts from [Hacker News](https://news.ycombinator.com/), a popular site where technology related stories (or 'posts') are voted and commented upon. The two types of posts I'll explore begin with either `Ask HN` or `Show HN`.\n",
    "\n",
    "Users submit `Ask HN` posts to ask the Hacker News community a specific question, such as \"What is the best online course you've ever taken?\" Likewise, users submit `Show HN` posts to show the Hacker News community a project, product, or just generally something interesting.\n",
    "\n",
    "**Goal of the project:** In this project, I'll compare these two types of posts to determine the following:\n",
    "\n",
    "- Do `Ask HN` or `Show HN` receive more comments on average?\n",
    "- Do posts created at a certain time receive more comments on average?\n",
    "\n",
    "It should be noted that the data set I am working with was reduced from almost 300,000 rows to approximately 20,000 rows by removing all submissions that did not receive any comments, and then randomly sampling from the remaining submissions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, I'll read in the data and remove the headers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at'],\n",
       " ['12224879',\n",
       "  'Interactive Dynamic Video',\n",
       "  'http://www.interactivedynamicvideo.com/',\n",
       "  '386',\n",
       "  '52',\n",
       "  'ne0phyte',\n",
       "  '8/4/2016 11:52'],\n",
       " ['10975351',\n",
       "  'How to Use Open Source and Shut the Fuck Up at the Same Time',\n",
       "  'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/',\n",
       "  '39',\n",
       "  '10',\n",
       "  'josep2',\n",
       "  '1/26/2016 19:30'],\n",
       " ['11964716',\n",
       "  \"Florida DJs May Face Felony for April Fools' Water Joke\",\n",
       "  'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/',\n",
       "  '2',\n",
       "  '1',\n",
       "  'vezycash',\n",
       "  '6/23/2016 22:20'],\n",
       " ['11919867',\n",
       "  'Technology ventures: From Idea to Enterprise',\n",
       "  'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429',\n",
       "  '3',\n",
       "  '1',\n",
       "  'hswarna',\n",
       "  '6/17/2016 0:01']]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read in the data.\n",
    "import csv\n",
    "\n",
    "f = open('hacker_news.csv')\n",
    "hn = list(csv.reader(f))\n",
    "hn[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Removing Headers from a List of Lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the first list in the inner lists contains the column headers, and the lists after contain the data for one row. In order to analyze this data, I need to first remove the row containing the column headers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at']\n",
      "[['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', \"Florida DJs May Face Felony for April Fools' Water Joke\", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01'], ['10301696', 'Note by Note: The Making of Steinway L1037 (2007)', 'http://www.nytimes.com/2007/11/07/movies/07stein.html?_r=0', '8', '2', 'walterbell', '9/30/2015 4:12']]\n"
     ]
    }
   ],
   "source": [
    "# Remove the headers.\n",
    "headers = hn[0]\n",
    "hn = hn[1:]\n",
    "print(headers)\n",
    "print(hn[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I can now see that the data set contains the title of the posts, the number of comments for each post, and the date the post was created."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting Ask HN and Show HN Posts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by exploring the number of comments for each type of post.\n",
    "\n",
    "First, I'll identify posts that begin with either `Ask HN` or `Show HN` and separate the data for those two types of posts into different lists. Separating the data makes it easier to analyze in the following steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1744\n",
      "1162\n",
      "17194\n"
     ]
    }
   ],
   "source": [
    "# Identify posts that begin with either `Ask HN` or `Show HN` and separate the data into different lists.\n",
    "ask_posts = []\n",
    "show_posts = []\n",
    "other_posts = []\n",
    "\n",
    "for post in hn:\n",
    "    title = post[1]\n",
    "    if title.lower().startswith(\"ask hn\"):\n",
    "        ask_posts.append(post)\n",
    "    elif title.lower().startswith(\"show hn\"):\n",
    "        show_posts.append(post)\n",
    "    else:\n",
    "        other_posts.append(post)\n",
    "        \n",
    "print(len(ask_posts))\n",
    "print(len(show_posts))\n",
    "print(len(other_posts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculating the Average Number of Comments for Ask HN and Show HN Posts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that I have separated ask posts and show posts into different lists, I'll calculate the average number of comments each type of post receives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14.038417431192661\n"
     ]
    }
   ],
   "source": [
    "# Calculate the average number of comments `Ask HN` posts receive.\n",
    "total_ask_comments = 0\n",
    "\n",
    "for post in ask_posts:\n",
    "    total_ask_comments += int(post[4])\n",
    "    \n",
    "avg_ask_comments = total_ask_comments / len(ask_posts)\n",
    "print(avg_ask_comments)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10.31669535283993\n"
     ]
    }
   ],
   "source": [
    "total_show_comments = 0\n",
    "\n",
    "for post in show_posts:\n",
    "    total_show_comments += int(post[4])\n",
    "    \n",
    "avg_show_comments = total_show_comments / len(show_posts)\n",
    "print(avg_show_comments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On average, ask posts in my sample receive approximately 14 comments, whereas show posts receive approximately 10. Since ask posts are more likely to receive comments, I'll focus my remaining analysis specifically on these posts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding the Amount of Ask Posts and Comments by Hour Created"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, I'll determine if I can maximize the amount of comments an ask post receives by creating it at a certain time. \n",
    "\n",
    "To do this, I will first find the amount of ask posts created during each hour of day, along with the number of comments those posts received."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'09': 251,\n",
       " '13': 1253,\n",
       " '10': 793,\n",
       " '14': 1416,\n",
       " '16': 1814,\n",
       " '23': 543,\n",
       " '12': 687,\n",
       " '17': 1146,\n",
       " '15': 4477,\n",
       " '21': 1745,\n",
       " '20': 1722,\n",
       " '02': 1381,\n",
       " '18': 1439,\n",
       " '03': 421,\n",
       " '05': 464,\n",
       " '19': 1188,\n",
       " '01': 683,\n",
       " '22': 479,\n",
       " '08': 492,\n",
       " '04': 337,\n",
       " '00': 447,\n",
       " '06': 397,\n",
       " '07': 267,\n",
       " '11': 641}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Calculate the amount of ask posts created during each hour of day and the number of comments received.\n",
    "import datetime as dt\n",
    "\n",
    "result_list = []\n",
    "\n",
    "for post in ask_posts:\n",
    "    result_list.append(\n",
    "        [post[6], int(post[4])]\n",
    "    )\n",
    "\n",
    "comments_by_hour = {}\n",
    "counts_by_hour = {}\n",
    "date_format = \"%m/%d/%Y %H:%M\"\n",
    "\n",
    "for each_row in result_list:\n",
    "    date = each_row[0]\n",
    "    comment = each_row[1]\n",
    "    time = dt.datetime.strptime(date, date_format).strftime(\"%H\")\n",
    "    if time in counts_by_hour:\n",
    "        comments_by_hour[time] += comment\n",
    "        counts_by_hour[time] += 1\n",
    "    else:\n",
    "        comments_by_hour[time] = comment\n",
    "        counts_by_hour[time] = 1\n",
    "\n",
    "comments_by_hour"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculating the Average Number of Comments for Ask HN Posts by Hour"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, I'll calculate the average amount of comments ask posts receive at each hour of the day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['09', 5.5777777777777775],\n",
       " ['13', 14.741176470588234],\n",
       " ['10', 13.440677966101696],\n",
       " ['14', 13.233644859813085],\n",
       " ['16', 16.796296296296298],\n",
       " ['23', 7.985294117647059],\n",
       " ['12', 9.41095890410959],\n",
       " ['17', 11.46],\n",
       " ['15', 38.5948275862069],\n",
       " ['21', 16.009174311926607],\n",
       " ['20', 21.525],\n",
       " ['02', 23.810344827586206],\n",
       " ['18', 13.20183486238532],\n",
       " ['03', 7.796296296296297],\n",
       " ['05', 10.08695652173913],\n",
       " ['19', 10.8],\n",
       " ['01', 11.383333333333333],\n",
       " ['22', 6.746478873239437],\n",
       " ['08', 10.25],\n",
       " ['04', 7.170212765957447],\n",
       " ['00', 8.127272727272727],\n",
       " ['06', 9.022727272727273],\n",
       " ['07', 7.852941176470588],\n",
       " ['11', 11.051724137931034]]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Calculate the average amount of comments `Ask HN` posts receive at each hour of the day.\n",
    "avg_by_hour = []\n",
    "\n",
    "for hr in comments_by_hour:\n",
    "    avg_by_hour.append([hr, comments_by_hour[hr] / counts_by_hour[hr]])\n",
    "\n",
    "avg_by_hour"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I now have the results I need, however, the current format makes it difficult to identify the hours with the highest values. So, before I wrap things up, I'll sort the list of lists and print the five highest values in a format that's easier to read."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sorting and Printing Values from a List of Lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[5.5777777777777775, '09'], [14.741176470588234, '13'], [13.440677966101696, '10'], [13.233644859813085, '14'], [16.796296296296298, '16'], [7.985294117647059, '23'], [9.41095890410959, '12'], [11.46, '17'], [38.5948275862069, '15'], [16.009174311926607, '21'], [21.525, '20'], [23.810344827586206, '02'], [13.20183486238532, '18'], [7.796296296296297, '03'], [10.08695652173913, '05'], [10.8, '19'], [11.383333333333333, '01'], [6.746478873239437, '22'], [10.25, '08'], [7.170212765957447, '04'], [8.127272727272727, '00'], [9.022727272727273, '06'], [7.852941176470588, '07'], [11.051724137931034, '11']]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[[38.5948275862069, '15'],\n",
       " [23.810344827586206, '02'],\n",
       " [21.525, '20'],\n",
       " [16.796296296296298, '16'],\n",
       " [16.009174311926607, '21'],\n",
       " [14.741176470588234, '13'],\n",
       " [13.440677966101696, '10'],\n",
       " [13.233644859813085, '14'],\n",
       " [13.20183486238532, '18'],\n",
       " [11.46, '17'],\n",
       " [11.383333333333333, '01'],\n",
       " [11.051724137931034, '11'],\n",
       " [10.8, '19'],\n",
       " [10.25, '08'],\n",
       " [10.08695652173913, '05'],\n",
       " [9.41095890410959, '12'],\n",
       " [9.022727272727273, '06'],\n",
       " [8.127272727272727, '00'],\n",
       " [7.985294117647059, '23'],\n",
       " [7.852941176470588, '07'],\n",
       " [7.796296296296297, '03'],\n",
       " [7.170212765957447, '04'],\n",
       " [6.746478873239437, '22'],\n",
       " [5.5777777777777775, '09']]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "swap_avg_by_hour = []\n",
    "\n",
    "for row in avg_by_hour:\n",
    "    swap_avg_by_hour.append([row[1], row[0]])\n",
    "    \n",
    "print(swap_avg_by_hour)\n",
    "\n",
    "sorted_swap = sorted(swap_avg_by_hour, reverse=True)\n",
    "\n",
    "sorted_swap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 5 Hours for 'Ask HN' Comments\n",
      "15:00: 38.59 average comments per post\n",
      "02:00: 23.81 average comments per post\n",
      "20:00: 21.52 average comments per post\n",
      "16:00: 16.80 average comments per post\n",
      "21:00: 16.01 average comments per post\n"
     ]
    }
   ],
   "source": [
    "# Sort the values and print the the 5 hours with the highest average comments.\n",
    "\n",
    "print(\"Top 5 Hours for 'Ask HN' Comments\")\n",
    "for avg, hr in sorted_swap[:5]:\n",
    "    print(\n",
    "        \"{}: {:.2f} average comments per post\".format(\n",
    "            dt.datetime.strptime(hr, \"%H\").strftime(\"%H:%M\"),avg\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The hour of the day that receives the most comments per post, on average, is 15:00, with an average of 38.59 comments per post. Furthermore, the number of comments (on posts) increased by about 60% between the hours with the highest and second highest average number of comments.\n",
    "\n",
    "Note that, according to the data set [documentation](https://www.kaggle.com/hacker-news/hacker-news-posts/home), the timezone used is Eastern Time in the US. So, one could also write 15:00 as 3:00 pm est."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this project, I analyzed ask posts and show posts to determine which type of post and time receive the most comments on average. Based on my analysis, in order to maximize the amount of comments a post receives, I'd recommend the post be categorized as an ask post and published between 15:00 and 16:00 (3:00 pm est - 4:00 pm est).\n",
    "\n",
    "However, it should be noted that the data set I analyzed excluded posts without any comments. With that in mind, it's more accurate to say that *of the posts that received comments*, ask posts received more comments on average **and** ask posts created between 15:00 and 16:00 (3:00 pm est - 4:00 pm est) received the most comments on average."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}