Scraper_Completed.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web Scraping with Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MIT Political Methodology Lab Workshop Series  \n",
    "Spring 2021  \n",
    "Andy Halterman"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scraping Overview\n",
    "\n",
    "Most news sites and similar web pages can be scraped through a three step process:\n",
    "\n",
    "1. given a link to an article, extract and format all the needed info from the page\n",
    "2. given an archive-type page of links, finding all the links on the page.\n",
    "3. iterating through each page of archives, scraping all the pages from it, and saving to disk\n",
    "\n",
    "We'll write one function for each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Libraries and setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# requests is for general HTTP loading\n",
    "import requests\n",
    "# BeautifulSoup is an HTML parser\n",
    "from bs4 import BeautifulSoup\n",
    "# JSON is a nice format for writing out\n",
    "# ujson can handle datetimes better and is a drop in replacement for the json module\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sometimes you'll need the regular expressions library and a date library\n",
    "import re\n",
    "import dateutil.parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Page scraper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll write a function here that takes in the URL of an article or page, extracts the information we want from HTML, and structures the output.\n",
    "\n",
    "Python things to learn:\n",
    "\n",
    "- calling methods from objects\n",
    "- what a dictionary looks like\n",
    "- how to define a function\n",
    "\n",
    "HTML things to learn: \n",
    "\n",
    "- Chrome inspector\n",
    "- what HTML tags look like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find the URL of an article to scrape\n",
    "url = \"https://reliefweb.int/report/afghanistan/more-200-displaced-families-receive-cash-assistance-laghman-province\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'<!DOCTYPE html>\\n<html lang=\"en\" dir=\"ltr\" prefix=\"fb: http://ogp.me/ns/fb# og: http://ogp.me/ns# article: http://ogp.me/ns/article#\" class=\"no-js\">\\n\\n<head>\\n  <meta charset=\\'UTF-8\\'>\\n  <title>More than 200 Displaced families Receive Cash Assistance in Laghman Province - Afghanistan | ReliefWeb</title>\\n  <script>document.documentElement.className = typeof document.addEventListener !== \\'undefined\\' ? \\'js\\' : \\'no-js\\';</script>\\n  <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\\n\\n  <link rel=\"apple-touch-icon\" sizes=\"180x180\" href=\"/apple-touch-icon.png\">\\n  <link rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/favicon-32x32.png\">\\n  <link rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/favicon-16x16.png\">\\n  <link rel=\"manifest\" href=\"/manifest.json\" crossOrigin=\"use-credentials\">\\n  <link rel=\"mask-icon\" href=\"/safari-pinned-tab.svg\" color=\"#5bbad5\">\\n  <meta name=\"msapplication-TileColor\" content=\"#055372\">\\n  <meta name=\"theme-color\" content=\"#ffffff\">\\n\\n    <link rel=\"preconnect\" href=\"https://www.googletagmanager.com\" crossorigin>\\n  <link rel=\"dns-prefetch\" href=\"https://www.googletagmanager.com\" crossorigin>\\n  \\n  <meta name=\"description\" content=\"English News and Press Release on Afghanistan about Shelter and Non-Food Items; published on 06 Jan 2018 by Govt. Afghanistan\" />\\n<link rel=\"canonical\" href=\"https://reliefweb.int/report/afghanistan/more-200-displaced-families-receive-cash-assistance-laghman-province\" />\\n<link rel=\"shortlink\" href=\"https://reliefweb.int/node/2404849\" />\\n<link rel=\"image_src\" href=\"https://reliefweb.int/profiles/reliefweb/themes/kobe/images/rw-logo-social-media.png\" />\\n<meta name=\"original-source\" content=\"http://morr.gov.af/en/news/70008\" />\\n<meta property=\"fb:app_id\" content=\"1916193535375038\" />\\n<meta property=\"og:description\" content=\"English News and Press Release on Afghanistan about Shelter and Non-Food Items; published on 06 Jan 2018 by Govt. Afghanistan\" />\\n<meta property=\"og:image\" content=\"https://reliefweb.int/profiles/reliefweb/themes/kobe/images/rw-logo-social-media.png\" />\\n<meta property=\"og:site_name\" content=\"ReliefWeb\" />\\n<meta property=\"og:title\" content=\"More than 200 Displaced families Receive Cash Assistance in Laghman Province - Afghanistan\" />\\n<meta property=\"og:type\" content=\"article\" />\\n<meta property=\"og:url\" content=\"https://reliefweb.int/report/afghanistan/more-200-displaced-families-receive-cash-assistance-laghman-province\" />\\n<meta name=\"twitter:card\" content=\"summary\" />\\n<meta name=\"twitter:site\" content=\"@reliefweb\" />\\n\\n  <!--[if lte IE 8]>\\n  <script>\\n  document.createElement(\"article\");\\n  document.createElement(\"aside\");\\n  document.createElement(\"footer\");\\n  document.createElement(\"header\");\\n  document.createElement(\"main\");\\n  document.createElement(\"nav\");\\n  document.createElement(\"section\");\\n  document.createElement(\"figure\");\\n  document.createElement(\"figcaption\");\\n  document.createElement(\"data\");\\n  document.createElement(\"time\");\\n  </script>\\n  <![endif]-->\\n\\n  <!--[if gte IE 8]><!-->\\n  <link type=\"text/css\" rel=\"stylesheet\" href=\"https://reliefweb.int/sites/reliefweb.int/files/css/css_yDVWuKTZoijzjjnaN-7ZBvZXc-4KNkC52DXpC-EbGns.css\" media=\"all\" />\\n\\n  <script>window.reliefweb = {};</script>\\n\\n  <script type=\"text/javascript\">\\n<!--//--><![CDATA[//><!--\\ndataLayer = [{\"dimension1\":\"Report\",\"dimension2\":\"News and Press Release\",\"dimension4\":\"English\",\"dimension5\":\"Afghanistan\",\"dimension6\":\"Afghanistan\",\"dimension7\":\"Government of Afghanistan\",\"dimension8\":\"Shelter and Non-Food Items\",\"dimension12\":\"anonymous user\",\"dimension13\":\"2018-01-08\",\"dimension20\":\"Published\"}];\\n//--><!]]>\\n</script>\\n  <!--<![endif]-->\\n\\n    <script type=\"text/javascript\">(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({\\'gtm.start\\':new Date().getTime(),event:\\'gtm.js\\'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!=\\'dataLayer\\'?\\'&l=\\'+l:\\'\\';j.async=true;j.src=\\'https://www.googletagmanager.com/gtm.js?id=\\'+i+dl;f.parentNode.insertBefore(j,f);})(window,document,\\'script\\',\\'dataLayer\\',\\'GTM-TLKXSWT\\');</script>\\n  </head>\\n<body class=\"not-logged-in page-node page-node-2404849 node-type-report page-entity-report\">\\n    <noscript><iframe src=\"https://www.googletagmanager.com/ns.html?id=GTM-TLKXSWT\"\\n  height=\"0\" width=\"0\" style=\"display:none;visibility:hidden\"></iframe></noscript>\\n  \\n    <a id=\"skip-link\" href=\"#main\" class=\"accessibility\">Skip to main content</a>\\n\\n    <!-- Site header -->\\n    <header id=\"header\">\\n  <div>\\n    <h1><a href=\"/\">ReliefWeb</a></h1>\\n\\n    <!-- Slogan -->\\n    <p id=\"site-slogan\">\\n      <!-- TEMPORARY - COVID-19 - disabled\\n      <span>Informing humanitarians worldwide 24/7 &mdash; <span>a service provided by <span class=\"ocha\">UN OCHA</span></span></span>\\n      -->\\n      <!-- TEMPORARY - COVID-19 -->\\n      <span><strong>RW COVID-19 page:</strong> <a href=\"/topics/covid-19\">Find latest updates on global humanitarian responses</a></span>\\n    </p>\\n\\n    <!-- Help and user menu -->\\n    <nav id=\"global-menu\" aria-labelledby=\"global-menu-title\">\\n      <h2 id=\"global-menu-title\" class=\"accessibility\">Global Menu</h2>\\n      <ul>\\n        <li>\\n          <a href=\"/help\" aria-describedby=\"hint-help\">Help</a>\\n          <p id=\"hint-help\" class=\"accessibility\">Find help on how to use the site, read terms and conditions, view the FAQs and API documentation.</p>\\n        </li>\\n                <li>\\n          <a href=\"/user/login\" aria-describedby=\"hint-login\">Log in / Register</a>\\n          <p id=\"hint-login\" class=\"accessibility\">Access your account or create a new one for additional features or to post job or training opportunities.</p>\\n        </li>\\n              </ul>\\n    </nav>\\n\\n    <!-- Site Navigation -->\\n    <nav id=\"navigation-menu\" aria-labelledby=\"navigation-menu-title\" data-toggable=\"Menu\">\\n      <h2 id=\"navigation-menu-title\" class=\"accessibility\">Site Navigation</h2>\\n      <ul>\\n        <li>\\n          <a href=\"/updates\" aria-describedby=\"hint-updates\" data-active>Updates</a>\\n          <p id=\"hint-updates\">Latest humanitarian reports, maps and infographics and full document archive.</p>\\n        </li>\\n        <li>\\n          <a href=\"/countries\" aria-describedby=\"hint-countries\">Countries</a>\\n          <p id=\"hint-countries\">List of countries covered by ReliefWeb.</p>\\n        </li>\\n        <li>\\n          <a href=\"/disasters\" aria-describedby=\"hint-disasters\">Disasters</a>\\n          <p id=\"hint-disasters\">List of alerts, ongoing and past disasters covered by ReliefWeb.</p>\\n        </li>\\n        <li>\\n          <a href=\"/organizations\" aria-describedby=\"hint-organizations\">Organizations</a>\\n          <p id=\"hint-organizations\">List of organizations that are actively providing ReliefWeb with content.</p>\\n        </li>\\n        <li>\\n          <a href=\"/topics\" aria-describedby=\"hint-topics\">Topics</a>\\n          <p id=\"hint-topics\">Curated pages dedicated to humanitarian themes and specific humanitarian crises.</p>\\n        </li>\\n        <li>\\n          <a href=\"/jobs\" aria-describedby=\"hint-jobs\">Jobs</a>\\n          <p id=\"hint-jobs\">Open job opportunities in the humanitarian field.</p>\\n        </li>\\n        <li>\\n          <a href=\"/training\" aria-describedby=\"hint-training\">Training</a>\\n          <p id=\"hint-training\">Open training opportunities in the humanitarian field.</p>\\n        </li>\\n      </ul>\\n    </nav>\\n\\n    <!-- Site search form -->\\n    <section id=\"site-search\" role=\"search\" aria-labelledby=\"site-search-title\" data-toggable=\"Search\" data-focus-target=\"site-search-input\">\\n      <h2 id=\"site-search-title\" class=\"accessibility\">Content Search</h2>\\n      <form action=\"/search/results\" method=\"GET\">\\n        <label for=\"site-search-input\">Search ReliefWeb</label>\\n        <input type=\"search\" name=\"search\" autocomplete=\"off\" placeholder=\"What are you looking for?\" size=\"30\" id=\"site-search-input\">\\n        <button type=\"submit\" value=\"search\"><span>Search</span></button>\\n      </form>\\n    </section>\\n  </div>\\n</header>\\n\\n    <hr>\\n\\n    <!-- Main page content -->\\n    <main id=\"main\">\\n      <!-- Local tasks -->\\n\\n<!-- Messages -->\\n\\n<!-- Moderation information -->\\n\\n<!-- Pager content -->\\n<article class=\"report document\">\\n  <header>\\n    <p class=\"country\">\\n            <a href=\"/country/afg\">Afghanistan</a>\\n                </p>\\n\\n    <h2 lang=\"en\">More than 200 Displaced families Receive Cash Assistance in Laghman Province</h2>\\n\\n    <dl class=\"meta core\">\\n            <dt class=\"format news-and-press-release\">Format</dt>\\n      <dd class=\"format news-and-press-release\">News and Press Release</dd>\\n      \\n      <dt class=\"source\">Source</dt>\\n      <dd class=\"source\">\\n        <ul>\\n                    <li data-last><a href=\"/organization/govt-afghanistan\">Govt. Afghanistan</a></li>\\n                            </ul>\\n      </dd>\\n\\n            <dt class=\"date posted\">Posted</dt>\\n      <dd class=\"date posted\"><time datetime=\"2018-01-08T03:52:43+00:00\">8 Jan 2018</time></dd>\\n      \\n            <dt class=\"date published\">Originally published</dt>\\n      <dd class=\"date published\"><time datetime=\"2018-01-06T00:00:00+00:00\">6 Jan 2018</time></dd>\\n      \\n            <dt class=\"origin\">Origin</dt>\\n      <dd class=\"origin\" data-last><a href=\"http://morr.gov.af/en/news/70008\" target=\"_blank\" rel=\"noopener\">View original</a></dd>\\n          </dl>\\n  </header>\\n\\n  <div class=\"content\">\\n    \\n    \\n        <div class=\"content\" lang=\"en\">\\n      <p>Directorate of Refugees and Repatriations(DoRR) of Laghman province with co-operation of DRC and CWSA have distributed for 203 displaced families which displaced from insecure places of Alingar, Kanda gul, districts and Khogiani district of Nangarhar province and returnees from Pakistan have placed in the central of this province.</p>\\n\\n<p>Thursday January 4, 2018 of the current year for 143 displaced families distributed 18000Af by DRC and for 60 displaced families distributed 100$ dolor by CWSA.</p>    </div>\\n    \\n      </div>\\n\\n  <footer>\\n    <dl class=\"meta\">\\n            <dt class=\"primary_country\">Primary country</dt>\\n      <dd class=\"primary_country\">\\n        <ul>\\n                    <li data-last><a href=\"/country/afg\">Afghanistan</a></li>\\n                  </ul>\\n      </dd>\\n            <dt class=\"source\">Source</dt>\\n      <dd class=\"source\">\\n        <ul>\\n                    <li data-last><a href=\"/organization/govt-afghanistan\">Government of Afghanistan</a></li>\\n                  </ul>\\n      </dd>\\n            <dt class=\"format\">Format</dt>\\n      <dd class=\"format\">\\n        <ul>\\n                    <li data-last><a href=\"/updates?advanced-search=%28F8%29\">News and Press Release</a></li>\\n                  </ul>\\n      </dd>\\n            <dt class=\"theme\">Theme</dt>\\n      <dd class=\"theme\">\\n        <ul>\\n                    <li data-last><a href=\"/updates?advanced-search=%28T4603%29\">Shelter and Non-Food Items</a></li>\\n                  </ul>\\n      </dd>\\n            <dt class=\"language\">Language</dt>\\n      <dd class=\"language\">\\n        <ul>\\n                    <li data-last><a href=\"/updates?advanced-search=%28L267%29\">English</a></li>\\n                  </ul>\\n      </dd>\\n          </dl>\\n\\n    <section id=\"share\">\\n  <h3 id=\"pager-title\" class=\"accessibility\">Share</h3>\\n  <ul>\\n        <li><a\\n      href=\"https://www.facebook.com/sharer.php?u=https%3A//reliefweb.int/report/afghanistan/more-200-displaced-families-receive-cash-assistance-laghman-province%3Futm_medium%3Dsocial%26utm_campaign%3Dshared%26utm_source%3Dfacebook.com&amp;t=More%20than%20200%20Displaced%20families%20Receive%20Cash%20Assistance%20in%20Laghman%20Province\"\\n      target=\"_blank\"\\n      rel=\"nofollow noopener noreferrer\"\\n      title=\"Share this on Facebook\"\\n    ><span class=\"accessibility\">Share this on Facebook</span></a></li>\\n        <li><a\\n      href=\"https://twitter.com/share?url=https%3A//reliefweb.int/report/afghanistan/more-200-displaced-families-receive-cash-assistance-laghman-province%3Futm_medium%3Dsocial%26utm_campaign%3Dshared%26utm_source%3Dtwitter.com&amp;text=More%20than%20200%20Displaced%20families%20Receive%20Cash%20Assistance%20in%20Laghman%20Province&amp;via=reliefweb\"\\n      target=\"_blank\"\\n      rel=\"nofollow noopener noreferrer\"\\n      title=\"Share this on Twitter\"\\n    ><span class=\"accessibility\">Share this on Twitter</span></a></li>\\n        <li><a\\n      href=\"https://www.linkedin.com/shareArticle?mini=true&amp;url=https%3A//reliefweb.int/report/afghanistan/more-200-displaced-families-receive-cash-assistance-laghman-province%3Futm_medium%3Dsocial%26utm_campaign%3Dshared%26utm_source%3Dlinkedin.com&amp;title=More%20than%20200%20Displaced%20families%20Receive%20Cash%20Assistance%20in%20Laghman%20Province&amp;summary=&amp;source=ReliefWeb\"\\n      target=\"_blank\"\\n      rel=\"nofollow noopener noreferrer\"\\n      title=\"Post this on LinkedIn\"\\n    ><span class=\"accessibility\">Post this on LinkedIn</span></a></li>\\n      </ul>\\n</section>\\n\\n      </footer>\\n</article>\\n\\n<section id=\"related-content\">\\n  <h2>Related Content</h2>\\n  <div class=\"articles\">\\n      <article  class=\"report\">\\n  <header>\\n        <p class=\"country\">\\n            <a href=\"/country/afg\">Afghanistan</a>\\n                </p>\\n    \\n    <h3 lang=\"en\" class=\"title\"><a href=\"https://reliefweb.int/report/afghanistan/afghanistan-2020-year-review-humanitarian-priorities-challenges-and-response\">Afghanistan: 2020 year in review Humanitarian priorities, challenges, and response</a></h3>\\n  </header>\\n\\n  \\n  \\n  <footer>\\n    <dl class=\"meta core\">\\n            <dt class=\"format news-and-press-release\">Format</dt>\\n      <dd class=\"format news-and-press-release\">News and Press Release</dd>\\n      \\n      <dt class=\"source\">Source</dt>\\n      <dd class=\"source\">\\n        <ul>\\n                    <li data-last><a href=\"/organization/ocha\">OCHA</a></li>\\n                            </ul>\\n      </dd>\\n\\n            <dt class=\"date posted\">Posted</dt>\\n      <dd class=\"date posted\">\\n        <time datetime=\"2021-03-25T15:21:29+00:00\">25 Mar 2021</time>\\n      </dd>\\n      \\n            <dt class=\"date published\">Originally published</dt>\\n      <dd class=\"date published\">\\n        <time datetime=\"2021-03-25T00:00:00+00:00\">25 Mar 2021</time>\\n      </dd>\\n          </dl>\\n  </footer>\\n</article>\\n      <article  class=\"report\">\\n  <header>\\n        <p class=\"country\">\\n            <a href=\"/country/afg\">Afghanistan</a>\\n                  <span>+ 2 more</span>\\n          </p>\\n    \\n    <h3 lang=\"en\" class=\"title\"><a href=\"https://reliefweb.int/report/afghanistan/afghanistan-weekly-humanitarian-update-15-21-march-2021\">Afghanistan: Weekly Humanitarian Update (15 - 21 March 2021)</a></h3>\\n  </header>\\n\\n  \\n  \\n  <footer>\\n    <dl class=\"meta core\">\\n            <dt class=\"format situation-report\">Format</dt>\\n      <dd class=\"format situation-report\">Situation Report</dd>\\n      \\n      <dt class=\"source\">Source</dt>\\n      <dd class=\"source\">\\n        <ul>\\n                    <li data-last><a href=\"/organization/ocha\">OCHA</a></li>\\n                            </ul>\\n      </dd>\\n\\n            <dt class=\"date posted\">Posted</dt>\\n      <dd class=\"date posted\">\\n        <time datetime=\"2021-03-24T14:53:59+00:00\">24 Mar 2021</time>\\n      </dd>\\n      \\n            <dt class=\"date published\">Originally published</dt>\\n      <dd class=\"date published\">\\n        <time datetime=\"2021-03-24T00:00:00+00:00\">24 Mar 2021</time>\\n      </dd>\\n          </dl>\\n  </footer>\\n</article>\\n      <article  class=\"report\">\\n  <header>\\n        <p class=\"country\">\\n            <a href=\"/country/afg\">Afghanistan</a>\\n                </p>\\n    \\n    <h3 lang=\"en\" class=\"title\"><a href=\"https://reliefweb.int/report/afghanistan/unhcr-afghanistan-core-relief-items-response-snapshot-01-january-28-february-2021\">UNHCR Afghanistan: Core Relief Items - Response Snapshot (01 January - 28 February 2021)</a></h3>\\n  </header>\\n\\n  \\n  \\n  <footer>\\n    <dl class=\"meta core\">\\n            <dt class=\"format infographic\">Format</dt>\\n      <dd class=\"format infographic\">Infographic</dd>\\n      \\n      <dt class=\"source\">Source</dt>\\n      <dd class=\"source\">\\n        <ul>\\n                    <li data-last><a href=\"/organization/unhcr\">UNHCR</a></li>\\n                            </ul>\\n      </dd>\\n\\n            <dt class=\"date posted\">Posted</dt>\\n      <dd class=\"date posted\">\\n        <time datetime=\"2021-03-23T16:31:43+00:00\">23 Mar 2021</time>\\n      </dd>\\n      \\n            <dt class=\"date published\">Originally published</dt>\\n      <dd class=\"date published\">\\n        <time datetime=\"2021-03-23T00:00:00+00:00\">23 Mar 2021</time>\\n      </dd>\\n          </dl>\\n  </footer>\\n</article>\\n      <article  class=\"report\">\\n  <header>\\n        <p class=\"country\">\\n            <a href=\"/country/afg\">Afghanistan</a>\\n                </p>\\n    \\n    <h3 lang=\"en\" class=\"title\"><a href=\"https://reliefweb.int/report/afghanistan/afghanistan-protection-cluster-response-snapshot-january-february-2021\">Afghanistan: Protection Cluster Response Snapshot - January - February 2021</a></h3>\\n  </header>\\n\\n  \\n  \\n  <footer>\\n    <dl class=\"meta core\">\\n            <dt class=\"format infographic\">Format</dt>\\n      <dd class=\"format infographic\">Infographic</dd>\\n      \\n      <dt class=\"source\">Sources</dt>\\n      <dd class=\"source\">\\n        <ul>\\n                    <li><a href=\"/organization/protection-cluster\">Protection Cluster</a></li>\\n                    <li data-last><a href=\"/organization/unhcr\">UNHCR</a></li>\\n                            </ul>\\n      </dd>\\n\\n            <dt class=\"date posted\">Posted</dt>\\n      <dd class=\"date posted\">\\n        <time datetime=\"2021-03-23T13:06:33+00:00\">23 Mar 2021</time>\\n      </dd>\\n      \\n            <dt class=\"date published\">Originally published</dt>\\n      <dd class=\"date published\">\\n        <time datetime=\"2021-03-23T00:00:00+00:00\">23 Mar 2021</time>\\n      </dd>\\n          </dl>\\n  </footer>\\n</article>\\n    </div>\\n</section>\\n    </main>\\n\\n    <hr>\\n\\n    <!-- Site footer -->\\n    <footer id=\"footer\">\\n  <div>\\n    <!-- Additional Links -->\\n    <section id=\"additional-links\" aria-labelledby=\"additional-links\">\\n      <h2 id=\"additional-links-title\" class=\"accessibility\">Additional links</h2>\\n      <ul>\\n        <li>\\n          <a href=\"/blog\" aria-describedby=\"hint-blog\">Blog</a>\\n          <p id=\"hint-blog\" class=\"accessibility\">ReliefWeb&#039;s blog</p>\\n        </li>\\n        <li>\\n          <a href=\"https://labs.reliefweb.int\" aria-describedby=\"hint-labs\">Labs</a>\\n          <p id=\"hint-labs\" class=\"accessibility\">ReliefWeb Labs projects explore new and emerging opportunities to improve information delivery to humanitarians.</p>\\n        </li>\\n        <li>\\n          <a href=\"/about\" aria-describedby=\"hint-about\">About Us</a>\\n          <p id=\"hint-about\" class=\"accessibility\">Learn more about ReliefWeb, leading online source for reliable and timely humanitarian information on global crises and disasters since 1996.</p>\\n        </li>\\n        <li>\\n          <a href=\"/terms-conditions\" aria-describedby=\"hint-terms-conditions\">Terms</a>\\n          <p id=\"hint-terms-conditions\" class=\"accessibility\">ReliefWeb&#039;s terms &amp; Conditions.</p>\\n        </li>\\n        <li>\\n          <a href=\"/contact\" aria-describedby=\"hint-contact\">Contact</a>\\n          <p id=\"hint-contact\" class=\"accessibility\">Contact us.</p>\\n        </li>\\n      </ul>\\n    </section>\\n\\n    <!-- Social media -->\\n    <section id=\"social-media\" aria-labelledby=\"social-media-title\">\\n      <h2 id=\"social-media-title\" class=\"accessibility\">Social media</h2>\\n      <ul>\\n        <li><a href=\"https://www.facebook.com/reliefweb\" target=\"_blank\" rel=\"noopener\"><span class=\"accessibility\">ReliefWeb on Facebook</span></a></li>\\n        <li><a href=\"https://twitter.com/reliefweb\" target=\"_blank\" rel=\"noopener\"><span class=\"accessibility\">ReliefWeb on Twitter</span></a></li>\\n        <li><a href=\"https://www.linkedin.com/company/reliefweb\" target=\"_blank\" rel=\"noopener\"><span class=\"accessibility\">ReliefWeb on LinkedIn</span></a></li>\\n      </ul>\\n    </section>\\n\\n    \\n    <!-- Other OCHA services -->\\n    <section id=\"ocha-services\" aria-labelledby=\"ocha-services-title\"  data-toggable=\"OCHA Services\">\\n      <h2 id=\"ocha-services-title\">OCHA Services</h2>\\n      <ul>\\n        <li>\\n          <div id=\"ocha-services-related-platforms\">Related Platforms</div>\\n          <ul aria-labelledby=\"ocha-services-related-platforms\">\\n            <li><a href=\"https://www.agendaforhumanity.org\" target=\"_blank\" rel=\"noopener\">Agenda for Humanity</a></li>\\n            <li><a href=\"https://cerf.un.org\" target=\"_blank\" rel=\"noopener\">Central Emergency Response Fund</a></li>\\n            <li><a href=\"https://redhum.org\" target=\"_blank\" rel=\"noopener\">Redhum</a></li>\\n          </ul>\\n        </li>\\n        <li>\\n          <div id=\"ocha-services-others\">Other OCHA Services</div>\\n          <ul aria-labelledby=\"ocha-services-others\">\\n            <li><a href=\"https://fts.unocha.org\" target=\"_blank\" rel=\"noopener\">Financial Tracking Service</a></li>\\n            <li><a href=\"https://data.humdata.org\" target=\"_blank\" rel=\"noopener\">Humanitarian Data Exchange</a></li>\\n            <li><a href=\"https://humanitarian.id\" target=\"_blank\" rel=\"noopener\">Humanitarian ID</a></li>\\n            <li><a href=\"https://humanitarianresponse.info\" target=\"_blank\" rel=\"noopener\">Humanitarian Response</a></li>\\n          </ul>\\n        </li>\\n        <li>\\n          <div id=\"ocha-services-others-continuation\" class=\"accessibility\">Other OCHA Services</div>\\n          <ul aria-labelledby=\"ocha-services-others-continuation\">\\n            <li><a href=\"https://interagencystandingcommittee.org \" target=\"_blank\" rel=\"noopener\">Inter-Agency Standing Committee</a></li>\\n            <li><a href=\"https://unocha.org\" target=\"_blank\" rel=\"noopener\">OCHA website</a></li>\\n            <li><a href=\"https://vosocc.unocha.org\" target=\"_blank\" rel=\"noopener\">Virtual OSOCC</a></li>\\n          </ul>\\n        </li>\\n        <li>\\n          <a href=\"https://www.unocha.org/ocha-digital-services\" target=\"_blank\" rel=\"noopener\">See All<span class=\"accessibility\"> OCHA services</span></a>\\n        </li>\\n      </ul>\\n    </section>\\n\\n    <!-- OCHA Mandate -->\\n    <div id=\"ocha-mandate\">\\n      <p>Service provided by <span class=\"ocha\">UN OCHA</span></p>\\n      <p>OCHA coordinates the global emergency response to save lives and protect people in humanitarian crises. We advocate for effective and principled humanitarian action by all, for all.</p>\\n    </div>\\n\\n    <!-- Copyright -->\\n    <div id=\"copyright\">\\n      <p><a href=\"https://reliefweb.int/terms-conditions\"><span class=\"accessibility\">ReliefWeb&#039;s Terms &amp; Conditions.</span></a></p>\\n      <p>&copy; 2021 all rights reserved.</p>\\n    </div>\\n  </div>\\n</footer>\\n\\n    <!-- Additional scripts -->\\n    <!--[if gte IE 8]><!-->\\n    <script type=\"text/javascript\" src=\"https://reliefweb.int/sites/reliefweb.int/files/js/js_J21rg4GS69wI1dtzSLH3K_aTmcAGRnjxMX7ULwPZTYE.js\"></script>\\n    <!--<![endif]-->\\n</body>\\n</html>\\n'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# download article page and get content\n",
    "page = requests.get(url)\n",
    "content = page.content\n",
    "content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "page.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert to BeautifulSoup\n",
    "soup = BeautifulSoup(content, \"lxml\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Andy asdf  \n",
      "Andy asdf\n"
     ]
    }
   ],
   "source": [
    "# quick aside to demonstrate \"strip\"\n",
    "txt = \"  Andy asdf  \"\n",
    "print(txt)\n",
    "print(txt.strip())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Directorate of Refugees and Repatriations(DoRR) of Laghman province with co-operation of DRC and CWSA have distributed for 203 displaced families which displaced from insecure places of Alingar, Kanda gul, districts and Khogiani district of Nangarhar province and returnees from Pakistan have placed in the central of this province.\n",
      "Thursday January 4, 2018 of the current year for 143 displaced families distributed 18000Af by DRC and for 60 displaced families distributed 100$ dolor by CWSA.\n"
     ]
    }
   ],
   "source": [
    "# extract body text from page HTML (we'll do this together)\n",
    "body = soup.find(\"div\", {\"class\" : \"content\"})\n",
    "#body.text\n",
    "print(body.text.strip())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "More than 200 Displaced families Receive Cash Assistance in Laghman Province\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<h2 lang=\"en\">More than 200 Displaced families Receive Cash Assistance in Laghman Province</h2>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# extract title from page HTML\n",
    "title = soup.find(\"article\").find(\"h2\")\n",
    "print(title.text)\n",
    "title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Govt. Afghanistan'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# extract author from page HTML\n",
    "author = soup.find(\"dd\", {\"class\" : \"source\"})\n",
    "author.text.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6 Jan 2018\n",
      "2018-01-06\n"
     ]
    }
   ],
   "source": [
    "# extract date from page HTML\n",
    "raw_date = soup.find(\"dd\", {\"class\" : \"date published\"})\n",
    "print(raw_date.text)\n",
    "# convert from raw text into a standardized form\n",
    "date = dateutil.parser.parse(raw_date.text)\n",
    "# Put it into a standard ISO format \n",
    "date = date.strftime(\"%Y-%m-%d\")\n",
    "print(date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# put it all together!\n",
    "def page_scraper(url):\n",
    "    \"\"\"Function to scrape a page\"\"\"\n",
    "    # Code to download and soupify the page\n",
    "    page = requests.get(url)\n",
    "    content = page.content\n",
    "    soup = BeautifulSoup(content, \"lxml\")\n",
    "    # All the code to extract pieces from the HTML\n",
    "    title = soup.find(\"article\").find(\"h2\")\n",
    "    title = title.text.strip()\n",
    "    body = soup.find(\"div\", {\"class\" : \"content\"})\n",
    "    body = body.text.strip()\n",
    "    raw_date = soup.find(\"dd\", {\"class\" : \"date published\"})\n",
    "    date = dateutil.parser.parse(raw_date.text)\n",
    "    date = date.strftime(\"%Y-%m-%d\")\n",
    "    author = soup.find(\"dd\", {\"class\" : \"source\"})\n",
    "    author = author.text.strip()\n",
    "    article = {\n",
    "        \"title\" : title,\n",
    "        \"body\" : body,\n",
    "        \"date\" : date,\n",
    "        \"author\" : author\n",
    "    }\n",
    "    return article"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'title': 'More than 200 Displaced families Receive Cash Assistance in Laghman Province',\n",
       " 'body': 'Directorate of Refugees and Repatriations(DoRR) of Laghman province with co-operation of DRC and CWSA have distributed for 203 displaced families which displaced from insecure places of Alingar, Kanda gul, districts and Khogiani district of Nangarhar province and returnees from Pakistan have placed in the central of this province.\\nThursday January 4, 2018 of the current year for 143 displaced families distributed 18000Af by DRC and for 60 displaced families distributed 100$ dolor by CWSA.',\n",
       " 'date': '2018-01-06',\n",
       " 'author': 'Govt. Afghanistan'}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test it\n",
    "page_scraper(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'title': 'Reconstruction Needed as Displaced Iraqis Continue to Return: IOM Iraq [EN/AR/KU]',\n",
       " 'body': 'IOM reports that thousands of Iraqi internally displaced persons (IDPs) continue to return home. © IOM\\n\\nAttachments\\n\\n\\n Download document (PDF | 96.27 KB | Arabic version)\\n\\n\\n Download document (PDF | 91.81 KB | Kurkish version)\\n\\n\\n\\n\\nKuwait – As the recent Kuwait International Conference for the Reconstruction of Iraq came to a close, IOM, the UN Migration Agency, reported that internally displaced persons (IDPs) in Iraq continue to return home by the thousands.\\nAccording to IOM’s latest Displacement Tracking Matrix (DTM) report, as of 31 January 2018 more than 3.3 million Iraqis have returned to their areas of origin, while nearly 2.5 million people continue to live in displacement. New IOM figures show that return movements are ongoing – in January another 125,000 returnees were identified – mainly to the four governorates of Ninewa, Salah al-Din, Kirkuk and Anbar.\\nSince the start of the crisis in early 2014, with ISIL later holding large parts of central Iraq and the subsequent conflict to retake these areas, nearly six million Iraqis have been displaced. Their communities have suffered widespread devastation and damage.\\nIn January 2018, for the first time in more than three years, there were more returnees than internally displaced people. In the last three years, Anbar Governorate has received the largest number of returnees in the country, due to improved security, rehabilitation of services and rebuilding of infrastructures.\\nIn January 2018, the three governorates reporting the biggest decreases in IDP numbers were Ninewa (- 6 per cent), Baghdad (- 12 per cent), and Anbar (- 17 per cent). Together, they account for almost two-thirds of the nationwide decrease of IDPs (more than 145,000). Of the remaining 2.47 million IDPs in Iraq, only half (51 per cent) are reportedly housed in private settings, while more than a quarter (26 per cent) still live in camps.\\nThough Ninewa Governorate accounts for two-thirds (nearly 84,000) of the new returnees identified in January, IOM has noted that not all returnees to Mosul city stay there.\\nDue to lack of security, services and livelihood opportunities in west Mosul, approximately 600 families returned in January to Haj Ali camp.\\nReturnees living in critical shelters, including informal settlements and unfinished buildings, are concentrated in four governorates – Diyala (21,500 individuals), Salah al-Din (12,400), Ninewa (7,500) and Kirkuk (800). Iraq’s urgent need for fast-track reconstruction to assist the reintegration of IDPs in their areas of origin was also discussed at the recent Kuwait International Conference for Reconstruction of Iraq this past week.\\n“As Iraq enters the recovery phase after three years of conflict, we should remember that real reconstruction of the country will not only be based on rebuilding infrastructure,” said IOM Iraq Chief of Mission Gerard Waite. “Provision of specialized support to all who survived the conflict is also needed, alongside reconstruction of infrastructure.”\\nDuring the Kuwait conference, the UN launched a two-year Recovery and Resilience Programme, in which IOM Iraq takes part, to assist the Iraqi government in addressing the multiple needs for rebuilding and reconstruction in the country. The UN Secretary General spoke about the programme in his speech at the conference.\\nIOM Iraq, in partnership with the Government of Iraq, is focusing its recovery assistance on areas of return. This assistance includes mobile community information centres, light infrastructure projects, housing rehabilitation, strengthening health facilities, relief kit distribution and livelihood support.\\nFor more information on displacement across Iraq, visit the IOM Iraq DTM Portal: http://iraqdtm.iom.int\\nThe latest DTM report, with data through 31 January is available at:\\nhttps://reliefweb.int/node/2466259\\nFor more information please contact Sandra Black at IOM Iraq, Tel: +964 751 234 2550, Email: sblack@iom.int \\n\\nInternational Organization for Migration\\nCopyright © IOM. All rights reserved.',\n",
       " 'date': '2018-02-20',\n",
       " 'author': 'IOM'}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test it on a different page\n",
    "page_scraper(\"https://reliefweb.int/report/iraq/reconstruction-needed-displaced-iraqis-continue-return-iom-iraq\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Link getter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second, we need to get the URLS of all the pages we want to scrape. We can do this by finding the directory pages, where the links are on it, and how to get all available directory pages.\n",
    "\n",
    "Python things to learn here:\n",
    "\n",
    "- getting values from dictionaries\n",
    "- for loops and list comprehensions\n",
    "- regex with the `re` library\n",
    "- basic string operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Function to get all the article links from a single directory page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://reliefweb.int/updates?advanced-search=%28PC13%29_%28F10%29&page={0}\".format(1)\n",
    "page = requests.get(url)\n",
    "content = page.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(content, \"lxml\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<a href=\"/country/afg\">Afghanistan</a>, <a href=\"https://reliefweb.int/report/afghanistan/return-undocumented-afghans-weekly-situation-report-05-11-feb-2021-enpashtodari\">Return of Undocumented Afghans - Weekly Situation Report (05 - 11 Feb 2021) [EN/Pashto/Dari]</a>, <a href=\"/organization/iom\">IOM</a>, <a href=\"/country/afg\">Afghanistan</a>, <a href=\"https://reliefweb.int/report/afghanistan/afghanistan-humanitarian-bulletin-issue-84-december-2020-february-2021\">Afghanistan Humanitarian Bulletin Issue 84 | December 2020 – February 2021</a>]\n"
     ]
    }
   ],
   "source": [
    "links_raw = soup.find(\"div\", {\"class\" : \"articles\"}).find_all(\"a\")\n",
    "print(links_raw[0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/country/afg',\n",
       " 'https://reliefweb.int/report/afghanistan/return-undocumented-afghans-weekly-situation-report-05-11-feb-2021-enpashtodari',\n",
       " '/organization/iom',\n",
       " '/country/afg',\n",
       " 'https://reliefweb.int/report/afghanistan/afghanistan-humanitarian-bulletin-issue-84-december-2020-february-2021']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# pull out just the links\n",
    "links_raw = [i['href'] for i in links_raw]\n",
    "links_raw[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://reliefweb.int/report/afghanistan/return-undocumented-afghans-weekly-situation-report-05-11-feb-2021-enpashtodari',\n",
       " 'https://reliefweb.int/report/afghanistan/afghanistan-humanitarian-bulletin-issue-84-december-2020-february-2021',\n",
       " 'https://reliefweb.int/report/afghanistan/afghanistan-covid-19-multi-sectoral-response-operational-situation-report-18',\n",
       " 'https://reliefweb.int/report/afghanistan/who-emro-weekly-epidemiological-monitor-volume-14-issue-no-5-31-january-2021',\n",
       " 'https://reliefweb.int/report/afghanistan/afghanistan-weekly-humanitarian-update-8-14-february-2021']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# uh oh! `links` is full of all sorts of garbage. Is there a term\n",
    "#  that we can search for to reliably pull out article links only?\n",
    "links = [i for i in links_raw if bool(re.search(\"/report\", i))]\n",
    "links[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If these weren't complete URLs we could use string operations plus a list comprehension to fix this:\n",
    "#links = [\"https://reliefweb.int\" + i for i in links]\n",
    "#links[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Put it all together into a function that takes in a \"page number\" \n",
    "#  and returns all the links to scrape from it.\n",
    "def page_to_link(page_num):\n",
    "    # how to use .format()\n",
    "    url = \"https://reliefweb.int/updates?advanced-search=%28PC13%29_%28F10%29&page={0}\".format(page_num)\n",
    "    # download the page\n",
    "    page = requests.get(url)\n",
    "    # get its content\n",
    "    content = page.content\n",
    "    # soupify\n",
    "    soup = BeautifulSoup(content, \"lxml\")\n",
    "    # pull out links\n",
    "    links_raw = soup.find(\"div\", {\"class\" : \"articles\"}).find_all(\"a\")\n",
    "    links = [i['href'] for i in links_raw]\n",
    "    # clean links\n",
    "    links = [i for i in links if bool(re.search(\"/report\", i))]\n",
    "    return links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://reliefweb.int/report/afghanistan/afghanistan-weekly-field-report-16-22-july-2018-enps',\n",
       " 'https://reliefweb.int/report/afghanistan/afghan-assisted-return-weekly-update-21-july-2018',\n",
       " 'https://reliefweb.int/report/afghanistan/afghanistan-humanitarian-response-wash-cluster-updates-cluster-achievements-june',\n",
       " 'https://reliefweb.int/report/afghanistan/who-afghanistan-situation-report-june-2018',\n",
       " 'https://reliefweb.int/report/afghanistan/iom-humanitarian-assistance-programme-hap-weekly-report-11-17-july-2018',\n",
       " 'https://reliefweb.int/report/afghanistan/afghanistan-weekly-field-report-9-15-july-2018-enps',\n",
       " 'https://reliefweb.int/report/afghanistan/afghan-assisted-return-weekly-update-14-july-2018',\n",
       " 'https://reliefweb.int/report/afghanistan/midyear-update-protection-civilians-armed-conflict-1-january-30-june-2018',\n",
       " 'https://reliefweb.int/report/afghanistan/iom-humanitarian-assistance-programme-hap-weekly-report-4-10-july-2018',\n",
       " 'https://reliefweb.int/report/afghanistan/afghanistan-complex-emergency-fact-sheet-3-fiscal-year-fy-2018',\n",
       " 'https://reliefweb.int/report/afghanistan/return-undocumented-afghans-weekly-situation-report-1-7-july-2018',\n",
       " 'https://reliefweb.int/report/afghanistan/afghanistan-weekly-field-report-2-8-july-2018-endari',\n",
       " 'https://reliefweb.int/report/afghanistan/afghan-assisted-return-weekly-update-7-july-2018',\n",
       " 'https://reliefweb.int/report/afghanistan/who-emro-weekly-epidemiological-monitor-volume-11-issue-26-1-july-2018',\n",
       " 'https://reliefweb.int/report/afghanistan/iom-humanitarian-assistance-programme-hap-weekly-report-27-june-2018-3-july-2018',\n",
       " 'https://reliefweb.int/report/afghanistan/iom-afghanistan-returns-iran-and-pakistan-funding-needs-july-2018',\n",
       " 'https://reliefweb.int/report/afghanistan/afghanistan-weekly-field-report-25-june-1-july-2018-enps',\n",
       " 'https://reliefweb.int/report/afghanistan/return-undocumented-afghans-weekly-situation-report-24-30-june-2018',\n",
       " 'https://reliefweb.int/report/afghanistan/afghan-assisted-return-weekly-update-30-june-2018',\n",
       " 'https://reliefweb.int/report/afghanistan/afghanistan-food-security-outlook-update-june-2018-january-2019']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "page_to_link(44)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting it together\n",
    "\n",
    "Now we have a function that'll take a page number for the archive page and return all the links.\n",
    "We have another function that'll take in an article URL and give us the structured content from the page.\n",
    "\n",
    "Let's put them together and download a (small!) range of stories.\n",
    "\n",
    "Note: let's be nice to the UN and not all download the whole thing at once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://reliefweb.int/report/afghanistan/return-undocumented-afghans-weekly-situation-report-22-28-january-2021'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get all the links we want to scrape\n",
    "all_links = []\n",
    "\n",
    "for num in range(1, 3):\n",
    "    lks = page_to_link(num)\n",
    "    all_links.extend(lks) # extend! not append.\n",
    "    \n",
    "len(all_links)\n",
    "\n",
    "all_links[12]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_content = []\n",
    "\n",
    "for link in all_links[10:20]: # be nice to reliefweb and only get some \n",
    "    try:\n",
    "        content = page_scraper(link)\n",
    "        all_content.append(content) # back to append!\n",
    "    except Exception as e:\n",
    "        # if something goes wrong, keep trucking,\n",
    "        #  but print out the link so we can diagnose it.\n",
    "        print(e)\n",
    "        print(link)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(all_content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'title': 'Afghanistan: Weekly Humanitarian Update (25 – 31 January 2021)',\n",
       " 'body': \"Attachments\\n\\n\\n Download document (PDF | 364.25 KB)\\n\\n\\n\\n\\nSouth: Hundreds of people displaced by ongoing fighting in Kandahar province\\nFighting between Afghan National Security Forces (ANSF) and a non-state armed group (NSAG) continued in Hilmand, Kandahar and Uruzgan provinces.\\nIn Kandahar, fighting continued mainly in Arghandab, Zheray and Panjwayi districts. Ongoing fighting displaced hundreds of people in Kandahar province, but the exact number of internally displaced persons (IDPs) is yet to be confirmed.\\nHumanitarian actors with coordination of provincial authorities are assessing the needs of IDPs and will provide them with immediate assistance. Farmers and agricultural activities continued to be affected by ongoing fighting. All movements on the main highway-1 connecting Hilmand to Kandahar provinces reportedly resumed, however improvised explosive devices (IEDs) along the highway continue to pose a threat.\\nIn Uruzgan province, clashes between ANSF and an NSAG continued along with the threat of IED attacks in Dehrawud, Gizab and Tirinkot districts. Two civilians were reportedly killed and eight others wounded by an IED detonation in Tirinkot district.\\nIn Hilmand province, fighting between ANSF and an NSAG continued in Nahr-eSaraj and Nawa-e-Barakzaiy districts. Targeted killings continued to be reported in Lashkargah city. The main road connecting Lashkargah to Grehk remained closed to all movements. Also, the threat of IEDs along main roads hindered civilian movement across Hilmand province. During the reporting period, two civilians were reportedly killed and 15 others wounded when their bus was hit by an IED on highway-1 connecting Hilmand to Kandahar provinces. During the reporting period, interagency assessment teams identified 1,660 people displaced by conflict to receive immediate humanitarian assistance in Kandahar province. On 27 January, 1,301 IDPs received humanitarian assistance in Kandahar province.\\nNorth-east: 30,653 people received humanitarian aid\\nFighting between ANSF and an NSAG continued in the north-east mainly in Shenwar and Kharuty areas of Kunduz city, Kunduz province, Nahr-e-Khushk, Dahna-e-Ghory and Dand-e-Shahabuddin areas of Pul-e-Khumri district of Baghlan province and Baharak district of Takhar province. Humanitarian assistance reached approximately 30,653 people affected by conflict in Badakhshan, Kunduz and Takhar provinces. Assessment teams identified 1,032 people displaced by conflict in Baghlan, Takhar and Kunduz provinces to receive humanitarian assistance in the coming days. \\n\\nUN Office for the Coordination of Humanitarian Affairs\\nTo learn more about OCHA's activities, please visit https://www.unocha.org/.\",\n",
       " 'date': '2021-02-03',\n",
       " 'author': 'OCHA'}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_content[4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving as CSV\n",
    "\n",
    "If you're going to work with your text next in R, a CSV is probably the most useful form to save your text in. To save it as a csv, we will convert into a `DataFrame` using `pandas`, a package for working with data in Python. `pandas` will then let us easily write it out to a CSV."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "content_df = pd.DataFrame(all_content)\n",
    "#content_df\n",
    "content_df.to_csv(\"all_content.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Saving as JSON\n",
    "\n",
    "We can also store is as a JSON file. JSON and dictionaries are almost equivalent, so it's a natural form to save a dict as a JSON file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "FILENAME = \"reliefweb.json\"\n",
    "\n",
    "with open(FILENAME, \"w\") as f:\n",
    "    json.dump(all_content, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read it back in\n",
    "\n",
    "If you want to load it back in later to analyze, you can do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "FILENAME = \"reliefweb.json\"\n",
    "\n",
    "with open(FILENAME, \"r\") as f:\n",
    "    loaded_content = json.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# is it the same?\n",
    "assert loaded_content[4] == all_content[4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(range(0, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}