22-Web-Scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d172de8e-787f-4af8-91f4-e73608ce7a63",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0101EN-SkillsNetwork/IDSNlogo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\">\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29becd48-4e00-407b-b7f2-21a80d14b3b6",
   "metadata": {},
   "source": [
    "# **Web Scraping Lab**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97d94674-ec88-4cf7-8949-3a5f6ee5c280",
   "metadata": {},
   "source": [
    "Estimated time needed: **30** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f75ebd49-03f8-4727-a623-0c5cf904f986",
   "metadata": {},
   "source": [
    "## Objectives\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6445a875-4e74-4f2b-ab4b-17c49a6a0a59",
   "metadata": {},
   "source": [
    "After completing this lab you will be:\n",
    "\n",
    "*   Familiar with the basics of the `BeautifulSoup` Python library\n",
    "*   Be able to scrape webpages for data and filter the data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59be9969-bd1f-4b3c-8362-172682d79a5b",
   "metadata": {},
   "source": [
    "<h2>Table of Contents</h2>\n",
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "    <ul>\n",
    "        <li>\n",
    "            <a href=\"https://bso/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0101ENSkillsNetwork19487395-2021-01-01\">Beautiful Soup Object</a>\n",
    "            <ul>\n",
    "                <li>Tag</li>\n",
    "                <li>Children, Parents, and Siblings</li>\n",
    "                <li>HTML Attributes</li>\n",
    "                <li>Navigable String</li>\n",
    "            </ul>\n",
    "        </li>\n",
    "     </ul>\n",
    "    <ul>\n",
    "        <li>\n",
    "            <a href=\"https://filter/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0101ENSkillsNetwork19487395-2021-01-01\">Filter</a>\n",
    "            <ul>\n",
    "                <li>find All</li>\n",
    "                <li>find </li>\n",
    "                <li>HTML Attributes</li>\n",
    "                <li>Navigable String</li>\n",
    "            </ul>\n",
    "        </li>\n",
    "     </ul>\n",
    "     <ul>\n",
    "        <li>\n",
    "            <a href=\"https://dscw/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0101ENSkillsNetwork19487395-2021-01-01\">Downloading And Scraping The Contents Of A Web</a>\n",
    "    </p>\n",
    "\n",
    "</div>\n",
    "\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4aaf6e7e-d0dc-4cee-b922-afb14843b8c4",
   "metadata": {},
   "source": [
    "For this lab, we are going to be using Python and several Python libraries. Some of these libraries might be installed in your lab environment or in SN Labs. Others may need to be installed by you. The cells below will install these libraries when executed.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69fea79c-e2fe-493b-9ee4-76684bc241c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install html5lib"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22f77610-3472-4dd3-bf8f-0c147539f954",
   "metadata": {},
   "source": [
    "**Note:- After running the above code cell, restart the kernel and don't run the above code cell after restarting the kernel.**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e55cb43-5b53-40f2-9ae2-b2c4210970a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install bs4\n",
    "#!pip install requests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86b3bacb-3723-4f8f-a054-aa332e572562",
   "metadata": {},
   "source": [
    "Import the required modules and functions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07aa45ec-8dc3-44a2-8d1b-3bc507184385",
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup # this module helps in web scrapping.\n",
    "import requests  # this module helps us to download a web page"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97ddb2e4-be09-4579-9093-ead66112d1dd",
   "metadata": {},
   "source": [
    "<h2 id=\"BSO\">Beautiful Soup Objects</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbebd99e-d084-4818-b272-80615b188043",
   "metadata": {},
   "source": [
    "Beautiful Soup is a Python library for pulling data out of HTML and XML files, we will focus on HTML files. This is accomplished by representing the HTML as a set of objects with methods used to parse the HTML.  We can navigate the HTML as a tree, and/or filter out what we are looking for.\n",
    "\n",
    "Consider the following HTML:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff40156d-bc7f-482a-885d-434622371cb0",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%html\n",
    "<!DOCTYPE html>\n",
    "<html>\n",
    "<head>\n",
    "<title>Page Title</title>\n",
    "</head>\n",
    "<body>\n",
    "<h3><b id='boldest'>Lebron James</b></h3>\n",
    "<p> Salary: $ 92,000,000 </p>\n",
    "<h3> Stephen Curry</h3>\n",
    "<p> Salary: $85,000, 000 </p>\n",
    "<h3> Kevin Durant </h3>\n",
    "<p> Salary: $73,200, 000</p>\n",
    "</body>\n",
    "</html>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "267e3844-6d86-46d0-b1cb-6be265940515",
   "metadata": {},
   "source": [
    "We can store it as a string in the variable HTML:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbc00d39-fbe8-4463-b9a9-459b00b4b56f",
   "metadata": {},
   "outputs": [],
   "source": [
    "html=\"<!DOCTYPE html><html><head><title>Page Title</title></head><body><h3><b id='boldest'>Lebron James</b></h3><p> Salary: $ 92,000,000 </p><h3> Stephen Curry</h3><p> Salary: $85,000, 000 </p><h3> Kevin Durant </h3><p> Salary: $73,200, 000</p></body></html>\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5574a4da-8340-44a5-826a-a4efb1695f26",
   "metadata": {},
   "source": [
    "To parse a document, pass it into the <code>BeautifulSoup</code> constructor. The <code>BeautifulSoup</code> object represents the document as a nested data structure:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80fbf11c-b487-4787-9775-a87f8ffaaed4",
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(html, 'html5lib')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd07a6da-2a8d-42c0-8cce-01b952bedba9",
   "metadata": {},
   "source": [
    "First, the document is converted to Unicode (similar to ASCII) and HTML entities are converted to Unicode characters. Beautiful Soup transforms a complex HTML document into a complex tree of Python objects. The <code>BeautifulSoup</code> object can create other types of objects. In this lab, we will cover <code>BeautifulSoup</code> and <code>Tag</code> objects, that for the purposes of this lab are identical. Finally, we will look at <code>NavigableString</code> objects.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74dfe2c7-722b-485d-a8d9-8fa3779c63af",
   "metadata": {},
   "source": [
    "We can use the method <code>prettify()</code> to display the HTML in the nested structure:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "050f2d38-e430-4f5f-86bf-017f0ad44f62",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b58e616-103a-4b0a-95ca-8152bffc4c0e",
   "metadata": {},
   "source": [
    "## Tags\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af676ea0-4466-41b1-a480-963f32901581",
   "metadata": {},
   "source": [
    "Let's say we want the  title of the page and the name of the top paid player. We can use the <code>Tag</code>. The <code>Tag</code> object corresponds to an HTML tag in the original document, for example, the tag title.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a90e3bcf-b42d-4cf8-89d0-dfe5b0d1a230",
   "metadata": {},
   "outputs": [],
   "source": [
    "tag_object=soup.title\n",
    "print(\"tag object:\",tag_object)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb2801ec-43e1-48bb-a8f3-ccf754824c2a",
   "metadata": {},
   "source": [
    "we can see the tag type <code>bs4.element.Tag</code>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1297b88-39fc-43b0-aaac-fb79e2fa1d12",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"tag object type:\",type(tag_object))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c6b8a12-a2fd-447b-a503-be5607d0d685",
   "metadata": {},
   "source": [
    "If there is more than one <code>Tag</code> with the same name, the first element with that <code>Tag</code> name is called. This corresponds to the most paid player:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4790392d-df03-4ad4-8a38-dffe154413ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "tag_object=soup.h3\n",
    "tag_object"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46594e03-81ad-4109-bd4c-87649202a2ec",
   "metadata": {},
   "source": [
    "Enclosed in the bold attribute <code>b</code>, it helps to use the tree representation. We can navigate down the tree using the child attribute to get the name.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4865f936-ef5a-4c5b-a9c3-a6d1753ad829",
   "metadata": {},
   "source": [
    "### Children, Parents, and Siblings\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cea84a77-c100-4d7b-ac1a-4907419090b8",
   "metadata": {},
   "source": [
    "As stated above, the <code>Tag</code> object is a tree of objects. We can access the child of the tag or navigate down the branch as follows:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc7b59fa-5740-43d7-83f5-ea6b244d775d",
   "metadata": {},
   "outputs": [],
   "source": [
    "tag_child =tag_object.b\n",
    "tag_child"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9abe5f8-a3e6-4661-9d2f-e6b7a291098e",
   "metadata": {},
   "source": [
    "You can access the parent with the <code> parent</code>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60ad9a6c-6b3d-403b-aec7-da669e20d899",
   "metadata": {},
   "outputs": [],
   "source": [
    "parent_tag=tag_child.parent\n",
    "parent_tag"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e61aeaac-6c72-485a-a498-374288a4cf52",
   "metadata": {},
   "source": [
    "this is identical to:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f0bb13e-ce4a-4d9d-8b3a-27d6d04a2393",
   "metadata": {},
   "outputs": [],
   "source": [
    "tag_object"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc89274a-98bc-4137-bb39-b0aad8296dd3",
   "metadata": {},
   "source": [
    "<code>tag_object</code> parent is the <code>body</code> element.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57f2dd7e-a81a-4a1a-9b5b-2dd1125517b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "tag_object.parent"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc81a82a-f55d-4b67-b822-31e62811c863",
   "metadata": {},
   "source": [
    "<code>tag_object</code> sibling is the <code>paragraph</code> element.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed2b9599-d1cb-4dca-a206-6633b3ff91f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "sibling_1=tag_object.next_sibling\n",
    "sibling_1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06f19df8-be41-4c53-88ed-b022d3a81859",
   "metadata": {},
   "source": [
    "`sibling_2` is the `header` element, which is also a sibling of both `sibling_1` and `tag_object`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c36cc5e-ea3e-427b-9806-39727d28c5e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "sibling_2=sibling_1.next_sibling\n",
    "sibling_2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a78427f6-d490-401b-8ed4-a86b1adb947a",
   "metadata": {},
   "source": [
    "<h3 id=\"first_question\">Exercise: <code>next_sibling</code></h3>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eb00342-1707-4218-abad-e01a2f664e6b",
   "metadata": {},
   "source": [
    "Use the object <code>sibling\\_2</code> and the method <code>next_sibling</code> to find the salary of Stephen Curry:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0ab11b7-ec53-4e08-8a33-b6372964235a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "77cc9bde-925f-4a38-964f-71a271cea567",
   "metadata": {},
   "source": [
    "<details><summary>Click here for the solution</summary>\n",
    "\n",
    "```\n",
    "sibling_2.next_sibling\n",
    "\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20f3f03d-2c19-4250-a4dd-d1f9602f1836",
   "metadata": {},
   "source": [
    "### HTML Attributes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8755461d-4fdb-442f-879f-82b02157ceba",
   "metadata": {},
   "source": [
    "If the tag has attributes, the tag <code>id=\"boldest\"</code> has an attribute <code>id</code> whose value is <code>boldest</code>. You can access a tag's attributes by treating the tag like a dictionary:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebb89194-b262-4d30-88ca-be01f6925c38",
   "metadata": {},
   "outputs": [],
   "source": [
    "tag_child['id']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "730f574e-689e-493e-aa49-1756deef9742",
   "metadata": {},
   "source": [
    "You can access that dictionary directly as <code>attrs</code>:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c63863c-b996-487b-9895-43f772dc8d1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "tag_child.attrs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e2182aa-0787-42b3-9dd3-1c2831896523",
   "metadata": {},
   "source": [
    "You can also work with Multi-valued attributes. Check out <a href=\"https://www.crummy.com/software/BeautifulSoup/bs4/doc/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0101ENSkillsNetwork19487395-2021-01-01\">\\[1]</a> for more.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73fa55e6-a111-455c-814f-83517dad3ec2",
   "metadata": {},
   "source": [
    "We can also obtain the content of the attribute of the <code>tag</code> using the Python <code>get()</code> method.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14d8b663-d3b9-4d19-bc57-bc643282d20a",
   "metadata": {},
   "outputs": [],
   "source": [
    "tag_child.get('id')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76d9a613-bcb8-4cbb-abef-39792dedc4bd",
   "metadata": {},
   "source": [
    "### Navigable String\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "124a0650-7e25-46f8-99f8-2963351178ad",
   "metadata": {},
   "source": [
    "A string corresponds to a bit of text or content within a tag. Beautiful Soup uses the <code>NavigableString</code> class to contain this text. In our HTML we can obtain the name of the first player by extracting the string of the <code>Tag</code> object <code>tag_child</code> as follows:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "287db3a7-4571-45e9-ad6e-1093e6c17e82",
   "metadata": {},
   "outputs": [],
   "source": [
    "tag_string=tag_child.string\n",
    "tag_string"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2600eb5-9f7f-486a-95d7-fe288cc70210",
   "metadata": {},
   "source": [
    "we can verify the type is Navigable String\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45e979f4-01f7-409e-9150-96f0dd60a451",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(tag_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93f7fd9c-005a-46ff-88fe-0461dbcdcbd2",
   "metadata": {},
   "source": [
    "A NavigableString is similar to a Python string or Unicode string. To be more precise, the main difference is that it also supports some <code>BeautifulSoup</code> features. We can convert it to string object in Python:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fdf5d29-d2e0-417b-9f93-a29835e0a304",
   "metadata": {},
   "outputs": [],
   "source": [
    "unicode_string = str(tag_string)\n",
    "unicode_string"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb605155-a3e3-469b-8e5f-b8ae06774ec7",
   "metadata": {},
   "source": [
    "<h2 id=\"filter\">Filter</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12111d69-8e1d-4d19-8a34-af53b0c966fb",
   "metadata": {},
   "source": [
    "Filters allow you to find complex patterns, the simplest filter is a string. In this section we will pass a string to a different filter method and Beautiful Soup will perform a match against that exact string. Consider the following HTML of rocket launches:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a95fc178-0f46-4176-be9d-4839fb268c29",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%html\n",
    "<table>\n",
    "  <tr>\n",
    "    <td id='flight' >Flight No</td>\n",
    "    <td>Launch site</td> \n",
    "    <td>Payload mass</td>\n",
    "   </tr>\n",
    "  <tr> \n",
    "    <td>1</td>\n",
    "    <td><a href='https://en.wikipedia.org/wiki/Florida'>Florida</a></td>\n",
    "    <td>300 kg</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>2</td>\n",
    "    <td><a href='https://en.wikipedia.org/wiki/Texas'>Texas</a></td>\n",
    "    <td>94 kg</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>3</td>\n",
    "    <td><a href='https://en.wikipedia.org/wiki/Florida'>Florida<a> </td>\n",
    "    <td>80 kg</td>\n",
    "  </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8ac9fae-596b-4874-a34d-b300e0dd495a",
   "metadata": {},
   "source": [
    "We can store it as a string in the variable <code>table</code>:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "902b4b89-007e-416a-83ec-191640545ab7",
   "metadata": {},
   "outputs": [],
   "source": [
    "table=\"<table><tr><td id='flight'>Flight No</td><td>Launch site</td> <td>Payload mass</td></tr><tr> <td>1</td><td><a href='https://en.wikipedia.org/wiki/Florida'>Florida<a></td><td>300 kg</td></tr><tr><td>2</td><td><a href='https://en.wikipedia.org/wiki/Texas'>Texas</a></td><td>94 kg</td></tr><tr><td>3</td><td><a href='https://en.wikipedia.org/wiki/Florida'>Florida<a> </td><td>80 kg</td></tr></table>\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5191e9a-b3bb-4c31-ae97-abc49e80c084",
   "metadata": {},
   "outputs": [],
   "source": [
    "table_bs = BeautifulSoup(table, 'html5lib')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5c7b9d1-89b9-4bd7-82df-5df169756cf0",
   "metadata": {},
   "source": [
    "## find All\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c078ee17-5d84-4bf3-b789-debac2154be0",
   "metadata": {},
   "source": [
    "The <code>find_all()</code> method looks through a tag's descendants and retrieves all descendants that match your filters.\n",
    "\n",
    "<p>\n",
    "The Method signature for <code>find_all(name, attrs, recursive, string, limit, **kwargs)<c/ode>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9022435e-0735-437d-9bb3-c17db7d90730",
   "metadata": {},
   "source": [
    "### Name\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f31f53fd-69e8-47c8-9a40-25994d8ee35c",
   "metadata": {},
   "source": [
    "When we set the <code>name</code> parameter to a tag name, the method will extract all the tags with that name and its children.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79aa9887-9716-4f12-a698-0c1dc4a78a9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "table_rows=table_bs.find_all('tr')\n",
    "table_rows"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35d00cff-23b4-4909-b8b5-1241255b128a",
   "metadata": {},
   "source": [
    "The result is a Python iterable just like a list, each element is a <code>tag</code> object:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59ab6f99-4a39-424d-8e47-a81d4e408a4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "first_row =table_rows[0]\n",
    "first_row"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a522e03e-0fa7-4cfb-90b5-e905c1fa21b0",
   "metadata": {},
   "source": [
    "The type is <code>tag</code>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a70f0810-845e-410b-8d7b-7f93ed1f0cbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(type(first_row))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3234c67-244e-43f3-ba83-ccd5ce544f36",
   "metadata": {},
   "source": [
    "we can obtain the child\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67032b46-397d-45ef-acbf-db24292a618e",
   "metadata": {},
   "outputs": [],
   "source": [
    "first_row.td"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f2c7e07-6890-40e4-9983-9aa999be3024",
   "metadata": {},
   "source": [
    "If we iterate through the list, each element corresponds to a row in the table:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b7f6b69-83f0-4957-89fb-d59d45f8e73d",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i,row in enumerate(table_rows):\n",
    "    print(\"row\",i,\"is\",row)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c831a4f8-e830-4d07-b585-5856fbc5e293",
   "metadata": {},
   "source": [
    "As <code>row</code> is a <code>cell</code> object, we can apply the method <code>find_all</code> to it and extract table cells in the object <code>cells</code> using the tag <code>td</code>, this is all the children with the name <code>td</code>. The result is a list, each element corresponds to a cell and is a <code>Tag</code> object, we can iterate through this list as well. We can extract the content using the <code>string</code> attribute.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad25bd52-6942-43fe-be36-8f8ca8f00c1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i,row in enumerate(table_rows):\n",
    "    print(\"row\",i)\n",
    "    cells=row.find_all('td')\n",
    "    for j,cell in enumerate(cells):\n",
    "        print('colunm',j,\"cell\",cell)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e517291-6216-4c07-8f72-6dfd3151cd11",
   "metadata": {},
   "source": [
    "If we use a list we can match against any item in that list.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "909519b1-747c-4edb-a439-d0a140d285cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_input=table_bs .find_all(name=[\"tr\", \"td\"])\n",
    "list_input"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb20f027-24d1-46e2-acd9-9230e2cb1a98",
   "metadata": {},
   "source": [
    "## Attributes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc6bf343-9843-4a4c-bac1-4ae5f6d7c65a",
   "metadata": {},
   "source": [
    "If the argument is not recognized it will be turned into a filter on the tag's attributes. For example with the <code>id</code> argument, Beautiful Soup will filter against each tag's <code>id</code> attribute. For example, the first <code>td</code> elements have a value of <code>id</code> of <code>flight</code>, therefore we can filter based on that <code>id</code> value.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c5ea194-a66b-41d2-8f51-7ab4385eeb8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "table_bs.find_all(id=\"flight\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56082368-f4ce-4521-bc46-2a0395fea341",
   "metadata": {},
   "source": [
    "We can find all the elements that have links to the Florida Wikipedia page:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d24c9f3-b786-4992-9448-1aafcd9b5d79",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_input=table_bs.find_all(href=\"https://en.wikipedia.org/wiki/Florida\")\n",
    "list_input"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1f2cd56-f42a-42ad-ae32-81f9178fb705",
   "metadata": {},
   "source": [
    "If we set the <code>href</code> attribute to True, regardless of what the value is, the code finds all tags with <code>href</code> value:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f0830f6-f873-4a89-8f8c-6ab0305afb02",
   "metadata": {},
   "outputs": [],
   "source": [
    "table_bs.find_all(href=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9d7e377-fd5c-4caf-9b60-3d8b9b61f96b",
   "metadata": {},
   "source": [
    "There are other methods for dealing with attributes and other related methods. Check out the following <a href='https://www.crummy.com/software/BeautifulSoup/bs4/doc/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0101ENSkillsNetwork19487395-2021-01-01#css-selectors'>link</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d266821b-a905-43ba-ba7c-a18f15005d13",
   "metadata": {},
   "source": [
    "<h3 id=\"exer_type\">Exercise: <code>find_all</code></h3>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6055aca-6157-4714-ab71-2a15bf7ecd94",
   "metadata": {},
   "source": [
    "Using the logic above, find all the elements without <code>href</code> value\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55ec1cfd-333d-40e8-a0a6-dc196adfd0ce",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3bc49a47-6bdd-45aa-8fbd-e12f1098efc5",
   "metadata": {},
   "source": [
    "<details><summary>Click here for the solution</summary>\n",
    "\n",
    "```\n",
    "table_bs.find_all(href=False)\n",
    "\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "909ffd92-80a4-45dd-b4c4-4f56959144d2",
   "metadata": {},
   "source": [
    "Using the soup object <code>soup</code>, find the element with the <code>id</code> attribute content set to <code>\"boldest\"</code>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bcb1c69-5589-4ed1-84ca-51e143966d28",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "edbe2121-1bb0-4062-b575-c1fe5ab5f0bd",
   "metadata": {},
   "source": [
    "<details><summary>Click here for the solution</summary>\n",
    "\n",
    "```\n",
    "soup.find_all(id=\"boldest\")\n",
    "\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fc2663f-d26d-4945-a63b-1cfbf71e3650",
   "metadata": {},
   "source": [
    "### string\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56996303-a7ac-4414-be80-bcfa06f9b3fd",
   "metadata": {},
   "source": [
    "With string you can search for strings instead of tags, where we find all the elments with Florida:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7109decd-d839-43b3-838c-b6e6c2a07be2",
   "metadata": {},
   "outputs": [],
   "source": [
    "table_bs.find_all(string=\"Florida\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8e8ff9d-5fab-4f99-8905-da118ccc850b",
   "metadata": {},
   "source": [
    "## find\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a3ab777-5463-46ab-aa66-4e72c1b08f69",
   "metadata": {},
   "source": [
    "The <code>find_all()</code> method scans the entire document looking for results. It’s useful if you are looking for one element, as you can use the <code>find()</code> method to find the first element in the document. Consider the following two tables:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a680e1a6-493e-42f3-b5e8-1469b1678112",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%html\n",
    "<h3>Rocket Launch </h3>\n",
    "\n",
    "<p>\n",
    "<table class='rocket'>\n",
    "  <tr>\n",
    "    <td>Flight No</td>\n",
    "    <td>Launch site</td> \n",
    "    <td>Payload mass</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>1</td>\n",
    "    <td>Florida</td>\n",
    "    <td>300 kg</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>2</td>\n",
    "    <td>Texas</td>\n",
    "    <td>94 kg</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>3</td>\n",
    "    <td>Florida </td>\n",
    "    <td>80 kg</td>\n",
    "  </tr>\n",
    "</table>\n",
    "</p>\n",
    "<p>\n",
    "\n",
    "<h3>Pizza Party  </h3>\n",
    "  \n",
    "    \n",
    "<table class='pizza'>\n",
    "  <tr>\n",
    "    <td>Pizza Place</td>\n",
    "    <td>Orders</td> \n",
    "    <td>Slices </td>\n",
    "   </tr>\n",
    "  <tr>\n",
    "    <td>Domino's Pizza</td>\n",
    "    <td>10</td>\n",
    "    <td>100</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>Little Caesars</td>\n",
    "    <td>12</td>\n",
    "    <td >144 </td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td>Papa John's </td>\n",
    "    <td>15 </td>\n",
    "    <td>165</td>\n",
    "  </tr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9176fc69-b5a9-4112-9278-5cc91b42d30e",
   "metadata": {},
   "source": [
    "We store the HTML as a Python string and assign <code>two_tables</code>:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a53b596-5e6c-496e-8bf5-a86ab0a2fb7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "two_tables=\"<h3>Rocket Launch </h3><p><table class='rocket'><tr><td>Flight No</td><td>Launch site</td> <td>Payload mass</td></tr><tr><td>1</td><td>Florida</td><td>300 kg</td></tr><tr><td>2</td><td>Texas</td><td>94 kg</td></tr><tr><td>3</td><td>Florida </td><td>80 kg</td></tr></table></p><p><h3>Pizza Party  </h3><table class='pizza'><tr><td>Pizza Place</td><td>Orders</td> <td>Slices </td></tr><tr><td>Domino's Pizza</td><td>10</td><td>100</td></tr><tr><td>Little Caesars</td><td>12</td><td >144 </td></tr><tr><td>Papa John's </td><td>15 </td><td>165</td></tr>\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fd5a31e-dc14-4ea3-8584-1bc33cf37bce",
   "metadata": {},
   "source": [
    "We create a <code>BeautifulSoup</code> object  <code>two_tables_bs</code>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f57a3381-8706-4a2e-84fb-c47e55928c0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "two_tables_bs= BeautifulSoup(two_tables, 'html.parser')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36cfcf86-331d-4254-a716-048695ce3811",
   "metadata": {},
   "source": [
    "We can find the first table using the tag name table\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d97f91bd-c3e4-4d12-a249-934b4d6f4129",
   "metadata": {},
   "outputs": [],
   "source": [
    "two_tables_bs.find(\"table\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f2eb5f3-8335-422b-acdf-92baefdd4b33",
   "metadata": {},
   "source": [
    "We can filter on the class attribute to find the second table, but because class is a keyword in Python, we add an underscore to differentiate them.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32f78e48-7dc6-41d9-afb6-a25a17bc37ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "two_tables_bs.find(\"table\",class_='pizza')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0d6141e-1857-43f4-8f6d-f36cda0d3880",
   "metadata": {},
   "source": [
    "<h2 id=\"DSCW\">Downloading And Scraping The Contents Of A Web Page</h2> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05c920a1-1cf5-4f12-aaba-776e44a44ae1",
   "metadata": {},
   "source": [
    "We Download the contents of the web page:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4a3966f-83bd-44fb-8a34-47fd336450ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"http://www.ibm.com\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f4cd19c-1251-4718-84e6-c4aa6f7da5b5",
   "metadata": {},
   "source": [
    "We use <code>get</code> to download the contents of the webpage in text format and store in a variable called <code>data</code>:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23d69a77-86a9-435d-b168-5f6f4440878b",
   "metadata": {},
   "outputs": [],
   "source": [
    "data  = requests.get(url).text "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0cfe971-c682-47bd-a585-e063ca414d93",
   "metadata": {},
   "source": [
    "We create a <code>BeautifulSoup</code> object using the <code>BeautifulSoup</code> constructor\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "807d5ab6-82c7-419a-8170-a1b2d07224b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(data,\"html5lib\")  # create a soup object using the variable 'data'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "add38a60-3a59-4c97-9fc6-c6eacc0367ee",
   "metadata": {},
   "source": [
    "Scrape all links\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9846895-e50d-45e4-b88a-092864241378",
   "metadata": {},
   "outputs": [],
   "source": [
    "for link in soup.find_all('a',href=True):  # in html anchor/link is represented by the tag <a>\n",
    "\n",
    "    print(link.get('href'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69f70bfc-f12f-4d39-a4bf-c910452515de",
   "metadata": {},
   "source": [
    "## Scrape  all images  Tags\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a19a6f1-60e8-4cf6-a3ae-5244039d405e",
   "metadata": {},
   "outputs": [],
   "source": [
    "for link in soup.find_all('img'):# in html image is represented by the tag <img>\n",
    "    print(link)\n",
    "    print(link.get('src'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26d6240b-41bb-4622-8c8a-7d12c7ef5ec8",
   "metadata": {},
   "source": [
    "### Scrape data from HTML tables\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f025de2-77e6-498d-ba53-9200fdb87c07",
   "metadata": {},
   "outputs": [],
   "source": [
    "#The below url contains an html table with data about colors and color codes.\n",
    "url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/datasets/HTMLColorCodes.html\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e29b1358-ba06-40ad-87f8-da535a6e7e5f",
   "metadata": {},
   "source": [
    "Before proceeding to scrape a web site, you need to examine the contents and the way data is organized on the website. Open the above url in your browser and check how many rows and columns there are in the color table.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fb9549d-17ba-4d00-91c1-1d217d0cf479",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the contents of the webpage in text format and store in a variable called data\n",
    "data  = requests.get(url).text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a513a87-6a75-4419-a710-dea72269725e",
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(data,\"html5lib\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4913f435-7352-4cc7-96bf-ad6f1ba1bc0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#find a html table in the web page\n",
    "table = soup.find('table') # in html table is represented by the tag <table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "381b085d-7642-44cb-8be4-f1a5fa6aedad",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Get all rows from the table\n",
    "for row in table.find_all('tr'): # in html table row is represented by the tag <tr>\n",
    "    # Get all columns in each row.\n",
    "    cols = row.find_all('td') # in html a column is represented by the tag <td>\n",
    "    color_name = cols[2].string # store the value in column 3 as color_name\n",
    "    color_code = cols[3].text # store the value in column 4 as color_code\n",
    "    print(\"{}--->{}\".format(color_name,color_code))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cebce67-9842-4893-920e-11ae2846cfe4",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d5e0340-b356-4ca9-bdaf-a24de8d98c54",
   "metadata": {},
   "source": [
    "Ramesh Sannareddy\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ab4eb5a-09ba-4f38-ab41-9c4b8a07b8cc",
   "metadata": {},
   "source": [
    "### Other Contributors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b473f637-c10e-4f38-9de4-cdb6cdac694f",
   "metadata": {},
   "source": [
    "Rav Ahuja\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "563810b9-65c8-4263-9cee-15d33ad99269",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c849df3-b0b8-4fad-842b-a54da2ff8d77",
   "metadata": {},
   "source": [
    "| Date (YYYY-MM-DD) | Version | Changed By            |          Change Description         |\n",
    "| ----------------- | ------- | ----------------------| ----------------------------------- |\n",
    "| 2023-06-11        | 1.0     | Akansha Yadav         |   Spell check                       |\n",
    "| 2020-10-17        | 0.1     | Joseph Santarcangelo  |  Created initial version of the lab |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d01d509-bd6a-4a89-b8e5-eadf2860d3f7",
   "metadata": {},
   "source": [
    "Copyright © 2020 IBM Corporation. This notebook and its source code are released under the terms of the [MIT License](https://cognitiveclass.ai/mit-license/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0101ENSkillsNetwork19487395-2021-01-01).\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}