allofplos_presentation.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<style type=\"text/css\">\n",
    ".reveal {\n",
    "    font-size: 225%;\n",
    "    }\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h1>Mining scientific articles</h1>\n",
    "\n",
    "<h2>with XML, Python, and allofplos</h2>\n",
    "<br>\n",
    "<br>\n",
    "<center> \n",
    "<h4> Elizabeth Seiver<br>\n",
    "<a href=\"http://plos.org\">plos.org</a>, <a href=\"http://twitter.com/tweetotaler\">@tweetotaler</a><br>\n",
    "PLOS and The Hacker Within<br>\n",
    "Wed November 29, 2017</h4>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Why Mine Scientific Articles?\n",
    "* Science articles represent scientific knowledge\n",
    "* XML is version of articles for machines, while PDF is for humans\n",
    "* Tool for meta-research and meta-science\n",
    "* Quickly identify sets of articles of interest\n",
    "* Identify research literature trends over time (study findings, jargon usage, citation networks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# PLOS corpus of articles\n",
    "* 220,000+ scientific articles from a wide array of research fields, focusing on the medical and life sciences\n",
    "* Since 2003\n",
    "* Open Access: free to read, free to re-use\n",
    "* Creative Commons license (CC-BY, CC0)\n",
    "* Many scientific articles are behind a paywall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Tutorial plan\n",
    "* Goal: to enable research questions about science articles\n",
    "* Will use JupyterHub and a sample corpus of 10,000 randomly selected PLOS articles\n",
    "* Won't be discussing research techniques (data analysis, natural language processing)\n",
    "* Assumes basic Python knowledge (lists, dictionaries, loops, conditionals, datetime)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Tutorial structure\n",
    "1. How to parse XML using allofplos and [lxml](http://lxml.de/tutorial.html)\n",
    "2. Basic structure of XML documents and [JATS standard](https://jats.nlm.nih.gov/)\n",
    "3. Example projects with the PLOS test corpus\n",
    "4. Hacking session: parse articles or contribute to the allofplos codebase  \n",
    "\n",
    "Exercises based on tutorial: https://github.com/eseiver/xml_tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# allofplos\n",
    "* Python package for both downloading and parsing PLOS XML articles\n",
    "* Turns PLOS XML articles into Python data structures\n",
    "* Doesn't require knowledge of XML to use\n",
    "* Focuses on article metadata (e.g., title, authors, date of publication)\n",
    "* Work in progress, so aspects of it may change"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# allofplos basics\n",
    "* Initialize an article object w/DOI or XML filename\n",
    "    * DOI (Digital Object Identifier) is a unique identifier for an online document/article\n",
    "    * All PLOS DOIs start with `\"10.1371/journal.\"`, like `\"10.1371/journal.pone.0185809\"`\n",
    "* allofplos XML files are named with last part of DOI, e.g. `\"journal.pone.0185809.xml\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Physician assessments of drug seeking behavior: A mixed methods study'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from allofplos import Article # if have run `pip install allofplos`\n",
    "# from article_class import Article  # if inside cloned GitHub directory\n",
    "\n",
    "# first instantiation of Article class by DOI\n",
    "article = Article('10.1371/journal.pone.0178690')\n",
    "article.title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'THPdb: Database of FDA-approved peptide and protein therapeutics'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# first instantiation of Article class by filename\n",
    "article = Article.from_filename('allofplos_xml/journal.pone.0181748.xml')\n",
    "article.title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'A checklist is associated with increased quality of reporting preclinical biomedical research: A systematic review'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# new article\n",
    "article.doi = '10.1371/journal.pone.0183591'\n",
    "article.title"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Notable properties\n",
    "Try printing or returning some of these values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Basic metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'10.1371/journal.pone.0183591'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.doi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'PLOS ONE'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.journal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.datetime(2017, 9, 13, 0, 0)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.pubdate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'A checklist is associated with increased quality of reporting preclinical biomedical research: A systematic review'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'fig-count': '3', 'page-count': '14', 'table-count': '2'}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4954"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.word_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Irreproducibility of preclinical biomedical research has gained recent attention. It is suggested that requiring authors to complete a checklist at the time of manuscript submission would improve the quality and transparency of scientific reporting, and ultimately enhance reproducibility. Whether a checklist \n"
     ]
    }
   ],
   "source": [
    "print(article.abstract[:310])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## People"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['contrib_initials', 'given_names', 'surname', 'group_name', 'ids', 'rid_dict', 'contrib_type', 'author_type', 'editor_type', 'email', 'affiliations', 'author_roles', 'footnotes'])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "contributor = article.contributors[0]\n",
    "contributor.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'affiliations': ['Division of Pulmonary, Allergy, and Critical Care Medicine, Department of Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania, United States of America'],\n",
       " 'author_roles': {'CASRAI CREDiT taxonomy': ['Conceptualization',\n",
       "   'Data curation',\n",
       "   'Formal analysis',\n",
       "   'Funding acquisition',\n",
       "   'Investigation',\n",
       "   'Writing – original draft',\n",
       "   'Writing – review & editing']},\n",
       " 'author_type': 'corresponding',\n",
       " 'contrib_initials': 'SH',\n",
       " 'contrib_type': 'author',\n",
       " 'editor_type': None,\n",
       " 'email': ['shan.workmd@gmail.com'],\n",
       " 'footnotes': ['Current address: Division of Pulmonary and Critical Care, Department of Medicine, Northwestern University, Chicago, Illinois, United States of America'],\n",
       " 'given_names': 'SeungHye',\n",
       " 'group_name': None,\n",
       " 'ids': [{'authenticated': 'true',\n",
       "   'id': 'http://orcid.org/0000-0001-5625-6337',\n",
       "   'id_type': 'orcid'}],\n",
       " 'rid_dict': {'aff': ['aff001'],\n",
       "  'corresp': ['cor001'],\n",
       "  'fn': ['currentaff001']},\n",
       " 'surname': 'Han'}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.authors[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'affiliations': ['Division of Pulmonary, Allergy, and Critical Care Medicine, Department of Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania, United States of America'],\n",
       "  'author_roles': {'CASRAI CREDiT taxonomy': ['Conceptualization',\n",
       "    'Data curation',\n",
       "    'Formal analysis',\n",
       "    'Funding acquisition',\n",
       "    'Investigation',\n",
       "    'Writing – original draft',\n",
       "    'Writing – review & editing']},\n",
       "  'author_type': 'corresponding',\n",
       "  'contrib_initials': 'SH',\n",
       "  'contrib_type': 'author',\n",
       "  'editor_type': None,\n",
       "  'email': ['shan.workmd@gmail.com'],\n",
       "  'footnotes': ['Current address: Division of Pulmonary and Critical Care, Department of Medicine, Northwestern University, Chicago, Illinois, United States of America'],\n",
       "  'given_names': 'SeungHye',\n",
       "  'group_name': None,\n",
       "  'ids': [{'authenticated': 'true',\n",
       "    'id': 'http://orcid.org/0000-0001-5625-6337',\n",
       "    'id_type': 'orcid'}],\n",
       "  'rid_dict': {'aff': ['aff001'],\n",
       "   'corresp': ['cor001'],\n",
       "   'fn': ['currentaff001']},\n",
       "  'surname': 'Han'}]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.corr_author"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'affiliations': ['Fraunhofer Research Institution of Marine Biotechnology, GERMANY'],\n",
       " 'author_roles': {None: ['Editor']},\n",
       " 'author_type': None,\n",
       " 'contrib_initials': 'JB',\n",
       " 'contrib_type': 'editor',\n",
       " 'editor_type': None,\n",
       " 'email': None,\n",
       " 'footnotes': [],\n",
       " 'given_names': 'Johannes',\n",
       " 'group_name': None,\n",
       " 'ids': [],\n",
       " 'rid_dict': {'aff': ['edit1']},\n",
       " 'surname': 'Boltze'}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.editor[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Article type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'research-article'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.type_  # JATS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Research Article'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.plostype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "article.proof  # whether an uncorrected proof/early version or not"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Local article file (more on this later)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/Elizabeth/PLOS_Corpus_Project/allofplos/allofplos/allofplos_xml/journal.pone.0183591.xml'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.local"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<lxml.etree._ElementTree at 0x10efa9dc8>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Element article at 0x1118cf348>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.root"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "article.xml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Other notable methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'accepted': datetime.datetime(2017, 8, 7, 0, 0),\n",
       " 'collection': datetime.datetime(2017, 1, 1, 0, 0),\n",
       " 'epub': datetime.datetime(2017, 9, 13, 0, 0),\n",
       " 'received': datetime.datetime(2017, 3, 19, 0, 0)}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.get_dates()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'works'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article.check_if_doi_resolves()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DOI: 10.1371/journal.pone.0183591\n",
       "Title: A checklist is associated with increased quality of reporting preclinical biomedical research: A systematic review"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(article)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "hideCode": false,
    "hideOutput": false,
    "hidePrompt": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [
     "hidecode"
    ]
   },
   "outputs": [],
   "source": [
    "from display_xml import XML\n",
    "import lxml.etree as et\n",
    "test_file = 'element.xml'\n",
    "parser = et.XMLParser(remove_blank_text=True)\n",
    "test_tree = et.parse(test_file, parser)\n",
    "test_root = test_tree.getroot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidePrompt": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Primer on XML elements\n",
    "## Why use lxml.etree for parsing XML documents?\n",
    "* JATS standard for scientific article XML is consistent\n",
    "* BeautifulSoup better for unreliable web documents\n",
    "* lxml.etree and BeautifulSoup each have a module of the other\n",
    "* The following XML element examples are derived from actual PLOS articles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Primer on XML elements\n",
    "## XML elements have four key properties in the lxml.etree library\n",
    "1. element.tag\n",
    "2. element.text\n",
    "3. element.attrib\n",
    "4. element.tail\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Example basic element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "hidecode"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div class=ae439bd2c-17db-48c2-a611-faebe69cb28f> \n",
       "        <style>\n",
       "            div.ae439bd2c-17db-48c2-a611-faebe69cb28f .hll { background-color: #ffffcc }\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .c { color: #408080; font-style: italic } /* Comment */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .err { border: 1px solid #FF0000 } /* Error */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .k { color: #008000; font-weight: bold } /* Keyword */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .o { color: #666666 } /* Operator */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .gd { color: #A00000 } /* Generic.Deleted */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .ge { font-style: italic } /* Generic.Emph */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .gr { color: #FF0000 } /* Generic.Error */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .gi { color: #00A000 } /* Generic.Inserted */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .go { color: #888888 } /* Generic.Output */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .gs { font-weight: bold } /* Generic.Strong */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .gt { color: #0044DD } /* Generic.Traceback */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .kp { color: #008000 } /* Keyword.Pseudo */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .kt { color: #B00040 } /* Keyword.Type */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .m { color: #666666 } /* Literal.Number */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .s { color: #BA2121 } /* Literal.String */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .na { color: #7D9029 } /* Name.Attribute */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .nb { color: #008000 } /* Name.Builtin */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .no { color: #880000 } /* Name.Constant */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .nd { color: #AA22FF } /* Name.Decorator */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .nf { color: #0000FF } /* Name.Function */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .nl { color: #A0A000 } /* Name.Label */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .nv { color: #19177C } /* Name.Variable */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .w { color: #bbbbbb } /* Text.Whitespace */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .mb { color: #666666 } /* Literal.Number.Bin */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .mf { color: #666666 } /* Literal.Number.Float */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .mh { color: #666666 } /* Literal.Number.Hex */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .mi { color: #666666 } /* Literal.Number.Integer */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .mo { color: #666666 } /* Literal.Number.Oct */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .sc { color: #BA2121 } /* Literal.String.Char */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .sx { color: #008000 } /* Literal.String.Other */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .ss { color: #19177C } /* Literal.String.Symbol */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .fm { color: #0000FF } /* Name.Function.Magic */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .vc { color: #19177C } /* Name.Variable.Class */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .vg { color: #19177C } /* Name.Variable.Global */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .vi { color: #19177C } /* Name.Variable.Instance */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .vm { color: #19177C } /* Name.Variable.Magic */\n",
       "div.ae439bd2c-17db-48c2-a611-faebe69cb28f .il { color: #666666 } /* Literal.Number.Integer.Long */\n",
       "        </style>\n",
       "        <div class=\"highlight\"><pre><span></span><span class=\"nt\">&lt;article-title&gt;</span>Why Most Published Research Findings Are False<span class=\"nt\">&lt;/article-title&gt;</span>\n",
       "</pre></div>\n",
       "\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<display_xml.XML at 0x1087e1358>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element = test_root.getchildren()[0]\n",
    "XML(element)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'article-title'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element.tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Why Most Published Research Findings Are False'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Basic element with attribute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "hidecode"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div class=a41941cb7-0ea0-4517-a60a-8b24828417d2> \n",
       "        <style>\n",
       "            div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .hll { background-color: #ffffcc }\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .c { color: #408080; font-style: italic } /* Comment */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .err { border: 1px solid #FF0000 } /* Error */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .k { color: #008000; font-weight: bold } /* Keyword */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .o { color: #666666 } /* Operator */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .gd { color: #A00000 } /* Generic.Deleted */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .ge { font-style: italic } /* Generic.Emph */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .gr { color: #FF0000 } /* Generic.Error */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .gi { color: #00A000 } /* Generic.Inserted */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .go { color: #888888 } /* Generic.Output */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .gs { font-weight: bold } /* Generic.Strong */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .gt { color: #0044DD } /* Generic.Traceback */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .kp { color: #008000 } /* Keyword.Pseudo */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .kt { color: #B00040 } /* Keyword.Type */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .m { color: #666666 } /* Literal.Number */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .s { color: #BA2121 } /* Literal.String */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .na { color: #7D9029 } /* Name.Attribute */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .nb { color: #008000 } /* Name.Builtin */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .no { color: #880000 } /* Name.Constant */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .nd { color: #AA22FF } /* Name.Decorator */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .nf { color: #0000FF } /* Name.Function */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .nl { color: #A0A000 } /* Name.Label */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .nv { color: #19177C } /* Name.Variable */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .w { color: #bbbbbb } /* Text.Whitespace */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .mb { color: #666666 } /* Literal.Number.Bin */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .mf { color: #666666 } /* Literal.Number.Float */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .mh { color: #666666 } /* Literal.Number.Hex */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .mi { color: #666666 } /* Literal.Number.Integer */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .mo { color: #666666 } /* Literal.Number.Oct */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .sc { color: #BA2121 } /* Literal.String.Char */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .sx { color: #008000 } /* Literal.String.Other */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .ss { color: #19177C } /* Literal.String.Symbol */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .fm { color: #0000FF } /* Name.Function.Magic */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .vc { color: #19177C } /* Name.Variable.Class */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .vg { color: #19177C } /* Name.Variable.Global */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .vi { color: #19177C } /* Name.Variable.Instance */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .vm { color: #19177C } /* Name.Variable.Magic */\n",
       "div.a41941cb7-0ea0-4517-a60a-8b24828417d2 .il { color: #666666 } /* Literal.Number.Integer.Long */\n",
       "        </style>\n",
       "        <div class=\"highlight\"><pre><span></span><span class=\"nt\">&lt;alt-title</span> <span class=\"na\">alt-title-type=</span><span class=\"s\">&quot;running-head&quot;</span><span class=\"nt\">&gt;</span>Essay<span class=\"nt\">&lt;/alt-title&gt;</span>\n",
       "</pre></div>\n",
       "\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<display_xml.XML at 0x10ab20208>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element = test_root.getchildren()[1]\n",
    "XML(element)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'alt-title'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element.tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Essay'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'alt-title-type': 'running-head'}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element.attrib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# for any text that comes directly after closing tag and before another tag\n",
    "element.tail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## An element attribute is a dictionary\n",
    "### Appears inside the element tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "hideCode": false,
    "hidePrompt": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div class=a0c5aec32-ebe4-43a0-97da-e15494171906> \n",
       "        <style>\n",
       "            div.a0c5aec32-ebe4-43a0-97da-e15494171906 .hll { background-color: #ffffcc }\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .c { color: #408080; font-style: italic } /* Comment */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .err { border: 1px solid #FF0000 } /* Error */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .k { color: #008000; font-weight: bold } /* Keyword */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .o { color: #666666 } /* Operator */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .gd { color: #A00000 } /* Generic.Deleted */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .ge { font-style: italic } /* Generic.Emph */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .gr { color: #FF0000 } /* Generic.Error */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .gi { color: #00A000 } /* Generic.Inserted */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .go { color: #888888 } /* Generic.Output */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .gs { font-weight: bold } /* Generic.Strong */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .gt { color: #0044DD } /* Generic.Traceback */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .kp { color: #008000 } /* Keyword.Pseudo */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .kt { color: #B00040 } /* Keyword.Type */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .m { color: #666666 } /* Literal.Number */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .s { color: #BA2121 } /* Literal.String */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .na { color: #7D9029 } /* Name.Attribute */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .nb { color: #008000 } /* Name.Builtin */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .no { color: #880000 } /* Name.Constant */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .nd { color: #AA22FF } /* Name.Decorator */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .nf { color: #0000FF } /* Name.Function */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .nl { color: #A0A000 } /* Name.Label */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .nv { color: #19177C } /* Name.Variable */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .w { color: #bbbbbb } /* Text.Whitespace */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .mb { color: #666666 } /* Literal.Number.Bin */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .mf { color: #666666 } /* Literal.Number.Float */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .mh { color: #666666 } /* Literal.Number.Hex */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .mi { color: #666666 } /* Literal.Number.Integer */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .mo { color: #666666 } /* Literal.Number.Oct */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .sc { color: #BA2121 } /* Literal.String.Char */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .sx { color: #008000 } /* Literal.String.Other */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .ss { color: #19177C } /* Literal.String.Symbol */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .fm { color: #0000FF } /* Name.Function.Magic */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .vc { color: #19177C } /* Name.Variable.Class */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .vg { color: #19177C } /* Name.Variable.Global */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .vi { color: #19177C } /* Name.Variable.Instance */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .vm { color: #19177C } /* Name.Variable.Magic */\n",
       "div.a0c5aec32-ebe4-43a0-97da-e15494171906 .il { color: #666666 } /* Literal.Number.Integer.Long */\n",
       "        </style>\n",
       "        <div class=\"highlight\"><pre><span></span><span class=\"nt\">&lt;alt-title</span> <span class=\"na\">alt-title-type=</span><span class=\"s\">&quot;running-head&quot;</span><span class=\"nt\">&gt;</span>Essay<span class=\"nt\">&lt;/alt-title&gt;</span>\n",
       "</pre></div>\n",
       "\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<display_xml.XML at 0x10ab20400>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "XML(element)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'alt-title-type': 'running-head'}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element.attrib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'running-head'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element.attrib['alt-title-type']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'running-head'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element.attrib.get('alt-title-type')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## XML elements can have sub-elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "hidecode"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div class=a47bc8735-c47b-4534-a56b-04b86d0b3c5e> \n",
       "        <style>\n",
       "            div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .hll { background-color: #ffffcc }\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .c { color: #408080; font-style: italic } /* Comment */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .err { border: 1px solid #FF0000 } /* Error */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .k { color: #008000; font-weight: bold } /* Keyword */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .o { color: #666666 } /* Operator */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .gd { color: #A00000 } /* Generic.Deleted */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .ge { font-style: italic } /* Generic.Emph */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .gr { color: #FF0000 } /* Generic.Error */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .gi { color: #00A000 } /* Generic.Inserted */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .go { color: #888888 } /* Generic.Output */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .gs { font-weight: bold } /* Generic.Strong */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .gt { color: #0044DD } /* Generic.Traceback */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .kp { color: #008000 } /* Keyword.Pseudo */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .kt { color: #B00040 } /* Keyword.Type */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .m { color: #666666 } /* Literal.Number */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .s { color: #BA2121 } /* Literal.String */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .na { color: #7D9029 } /* Name.Attribute */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .nb { color: #008000 } /* Name.Builtin */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .no { color: #880000 } /* Name.Constant */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .nd { color: #AA22FF } /* Name.Decorator */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .nf { color: #0000FF } /* Name.Function */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .nl { color: #A0A000 } /* Name.Label */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .nv { color: #19177C } /* Name.Variable */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .w { color: #bbbbbb } /* Text.Whitespace */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .mb { color: #666666 } /* Literal.Number.Bin */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .mf { color: #666666 } /* Literal.Number.Float */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .mh { color: #666666 } /* Literal.Number.Hex */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .mi { color: #666666 } /* Literal.Number.Integer */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .mo { color: #666666 } /* Literal.Number.Oct */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .sc { color: #BA2121 } /* Literal.String.Char */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .sx { color: #008000 } /* Literal.String.Other */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .ss { color: #19177C } /* Literal.String.Symbol */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .fm { color: #0000FF } /* Name.Function.Magic */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .vc { color: #19177C } /* Name.Variable.Class */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .vg { color: #19177C } /* Name.Variable.Global */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .vi { color: #19177C } /* Name.Variable.Instance */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .vm { color: #19177C } /* Name.Variable.Magic */\n",
       "div.a47bc8735-c47b-4534-a56b-04b86d0b3c5e .il { color: #666666 } /* Literal.Number.Integer.Long */\n",
       "        </style>\n",
       "        <div class=\"highlight\"><pre><span></span><span class=\"nt\">&lt;title-group&gt;</span>\n",
       "  <span class=\"nt\">&lt;article-title&gt;</span>Why Most Published Research Findings Are False<span class=\"nt\">&lt;/article-title&gt;</span>\n",
       "  <span class=\"nt\">&lt;alt-title</span> <span class=\"na\">alt-title-type=</span><span class=\"s\">&quot;running-head&quot;</span><span class=\"nt\">&gt;</span>Essay<span class=\"nt\">&lt;/alt-title&gt;</span>\n",
       "<span class=\"nt\">&lt;/title-group&gt;</span>\n",
       "</pre></div>\n",
       "\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<display_xml.XML at 0x10ab206a0>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element = test_root\n",
    "XML(element)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Element article-title at 0x10ab16888>, <Element alt-title at 0x1087d06c8>]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to find direct descendants; don't need to know their tags\n",
    "element.getchildren()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'article-title'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_element = element.getchildren()[0]\n",
    "new_element.tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Element title-group at 0x10ab16588>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to find direct ancestor; don't need to know its tag\n",
    "new_element.getparent()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Finding sub-elements by name with xpath\n",
    "### Example element: Creative Commons License"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "hidecode"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div class=ae0944246-a0b4-4a63-83e3-0058c4ce4020> \n",
       "        <style>\n",
       "            div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .hll { background-color: #ffffcc }\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .c { color: #228B22 } /* Comment */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .err { color: #a61717; background-color: #e3d2d2 } /* Error */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .k { color: #8B008B; font-weight: bold } /* Keyword */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .ch { color: #228B22 } /* Comment.Hashbang */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .cm { color: #228B22 } /* Comment.Multiline */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .cp { color: #1e889b } /* Comment.Preproc */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .cpf { color: #228B22 } /* Comment.PreprocFile */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .c1 { color: #228B22 } /* Comment.Single */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .cs { color: #8B008B; font-weight: bold } /* Comment.Special */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .gd { color: #aa0000 } /* Generic.Deleted */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .ge { font-style: italic } /* Generic.Emph */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .gr { color: #aa0000 } /* Generic.Error */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .gi { color: #00aa00 } /* Generic.Inserted */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .go { color: #888888 } /* Generic.Output */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .gp { color: #555555 } /* Generic.Prompt */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .gs { font-weight: bold } /* Generic.Strong */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .gt { color: #aa0000 } /* Generic.Traceback */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .kc { color: #8B008B; font-weight: bold } /* Keyword.Constant */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .kd { color: #8B008B; font-weight: bold } /* Keyword.Declaration */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .kn { color: #8B008B; font-weight: bold } /* Keyword.Namespace */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .kp { color: #8B008B; font-weight: bold } /* Keyword.Pseudo */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .kr { color: #8B008B; font-weight: bold } /* Keyword.Reserved */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .kt { color: #00688B; font-weight: bold } /* Keyword.Type */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .m { color: #B452CD } /* Literal.Number */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .s { color: #CD5555 } /* Literal.String */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .na { color: #658b00 } /* Name.Attribute */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .nb { color: #658b00 } /* Name.Builtin */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .nc { color: #008b45; font-weight: bold } /* Name.Class */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .no { color: #00688B } /* Name.Constant */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .nd { color: #707a7c } /* Name.Decorator */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .ne { color: #008b45; font-weight: bold } /* Name.Exception */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .nf { color: #008b45 } /* Name.Function */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .nn { color: #008b45; text-decoration: underline } /* Name.Namespace */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .nt { color: #8B008B; font-weight: bold } /* Name.Tag */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .nv { color: #00688B } /* Name.Variable */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .ow { color: #8B008B } /* Operator.Word */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .w { color: #bbbbbb } /* Text.Whitespace */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .mb { color: #B452CD } /* Literal.Number.Bin */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .mf { color: #B452CD } /* Literal.Number.Float */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .mh { color: #B452CD } /* Literal.Number.Hex */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .mi { color: #B452CD } /* Literal.Number.Integer */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .mo { color: #B452CD } /* Literal.Number.Oct */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .sa { color: #CD5555 } /* Literal.String.Affix */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .sb { color: #CD5555 } /* Literal.String.Backtick */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .sc { color: #CD5555 } /* Literal.String.Char */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .dl { color: #CD5555 } /* Literal.String.Delimiter */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .sd { color: #CD5555 } /* Literal.String.Doc */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .s2 { color: #CD5555 } /* Literal.String.Double */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .se { color: #CD5555 } /* Literal.String.Escape */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .sh { color: #1c7e71; font-style: italic } /* Literal.String.Heredoc */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .si { color: #CD5555 } /* Literal.String.Interpol */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .sx { color: #cb6c20 } /* Literal.String.Other */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .sr { color: #1c7e71 } /* Literal.String.Regex */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .s1 { color: #CD5555 } /* Literal.String.Single */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .ss { color: #CD5555 } /* Literal.String.Symbol */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .bp { color: #658b00 } /* Name.Builtin.Pseudo */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .fm { color: #008b45 } /* Name.Function.Magic */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .vc { color: #00688B } /* Name.Variable.Class */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .vg { color: #00688B } /* Name.Variable.Global */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .vi { color: #00688B } /* Name.Variable.Instance */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .vm { color: #00688B } /* Name.Variable.Magic */\n",
       "div.ae0944246-a0b4-4a63-83e3-0058c4ce4020 .il { color: #B452CD } /* Literal.Number.Integer.Long */\n",
       "        </style>\n",
       "        <div class=\"highlight\"><pre><span></span><span class=\"nt\">&lt;license</span> <span class=\"na\">xmlns:xlink=</span><span class=\"s\">&quot;http://www.w3.org/1999/xlink&quot;</span> <span class=\"na\">xmlns:mml=</span><span class=\"s\">&quot;http://www.w3.org/1998/Math/MathML&quot;</span> <span class=\"na\">xlink:href=</span><span class=\"s\">&quot;http://creativecommons.org/licenses/by/4.0/&quot;</span> <span class=\"na\">xlink:type=</span><span class=\"s\">&quot;simple&quot;</span><span class=\"nt\">&gt;</span>\n",
       "  <span class=\"nt\">&lt;license-p&gt;</span>This is an open access article distributed under the terms of the <span class=\"nt\">&lt;ext-link</span> <span class=\"na\">ext-link-type=</span><span class=\"s\">&quot;uri&quot;</span> <span class=\"na\">xlink:href=</span><span class=\"s\">&quot;http://creativecommons.org/licenses/by/4.0/&quot;</span> <span class=\"na\">xlink:type=</span><span class=\"s\">&quot;simple&quot;</span><span class=\"nt\">&gt;</span>Creative Commons Attribution License<span class=\"nt\">&lt;/ext-link&gt;</span>, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.<span class=\"nt\">&lt;/license-p&gt;</span>\n",
       "<span class=\"nt\">&lt;/license&gt;</span>\n",
       "</pre></div>\n",
       "\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<display_xml.XML at 0x1087c3f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "parser = et.XMLParser(remove_blank_text=True)\n",
    "test_file = 'element_2.xml'\n",
    "test_tree = et.parse(test_file, parser)\n",
    "license = test_tree.xpath('.//license')[0]\n",
    "display(XML(license, style='perldoc'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### Xpath returns a list of search results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Element license-p at 0x10ab47488>]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# search direct descendants by name\n",
    "license.xpath('./license-p')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Element ext-link at 0x10ab445c8>]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# search descendants of direct descendants\n",
    "license.xpath('./license-p/ext-link')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Element ext-link at 0x10ab445c8>]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# search ALL descendants\n",
    "license.xpath('.//ext-link')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Warning 1: Multiple elements can have the same xpath location\n",
    "### Remember that it always returns a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "hidecode"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div class=a8f097436-5415-4276-94ec-1a4d59f009b2> \n",
       "        <style>\n",
       "            div.a8f097436-5415-4276-94ec-1a4d59f009b2 .hll { background-color: #ffffcc }\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .c { color: #408080; font-style: italic } /* Comment */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .err { border: 1px solid #FF0000 } /* Error */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .k { color: #008000; font-weight: bold } /* Keyword */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .o { color: #666666 } /* Operator */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .ch { color: #408080; font-style: italic } /* Comment.Hashbang */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .cm { color: #408080; font-style: italic } /* Comment.Multiline */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .cp { color: #BC7A00 } /* Comment.Preproc */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .cpf { color: #408080; font-style: italic } /* Comment.PreprocFile */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .c1 { color: #408080; font-style: italic } /* Comment.Single */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .cs { color: #408080; font-style: italic } /* Comment.Special */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .gd { color: #A00000 } /* Generic.Deleted */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .ge { font-style: italic } /* Generic.Emph */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .gr { color: #FF0000 } /* Generic.Error */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .gi { color: #00A000 } /* Generic.Inserted */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .go { color: #888888 } /* Generic.Output */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .gp { color: #000080; font-weight: bold } /* Generic.Prompt */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .gs { font-weight: bold } /* Generic.Strong */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .gt { color: #0044DD } /* Generic.Traceback */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .kc { color: #008000; font-weight: bold } /* Keyword.Constant */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .kp { color: #008000 } /* Keyword.Pseudo */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .kt { color: #B00040 } /* Keyword.Type */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .m { color: #666666 } /* Literal.Number */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .s { color: #BA2121 } /* Literal.String */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .na { color: #7D9029 } /* Name.Attribute */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .nb { color: #008000 } /* Name.Builtin */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .nc { color: #0000FF; font-weight: bold } /* Name.Class */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .no { color: #880000 } /* Name.Constant */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .nd { color: #AA22FF } /* Name.Decorator */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .ni { color: #999999; font-weight: bold } /* Name.Entity */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .ne { color: #D2413A; font-weight: bold } /* Name.Exception */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .nf { color: #0000FF } /* Name.Function */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .nl { color: #A0A000 } /* Name.Label */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .nt { color: #008000; font-weight: bold } /* Name.Tag */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .nv { color: #19177C } /* Name.Variable */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .w { color: #bbbbbb } /* Text.Whitespace */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .mb { color: #666666 } /* Literal.Number.Bin */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .mf { color: #666666 } /* Literal.Number.Float */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .mh { color: #666666 } /* Literal.Number.Hex */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .mi { color: #666666 } /* Literal.Number.Integer */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .mo { color: #666666 } /* Literal.Number.Oct */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .sa { color: #BA2121 } /* Literal.String.Affix */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .sb { color: #BA2121 } /* Literal.String.Backtick */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .sc { color: #BA2121 } /* Literal.String.Char */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .dl { color: #BA2121 } /* Literal.String.Delimiter */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .s2 { color: #BA2121 } /* Literal.String.Double */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .sh { color: #BA2121 } /* Literal.String.Heredoc */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .sx { color: #008000 } /* Literal.String.Other */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .sr { color: #BB6688 } /* Literal.String.Regex */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .s1 { color: #BA2121 } /* Literal.String.Single */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .ss { color: #19177C } /* Literal.String.Symbol */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .bp { color: #008000 } /* Name.Builtin.Pseudo */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .fm { color: #0000FF } /* Name.Function.Magic */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .vc { color: #19177C } /* Name.Variable.Class */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .vg { color: #19177C } /* Name.Variable.Global */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .vi { color: #19177C } /* Name.Variable.Instance */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .vm { color: #19177C } /* Name.Variable.Magic */\n",
       "div.a8f097436-5415-4276-94ec-1a4d59f009b2 .il { color: #666666 } /* Literal.Number.Integer.Long */\n",
       "        </style>\n",
       "        <div class=\"highlight\"><pre><span></span><span class=\"nt\">&lt;contrib-group&gt;</span>\n",
       "  <span class=\"nt\">&lt;contrib</span> <span class=\"na\">contrib-type=</span><span class=\"s\">&quot;author&quot;</span><span class=\"nt\">&gt;</span>\n",
       "    <span class=\"nt\">&lt;name</span> <span class=\"na\">name-style=</span><span class=\"s\">&quot;western&quot;</span><span class=\"nt\">&gt;</span>\n",
       "      <span class=\"nt\">&lt;surname&gt;</span>dos Santos<span class=\"nt\">&lt;/surname&gt;</span>\n",
       "      <span class=\"nt\">&lt;given-names&gt;</span>Renato Vieira<span class=\"nt\">&lt;/given-names&gt;</span>\n",
       "    <span class=\"nt\">&lt;/name&gt;</span>\n",
       "  <span class=\"nt\">&lt;/contrib&gt;</span>\n",
       "  <span class=\"nt\">&lt;contrib</span> <span class=\"na\">contrib-type=</span><span class=\"s\">&quot;author&quot;</span><span class=\"nt\">&gt;</span>\n",
       "    <span class=\"nt\">&lt;name</span> <span class=\"na\">name-style=</span><span class=\"s\">&quot;western&quot;</span><span class=\"nt\">&gt;</span>\n",
       "      <span class=\"nt\">&lt;surname&gt;</span>da Silva<span class=\"nt\">&lt;/surname&gt;</span>\n",
       "      <span class=\"nt\">&lt;given-names&gt;</span>Linaena Mericy<span class=\"nt\">&lt;/given-names&gt;</span>\n",
       "    <span class=\"nt\">&lt;/name&gt;</span>\n",
       "  <span class=\"nt\">&lt;/contrib&gt;</span>\n",
       "<span class=\"nt\">&lt;/contrib-group&gt;</span>\n",
       "</pre></div>\n",
       "\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<display_xml.XML at 0x10ab20b38>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_to_display4 = \"\"\"<contrib-group>\n",
    "<contrib contrib-type=\"author\"><name name-style=\"western\"><surname>dos Santos</surname><given-names>Renato Vieira</given-names></name></contrib>\n",
    "<contrib contrib-type=\"author\"><name name-style=\"western\"><surname>da Silva</surname><given-names>Linaena Mericy</given-names></name></contrib>\n",
    "</contrib-group>\"\"\"\n",
    "element = et.fromstring(text_to_display4)\n",
    "XML(text_to_display4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Element contrib at 0x10ab44608>, <Element contrib at 0x10ab4d508>]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "element.xpath('./contrib')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Warning 2: element.text doesn't always work\n",
    "### When in doubt, use `lxml.etree.tostring()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "hidecode"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div class=a548cae77-ec4e-4753-af6e-ccbb6b022962> \n",
       "        <style>\n",
       "            div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .hll { background-color: #ffffcc }\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .c { color: #228B22 } /* Comment */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .err { color: #a61717; background-color: #e3d2d2 } /* Error */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .k { color: #8B008B; font-weight: bold } /* Keyword */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .ch { color: #228B22 } /* Comment.Hashbang */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .cm { color: #228B22 } /* Comment.Multiline */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .cp { color: #1e889b } /* Comment.Preproc */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .cpf { color: #228B22 } /* Comment.PreprocFile */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .c1 { color: #228B22 } /* Comment.Single */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .cs { color: #8B008B; font-weight: bold } /* Comment.Special */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .gd { color: #aa0000 } /* Generic.Deleted */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .ge { font-style: italic } /* Generic.Emph */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .gr { color: #aa0000 } /* Generic.Error */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .gh { color: #000080; font-weight: bold } /* Generic.Heading */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .gi { color: #00aa00 } /* Generic.Inserted */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .go { color: #888888 } /* Generic.Output */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .gp { color: #555555 } /* Generic.Prompt */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .gs { font-weight: bold } /* Generic.Strong */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .gu { color: #800080; font-weight: bold } /* Generic.Subheading */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .gt { color: #aa0000 } /* Generic.Traceback */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .kc { color: #8B008B; font-weight: bold } /* Keyword.Constant */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .kd { color: #8B008B; font-weight: bold } /* Keyword.Declaration */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .kn { color: #8B008B; font-weight: bold } /* Keyword.Namespace */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .kp { color: #8B008B; font-weight: bold } /* Keyword.Pseudo */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .kr { color: #8B008B; font-weight: bold } /* Keyword.Reserved */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .kt { color: #00688B; font-weight: bold } /* Keyword.Type */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .m { color: #B452CD } /* Literal.Number */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .s { color: #CD5555 } /* Literal.String */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .na { color: #658b00 } /* Name.Attribute */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .nb { color: #658b00 } /* Name.Builtin */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .nc { color: #008b45; font-weight: bold } /* Name.Class */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .no { color: #00688B } /* Name.Constant */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .nd { color: #707a7c } /* Name.Decorator */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .ne { color: #008b45; font-weight: bold } /* Name.Exception */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .nf { color: #008b45 } /* Name.Function */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .nn { color: #008b45; text-decoration: underline } /* Name.Namespace */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .nt { color: #8B008B; font-weight: bold } /* Name.Tag */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .nv { color: #00688B } /* Name.Variable */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .ow { color: #8B008B } /* Operator.Word */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .w { color: #bbbbbb } /* Text.Whitespace */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .mb { color: #B452CD } /* Literal.Number.Bin */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .mf { color: #B452CD } /* Literal.Number.Float */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .mh { color: #B452CD } /* Literal.Number.Hex */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .mi { color: #B452CD } /* Literal.Number.Integer */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .mo { color: #B452CD } /* Literal.Number.Oct */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .sa { color: #CD5555 } /* Literal.String.Affix */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .sb { color: #CD5555 } /* Literal.String.Backtick */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .sc { color: #CD5555 } /* Literal.String.Char */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .dl { color: #CD5555 } /* Literal.String.Delimiter */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .sd { color: #CD5555 } /* Literal.String.Doc */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .s2 { color: #CD5555 } /* Literal.String.Double */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .se { color: #CD5555 } /* Literal.String.Escape */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .sh { color: #1c7e71; font-style: italic } /* Literal.String.Heredoc */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .si { color: #CD5555 } /* Literal.String.Interpol */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .sx { color: #cb6c20 } /* Literal.String.Other */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .sr { color: #1c7e71 } /* Literal.String.Regex */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .s1 { color: #CD5555 } /* Literal.String.Single */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .ss { color: #CD5555 } /* Literal.String.Symbol */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .bp { color: #658b00 } /* Name.Builtin.Pseudo */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .fm { color: #008b45 } /* Name.Function.Magic */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .vc { color: #00688B } /* Name.Variable.Class */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .vg { color: #00688B } /* Name.Variable.Global */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .vi { color: #00688B } /* Name.Variable.Instance */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .vm { color: #00688B } /* Name.Variable.Magic */\n",
       "div.a548cae77-ec4e-4753-af6e-ccbb6b022962 .il { color: #B452CD } /* Literal.Number.Integer.Long */\n",
       "        </style>\n",
       "        <div class=\"highlight\"><pre><span></span><span class=\"nt\">&lt;license</span> <span class=\"na\">xmlns:xlink=</span><span class=\"s\">&quot;http://www.w3.org/1999/xlink&quot;</span> <span class=\"na\">xmlns:mml=</span><span class=\"s\">&quot;http://www.w3.org/1998/Math/MathML&quot;</span> <span class=\"na\">xlink:href=</span><span class=\"s\">&quot;http://creativecommons.org/licenses/by/4.0/&quot;</span> <span class=\"na\">xlink:type=</span><span class=\"s\">&quot;simple&quot;</span><span class=\"nt\">&gt;</span>\n",
       "  <span class=\"nt\">&lt;license-p&gt;</span>This is an open access article distributed under the terms of the <span class=\"nt\">&lt;ext-link</span> <span class=\"na\">ext-link-type=</span><span class=\"s\">&quot;uri&quot;</span> <span class=\"na\">xlink:href=</span><span class=\"s\">&quot;http://creativecommons.org/licenses/by/4.0/&quot;</span> <span class=\"na\">xlink:type=</span><span class=\"s\">&quot;simple&quot;</span><span class=\"nt\">&gt;</span>Creative Commons Attribution License<span class=\"nt\">&lt;/ext-link&gt;</span>, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.<span class=\"nt\">&lt;/license-p&gt;</span>\n",
       "<span class=\"nt\">&lt;/license&gt;</span>\n",
       "</pre></div>\n",
       "\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<display_xml.XML at 0x10ab20cf8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(XML(license, style='perldoc'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    }
   ],
   "source": [
    "print(license.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.\n"
     ]
    }
   ],
   "source": [
    "import lxml.etree as et\n",
    "license_text = et.tostring(license, method='text', encoding='unicode')\n",
    "print(license_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Quick quiz before we move on!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "hidecode"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "<contrib contrib-type=\"author\" equal-contrib=\"yes\">\n",
      "    <name name-style=\"western\">\n",
      "        <surname>Chen</surname>\n",
      "        <given-names>Ximing</given-names>\n",
      "    </name>plossy\n",
      "</contrib>\n"
     ]
    }
   ],
   "source": [
    "text_to_display3 = \"\"\"\n",
    "<contrib contrib-type=\"author\" equal-contrib=\"yes\">\n",
    "    <name name-style=\"western\">\n",
    "        <surname>Chen</surname>\n",
    "        <given-names>Ximing</given-names>\n",
    "    </name>plossy\n",
    "</contrib>\"\"\"\n",
    "element = et.fromstring(text_to_display3, parser)\n",
    "print(text_to_display3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* element.tag?\n",
    "* element.attrib?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "hideCode": true,
    "hidePrompt": false,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "hidecode"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "element.tag: contrib\n",
      "element.attrib: {'contrib-type': 'author', 'equal-contrib': 'yes'}\n"
     ]
    }
   ],
   "source": [
    "print('element.tag: {}'.format(element.tag))\n",
    "print('element.attrib: {}'.format(element.attrib))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Quick quiz before we move on!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "hidecode"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "<contrib contrib-type=\"author\" equal-contrib=\"yes\">\n",
      "    <name name-style=\"western\">\n",
      "        <surname>Chen</surname>\n",
      "        <given-names>Ximing</given-names>\n",
      "    </name>plossy\n",
      "</contrib>\n"
     ]
    }
   ],
   "source": [
    "text_to_display3 = \"\"\"\n",
    "<contrib contrib-type=\"author\" equal-contrib=\"yes\">\n",
    "    <name name-style=\"western\">\n",
    "        <surname>Chen</surname>\n",
    "        <given-names>Ximing</given-names>\n",
    "    </name>plossy\n",
    "</contrib>\"\"\"\n",
    "element = et.fromstring(text_to_display3, parser)\n",
    "print(text_to_display3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "new_element = element.xpath('./name')[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* new_element.tag?\n",
    "* new_element.attrib?\n",
    "* new_element.tail?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "hideCode": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "hidecode"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "new_element.tag: name\n",
      "new_element.attrib: {'name-style': 'western'}\n",
      "new_element.tail: plossy\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('new_element.tag: {}'.format(new_element.tag))\n",
    "print('new_element.attrib: {}'.format(new_element.attrib))\n",
    "print('new_element.tail: {}'.format(new_element.tail))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Using lxml.etree and allofplos Article class to parse XML files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/Elizabeth/PLOS_Corpus_Project/allofplos/allofplos/allofplos_xml/journal.pone.0183591.xml'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from allofplos import Article\n",
    "doi = '10.1371/journal.pone.0183591'\n",
    "article = Article(doi)\n",
    "article.filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Is the article XML file locally stored?\n",
    "article.local"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Element body at 0x10b176a48>]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree = article.tree\n",
    "tree.xpath('./body')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Element license at 0x10ab58488>]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xml_root = article.root\n",
    "xml_root.xpath('.//license')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'{http://www.w3.org/1999/xlink}href': 'http://creativecommons.org/licenses/by/4.0/', '{http://www.w3.org/1999/xlink}type': 'simple'}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "license = xml_root.xpath('.//license')[0]\n",
    "license.attrib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Tying it all together!\n",
    "## Project example: which articles use PCR in their Methods section?\n",
    "## First, on finding elements with xpath searching tag AND attribute\n",
    "* Body of article is divided into sections ('sec')\n",
    "* Method section attribute of note: `{'sec-type': 'materials|methods'}` or `{'sec-type': 'methods'}`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<Element sec at 0x10b176b48>]\n"
     ]
    }
   ],
   "source": [
    "methods_sections = xml_root.xpath(\"//sec[@sec-type='materials|methods']\")\n",
    "print(methods_sections)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['10.1371/journal.pone.0128195', '10.1371/journal.pone.0165464', '10.1371/journal.pone.0136574', '10.1371/journal.pone.0072749', '10.1371/journal.pone.0060101']\n"
     ]
    }
   ],
   "source": [
    "from allofplos.samples.corpus_analysis import get_random_list_of_dois\n",
    "from allofplos.article_class import Article\n",
    "import lxml.etree as et\n",
    "\n",
    "# First get list of articles/DOIs\n",
    "dois = get_random_list_of_dois(count=50)\n",
    "pcr_list = []\n",
    "# Initialize first article object\n",
    "article = Article(dois[0])\n",
    "\n",
    "for doi in dois:\n",
    "    # Step 1: create new article object\n",
    "    article.doi = doi\n",
    "    xml_root = article.root\n",
    "    # Step 2: find Method sections\n",
    "    methods_sections = xml_root.xpath(\"//sec[@sec-type='materials|methods']\")\n",
    "    if not methods_sections:\n",
    "        methods_sections = xml_root.xpath(\"//sec[@sec-type='methods']\")\n",
    "    for sec in methods_sections:\n",
    "        # Step 3: turn the method sections into strings\n",
    "        method_string = et.tostring(sec, method='text', encoding='unicode')\n",
    "        # Step 4: add DOI if 'PCR' in string\n",
    "        if 'PCR' in method_string:\n",
    "            pcr_list.append(article.doi)\n",
    "            break\n",
    "        else:\n",
    "            pass\n",
    "\n",
    "print(pcr_list[0:5])"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "12px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "threshold": 4,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}