parsing10k.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:navy\"> Intro\n",
    "\n",
    "\n",
    "In this notebook we will apply REGEX & BeautifulSoup to find useful financial information in 10-Ks. In particular, we will extract text from Items 1A, 7, and 7A of 10-K."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:navy\"> STEP 1 : Import Libraries\n",
    "\n",
    "Note that, we will need parser for BeautifulSoup, there are many parsers, we will be using 'lxml' which can be pre-installed as follows & it help BeatifulSoup read HTML, XML documents:\n",
    "\n",
    "!pip install lxml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import requests to retrive Web Urls example HTML. TXT \n",
    "import requests\n",
    "\n",
    "# Import BeautifulSoup\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# import re module for REGEXes\n",
    "import re\n",
    "\n",
    "# import pandas\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:navy\"> STEP 2 : Get Apple's [AAPL] 2018 10-K \n",
    "\n",
    "Though we are using AAPL as example 10-K here, the pipeline being built is generic & can be used for other companies 10-K\n",
    " \n",
    "[SEC Website URL for 10-K (TEXT version)](https://www.sec.gov/Archives/edgar/data/320193/000032019318000145/0000320193-18-000145.txt)\n",
    "\n",
    "[SEC Website URL for 10-K (HTML version)](https://www.sec.gov/Archives/edgar/data/320193/000032019318000145/a10-k20189292018.htm)\n",
    "It will be good to view/study along in html format to see how the below code would apply.\n",
    "\n",
    "All the documents can be easily ssearched via CIK or company details via [SEC's search tool](https://www.sec.gov/cgi-bin/browse-edgar?CIK=0000320193&owner=exclude&action=getcompany&Find=Search)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the HTML data from the 2018 10-K from Apple\n",
    "r = requests.get('https://www.sec.gov/Archives/edgar/data/320193/000032019318000145/0000320193-18-000145.txt')\n",
    "raw_10k = r.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we print the `raw_10k` string we will see that it has many sections. In the code below, we print part of the `raw_10k` string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<SEC-DOCUMENT>0000320193-18-000145.txt : 20181105\n",
      "<SEC-HEADER>0000320193-18-000145.hdr.sgml : 20181105\n",
      "<ACCEPTANCE-DATETIME>20181105080140\n",
      "ACCESSION NUMBER:\t\t0000320193-18-000145\n",
      "CONFORMED SUBMISSION TYPE:\t10-K\n",
      "PUBLIC DOCUMENT COUNT:\t\t88\n",
      "CONFORMED PERIOD OF REPORT:\t20180929\n",
      "FILED AS OF DATE:\t\t20181105\n",
      "DATE AS OF CHANGE:\t\t20181105\n",
      "\n",
      "FILER:\n",
      "\n",
      "\tCOMPANY DATA:\t\n",
      "\t\tCOMPANY CONFORMED NAME:\t\t\tAPPLE INC\n",
      "\t\tCENTRAL INDEX KEY:\t\t\t0000320193\n",
      "\t\tSTANDARD INDUSTRIAL CLASSIFICATION:\tELECTRONIC COMPUTERS [3571]\n",
      "\t\tIRS NUMBER:\t\t\t\t942404110\n",
      "\t\tSTATE OF INCORPORATION:\t\t\tCA\n",
      "\t\tFISCAL YEAR END:\t\t\t0930\n",
      "\n",
      "\tFILING VALUES:\n",
      "\t\tFORM TYPE:\t\t10-K\n",
      "\t\tSEC ACT:\t\t1934 Act\n",
      "\t\tSEC FILE NUMBER:\t001-36743\n",
      "\t\tFILM NUMBER:\t\t181158788\n",
      "\n",
      "\tBUSINESS ADDRESS:\t\n",
      "\t\tSTREET 1:\t\tONE APPLE PARK WAY\n",
      "\t\tCITY:\t\t\tCUPERTINO\n",
      "\t\tSTATE:\t\t\tCA\n",
      "\t\tZIP:\t\t\t95014\n",
      "\t\tBUSINESS PHONE:\t\t(408) 996-1010\n",
      "\n",
      "\tMAIL ADDRESS:\t\n",
      "\t\tSTREET 1:\t\tONE APPLE PARK WAY\n",
      "\t\tCITY:\t\t\tCUPERTINO\n",
      "\t\tSTATE:\t\t\tCA\n",
      "\t\tZIP:\t\t\t95014\n",
      "\n",
      "\tFORMER COMPANY:\t\n",
      "\t\tFORMER CONFORMED NAME:\tAPPLE COMPUTER INC\n",
      "\t\tDATE OF NAME CHANGE:\t19970808\n",
      "</SEC-HEADER>\n",
      "<DOCUMENT>\n",
      "<TYPE>10-K\n",
      "<SEQUENCE>1\n",
      "<FILENAME>a10-k20189292018.htm\n",
      "<DESCRIPTION>10-K\n",
      "<TEXT>\n",
      "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\" \"http://www.w3.org/TR/html4/loose.dtd\">\n",
      "<html>\n",
      "\t<head>\n",
      "\t\t<!-- Document created using Wdesk 1 -->\n",
      "\t\t<!-- Copyright\n"
     ]
    }
   ],
   "source": [
    "print(raw_10k[0:1300])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:navy\"> STEP 3 : Apply REGEXes to find 10-K Section from the document\n",
    "\n",
    "For our purposes, we are only interested in the sections that contain the 10-K information. All the sections, including the 10-K are contained within the `<DOCUMENT>` and `</DOCUMENT>` tags. Each section within the document tags is clearly marked by a `<TYPE>` tag followed by the name of the section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Regex to find <DOCUMENT> tags\n",
    "doc_start_pattern = re.compile(r'<DOCUMENT>')\n",
    "doc_end_pattern = re.compile(r'</DOCUMENT>')\n",
    "# Regex to find <TYPE> tag prceeding any characters, terminating at new line\n",
    "type_pattern = re.compile(r'<TYPE>[^\\n]+')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define Span Indices using REGEXes\n",
    "\n",
    "Now, that we have the regexes defined, we will use the `.finditer()` method to match the regexes in the `raw_10k`. In the code below, we will create 3 lists:\n",
    "\n",
    "1. A list that holds the `.end()` index of each match of `doc_start_pattern`\n",
    "\n",
    "2. A list that holds the `.start()` index of each match of `doc_end_pattern`\n",
    "\n",
    "3. A list that holds the name of section from each match of `type_pattern`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create 3 lists with the span indices for each regex\n",
    "\n",
    "### There are many <Document> Tags in this text file, each as specific exhibit like 10-K, EX-10.17 etc\n",
    "### First filter will give us document tag start <end> and document tag end's <start> \n",
    "### We will use this to later grab content in between these tags\n",
    "doc_start_is = [x.end() for x in doc_start_pattern.finditer(raw_10k)]\n",
    "doc_end_is = [x.start() for x in doc_end_pattern.finditer(raw_10k)]\n",
    "\n",
    "### Type filter is interesting, it looks for <TYPE> with Not flag as new line, ie terminare there, with + sign\n",
    "### to look for any char afterwards until new line \\n. This will give us <TYPE> followed Section Name like '10-K'\n",
    "### Once we have have this, it returns String Array, below line will with find content after <TYPE> ie, '10-K' \n",
    "### as section names\n",
    "doc_types = [x[len('<TYPE>'):] for x in type_pattern.findall(raw_10k)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a Dictionary for the 10-K\n",
    "\n",
    "In the code below, we will create a dictionary which has the key `10-K` and as value the contents of the `10-K` section found above. To do this, we will create a loop, to go through all the sections found above, and if the section type is `10-K` then save it to the dictionary. Use the indices in  `doc_start_is` and `doc_end_is`to slice the `raw_10k` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "document = {}\n",
    "\n",
    "# Create a loop to go through each section type and save only the 10-K section in the dictionary\n",
    "for doc_type, doc_start, doc_end in zip(doc_types, doc_start_is, doc_end_is):\n",
    "    if doc_type == '10-K':\n",
    "        document[doc_type] = raw_10k[doc_start:doc_end]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n<TYPE>10-K\\n<SEQUENCE>1\\n<FILENAME>a10-k20189292018.htm\\n<DESCRIPTION>10-K\\n<TEXT>\\n<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\" \"http://www.w3.org/TR/html4/loose.dtd\">\\n<html>\\n\\t<head>\\n\\t\\t<!-- Document created using Wdesk 1 -->\\n\\t\\t<!-- Copyright 2018 Workiva -->\\n\\t\\t<title>Document</title>\\n\\t</head>\\n\\t<body style=\"font-family:Times New Roman;font-size:10pt;\">\\n<div><a name=\"s3540C27286EF5B0DA103CC59028B96BE\"></a></div><div style=\"line-height:120%;text-align:center;font-size:10pt;\"><div sty'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# display excerpt the document\n",
    "document['10-K'][0:500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:navy\"> STEP 3 : Apply REGEXes to find Item 1A, 7, and 7A under 10-K Section \n",
    "\n",
    "The items in this `document` can be found in four different patterns. For example Item 1A can be found in either of the following patterns:\n",
    "\n",
    "1. `>Item 1A`\n",
    "\n",
    "2. `>Item&#160;1A` \n",
    "\n",
    "3. `>Item&nbsp;1A`\n",
    "\n",
    "4. `ITEM 1A` \n",
    "\n",
    "In the code below we will write a single regular expression that can match all four patterns for Items 1A, 7, and 7A. Then use the `.finditer()` method to match the regex to `document['10-K']`.\n",
    "\n",
    "Note that Item 1B & Item 8 are added to find out end of section Item 1A & Item 7A subsequently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<re.Match object; span=(38318, 38327), match='>Item 1A.'>\n",
      "<re.Match object; span=(39347, 39356), match='>Item 1B.'>\n",
      "<re.Match object; span=(46148, 46156), match='>Item 7.'>\n",
      "<re.Match object; span=(47281, 47290), match='>Item 7A.'>\n",
      "<re.Match object; span=(48357, 48365), match='>Item 8.'>\n",
      "<re.Match object; span=(119131, 119140), match='>Item 1A.'>\n",
      "<re.Match object; span=(197023, 197032), match='>Item 1B.'>\n",
      "<re.Match object; span=(333318, 333326), match='>Item 7.'>\n",
      "<re.Match object; span=(729984, 729993), match='>Item 7A.'>\n",
      "<re.Match object; span=(741774, 741782), match='>Item 8.'>\n"
     ]
    }
   ],
   "source": [
    "# Write the regex\n",
    "regex = re.compile(r'(>Item(\\s|&#160;|&nbsp;)(1A|1B|7A|7|8)\\.{0,1})|(ITEM\\s(1A|1B|7A|7|8))')\n",
    "\n",
    "# Use finditer to math the regex\n",
    "matches = regex.finditer(document['10-K'])\n",
    "\n",
    "# Write a for loop to print the matches\n",
    "for match in matches:\n",
    "    print(match)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that each item is matched twice. This is because each item appears first in the index and then in the corresponding section. We will now have to remove the matches that correspond to the index. We will do this using Pandas in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the code below we will create a pandas dataframe with the following column names: `'item','start','end'`. In the `item` column save the `match.group()` in lower case letters, in the ` start` column save the `match.start()`, and in the `end` column save the ``match.end()`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item</th>\n",
       "      <th>start</th>\n",
       "      <th>end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&gt;item 1a.</td>\n",
       "      <td>38318</td>\n",
       "      <td>38327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&gt;item 1b.</td>\n",
       "      <td>39347</td>\n",
       "      <td>39356</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&gt;item 7.</td>\n",
       "      <td>46148</td>\n",
       "      <td>46156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&gt;item 7a.</td>\n",
       "      <td>47281</td>\n",
       "      <td>47290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>&gt;item 8.</td>\n",
       "      <td>48357</td>\n",
       "      <td>48365</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        item  start    end\n",
       "0  >item 1a.  38318  38327\n",
       "1  >item 1b.  39347  39356\n",
       "2   >item 7.  46148  46156\n",
       "3  >item 7a.  47281  47290\n",
       "4   >item 8.  48357  48365"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Matches\n",
    "matches = regex.finditer(document['10-K'])\n",
    "\n",
    "# Create the dataframe\n",
    "test_df = pd.DataFrame([(x.group(), x.start(), x.end()) for x in matches])\n",
    "\n",
    "test_df.columns = ['item', 'start', 'end']\n",
    "test_df['item'] = test_df.item.str.lower()\n",
    "\n",
    "# Display the dataframe\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Eliminate Unnecessary Characters\n",
    "\n",
    "As we can see, our dataframe, in particular the `item` column, contains some unnecessary characters such as `>` and periods `.`. In some cases, we will also get unicode characters such as `&#160;` and `&nbsp;`. In the code below, we will use the Pandas dataframe method `.replace()` with the keyword `regex=True` to replace all whitespaces, the above mentioned unicode characters, the `>` character, and the periods from our dataframe. We want to do this because we want to use the `item` column as our dataframe index later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item</th>\n",
       "      <th>start</th>\n",
       "      <th>end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>item1a</td>\n",
       "      <td>38318</td>\n",
       "      <td>38327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>item1b</td>\n",
       "      <td>39347</td>\n",
       "      <td>39356</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>item7</td>\n",
       "      <td>46148</td>\n",
       "      <td>46156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>item7a</td>\n",
       "      <td>47281</td>\n",
       "      <td>47290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>item8</td>\n",
       "      <td>48357</td>\n",
       "      <td>48365</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     item  start    end\n",
       "0  item1a  38318  38327\n",
       "1  item1b  39347  39356\n",
       "2   item7  46148  46156\n",
       "3  item7a  47281  47290\n",
       "4   item8  48357  48365"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get rid of unnesesary charcters from the dataframe\n",
    "test_df.replace('&#160;',' ',regex=True,inplace=True)\n",
    "test_df.replace('&nbsp;',' ',regex=True,inplace=True)\n",
    "test_df.replace(' ','',regex=True,inplace=True)\n",
    "test_df.replace('\\.','',regex=True,inplace=True)\n",
    "test_df.replace('>','',regex=True,inplace=True)\n",
    "\n",
    "# display the dataframe\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove Duplicates\n",
    "\n",
    "Now that we have removed all unnecessary characters form our dataframe, we can go ahead and remove the Item matches that correspond to the index. In the code below we will use the Pandas dataframe `.drop_duplicates()` method to only keep the last Item matches in the dataframe and drop the rest. Just as precaution ensure that the `start` column is sorted in ascending order before dropping the duplicates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item</th>\n",
       "      <th>start</th>\n",
       "      <th>end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>item1a</td>\n",
       "      <td>119131</td>\n",
       "      <td>119140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>item1b</td>\n",
       "      <td>197023</td>\n",
       "      <td>197032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>item7</td>\n",
       "      <td>333318</td>\n",
       "      <td>333326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>item7a</td>\n",
       "      <td>729984</td>\n",
       "      <td>729993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>item8</td>\n",
       "      <td>741774</td>\n",
       "      <td>741782</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     item   start     end\n",
       "5  item1a  119131  119140\n",
       "6  item1b  197023  197032\n",
       "7   item7  333318  333326\n",
       "8  item7a  729984  729993\n",
       "9   item8  741774  741782"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop duplicates\n",
    "pos_dat = test_df.sort_values('start', ascending=True).drop_duplicates(subset=['item'], keep='last')\n",
    "\n",
    "# Display the dataframe\n",
    "pos_dat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set Item to Index\n",
    "\n",
    "In the code below use the Pandas dataframe `.set_index()` method to set the `item`  column as the index of our dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>start</th>\n",
       "      <th>end</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>item</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>item1a</th>\n",
       "      <td>119131</td>\n",
       "      <td>119140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>item1b</th>\n",
       "      <td>197023</td>\n",
       "      <td>197032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>item7</th>\n",
       "      <td>333318</td>\n",
       "      <td>333326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>item7a</th>\n",
       "      <td>729984</td>\n",
       "      <td>729993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>item8</th>\n",
       "      <td>741774</td>\n",
       "      <td>741782</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         start     end\n",
       "item                  \n",
       "item1a  119131  119140\n",
       "item1b  197023  197032\n",
       "item7   333318  333326\n",
       "item7a  729984  729993\n",
       "item8   741774  741782"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set item as the dataframe index\n",
    "pos_dat.set_index('item', inplace=True)\n",
    "\n",
    "# display the dataframe\n",
    "pos_dat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Get The Financial Information From Each Item </b>\n",
    "\n",
    "The above dataframe contains the starting and end index of each match for Items 1A, 7, and 7A. In the code below, we will save all the text from the starting index of `item1a` till the starting index of `item1b` into a variable called `item_1a_raw`. Similarly, save all the text from the starting index of `item7` till the starting index of `item7a` into a variable called `item_7_raw`. Finally,  save all the text from the starting index of `item7a` till the starting of `item8` into a variable called `item_7a_raw`. We can accomplish all of this by making the correct slices of `document['10-K']`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get Item 1a\n",
    "item_1a_raw = document['10-K'][pos_dat['start'].loc['item1a']:pos_dat['start'].loc['item1b']]\n",
    "\n",
    "# Get Item 7\n",
    "item_7_raw = document['10-K'][pos_dat['start'].loc['item7']:pos_dat['start'].loc['item7a']]\n",
    "\n",
    "# Get Item 7a\n",
    "item_7a_raw = document['10-K'][pos_dat['start'].loc['item7a']:pos_dat['start'].loc['item8']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have each item saved into a separate variable we can view them separately. For illustration purposes we will display Item 1a, but the other items will look similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'>Item 1A.</font></div></td><td style=\"vertical-align:top;\"><div style=\"line-height:120%;text-align:justify;font-size:9pt;\"><font style=\"font-family:Helvetica,sans-serif;font-size:9pt;font-weight:bold;\">Risk Factors</font></div></td></tr></table><div style=\"line-height:120%;padding-top:8px;text-align:justify;font-size:9pt;\"><font style=\"font-family:Helvetica,sans-serif;font-size:9pt;\">The following discussion of risk factors contains forward-looking statements. These risk factors may be important to understanding other statements in this Form 10-K. The following information should be read in conjunction with Part II, Item&#160;7, &#8220;Management&#8217;s Discussion and Analysis of Financial Condition and Results of Operations&#8221; and the consolidated financial statements and related notes in Part II, Item&#160;8, &#8220;Financial Statements and Supplementary Data&#8221; of this Form 10-K.</font></div><div style=\"line-height:120%;padding-top:16px;text-align:justify;font-size:9pt;\"><f'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "item_1a_raw[0:1000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the items looks pretty messy, they contain HTML tags, Unicode characters, etc... Before we can do a proper Natural Language Processing in these items we need to clean them up. This means we need to remove all HTML Tags, unicode characters, etc... In principle we could do this using regex substitutions as we learned previously, but his can be rather difficult. Luckily, packages already exist that can do all the cleaning for us, such as **Beautifulsoup**, let's make use of this to refine the extracted text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:navy\"> STEP 4 : Apply BeautifulSoup to refine the content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "### First convert the raw text we have to exrtacted to BeautifulSoup object \n",
    "item_1a_content = BeautifulSoup(item_1a_raw, 'lxml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html>\n",
      " <body>\n",
      "  <p>\n",
      "   &gt;Item 1A.\n",
      "  </p>\n",
      "  <td style=\"vertical-align:top;\">\n",
      "   <div style=\"line-height:120%;text-align:justify;font-size:9pt;\">\n",
      "    <font style=\"font-family:Helvetica,sans-serif;font-size:9pt;font-weight:bold;\">\n",
      "     Risk Factors\n",
      "    </font>\n",
      "   </div>\n",
      "  </td>\n",
      "  <div style=\"line-height:120%;padding-top:8px;text-align:justify;font-size:9pt;\">\n",
      "   <font style=\"font-family:Helvetica,sans-serif;font-size:9pt;\">\n",
      "    The following discussion of risk factors contains forward-looking statements. These risk factors may be important to understanding other statements in this Form 10-K. The following information should be read in conjunction with Part II, Item 7, “Management’s Discussion and Analysis of Financial Condition and Results of Operations” and the consolidated financial statements and related notes in Part II, Item 8, “Financial Statements and Supplementary Data” of this Form 10-K.\n",
      "   </font>\n",
      "  </div>\n",
      "  <div style=\"line-height:120%;padding-top:16px;text-align:justify;fon\n"
     ]
    }
   ],
   "source": [
    "### By just applying .pretiffy() we see that raw text start to look oragnized, as BeautifulSoup\n",
    "### apply indentation according to the HTML Tag tree structure\n",
    "print(item_1a_content.prettify()[0:1000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ">Item 1A.\n",
      "\n",
      "Risk Factors\n",
      "\n",
      "The following discussion of risk factors contains forward-looking statements. These risk factors may be important to understanding other statements in this Form 10-K. The following information should be read in conjunction with Part II, Item 7, “Management’s Discussion and Analysis of Financial Condition and Results of Operations” and the consolidated financial statements and related notes in Part II, Item 8, “Financial Statements and Supplementary Data” of this Form 10-K.\n",
      "\n",
      "The business, financial condition and operating results of the Company can be affected by a number of factors, whether currently known or unknown, including but not limited to those described below, any one or more of which could, directly or indirectly, cause the Company’s actual financial condition and operating results to vary materially from past, or from anticipated future, financial condition and operating results. Any of these factors, in whole or in part, could materially and adversely affect the Company’s business, financial condition, operating results and stock price.\n",
      "\n",
      "Because of the following factors, as well as other factors affecting the Company’s financial condition and operating results, past financial performance should not be considered to be a reliable indicator of future performance, and investors should not use historical trends to anticipate results or trends in future periods.\n",
      "\n",
      "Global and regional economic conditions could materially adversely affect the Company’s business, results of operations, financial condition and growth\n",
      "\n",
      ".\n",
      "\n",
      "The Company has international operations with sales outside the U.S. representing a majority of the Company’s total net sales. In addition, a majority of the Company’s supply chain, and its manufacturing and assembly activities, are located outside the U.S. As a result, the Company’s operations and performance depend significantly on global and regional economic conditions.\n",
      "\n",
      "Adverse macroeconomic conditions, including inflation, slower growth or recession, new or increased tariffs, changes to fiscal and monetary policy, tighter credit, higher interest rates, high unemployment and currency fluctuations could materially adversely affect demand for the Company’s products and services. In addition, consumer confidence and spending could be adversely affected in response to financial market volatility, negative financial news, conditions in the real estate and mortgage markets, declines in income or asset values, changes to fuel and other energy costs, labor and healthcare costs and other economic factors. \n",
      "\n",
      "In addition to an adverse impact on demand for the Company’s products, uncertainty about, or a decline in, global or regional economic conditions could have a significant impact on the Company’s suppliers, contract manufacturers, logistics providers, distributors, cellular network carriers and other channel partners. Potential effects include financial instability; inability to obtain credit to finance operations and purchases of the Company’s products; and insolvency.\n",
      "\n",
      "A downturn in the economic environment could also lead to increased credit and collectibility risk on the Company’s trade receivables; the failure of derivative counterparties and other financial institutions; limitations on the Company’s ability to issue new debt; reduced liquidity; and declines in the fair value of the Company’s financial instruments. These and other economic factors could materially adversely affect the Company’s business, results of operations, financial condition and growth.\n",
      "\n",
      "Global markets for the Company’s products and services are highly competitive and subject to rapid technological change, and the Company may be unable to compete effectively in these markets.\n",
      "\n",
      "The Company’s products and services are offered in highly competitive global markets characterized by aggressive price competition and resulting downward pressure on gross margins, frequent introduction of new products and services, short product life cycles, evolving industry standards, continual improvement in product price/performance characteristics, rapid adoption of technological advancements by competitors and price sensitivity on the part of consumers and businesses.\n",
      "\n",
      "The Company’s ability to compete successfully depends heavily on its ability to ensure a continuing and timely introduction of innovative new products, services and technologies to the marketplace. The Company believes it is unique in that it designs and develops nearly the entire solution for its products, including the hardware, operating system, numerous software applications and related services. As a result, the Company must make significant investments in R&D. There can be no assurance that these investments will achieve expected returns, and the Company may not be able to develop and market new products and services successfully.\n",
      "\n",
      "The Company currently holds a significant number of patents and copyrights and has registered, and applied to register, numerous patents, trademarks and service marks. In contrast, many of the Company’s competitors seek to compete primarily through aggressive pricing and very low cost structures, and emulating the Company’s products and infringing on its intellectual property. If the Company is unable to continue to develop and sell innovative new products with attractive margins or if competitors infringe on the Company’s intellectual property, the Company’s ability to maintain a competitive advantage could be adversely affected.\n",
      "\n",
      "Apple Inc. | 2018 Form 10-K | \n",
      "\n",
      "8\n",
      "\n",
      "The Company has a minority market share in the global smartphone, tablet and personal computer markets. The Company faces substantial competition in these markets from companies that have significant technical, marketing, distribution and other resources, as well as established hardware, software and digital content supplier relationships. In addition, some of the Company’s competitors have broader product lines, lower-priced products and a larger installed base of active devices. Competition has been particularly intense as competitors have aggressively cut prices and lowered product margins. Certain competitors may have the resources, experience or cost structures to provide products at little or no profit or even at a loss.\n",
      "\n",
      "Additionally, the Company faces significant competition as competitors attempt to imitate the Company’s product features and applications within their own products or, alternatively, collaborate with each other to offer solutions that are more competitive than those they currently offer. The Company also expects competition to intensify as competitors attempt to imitate the Company’s approach to providing components seamlessly within their individual offerings or work collaboratively to offer integrated solutions.\n",
      "\n",
      "Some of the markets in which the Company competes, including the market for personal computers, have from time to time experienced little to no growth or contracted. In addition, an increasing number of internet-enabled devices that include software applications and are smaller, simpler and cheaper than traditional personal computers compete with some of the Company’s existing products.\n",
      "\n",
      "The Company’s services also face substantial competition, including from companies that have significant resources and experience and have established service offerings with large customer bases. The Company competes with business models that provide content to users for free. The Company also competes with illegitimate means to obtain third-party digital content and applications.\n",
      "\n",
      "The Company’s financial condition and operating results depend substantially on the Company’s ability to continually improve its products and services in order to maintain their functional and design advantages. There can be no assurance the Company will be able to continue to provide products and services that compete effectively.\n",
      "\n",
      "To remain competitive and stimulate customer demand, the Company must successfully manage frequent introductions and transitions of products and services.\n",
      "\n",
      "Due to the highly volatile and competitive nature of the industries in which the Company competes, the Company must continually introduce new products, services and technologies, enhance existing products and services, effectively stimulate customer demand for new and upgraded products and services and successfully manage the transition to these new and upgraded products and services. The success of new product and service introductions depends on a number of factors including, but not limited to, timely and successful development, market acceptance, the Company’s ability to manage the risks associated with new product production ramp-up issues, the availability of application software for new products, the effective management of purchase commitments and inventory levels in line with anticipated product demand, the availability of products in appropriate quantities and at expected costs to meet anticipated demand and the risk that new products and services may have quality or other defects or deficiencies. Accordingly, the Company cannot determine in advance the ultimate effect of new product and service introductions and transitions.\n",
      "\n",
      "The Company depends on the performance of carriers, wholesalers, retailers and other resellers.\n",
      "\n",
      "The Company distributes its products through cellular network carriers, wholesalers, retailers and resellers, many of whom distribute products from competing manufacturers. The Company also sells its products and third-party products in most of its major markets directly to education, enterprise and government customers and consumers and small and mid-sized businesses through its retail and online stores.\n",
      "\n",
      "Some carriers providing cellular network service for iPhone offer financing, installment payment plans or subsidies for users’ purchases of the device. There is no assurance that such offers will be continued at all or in the same amounts upon renewal of the Company’s agreements with these carriers or in agreements the Company enters into with new carriers.\n",
      "\n",
      "The Company has invested and will continue to invest in programs to enhance reseller sales, including staffing selected resellers’ stores with Company employees and contractors, and improving product placement displays. These programs could require a substantial investment while providing no assurance of return or incremental revenue. The financial condition of these resellers could weaken, these resellers could stop distributing the Company’s products, or uncertainty regarding demand for some or all of the Company’s products could cause resellers to reduce their ordering and marketing of the Company’s products.\n",
      "\n",
      "Apple Inc. | 2018 Form 10-K | \n",
      "\n",
      "9\n",
      "\n",
      "The Company faces substantial inventory and other asset risk in addition to purchase commitment cancellation risk.\n",
      "\n",
      "The Company records a write-down for product and component inventories that have become obsolete or exceed anticipated demand, or for which cost exceeds net realizable value. The Company also accrues necessary cancellation fee reserves for orders of excess products and components. The Company reviews long-lived assets, including capital assets held at its suppliers’ facilities and inventory prepayments, for impairment whenever events or circumstances indicate the assets may not be recoverable. If the Company determines that an impairment has occurred, it records a write-down equal to the amount by which the carrying value of the asset exceeds its fair value. Although the Company believes its inventory, capital assets, inventory prepayments and other assets and purchase commitments are currently recoverable, no assurance can be given that the Company will not incur write-downs, fees, impairments and other charges given the rapid and unpredictable pace of product obsolescence in the industries in which the Company competes.\n",
      "\n",
      "The Company orders components for its products and builds inventory in advance of product announcements and shipments. Manufacturing purchase obligations cover the Company’s forecasted component and manufacturing requirements, typically for periods up to \n",
      "\n",
      "150 days\n",
      "\n",
      ". Because the Company’s markets are volatile, competitive and subject to rapid technology and price changes, there is a risk the Company will forecast incorrectly and order or produce excess or insufficient amounts of components or products, or not fully utilize firm purchase commitments.\n",
      "\n",
      "Future operating results depend upon the Company’s ability to obtain components in sufficient quantities on commercially reasonable terms.\n",
      "\n",
      "Because the Company currently obtains certain components from single or limited sources, the Company is subject to significant supply and pricing risks. Many components, including those that are available from multiple sources, are at times subject to industry-wide shortages and significant commodity pricing fluctuations that could materially adversely affect the Company’s financial condition and operating results. While the Company has entered into agreements for the supply of many components, there can be no assurance that the Company will be able to extend or renew these agreements on similar terms, or at all. Component suppliers may suffer from poor financial conditions, which can lead to business failure for the supplier or consolidation within a particular industry, further limiting the Company’s ability to obtain sufficient quantities of components on commercially reasonable terms. The effects of global or regional economic conditions on the Company’s suppliers, described in \n",
      "\n",
      "“\n",
      "\n",
      "Global and regional economic conditions could materially adversely affect the Company’s business, results of operations, financial condition and growth\n",
      "\n",
      "”\n",
      "\n",
      " above, also could affect the Company’s ability to obtain components\n",
      "\n",
      ". \n",
      "\n",
      "Therefore, the Company remains subject to significant risks of supply shortages and price increases that could materially adversely affect its financial condition and operating results.\n",
      "\n",
      "The Company’s new products often utilize custom components available from only one source. When a component or product uses new technologies, initial capacity constraints may exist until the suppliers’ yields have matured or manufacturing capacity has increased. Continued availability of these components at acceptable prices, or at all, may be affected for any number of reasons, including if suppliers decide to concentrate on the production of common components instead of components customized to meet the Company’s requirements. If the Company’s supply of components for a new or existing product were delayed or constrained, or if an outsourcing partner delayed shipments of completed products to the Company, the Company’s financial condition and operating results could be materially adversely affected. The Company’s business and financial performance could also be materially adversely affected depending on the time required to obtain sufficient quantities from the original source, or to identify and obtain sufficient quantities from an alternative source.\n",
      "\n",
      "The Company depends on component and product manufacturing and logistical services provided by outsourcing partners, many of which are located outside of the U.S.\n",
      "\n",
      "Substantially all of the Company’s manufacturing is performed in whole or in part by outsourcing partners located primarily in Asia. A significant concentration of this manufacturing is currently performed by a small number of outsourcing partners, often in single locations. The Company has also outsourced much of its transportation and logistics management. While these arrangements may lower operating costs, they also reduce the Company’s direct control over production and distribution. Such diminished control may have an adverse effect on the quality or quantity of products or services, or the Company’s flexibility to respond to changing conditions. Although arrangements with these partners may contain provisions for warranty expense reimbursement, the Company may remain responsible to the consumer for warranty service in the event of product defects and could experience an unanticipated product defect or warranty liability. While the Company relies on its partners to adhere to its supplier code of conduct, material violations of the supplier code of conduct could occur.\n",
      "\n",
      "The Company relies on single-sourced outsourcing partners in the U.S., Asia and Europe to supply and manufacture many components, and on outsourcing partners primarily located in Asia, for final assembly of substantially all of the Company’s hardware products. Any failure of these partners to perform may have a negative impact on the Company’s cost or supply of components or finished goods. In addition, manufacturing or logistics in these locations or transit to final destinations may be disrupted for a variety of reasons including, but not limited to, natural and man-made disasters, information technology system failures, commercial disputes, military actions, economic, business, labor, environmental, public health or political issues, or international trade disputes.\n",
      "\n",
      "Apple Inc. | 2018 Form 10-K | \n",
      "\n",
      "10\n",
      "\n",
      "The Company has invested in manufacturing process equipment, much of which is held at certain of its outsourcing partners, and has made prepayments to certain of its suppliers associated with long-term supply agreements. While these arrangements help ensure the supply of components and finished goods, if these outsourcing partners or suppliers experience severe financial problems or other disruptions in their business, such continued supply could be reduced or terminated and the recoverability of manufacturing process equipment or prepayments could be negatively impacted.\n",
      "\n",
      "The Company’s products and services may be affected from time to time by design and manufacturing defects that could materially adversely affect the Company’s business and result in harm to the Company’s reputation.\n",
      "\n",
      "The Company offers complex hardware and software products and services that can be affected by design and manufacturing defects. Sophisticated operating system software and applications, such as those offered by the Company, often have issues that can unexpectedly interfere with the intended operation of hardware or software products. Defects may also exist in components and products the Company purchases from third parties. Component defects could make the Company’s products unsafe and create a risk of environmental or property damage and personal injury. These risks may increase as the Company’s products are introduced into specialized applications, including healthcare. In addition, the Company’s service offerings may have quality issues and from time to time experience outages, service slowdowns or errors. As a result, the Company’s services may not perform as anticipated and may not meet customer expectations. There can be no assurance the Company will be able to detect and fix all issues and defects in the hardware, software and services it offers. Failure to do so could result in widespread technical and performance issues affecting the Company’s products and services. In addition, the Company may be exposed to product liability claims, recalls, product replacements or modifications, write-offs of inventory, property, plant and equipment, and/or intangible assets, and significant warranty and other expenses, including litigation costs and regulatory fines. Quality problems could also adversely affect the experience for users of the Company’s products and services, and result in harm to the Company’s reputation, loss of competitive advantage, poor market acceptance, reduced demand for products and services, delay in new product and services introductions and lost revenue.\n",
      "\n",
      "The Company relies on access to third-party digital content, which may not be available to the Company on commercially reasonable terms or at all.\n",
      "\n",
      "The Company contracts with numerous third parties to offer their digital content to customers. This includes the right to sell currently available music, movies, TV shows and books. The licensing or other distribution arrangements with these third parties are for relatively short terms and do not guarantee the continuation or renewal of these arrangements on reasonable terms, if at all. Some third-party content providers and distributors currently or in the future may offer competing products and services, and could take action to make it more difficult or impossible for the Company to license or otherwise distribute their content in the future. Other content owners, providers or distributors may seek to limit the Company’s access to, or increase the cost of, such content. The Company may be unable to continue to offer a wide variety of content at reasonable prices with acceptable usage rules, or continue to expand its geographic reach. Failure to obtain the right to make third-party digital content available, or to make such content available on commercially reasonable terms, could have a material adverse impact on the Company’s financial condition and operating results.\n",
      "\n",
      "Some third-party digital content providers require the Company to provide digital rights management and other security solutions. If requirements change, the Company may have to develop or license new technology to provide these solutions. There is no assurance the Company will be able to develop or license such solutions at a reasonable cost and in a timely manner. In addition, certain countries have passed or may propose and adopt legislation that would force the Company to license its digital rights management, which could lessen the protection of content and subject it to piracy and also could negatively affect arrangements with the Company’s content providers.\n",
      "\n",
      "The Company’s future performance depends in part on support from third-party software developers.\n",
      "\n",
      "The Company believes decisions by customers to purchase its hardware products depend in part on the availability of third-party software applications and services. There is no assurance that third-party developers will continue to develop and maintain software applications and services for the Company’s products. If third-party software applications and services cease to be developed and maintained for the Company’s products, customers may choose not to buy the Company’s products.\n",
      "\n",
      "The Company believes the availability of third-party software applications and services for its products depends in part on the developers’ perception and analysis of the relative benefits of developing, maintaining and upgrading such software and services for the Company’s products compared to competitors’ platforms, such as Android for smartphones and tablets and Windows for personal computers. This analysis may be based on factors such as the market position of the Company and its products, the anticipated revenue that may be generated, expected future growth of product sales and the costs of developing such applications and services.\n",
      "\n",
      "Apple Inc. | 2018 Form 10-K | \n",
      "\n",
      "11\n",
      "\n",
      "The Company’s minority market share in the global smartphone, tablet and personal computer markets could make developers less inclined to develop or upgrade software for the Company’s products and more inclined to devote their resources to developing and upgrading software for competitors’ products with larger market share. If developers focus their efforts on these competing platforms, the availability and quality of applications for the Company’s devices may suffer.\n",
      "\n",
      "The Company relies on the continued availability and development of compelling and innovative software applications for its products. The Company’s products and operating systems are subject to rapid technological change, and if third-party developers are unable to or choose not to keep up with this pace of change, third-party applications might not take advantage of these changes to deliver improved customer experiences or might not operate correctly and may result in dissatisfied customers.\n",
      "\n",
      "The Company sells and delivers third-party applications for its products through the App Store, Mac App Store and TV App Store. The Company retains a commission from sales through these platforms. If developers reduce their use of these platforms to distribute their applications and offer in-app purchases to customers, then the volume of sales, and the commission that the Company earns on those sales, would decrease.\n",
      "\n",
      "The Company relies on access to third-party intellectual property, which may not be available to the Company on commercially reasonable terms or at all.\n",
      "\n",
      "Many of the Company’s products include third-party intellectual property, which requires licenses from those third parties. Based on past experience and industry practice, the Company believes such licenses generally can be obtained on reasonable terms. There is, however, no assurance that the necessary licenses can be obtained on acceptable terms or at all. Failure to obtain the right to use third-party intellectual property, or to use such intellectual property on commercially reasonable terms, could preclude the Company from selling certain products or services, or otherwise have a material adverse impact on the Company’s financial condition and operating results.\n",
      "\n",
      "The Company could be impacted by unfavorable results of legal proceedings, such as being found to have infringed on intellectual property rights\n",
      "\n",
      ".\n",
      "\n",
      "The Company is subject to various legal proceedings and claims that have arisen in the ordinary course of business and have not yet been fully resolved, and new claims may arise in the future. In addition, agreements entered into by the Company sometimes include indemnification provisions which may subject the Company to costs and damages in the event of a claim against an indemnified third party.\n",
      "\n",
      "Claims against the Company based on allegations of patent infringement or other violations of intellectual property rights have generally increased over time and may continue to increase. In particular, the Company has historically faced a significant number of patent claims relating to its cellular-enabled products, and new claims may arise in the future. For example, technology and other patent-holding companies frequently assert their patents and seek royalties and often enter into litigation based on allegations of patent infringement or other violations of intellectual property rights. The Company is vigorously defending infringement actions in courts in a number of U.S. jurisdictions and before the U.S. International Trade Commission, as well as internationally in various countries. The plaintiffs in these actions frequently seek injunctions and substantial damages.\n",
      "\n",
      "Regardless of the merit of particular claims, litigation may be expensive, time consuming, disruptive to the Company’s operations and distracting to management. In recognition of these considerations, the Company may enter into licensing agreements or other arrangements to settle litigation and resolve such disputes. No assurance can be given that such agreements can be obtained on acceptable terms or that litigation will not occur. These agreements may also significantly increase the Company’s operating expenses.\n",
      "\n",
      "Except as described in Part II, Item 8 of this Form 10-K in the Notes to Consolidated Financial Statements in Note 9, “Commitments and Contingencies” under the heading “Contingencies,” in the opinion of management, there was not at least a reasonable possibility the Company may have incurred a material loss, or a material loss in excess of a recorded accrual, with respect to loss contingencies for asserted legal and other claims, including matters related to infringement of intellectual property rights.\n",
      "\n",
      "The outcome of litigation is inherently uncertain. If one or more legal matters were resolved against the Company or an indemnified third party in a reporting period for amounts in excess of management’s expectations, the Company’s financial condition and operating results for that reporting period could be materially adversely affected. Further, such an outcome could result in significant compensatory, punitive or trebled monetary damages, disgorgement of revenue or profits, remedial corporate measures or injunctive relief against the Company that could materially adversely affect its financial condition and operating results.\n",
      "\n",
      "While the Company maintains insurance coverage for certain types of claims, such insurance coverage may be insufficient to cover all losses or all types of claims that may arise.\n",
      "\n",
      "Apple Inc. | 2018 Form 10-K | \n",
      "\n",
      "12\n",
      "\n",
      "The Company is subject to laws and regulations worldwide, changes to which could increase the Company’s costs and individually or in the aggregate adversely affect the Company’s business.\n",
      "\n",
      "The Company is subject to laws and regulations affecting its domestic and international operations in a number of areas. These U.S. and foreign laws and regulations affect the Company’s activities in areas including, but not limited to, labor, advertising, digital content, consumer protection, real estate, billing, e-commerce, promotions, quality of services, telecommunications, mobile communications and media, television, intellectual property ownership and infringement, tax, import and export requirements, anti-corruption, foreign exchange controls and cash repatriation restrictions, data privacy and data localization requirements, anti-competition, environmental, health and safety.\n",
      "\n",
      "By way of example, laws and regulations related to mobile communications and media devices in the many jurisdictions in which the Company operates are extensive and subject to change. Such changes could include, among others, restrictions on the production, manufacture, distribution and use of devices, locking devices to a carrier’s network, or mandating the use of devices on more than one carrier’s network. These devices are also subject to certification and regulation by governmental and standardization bodies, as well as by cellular network carriers for use on their networks. These certification processes are extensive and time consuming, and could result in additional testing requirements, product modifications, or delays in product shipment dates, or could preclude the Company from selling certain products.\n",
      "\n",
      "Compliance with these laws, regulations and similar requirements may be onerous and expensive, and they may be inconsistent from jurisdiction to jurisdiction, further increasing the cost of compliance and doing business. Any such costs, which may rise in the future as a result of changes in these laws and regulations or in their interpretation, could individually or in the aggregate make the Company’s products and services less attractive to the Company’s customers, delay the introduction of new products in one or more regions, or cause the Company to change or limit its business practices. The Company has implemented policies and procedures designed to ensure compliance with applicable laws and regulations, but there can be no assurance that the Company’s employees, contractors, or agents will not violate such laws and regulations or the Company’s policies and procedures.\n",
      "\n",
      "The Company’s business is subject to the risks of international operations.\n",
      "\n",
      "The Company derives a majority of its revenue and earnings from its international operations. Compliance with applicable U.S. and foreign laws and regulations, such as import and export requirements, anti-corruption laws, tax laws, foreign exchange controls and cash repatriation restrictions, data privacy and data localization requirements, environmental laws, labor laws and anti-competition regulations, increases the costs of doing business in foreign jurisdictions. Although the Company has implemented policies and procedures to comply with these laws and regulations, a violation by the Company’s employees, contractors or agents could nevertheless occur. In some cases, compliance with the laws and regulations of one country could violate the laws and regulations of another country. Violations of these laws and regulations could materially adversely affect the Company’s brand, international growth efforts and business.\n",
      "\n",
      "The Company also could be significantly affected by other risks associated with international activities including, but not limited to, economic and labor conditions, increased duties, taxes and other costs, political instability and international trade disputes. Gross margins on the Company’s products in foreign countries, and on products that include components obtained from foreign suppliers, could be materially adversely affected by international trade regulations, including duties, tariffs and antidumping penalties. The Company is also exposed to credit and collectibility risk on its trade receivables with customers in certain international markets. There can be no assurance the Company can effectively limit its credit risk and avoid losses.\n",
      "\n",
      "The Company’s retail stores have required and will continue to require a substantial investment and commitment of resources and are subject to numerous risks and uncertainties.\n",
      "\n",
      "The Company’s retail stores have required substantial investment in equipment and leasehold improvements, information systems, inventory and personnel. The Company also has entered into substantial operating lease commitments for retail space. Certain stores have been designed and built to serve as high-profile venues to promote brand awareness. Because of their unique design elements, locations and size, these stores require substantially more investment than the Company’s more typical retail stores. Due to the high cost structure associated with the Company’s retail stores, a decline in sales or the closure or poor performance of individual or multiple stores could result in significant lease termination costs, write-offs of equipment and leasehold improvements and severance costs.\n",
      "\n",
      "The Company’s retail operations are subject to many factors that pose risks and uncertainties and could adversely impact the Company’s financial condition and operating results, including macro-economic factors that could have an adverse effect on general retail activity. Other factors include, but are not limited to, the Company’s ability to manage costs associated with retail store construction and operation; manage relationships with existing retail partners; manage costs associated with fluctuations in the value of retail inventory; and obtain and renew leases in quality retail locations at a reasonable cost.\n",
      "\n",
      "Apple Inc. | 2018 Form 10-K | \n",
      "\n",
      "13\n",
      "\n",
      "Investment in new business strategies and acquisitions could disrupt the Company’s ongoing business and present risks not originally contemplated.\n",
      "\n",
      "The Company has invested, and in the future may invest, in new business strategies or acquisitions. Such endeavors may involve significant risks and uncertainties, including distraction of management from current operations, greater than expected liabilities and expenses, inadequate return of capital and unidentified issues not discovered in the Company’s due diligence. These new ventures are inherently risky and may not be successful.\n",
      "\n",
      "The Company’s business and reputation may be impacted by information technology system failures or network disruptions.\n",
      "\n",
      "The Company may be subject to information technology system failures or network disruptions caused by natural disasters, accidents, power disruptions, telecommunications failures, acts of terrorism or war, computer viruses, physical or electronic break-ins, or other events or disruptions. System redundancy and other continuity measures may be ineffective or inadequate, and the Company’s business continuity and disaster recovery planning may not be sufficient for all eventualities. Such failures or disruptions could adversely impact the Company’s business by, among other things, preventing access to the Company’s online services, interfering with customer transactions or impeding the manufacturing and shipping of the Company’s products. These events could materially adversely affect the Company’s reputation, financial condition and operating results.\n",
      "\n",
      "There may be losses or unauthorized access to or releases of confidential information, including personally identifiable information, that could subject the Company to significant reputational, financial, legal and operational consequences.\n",
      "\n",
      "The Company’s business requires it to use and store confidential information including, among other things, personally identifiable information (“PII”) with respect to the Company’s customers and employees. The Company devotes significant resources to network and data security, including through the use of encryption and other security measures intended to protect its systems and data. But these measures cannot provide absolute security, and losses or unauthorized access to or releases of confidential information occur and could materially adversely affect the Company’s reputation, financial condition and operating results.\n",
      "\n",
      "The Company’s business also requires it to share confidential information with suppliers and other third parties. Although the Company takes steps to secure confidential information that is provided to third parties, such measures are not always effective and losses or unauthorized access to or releases of confidential information occur and could materially adversely affect the Company’s reputation, financial condition and operating results.\n",
      "\n",
      "For example, the Company may experience a security breach impacting the Company’s information technology systems that compromises the confidentiality, integrity or availability of confidential information. Such an incident could, among other things, impair the Company’s ability to attract and retain customers for its products and services, impact the Company’s stock price, materially damage supplier relationships, and expose the Company to litigation or government investigations, which could result in penalties, fines or judgments against the Company.\n",
      "\n",
      "Although malicious attacks perpetrated to gain access to confidential information, including PII, affect many companies across various industries, the Company is at a relatively greater risk of being targeted because of its high profile and the value of the confidential information it creates, owns, manages, stores and processes.\n",
      "\n",
      "The Company has implemented systems and processes intended to secure its information technology systems and prevent unauthorized access to or loss of sensitive data, including through the use of encryption and authentication technologies. As with all companies, these security measures may not be sufficient for all eventualities and may be vulnerable to hacking, employee error, malfeasance, system error, faulty password management or other irregularities. For example, third parties attempt to fraudulently induce employees or customers into disclosing user names, passwords or other sensitive information, which may in turn be used to access the Company’s information technology systems. To help protect customers and the Company, the Company monitors its services and systems for unusual activity and may freeze accounts under suspicious circumstances, which, among other things, may result in the delay or loss of customer orders or impede customer access to the Company’s products and services.\n",
      "\n",
      "In addition to the risks relating to general confidential information described above, the Company may also be subject to specific obligations relating to health data and payment card data. Health data may be subject to additional privacy, security and breach notification requirements, and the Company may be subject to audit by governmental authorities regarding the Company’s compliance with these obligations. If the Company fails to adequately comply with these rules and requirements, or if health data is handled in a manner not permitted by law or under the Company’s agreements with healthcare institutions, the Company could be subject to litigation or government investigations, may be liable for associated investigatory expenses, and could also incur significant fees or fines.\n",
      "\n",
      "Apple Inc. | 2018 Form 10-K | \n",
      "\n",
      "14\n",
      "\n",
      "Under payment card rules and obligations, if cardholder information is potentially compromised, the Company could be liable for associated investigatory expenses and could also incur significant fees or fines if the Company fails to follow payment card industry data security standards. The Company could also experience a significant increase in payment card transaction costs or lose the ability to process payment cards if it fails to follow payment card industry data security standards, which would materially adversely affect the Company’s reputation, financial condition and operating results.\n",
      "\n",
      "While the Company maintains insurance coverage that is intended to address certain aspects of data security risks, such insurance coverage may be insufficient to cover all losses or all types of claims that may arise.\n",
      "\n",
      "The Company’s business is subject to a variety of U.S. and international laws, rules, policies and other obligations regarding data protection.\n",
      "\n",
      "The Company is subject to federal, state and international laws relating to the collection, use, retention, security and transfer of PII. In many cases, these laws apply not only to third-party transactions, but also may restrict transfers of PII among the Company and its international subsidiaries. Several jurisdictions have passed laws in this area, and other jurisdictions are considering imposing additional restrictions. These laws continue to develop and may be inconsistent from jurisdiction to jurisdiction. Complying with emerging and changing international requirements may cause the Company to incur substantial costs or require the Company to change its business practices. Noncompliance could result in significant penalties or legal liability.\n",
      "\n",
      "The Company makes statements about its use and disclosure of PII through its privacy policy, information provided on its website and press statements. Any failure by the Company to comply with these public statements or with other federal, state or international privacy-related or data protection laws and regulations could result in proceedings against the Company by governmental entities or others. In addition to reputational impacts, penalties could include ongoing audit requirements and significant legal liability.\n",
      "\n",
      "The Company’s success depends largely on the continued service and availability of key personnel.\n",
      "\n",
      "Much of the Company’s future success depends on the continued availability and service of key personnel, including its Chief Executive Officer, executive team and other highly skilled employees. Experienced personnel in the technology industry are in high demand and competition for their talents is intense, especially in Silicon Valley, where most of the Company’s key personnel are located.\n",
      "\n",
      "The Company’s business may be impacted by political events, international trade disputes, war, terrorism, natural disasters, public health issues, industrial accidents and other business interruptions.\n",
      "\n",
      "Political events, international trade disputes, war, terrorism, natural disasters, public health issues, industrial accidents and other business interruptions could harm or disrupt international commerce and the global economy, and could have a material adverse effect on the Company and its customers, suppliers, contract manufacturers, logistics providers, distributors, cellular network carriers and other channel partners.\n",
      "\n",
      "International trade disputes could result in tariffs and other protectionist measures that could adversely affect the Company’s business. Tariffs could increase the cost of the Company’s products and the components and raw materials that go into making them. These increased costs could adversely impact the gross margin that the Company earns on its products. Tariffs could also make the Company’s products more expensive for customers, which could make the Company’s products less competitive and reduce consumer demand. Countries may also adopt other protectionist measures that could limit the Company’s ability to offer its products and services. Political uncertainty surrounding international trade disputes and protectionist measures could also have a negative effect on consumer confidence and spending, which could adversely affect the Company’s business.\n",
      "\n",
      "Many of the Company’s operations and facilities as well as critical business operations of the Company’s suppliers and contract manufacturers are in locations that are prone to earthquakes and other natural disasters. In addition, such operations and facilities are subject to the risk of interruption by fire, power shortages, nuclear power plant accidents and other industrial accidents, terrorist attacks and other hostile acts, labor disputes, public health issues and other events beyond the Company’s control. Global climate change could result in certain types of natural disasters occurring more frequently or with more intense effects. Such events could make it difficult or impossible for the Company to manufacture and deliver products to its customers, create delays and inefficiencies in the Company’s supply and manufacturing chain, and result in slowdowns and outages to the Company’s service offerings. Following an interruption to its business, the Company could require substantial recovery time, experience significant expenditures in order to resume operations, and lose significant revenue. Because the Company relies on single or limited sources for the supply and manufacture of many critical components, a business interruption affecting such sources would exacerbate any negative consequences to the Company.\n",
      "\n",
      "Apple Inc. | 2018 Form 10-K | \n",
      "\n",
      "15\n",
      "\n",
      "The Company’s operations are also subject to the risks of industrial accidents at its suppliers and contract manufacturers. While the Company’s suppliers are required to maintain safe working environments and operations, an industrial accident could occur and could result in disruption to the Company’s business and harm to the Company’s reputation. Should major public health issues, including pandemics, arise, the Company could be adversely affected by more stringent employee travel restrictions, additional limitations in freight services, governmental actions limiting the movement of products between regions, delays in production ramps of new products and disruptions in the operations of the Company’s suppliers and contract manufacturers.\n",
      "\n",
      "The Company expects its quarterly revenue and operating results to fluctuate.\n",
      "\n",
      "The Company’s profit margins vary across its products, services, geographic segments and distribution channels. For example, gross margins on the Company’s hardware products vary across product lines and can change over time as a result of product transitions, pricing and configuration changes, and component, warranty and other cost fluctuations. The Company’s financial results may be materially adversely impacted as a result of shifts in the mix of products and services that the Company sells; shifts in the geographic, currency or channel mix of the Company’s sales; component cost increases; price competition; or the introduction of new products, including new products with higher cost structures.\n",
      "\n",
      "The Company has typically experienced higher net sales in its first quarter compared to other quarters due in part to seasonal holiday demand. Additionally, new product introductions can significantly impact net sales, product costs and operating expenses. Further, the Company generates a majority of its net sales from a single product and a decline in demand for that product could significantly impact quarterly net sales. The Company could also be subject to unexpected developments, such as lower-than-anticipated demand for the Company’s products, issues with new product introductions, information technology system failures or network disruptions, or failure of one of the Company’s logistics, components supply, or manufacturing partners.\n",
      "\n",
      "The Company’s stock price is subject to volatility.\n",
      "\n",
      "The Company’s stock price has experienced substantial price volatility in the past and may continue to do so in the future. Additionally, the Company, the technology industry and the stock market as a whole have experienced extreme stock price and volume fluctuations that have affected stock prices in ways that may have been unrelated to these companies’ operating performance. Price volatility over a given period may cause the average price at which the Company repurchases its own stock to exceed the stock’s price at a given point in time. The Company believes its stock price should reflect expectations of future growth and profitability. The Company also believes its stock price should reflect expectations that its cash dividend will continue at current levels or grow and that its current share repurchase program will be fully consummated. Future dividends are subject to declaration by the Company’s Board of Directors, and the Company’s share repurchase program does not obligate it to acquire any specific number of shares. If the Company fails to meet expectations related to future growth, profitability, dividends, share repurchases or other market expectations, its stock price may decline significantly, which could have a material adverse impact on investor confidence and employee retention.\n",
      "\n",
      "The Company’s financial performance is subject to risks associated with changes in the value of the U.S. dollar relative to local currencies.\n",
      "\n",
      "The Company’s primary exposure to movements in foreign currency exchange rates relates to non–U.S. dollar–denominated sales and operating expenses worldwide. Gross margins on the Company’s products in foreign countries and on products that include components obtained from foreign suppliers could be materially adversely affected by foreign currency exchange rate fluctuations.\n",
      "\n",
      "Weakening of foreign currencies relative to the U.S. dollar adversely affects the U.S. dollar value of the Company’s foreign currency–denominated sales and earnings, and generally leads the Company to raise international pricing, potentially reducing demand for the Company’s products. In some circumstances, for competitive or other reasons, the Company may decide not to raise international pricing to offset the U.S. dollar’s strengthening, which would adversely affect the U.S. dollar value of the gross margins the Company earns on foreign currency–denominated sales.\n",
      "\n",
      "Conversely, a strengthening of foreign currencies relative to the U.S. dollar, while generally beneficial to the Company’s foreign currency–denominated sales and earnings, could cause the Company to reduce international pricing and incur losses on its foreign currency derivative instruments, thereby limiting the benefit. Additionally, strengthening of foreign currencies may increase the Company’s cost of product components denominated in those currencies, thus adversely affecting gross margins.\n",
      "\n",
      "The Company uses derivative instruments, such as foreign currency forward and option contracts, to hedge certain exposures to fluctuations in foreign currency exchange rates. The use of such hedging activities may not be effective to offset any, or more than a portion, of the adverse financial effects of unfavorable movements in foreign exchange rates over the limited time the hedges are in place.\n",
      "\n",
      "Apple Inc. | 2018 Form 10-K | \n",
      "\n",
      "16\n",
      "\n",
      "The Company is exposed to credit risk and fluctuations in the market values of its investment portfolio.\n",
      "\n",
      "The Company’s investments can be negatively affected by liquidity, credit deterioration, financial results, market and economic conditions, political risk, sovereign risk, interest rate fluctuations or other factors. As a result, the value and liquidity of the Company’s cash, cash equivalents and marketable securities may fluctuate substantially. Therefore, although the Company has not realized any significant losses on its cash, cash equivalents and marketable securities, future fluctuations in their value could result in significant realized losses and could have a material adverse impact on the Company’s financial condition and operating results.\n",
      "\n",
      "The Company is exposed to credit risk on its trade accounts receivable, vendor non-trade receivables and prepayments related to long-term supply agreements, and this risk is heightened during periods when economic conditions worsen.\n",
      "\n",
      "The Company distributes its products through third-party cellular network carriers, wholesalers, retailers and resellers. The Company also sells its products directly to small and mid-sized businesses and education, enterprise and government customers. A substantial majority of the Company’s outstanding trade receivables are not covered by collateral, third-party bank support or financing arrangements, or credit insurance. The Company’s exposure to credit and collectibility risk on its trade receivables is higher in certain international markets and its ability to mitigate such risks may be limited. The Company also has unsecured vendor non-trade receivables resulting from purchases of components by outsourcing partners and other vendors that manufacture sub-assemblies or assemble final products for the Company. In addition, the Company has made prepayments associated with long-term supply agreements to secure supply of inventory components. As of \n",
      "\n",
      "September 29, 2018\n",
      "\n",
      ", a significant portion of the Company’s trade receivables was concentrated within cellular network carriers, and its vendor non-trade receivables and prepayments related to long-term supply agreements were concentrated among a few individual vendors located primarily in Asia. While the Company has procedures to monitor and limit exposure to credit risk on its trade and vendor non-trade receivables, as well as long-term prepayments, there can be no assurance such procedures will effectively limit its credit risk and avoid losses.\n",
      "\n",
      "The Company could be subject to changes in its tax rates, the adoption of new U.S. or international tax legislation or exposure to additional tax liabilities.\n",
      "\n",
      "The Company is subject to taxes in the U.S. and numerous foreign jurisdictions, including Ireland, where a number of the Company’s subsidiaries are organized. Due to economic and political conditions, tax rates in various jurisdictions may be subject to significant change. The Company’s effective tax rates could be affected by changes in the mix of earnings in countries with differing statutory tax rates, changes in the valuation of deferred tax assets and liabilities, or changes in tax laws or their interpretation, including in the U.S. and Ireland.\n",
      "\n",
      "The Company is also subject to the examination of its tax returns and other tax matters by the U.S. Internal Revenue Service (the “IRS”) and other tax authorities and governmental bodies. The Company regularly assesses the likelihood of an adverse outcome resulting from these examinations to determine the adequacy of its provision for taxes. There can be no assurance as to the outcome of these examinations. If the Company’s effective tax rates were to increase, particularly in the U.S. or Ireland, or if the ultimate determination of the Company’s taxes owed is for an amount in excess of amounts previously accrued, the Company’s financial condition, operating results and cash flows could be materially adversely affected.\n"
     ]
    }
   ],
   "source": [
    "### Our goal is though to remove html tags and see the content\n",
    "### Method get_text() is what we need, \\n\\n is optional, I just added this to read text \n",
    "### more cleanly, it's basically new line character between sections. \n",
    "print(item_1a_content.get_text(\"\\n\\n\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:navy\"> Summary..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have seen that simply applying REGEX & BeautifulSoup combination we can form a very powerful combination extracting/scarpping content from any web-content very easily. Having said this, not all 10-Ks are well crafted HTML, TEXT formats, example older 10-Ks, hence there may be adjustments needed to adopt to the circumstances."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}