Project_Report_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project Report I: Data Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Report Brief**<br>\n",
    "In this progress report, I'm hoping to do as much of the data processing as I can, although I don't expect to finish. The breakdown of my project likely means that the heaviest proportion of work will fall on data cleaning/organization, and then EDA, with visualization and analysis being the least intensive section. This is because my project is product-centric--first and foremost, I'm trying to make a *thing* that can classify data.\n",
    "___\n",
    "**Goals**\n",
    "\n",
    "Here's a list of things I'm hoping to accomplish in the first portion of my project:\n",
    "1. ~Fix the Markdown formatting for my README and other files~\n",
    "2. ~Add data sample to GitHub~\n",
    "3. read in data from different folders/sources **unsuccessful**\n",
    "4. create a sample DF with a few columns of useful data \n",
    "    1. text\n",
    "    2. folder it is in\n",
    "    3. length of email\n",
    "    4. to/from\n",
    "5. visualize some of the data with a few statistical tests or plots\n",
    "\n",
    "B and D will likely be fairly difficult, and just reading in the data isn't easy, so I'm really aiming to accomplish 1, 2, 3, and 4. Lets see how it goes!\n",
    "___\n",
    "**Sharing Data**<br>\n",
    "In regards to sharing data, there is a bit of a problem in that the source from which I took my data (http://www.cs.cmu.edu/~enron/) doesn't make mention of any type of licensing. I'm going to need to look into this further, but I'm tentatively planning on my data sample being just one (anonymized) person's emails, since this is the most natural data member of the corpus. This will clearly demonstrate the shape and scope of the corpus without taking up too much space or compromising privacy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Reading In Data**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "imports, setting up file path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.corpus import PlaintextCorpusReader as cr\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "628\n"
     ]
    }
   ],
   "source": [
    "corpus_root = '../../../../../Enron_Emails/maildir/'\n",
    "all_allen_p = 'allen-p/all_documents'\n",
    "all_files = cr(corpus_root + all_allen_p, \".*\")\n",
    "print(len(all_files.fileids()))\n",
    "#needs a way to go one in, regardless of name\n",
    "#plaintext reader?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "prepping email for NotePad-level readability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"\"\n",
    "file = open(corpus_root + all_allen_p + \"/1\")\n",
    "for line in file:\n",
    "    text += line\n",
    "file.close()\n",
    "email = text.split('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Message-ID: <29790972.1075855665306.JavaMail.evans@thyme>',\n",
       " 'Date: Wed, 13 Dec 2000 18:41:00 -0800 (PST)',\n",
       " 'From: 1.11913372.-2@multexinvestornetwork.com',\n",
       " 'To: pallen@enron.com',\n",
       " \"Subject: December 14, 2000 - Bear Stearns' predictions for telecom in Latin\",\n",
       " ' America',\n",
       " 'Mime-Version: 1.0',\n",
       " 'Content-Type: text/plain; charset=us-ascii',\n",
       " 'Content-Transfer-Encoding: 7bit',\n",
       " 'X-From: Multex Investor <1.11913372.-2@multexinvestornetwork.com>',\n",
       " 'X-To: <pallen@enron.com>',\n",
       " 'X-cc: ',\n",
       " 'X-bcc: ',\n",
       " 'X-Folder: \\\\Phillip_Allen_Dec2000\\\\Notes Folders\\\\All documents',\n",
       " 'X-Origin: Allen-P',\n",
       " 'X-FileName: pallen.nsf',\n",
       " '',\n",
       " \"In today's Daily Update you'll find free reports on\",\n",
       " 'America Online (AOL), Divine Interventures (DVIN),',\n",
       " 'and 3M (MMM); reports on the broadband space, Latin',\n",
       " 'American telecom, and more.',\n",
       " '',\n",
       " \"For free research, editor's picks, and more come to the Daily Investor:\",\n",
       " 'http://www.multexinvestor.com/AF004627/magazinecover.asp?promo=unl&d=20001214#',\n",
       " 'investor',\n",
       " '',\n",
       " '***************************************************************',\n",
       " 'You are receiving this mail because you have registered for',\n",
       " 'Multex Investor. To unsubscribe, see bottom of this message.',\n",
       " '***************************************************************',\n",
       " '',\n",
       " '======================== Sponsored by =========================',\n",
       " 'Would you own just the energy stocks in the S&P 500?',\n",
       " 'Select Sector SPDRs divides the S&P 500 into nine sector index funds.',\n",
       " 'Pick and choose just the pieces of the S&P 500 you like best.',\n",
       " 'http://www.spdrindex.com',\n",
       " '===============================================================',\n",
       " '',\n",
       " \"Featured in today's edition of the Daily Update:\",\n",
       " '',\n",
       " \"1. SPECIAL ANNOUNCEMENT: Treat yourself to Multex Investor's NEW Personal\",\n",
       " 'Finance Channel to take advantage of top-notch content and tools  FREE.',\n",
       " '',\n",
       " '2. DAILY FREE SPONSOR REPORT: Robertson Stephens maintains a \"buy\" rating',\n",
       " 'on Divine Interventures (DVIN).',\n",
       " '',\n",
       " '3. FREE RESEARCH REPORT: Jefferies & Co. rates America Online (AOL) a',\n",
       " '\"buy,\" saying projected growth remains in place.',\n",
       " '',\n",
       " \"4. ASK THE ANALYST: Morgan Stanley Dean Witter's Lew Smith in the Analyst\",\n",
       " 'Corner',\n",
       " '',\n",
       " \"5. HOT REPORT: Oscar Gruss & Son's most recent issue of its Broadband\",\n",
       " 'Brief reports the latest developments in the broadband space.',\n",
       " '',\n",
       " \"6. EDITOR'S PICK: Bear Stearns measures the impact of broadband and the\",\n",
       " 'Internet on telecom in Latin America.',\n",
       " '',\n",
       " \"7. FREE STOCK SNAPSHOT: The current analysts' consensus rates 3M (MMM), a\",\n",
       " '\"buy/hold.\"',\n",
       " '',\n",
       " '8. JOIN THE MARKETBUZZ: where top financial industry professionals answer',\n",
       " \"your questions and offer insights every market day from noon 'til 2:00\",\n",
       " 'p.m. ET.',\n",
       " '',\n",
       " '9. TRANSCRIPTS FROM WALL STREET: Ash Rajan, senior vice president and',\n",
       " 'market analyst with Prudential Securities, answers questions about the',\n",
       " 'market.',\n",
       " '',\n",
       " '======================== Sponsored by =========================',\n",
       " 'Profit From AAII\\'s \"Cash Rich\" Stock Screen - 46% YTD Return',\n",
       " '',\n",
       " 'With so much market volatility, how did AAII\\'s \"Cash Rich\"',\n",
       " 'Stock Screen achieve such stellar returns?  Find the answer by',\n",
       " 'taking a free trial membership from the American Association',\n",
       " 'of Individual Investors and using our FREE Stock Screen service at:',\n",
       " 'http://subs.aaii.com/c/go/XAAI/MTEX1B-aaiitU1?s=S900',\n",
       " '===============================================================',\n",
       " '',\n",
       " '1. NEW ON MULTEX INVESTOR',\n",
       " 'Take charge of your personal finances',\n",
       " '',\n",
       " 'Do you have endless hours of free time to keep your financial house in',\n",
       " \"order? We didn't think so. That's why you need to treat yourself to Multex\",\n",
       " \"Investor's NEW Personal Finance Channel to take advantage of top-notch\",\n",
       " 'content and tools  FREE.',\n",
       " 'Click here for more information.',\n",
       " 'http://www.multexpf.com?mktg=sgpftx4&promo=unl&t=10&d=20001214',\n",
       " '',\n",
       " '',\n",
       " '2. DAILY FREE SPONSOR REPORT',\n",
       " 'Divine Interventures (DVIN)',\n",
       " '',\n",
       " 'Robertson Stephens maintains a \"buy\" rating on Divine Interventures, an',\n",
       " 'incubator focused on infrastructure services and business-to-business',\n",
       " \"(B2B) exchanges. Register for Robertson Stephens' free-research trial to\",\n",
       " 'access this report.',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/Download.asp?docid=5018549&sid=9&promo=unl&t=12&',\n",
       " 'd=20001214',\n",
       " '',\n",
       " '',\n",
       " '3. FREE RESEARCH REPORT',\n",
       " \"Hold 'er steady -- America Online (AOL)\",\n",
       " '',\n",
       " \"AOL's projected growth and proposed merger with Time Warner (TWX) both\",\n",
       " 'remain in place, says Jefferies & Co., which maintains a \"buy\" rating on',\n",
       " 'AOL. In the report, which is free for a limited time, analysts are',\n",
       " 'confident the deal will close soon.',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/AF004627/magazinecover.asp?promo=unl&t=11&d=2000',\n",
       " '1214',\n",
       " '',\n",
       " '',\n",
       " '4. TODAY IN THE ANALYST CORNER',\n",
       " 'Following market trends',\n",
       " '',\n",
       " \"Morgan Stanley Dean Witter's Lew Smith sees strong underlying trends\",\n",
       " 'guiding future market performance. What trends does he point to, and what',\n",
       " 'stocks and sectors does he see benefiting from his premise?',\n",
       " '',\n",
       " \"Here is your opportunity to gain free access to Morgan Stanley's research.\",\n",
       " 'Simply register and submit a question below. You will then have a free',\n",
       " \"trial membership to this top Wall Street firms' research!  Lew Smith will\",\n",
       " 'be in the Analyst Corner only until 5 p.m. ET Thurs., Dec. 14, so be sure',\n",
       " 'to ask your question now.',\n",
       " 'Ask the analyst.',\n",
       " 'http://www.multexinvestor.com/ACHome.asp?promo=unl&t=1&d=20001214',\n",
       " '',\n",
       " '',\n",
       " \"5. WHAT'S HOT? RESEARCH REPORTS FROM MULTEX INVESTOR'S HOT LIST\",\n",
       " 'Breaking the bottleneck -- An update on the broadband space',\n",
       " '',\n",
       " \"Oscar Gruss & Son's most recent issue of its Broadband Brief reports the\",\n",
       " 'latest developments in the broadband space, with coverage of Adaptive',\n",
       " 'Broadband (ADAP), Broadcom (BRCM), Efficient Networks (EFNT), and others',\n",
       " '(report for purchase - $25).',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/Download.asp?docid=5149041&promo=unl&t=4&d=20001',\n",
       " '214',\n",
       " '',\n",
       " '======================== Sponsored by =========================',\n",
       " 'Get Red Herring insight into hot IPOs, investing strategies,',\n",
       " 'stocks to watch, future technologies, and more. FREE',\n",
       " 'E-newsletters from Redherring.com provide more answers,',\n",
       " 'analysis and opinion to help you make more strategic',\n",
       " 'investing decisions. Subscribe today',\n",
       " 'http://www.redherring.com/jump/om/i/multex/email2/subscribe/47.html',\n",
       " '===============================================================',\n",
       " '',\n",
       " \"6. EDITOR'S PICK: CURRENT RESEARCH FROM THE CUTTING EDGE\",\n",
       " \"Que pasa? -- Predicting telecom's future in Latin America\",\n",
       " '',\n",
       " 'Bear Stearns measures the impact of broadband and the Internet on telecom',\n",
       " 'in Latin America, saying incumbent local-exchange carriers (ILECs) are',\n",
       " 'ideally positioned to benefit from the growth of Internet and data',\n",
       " 'services (report for purchase - $150).',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/Download.asp?docid=5140995&promo=unl&t=8&d=20001',\n",
       " '214',\n",
       " '',\n",
       " '',\n",
       " '7. FREE STOCK SNAPSHOT',\n",
       " '3M (MMM)',\n",
       " '',\n",
       " 'The current analysts\\' consensus rates 3M, a \"buy/hold.\" Analysts expect',\n",
       " 'the industrial product manufacturer to earn $4.76 per share in 2000 and',\n",
       " '$5.26 per share in 2001.',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/Download.asp?docid=1346414&promo=unl&t=3&d=20001',\n",
       " '214',\n",
       " '',\n",
       " '',\n",
       " '8. JOIN THE MARKETBUZZ!',\n",
       " 'Check out SageOnline',\n",
       " '',\n",
       " 'where top financial industry professionals answer your questions and offer',\n",
       " \"insights every market day from noon 'til 2:00 p.m. ET.\",\n",
       " 'Click here.',\n",
       " 'http://multexinvestor.sageonline.com/page2.asp?id=9512&ps=1&s=2&mktg=evn&promo',\n",
       " '=unl&t=24&d=20001214',\n",
       " '',\n",
       " '',\n",
       " \"9. TRANSCRIPTS FROM WALL STREET'S GURUS\",\n",
       " \"Prudential Securities' Ash Rajan\",\n",
       " '',\n",
       " 'In this SageOnline transcript from a chat that took place earlier this',\n",
       " 'week, Ash Rajan, senior vice president and market analyst with Prudential',\n",
       " 'Securities, answers questions about tech, retail, finance, and the outlook',\n",
       " 'for the general market.',\n",
       " 'Click here.',\n",
       " 'http://multexinvestor.sageonline.com/transcript.asp?id=10403&ps=1&s=8&mktg=trn',\n",
       " '&promo=unl&t=13&d=20001214',\n",
       " '',\n",
       " '===================================================================',\n",
       " 'Please send your questions and comments to mailto:investor.help@multex.com',\n",
       " '',\n",
       " \"If you'd like to learn more about Multex Investor, please visit:\",\n",
       " 'http://www.multexinvestor.com/welcome.asp',\n",
       " '',\n",
       " \"If you can't remember your password and/or your user name, click here:\",\n",
       " 'http://www.multexinvestor.com/lostinfo.asp',\n",
       " '',\n",
       " 'If you want to update your email address, please click on the url below:',\n",
       " 'http://www.multexinvestor.com/edituinfo.asp',\n",
       " '===================================================================',\n",
       " 'To remove yourself from the mailing list for the Daily Update, please',\n",
       " 'REPLY to THIS email message with the word UNSUBSCRIBE in the subject',\n",
       " 'line. To remove yourself from all Multex Investor mailings, including',\n",
       " 'the Daily Update and The Internet Analyst, please respond with the',\n",
       " 'words NO EMAIL in the subject line.',\n",
       " '',\n",
       " 'You may also unsubscribe on the account update page at:',\n",
       " 'http://www.multexinvestor.com/edituinfo.asp',\n",
       " '===================================================================',\n",
       " 'Please email advertising inquiries to us at mailto:advertise@multex.com.',\n",
       " '',\n",
       " 'For information on becoming an affiliate click here: ',\n",
       " 'http://www.multexinvestor.com/Affiliates/home.asp?promo=unl',\n",
       " '',\n",
       " 'Be sure to check out one of our other newsletters, The Internet Analyst by',\n",
       " 'Multex.com. The Internet Analyst informs, educates, and entertains you with',\n",
       " 'usable investment data, ideas, experts, and info about the Internet industry.',\n",
       " \"To see this week's issue, click here: http://www.theinternetanalyst.com\",\n",
       " '',\n",
       " 'If you are not 100% satisfied with a purchase you make on Multex',\n",
       " 'Investor, we will refund your money.']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "email"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HUGE problem. any categories on multiple lines are split too. no good!\n",
    "\n",
    "**my fix**: lets loop through the list, and if we find an element in the list that starts with whitespace, we append it to the previous element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Message-ID: <29790972.1075855665306.JavaMail.evans@thyme>',\n",
       " 'Date: Wed, 13 Dec 2000 18:41:00 -0800 (PST)',\n",
       " 'From: 1.11913372.-2@multexinvestornetwork.com',\n",
       " 'To: pallen@enron.com',\n",
       " \"Subject: December 14, 2000 - Bear Stearns' predictions for telecom in Latin America\",\n",
       " 'Mime-Version: 1.0',\n",
       " 'Content-Type: text/plain; charset=us-ascii',\n",
       " 'Content-Transfer-Encoding: 7bit',\n",
       " 'X-From: Multex Investor <1.11913372.-2@multexinvestornetwork.com>',\n",
       " 'X-To: <pallen@enron.com>',\n",
       " 'X-cc: ',\n",
       " 'X-bcc: ',\n",
       " 'X-Folder: \\\\Phillip_Allen_Dec2000\\\\Notes Folders\\\\All documents',\n",
       " 'X-Origin: Allen-P',\n",
       " 'X-FileName: pallen.nsf',\n",
       " '',\n",
       " \"In today's Daily Update you'll find free reports on\",\n",
       " 'America Online (AOL), Divine Interventures (DVIN),',\n",
       " 'and 3M (MMM); reports on the broadband space, Latin',\n",
       " 'American telecom, and more.',\n",
       " '',\n",
       " \"For free research, editor's picks, and more come to the Daily Investor:\",\n",
       " 'http://www.multexinvestor.com/AF004627/magazinecover.asp?promo=unl&d=20001214#',\n",
       " 'investor',\n",
       " '',\n",
       " '***************************************************************',\n",
       " 'You are receiving this mail because you have registered for',\n",
       " 'Multex Investor. To unsubscribe, see bottom of this message.',\n",
       " '***************************************************************',\n",
       " '',\n",
       " '======================== Sponsored by =========================',\n",
       " 'Would you own just the energy stocks in the S&P 500?',\n",
       " 'Select Sector SPDRs divides the S&P 500 into nine sector index funds.',\n",
       " 'Pick and choose just the pieces of the S&P 500 you like best.',\n",
       " 'http://www.spdrindex.com',\n",
       " '===============================================================',\n",
       " '',\n",
       " \"Featured in today's edition of the Daily Update:\",\n",
       " '',\n",
       " \"1. SPECIAL ANNOUNCEMENT: Treat yourself to Multex Investor's NEW Personal\",\n",
       " 'Finance Channel to take advantage of top-notch content and tools  FREE.',\n",
       " '',\n",
       " '2. DAILY FREE SPONSOR REPORT: Robertson Stephens maintains a \"buy\" rating',\n",
       " 'on Divine Interventures (DVIN).',\n",
       " '',\n",
       " '3. FREE RESEARCH REPORT: Jefferies & Co. rates America Online (AOL) a',\n",
       " '\"buy,\" saying projected growth remains in place.',\n",
       " '',\n",
       " \"4. ASK THE ANALYST: Morgan Stanley Dean Witter's Lew Smith in the Analyst\",\n",
       " 'Corner',\n",
       " '',\n",
       " \"5. HOT REPORT: Oscar Gruss & Son's most recent issue of its Broadband\",\n",
       " 'Brief reports the latest developments in the broadband space.',\n",
       " '',\n",
       " \"6. EDITOR'S PICK: Bear Stearns measures the impact of broadband and the\",\n",
       " 'Internet on telecom in Latin America.',\n",
       " '',\n",
       " \"7. FREE STOCK SNAPSHOT: The current analysts' consensus rates 3M (MMM), a\",\n",
       " '\"buy/hold.\"',\n",
       " '',\n",
       " '8. JOIN THE MARKETBUZZ: where top financial industry professionals answer',\n",
       " \"your questions and offer insights every market day from noon 'til 2:00\",\n",
       " 'p.m. ET.',\n",
       " '',\n",
       " '9. TRANSCRIPTS FROM WALL STREET: Ash Rajan, senior vice president and',\n",
       " 'market analyst with Prudential Securities, answers questions about the',\n",
       " 'market.',\n",
       " '',\n",
       " '======================== Sponsored by =========================',\n",
       " 'Profit From AAII\\'s \"Cash Rich\" Stock Screen - 46% YTD Return',\n",
       " '',\n",
       " 'With so much market volatility, how did AAII\\'s \"Cash Rich\"',\n",
       " 'Stock Screen achieve such stellar returns?  Find the answer by',\n",
       " 'taking a free trial membership from the American Association',\n",
       " 'of Individual Investors and using our FREE Stock Screen service at:',\n",
       " 'http://subs.aaii.com/c/go/XAAI/MTEX1B-aaiitU1?s=S900',\n",
       " '===============================================================',\n",
       " '',\n",
       " '1. NEW ON MULTEX INVESTOR',\n",
       " 'Take charge of your personal finances',\n",
       " '',\n",
       " 'Do you have endless hours of free time to keep your financial house in',\n",
       " \"order? We didn't think so. That's why you need to treat yourself to Multex\",\n",
       " \"Investor's NEW Personal Finance Channel to take advantage of top-notch\",\n",
       " 'content and tools  FREE.',\n",
       " 'Click here for more information.',\n",
       " 'http://www.multexpf.com?mktg=sgpftx4&promo=unl&t=10&d=20001214',\n",
       " '',\n",
       " '',\n",
       " '2. DAILY FREE SPONSOR REPORT',\n",
       " 'Divine Interventures (DVIN)',\n",
       " '',\n",
       " 'Robertson Stephens maintains a \"buy\" rating on Divine Interventures, an',\n",
       " 'incubator focused on infrastructure services and business-to-business',\n",
       " \"(B2B) exchanges. Register for Robertson Stephens' free-research trial to\",\n",
       " 'access this report.',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/Download.asp?docid=5018549&sid=9&promo=unl&t=12&',\n",
       " 'd=20001214',\n",
       " '',\n",
       " '',\n",
       " '3. FREE RESEARCH REPORT',\n",
       " \"Hold 'er steady -- America Online (AOL)\",\n",
       " '',\n",
       " \"AOL's projected growth and proposed merger with Time Warner (TWX) both\",\n",
       " 'remain in place, says Jefferies & Co., which maintains a \"buy\" rating on',\n",
       " 'AOL. In the report, which is free for a limited time, analysts are',\n",
       " 'confident the deal will close soon.',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/AF004627/magazinecover.asp?promo=unl&t=11&d=2000',\n",
       " '1214',\n",
       " '',\n",
       " '',\n",
       " '4. TODAY IN THE ANALYST CORNER',\n",
       " 'Following market trends',\n",
       " '',\n",
       " \"Morgan Stanley Dean Witter's Lew Smith sees strong underlying trends\",\n",
       " 'guiding future market performance. What trends does he point to, and what',\n",
       " 'stocks and sectors does he see benefiting from his premise?',\n",
       " '',\n",
       " \"Here is your opportunity to gain free access to Morgan Stanley's research.\",\n",
       " 'Simply register and submit a question below. You will then have a free',\n",
       " \"trial membership to this top Wall Street firms' research!  Lew Smith will\",\n",
       " 'be in the Analyst Corner only until 5 p.m. ET Thurs., Dec. 14, so be sure',\n",
       " 'to ask your question now.',\n",
       " 'Ask the analyst.',\n",
       " 'http://www.multexinvestor.com/ACHome.asp?promo=unl&t=1&d=20001214',\n",
       " '',\n",
       " '',\n",
       " \"5. WHAT'S HOT? RESEARCH REPORTS FROM MULTEX INVESTOR'S HOT LIST\",\n",
       " 'Breaking the bottleneck -- An update on the broadband space',\n",
       " '',\n",
       " \"Oscar Gruss & Son's most recent issue of its Broadband Brief reports the\",\n",
       " 'latest developments in the broadband space, with coverage of Adaptive',\n",
       " 'Broadband (ADAP), Broadcom (BRCM), Efficient Networks (EFNT), and others',\n",
       " '(report for purchase - $25).',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/Download.asp?docid=5149041&promo=unl&t=4&d=20001',\n",
       " '214',\n",
       " '',\n",
       " '======================== Sponsored by =========================',\n",
       " 'Get Red Herring insight into hot IPOs, investing strategies,',\n",
       " 'stocks to watch, future technologies, and more. FREE',\n",
       " 'E-newsletters from Redherring.com provide more answers,',\n",
       " 'analysis and opinion to help you make more strategic',\n",
       " 'investing decisions. Subscribe today',\n",
       " 'http://www.redherring.com/jump/om/i/multex/email2/subscribe/47.html',\n",
       " '===============================================================',\n",
       " '',\n",
       " \"6. EDITOR'S PICK: CURRENT RESEARCH FROM THE CUTTING EDGE\",\n",
       " \"Que pasa? -- Predicting telecom's future in Latin America\",\n",
       " '',\n",
       " 'Bear Stearns measures the impact of broadband and the Internet on telecom',\n",
       " 'in Latin America, saying incumbent local-exchange carriers (ILECs) are',\n",
       " 'ideally positioned to benefit from the growth of Internet and data',\n",
       " 'services (report for purchase - $150).',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/Download.asp?docid=5140995&promo=unl&t=8&d=20001',\n",
       " '214',\n",
       " '',\n",
       " '',\n",
       " '7. FREE STOCK SNAPSHOT',\n",
       " '3M (MMM)',\n",
       " '',\n",
       " 'The current analysts\\' consensus rates 3M, a \"buy/hold.\" Analysts expect',\n",
       " 'the industrial product manufacturer to earn $4.76 per share in 2000 and',\n",
       " '$5.26 per share in 2001.',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/Download.asp?docid=1346414&promo=unl&t=3&d=20001',\n",
       " '214',\n",
       " '',\n",
       " '',\n",
       " '8. JOIN THE MARKETBUZZ!',\n",
       " 'Check out SageOnline',\n",
       " '',\n",
       " 'where top financial industry professionals answer your questions and offer',\n",
       " \"insights every market day from noon 'til 2:00 p.m. ET.\",\n",
       " 'Click here.',\n",
       " 'http://multexinvestor.sageonline.com/page2.asp?id=9512&ps=1&s=2&mktg=evn&promo',\n",
       " '=unl&t=24&d=20001214',\n",
       " '',\n",
       " '',\n",
       " \"9. TRANSCRIPTS FROM WALL STREET'S GURUS\",\n",
       " \"Prudential Securities' Ash Rajan\",\n",
       " '',\n",
       " 'In this SageOnline transcript from a chat that took place earlier this',\n",
       " 'week, Ash Rajan, senior vice president and market analyst with Prudential',\n",
       " 'Securities, answers questions about tech, retail, finance, and the outlook',\n",
       " 'for the general market.',\n",
       " 'Click here.',\n",
       " 'http://multexinvestor.sageonline.com/transcript.asp?id=10403&ps=1&s=8&mktg=trn',\n",
       " '&promo=unl&t=13&d=20001214',\n",
       " '',\n",
       " '===================================================================',\n",
       " 'Please send your questions and comments to mailto:investor.help@multex.com',\n",
       " '',\n",
       " \"If you'd like to learn more about Multex Investor, please visit:\",\n",
       " 'http://www.multexinvestor.com/welcome.asp',\n",
       " '',\n",
       " \"If you can't remember your password and/or your user name, click here:\",\n",
       " 'http://www.multexinvestor.com/lostinfo.asp',\n",
       " '',\n",
       " 'If you want to update your email address, please click on the url below:',\n",
       " 'http://www.multexinvestor.com/edituinfo.asp',\n",
       " '===================================================================',\n",
       " 'To remove yourself from the mailing list for the Daily Update, please',\n",
       " 'REPLY to THIS email message with the word UNSUBSCRIBE in the subject',\n",
       " 'line. To remove yourself from all Multex Investor mailings, including',\n",
       " 'the Daily Update and The Internet Analyst, please respond with the',\n",
       " 'words NO EMAIL in the subject line.',\n",
       " '',\n",
       " 'You may also unsubscribe on the account update page at:',\n",
       " 'http://www.multexinvestor.com/edituinfo.asp',\n",
       " '===================================================================',\n",
       " 'Please email advertising inquiries to us at mailto:advertise@multex.com.',\n",
       " '',\n",
       " 'For information on becoming an affiliate click here: ',\n",
       " 'http://www.multexinvestor.com/Affiliates/home.asp?promo=unl',\n",
       " '',\n",
       " 'Be sure to check out one of our other newsletters, The Internet Analyst by',\n",
       " 'Multex.com. The Internet Analyst informs, educates, and entertains you with',\n",
       " 'usable investment data, ideas, experts, and info about the Internet industry.',\n",
       " \"To see this week's issue, click here: http://www.theinternetanalyst.com\",\n",
       " '',\n",
       " 'If you are not 100% satisfied with a purchase you make on Multex',\n",
       " 'Investor, we will refund your money.']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count = 0\n",
    "for line in email:\n",
    "    if line == '':\n",
    "        pass\n",
    "    else:\n",
    "        if line[0].isspace():\n",
    "            email[count-1] += line\n",
    "            del email[count]\n",
    "            print(count)\n",
    "    count += 1\n",
    "email\n",
    "#this is significantly better! we now have our completed subject \n",
    "#we simply need to remove the extra line under the governing category (Subject)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**that didn't work**<br>\n",
    "so we've got a problem on our hands. I've spent an hour or so on this and need to move on and come back later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "MessageID_tup = email[0].split(\":\")\n",
    "Date_tup = email[1].split(\":\")\n",
    "From_tup = email[2].split(\":\")\n",
    "To_tup = email[3].split(\":\")\n",
    "Subject_tup = email[4].split(\":\") \n",
    "MimeVers_tup = email[5].split(\":\")\n",
    "ContentType_tup = email[6].split(\":\")\n",
    "Encoding_tup = email[7].split(\":\")\n",
    "XFrom_tup = email[8].split(\":\")\n",
    "XTo_tup = email[9].split(\":\")\n",
    "CC_tup = email[10].split(\":\")\n",
    "BCC_tup = email[11].split(\":\")\n",
    "XFolder_tup = email[12].split(\":\")\n",
    "XOrigin_tup = email[13].split(\":\")\n",
    "XFilename_tup = email[14].split(\":\")\n",
    "header = [MessageID_tup, Date_tup, From_tup, To_tup, Subject_tup, MimeVers_tup, ContentType_tup, Encoding_tup, \n",
    "         XFrom_tup, XTo_tup, CC_tup, BCC_tup, XFolder_tup, XOrigin_tup, XFilename_tup]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Message-ID', ' <29790972.1075855665306.JavaMail.evans@thyme>']\n",
      "['Date', ' Wed, 13 Dec 2000 18', '41', '00 -0800 (PST)']\n",
      "['From', ' 1.11913372.-2@multexinvestornetwork.com']\n",
      "['To', ' pallen@enron.com']\n",
      "['Subject', \" December 14, 2000 - Bear Stearns' predictions for telecom in Latin America\"]\n",
      "['Content-Type', ' text/plain; charset=us-ascii']\n",
      "['Content-Transfer-Encoding', ' 7bit']\n",
      "['X-From', ' Multex Investor <1.11913372.-2@multexinvestornetwork.com>']\n",
      "['X-To', ' <pallen@enron.com>']\n",
      "['X-cc', ' ']\n",
      "['X-bcc', ' ']\n",
      "['X-Folder', ' \\\\Phillip_Allen_Dec2000\\\\Notes Folders\\\\All documents']\n",
      "['X-Origin', ' Allen-P']\n",
      "['X-FileName', ' pallen.nsf']\n",
      "['']\n"
     ]
    }
   ],
   "source": [
    "for cat in header:\n",
    "    print(cat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['America Online (AOL), Divine Interventures (DVIN),',\n",
       " 'and 3M (MMM); reports on the broadband space, Latin',\n",
       " 'American telecom, and more.',\n",
       " '',\n",
       " \"For free research, editor's picks, and more come to the Daily Investor:\",\n",
       " 'http://www.multexinvestor.com/AF004627/magazinecover.asp?promo=unl&d=20001214#',\n",
       " 'investor',\n",
       " '',\n",
       " '***************************************************************',\n",
       " 'You are receiving this mail because you have registered for',\n",
       " 'Multex Investor. To unsubscribe, see bottom of this message.',\n",
       " '***************************************************************',\n",
       " '',\n",
       " '======================== Sponsored by =========================',\n",
       " 'Would you own just the energy stocks in the S&P 500?',\n",
       " 'Select Sector SPDRs divides the S&P 500 into nine sector index funds.',\n",
       " 'Pick and choose just the pieces of the S&P 500 you like best.',\n",
       " 'http://www.spdrindex.com',\n",
       " '===============================================================',\n",
       " '',\n",
       " \"Featured in today's edition of the Daily Update:\",\n",
       " '',\n",
       " \"1. SPECIAL ANNOUNCEMENT: Treat yourself to Multex Investor's NEW Personal\",\n",
       " 'Finance Channel to take advantage of top-notch content and tools  FREE.',\n",
       " '',\n",
       " '2. DAILY FREE SPONSOR REPORT: Robertson Stephens maintains a \"buy\" rating',\n",
       " 'on Divine Interventures (DVIN).',\n",
       " '',\n",
       " '3. FREE RESEARCH REPORT: Jefferies & Co. rates America Online (AOL) a',\n",
       " '\"buy,\" saying projected growth remains in place.',\n",
       " '',\n",
       " \"4. ASK THE ANALYST: Morgan Stanley Dean Witter's Lew Smith in the Analyst\",\n",
       " 'Corner',\n",
       " '',\n",
       " \"5. HOT REPORT: Oscar Gruss & Son's most recent issue of its Broadband\",\n",
       " 'Brief reports the latest developments in the broadband space.',\n",
       " '',\n",
       " \"6. EDITOR'S PICK: Bear Stearns measures the impact of broadband and the\",\n",
       " 'Internet on telecom in Latin America.',\n",
       " '',\n",
       " \"7. FREE STOCK SNAPSHOT: The current analysts' consensus rates 3M (MMM), a\",\n",
       " '\"buy/hold.\"',\n",
       " '',\n",
       " '8. JOIN THE MARKETBUZZ: where top financial industry professionals answer',\n",
       " \"your questions and offer insights every market day from noon 'til 2:00\",\n",
       " 'p.m. ET.',\n",
       " '',\n",
       " '9. TRANSCRIPTS FROM WALL STREET: Ash Rajan, senior vice president and',\n",
       " 'market analyst with Prudential Securities, answers questions about the',\n",
       " 'market.',\n",
       " '',\n",
       " '======================== Sponsored by =========================',\n",
       " 'Profit From AAII\\'s \"Cash Rich\" Stock Screen - 46% YTD Return',\n",
       " '',\n",
       " 'With so much market volatility, how did AAII\\'s \"Cash Rich\"',\n",
       " 'Stock Screen achieve such stellar returns?  Find the answer by',\n",
       " 'taking a free trial membership from the American Association',\n",
       " 'of Individual Investors and using our FREE Stock Screen service at:',\n",
       " 'http://subs.aaii.com/c/go/XAAI/MTEX1B-aaiitU1?s=S900',\n",
       " '===============================================================',\n",
       " '',\n",
       " '1. NEW ON MULTEX INVESTOR',\n",
       " 'Take charge of your personal finances',\n",
       " '',\n",
       " 'Do you have endless hours of free time to keep your financial house in',\n",
       " \"order? We didn't think so. That's why you need to treat yourself to Multex\",\n",
       " \"Investor's NEW Personal Finance Channel to take advantage of top-notch\",\n",
       " 'content and tools  FREE.',\n",
       " 'Click here for more information.',\n",
       " 'http://www.multexpf.com?mktg=sgpftx4&promo=unl&t=10&d=20001214',\n",
       " '',\n",
       " '',\n",
       " '2. DAILY FREE SPONSOR REPORT',\n",
       " 'Divine Interventures (DVIN)',\n",
       " '',\n",
       " 'Robertson Stephens maintains a \"buy\" rating on Divine Interventures, an',\n",
       " 'incubator focused on infrastructure services and business-to-business',\n",
       " \"(B2B) exchanges. Register for Robertson Stephens' free-research trial to\",\n",
       " 'access this report.',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/Download.asp?docid=5018549&sid=9&promo=unl&t=12&',\n",
       " 'd=20001214',\n",
       " '',\n",
       " '',\n",
       " '3. FREE RESEARCH REPORT',\n",
       " \"Hold 'er steady -- America Online (AOL)\",\n",
       " '',\n",
       " \"AOL's projected growth and proposed merger with Time Warner (TWX) both\",\n",
       " 'remain in place, says Jefferies & Co., which maintains a \"buy\" rating on',\n",
       " 'AOL. In the report, which is free for a limited time, analysts are',\n",
       " 'confident the deal will close soon.',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/AF004627/magazinecover.asp?promo=unl&t=11&d=2000',\n",
       " '1214',\n",
       " '',\n",
       " '',\n",
       " '4. TODAY IN THE ANALYST CORNER',\n",
       " 'Following market trends',\n",
       " '',\n",
       " \"Morgan Stanley Dean Witter's Lew Smith sees strong underlying trends\",\n",
       " 'guiding future market performance. What trends does he point to, and what',\n",
       " 'stocks and sectors does he see benefiting from his premise?',\n",
       " '',\n",
       " \"Here is your opportunity to gain free access to Morgan Stanley's research.\",\n",
       " 'Simply register and submit a question below. You will then have a free',\n",
       " \"trial membership to this top Wall Street firms' research!  Lew Smith will\",\n",
       " 'be in the Analyst Corner only until 5 p.m. ET Thurs., Dec. 14, so be sure',\n",
       " 'to ask your question now.',\n",
       " 'Ask the analyst.',\n",
       " 'http://www.multexinvestor.com/ACHome.asp?promo=unl&t=1&d=20001214',\n",
       " '',\n",
       " '',\n",
       " \"5. WHAT'S HOT? RESEARCH REPORTS FROM MULTEX INVESTOR'S HOT LIST\",\n",
       " 'Breaking the bottleneck -- An update on the broadband space',\n",
       " '',\n",
       " \"Oscar Gruss & Son's most recent issue of its Broadband Brief reports the\",\n",
       " 'latest developments in the broadband space, with coverage of Adaptive',\n",
       " 'Broadband (ADAP), Broadcom (BRCM), Efficient Networks (EFNT), and others',\n",
       " '(report for purchase - $25).',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/Download.asp?docid=5149041&promo=unl&t=4&d=20001',\n",
       " '214',\n",
       " '',\n",
       " '======================== Sponsored by =========================',\n",
       " 'Get Red Herring insight into hot IPOs, investing strategies,',\n",
       " 'stocks to watch, future technologies, and more. FREE',\n",
       " 'E-newsletters from Redherring.com provide more answers,',\n",
       " 'analysis and opinion to help you make more strategic',\n",
       " 'investing decisions. Subscribe today',\n",
       " 'http://www.redherring.com/jump/om/i/multex/email2/subscribe/47.html',\n",
       " '===============================================================',\n",
       " '',\n",
       " \"6. EDITOR'S PICK: CURRENT RESEARCH FROM THE CUTTING EDGE\",\n",
       " \"Que pasa? -- Predicting telecom's future in Latin America\",\n",
       " '',\n",
       " 'Bear Stearns measures the impact of broadband and the Internet on telecom',\n",
       " 'in Latin America, saying incumbent local-exchange carriers (ILECs) are',\n",
       " 'ideally positioned to benefit from the growth of Internet and data',\n",
       " 'services (report for purchase - $150).',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/Download.asp?docid=5140995&promo=unl&t=8&d=20001',\n",
       " '214',\n",
       " '',\n",
       " '',\n",
       " '7. FREE STOCK SNAPSHOT',\n",
       " '3M (MMM)',\n",
       " '',\n",
       " 'The current analysts\\' consensus rates 3M, a \"buy/hold.\" Analysts expect',\n",
       " 'the industrial product manufacturer to earn $4.76 per share in 2000 and',\n",
       " '$5.26 per share in 2001.',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/Download.asp?docid=1346414&promo=unl&t=3&d=20001',\n",
       " '214',\n",
       " '',\n",
       " '',\n",
       " '8. JOIN THE MARKETBUZZ!',\n",
       " 'Check out SageOnline',\n",
       " '',\n",
       " 'where top financial industry professionals answer your questions and offer',\n",
       " \"insights every market day from noon 'til 2:00 p.m. ET.\",\n",
       " 'Click here.',\n",
       " 'http://multexinvestor.sageonline.com/page2.asp?id=9512&ps=1&s=2&mktg=evn&promo',\n",
       " '=unl&t=24&d=20001214',\n",
       " '',\n",
       " '',\n",
       " \"9. TRANSCRIPTS FROM WALL STREET'S GURUS\",\n",
       " \"Prudential Securities' Ash Rajan\",\n",
       " '',\n",
       " 'In this SageOnline transcript from a chat that took place earlier this',\n",
       " 'week, Ash Rajan, senior vice president and market analyst with Prudential',\n",
       " 'Securities, answers questions about tech, retail, finance, and the outlook',\n",
       " 'for the general market.',\n",
       " 'Click here.',\n",
       " 'http://multexinvestor.sageonline.com/transcript.asp?id=10403&ps=1&s=8&mktg=trn',\n",
       " '&promo=unl&t=13&d=20001214',\n",
       " '',\n",
       " '===================================================================',\n",
       " 'Please send your questions and comments to mailto:investor.help@multex.com',\n",
       " '',\n",
       " \"If you'd like to learn more about Multex Investor, please visit:\",\n",
       " 'http://www.multexinvestor.com/welcome.asp',\n",
       " '',\n",
       " \"If you can't remember your password and/or your user name, click here:\",\n",
       " 'http://www.multexinvestor.com/lostinfo.asp',\n",
       " '',\n",
       " 'If you want to update your email address, please click on the url below:',\n",
       " 'http://www.multexinvestor.com/edituinfo.asp',\n",
       " '===================================================================',\n",
       " 'To remove yourself from the mailing list for the Daily Update, please',\n",
       " 'REPLY to THIS email message with the word UNSUBSCRIBE in the subject',\n",
       " 'line. To remove yourself from all Multex Investor mailings, including',\n",
       " 'the Daily Update and The Internet Analyst, please respond with the',\n",
       " 'words NO EMAIL in the subject line.',\n",
       " '',\n",
       " 'You may also unsubscribe on the account update page at:',\n",
       " 'http://www.multexinvestor.com/edituinfo.asp',\n",
       " '===================================================================',\n",
       " 'Please email advertising inquiries to us at mailto:advertise@multex.com.',\n",
       " '',\n",
       " 'For information on becoming an affiliate click here: ',\n",
       " 'http://www.multexinvestor.com/Affiliates/home.asp?promo=unl',\n",
       " '',\n",
       " 'Be sure to check out one of our other newsletters, The Internet Analyst by',\n",
       " 'Multex.com. The Internet Analyst informs, educates, and entertains you with',\n",
       " 'usable investment data, ideas, experts, and info about the Internet industry.',\n",
       " \"To see this week's issue, click here: http://www.theinternetanalyst.com\",\n",
       " '',\n",
       " 'If you are not 100% satisfied with a purchase you make on Multex',\n",
       " 'Investor, we will refund your money.']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "email[16:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "now when we print the tuples, we get matching pairs. This will only be the case when I fix that algorithm (hard coded for now)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['America Online (AOL), Divine Interventures (DVIN),',\n",
       " 'and 3M (MMM); reports on the broadband space, Latin',\n",
       " 'American telecom, and more.',\n",
       " \"For free research, editor's picks, and more come to the Daily Investor:\",\n",
       " 'http://www.multexinvestor.com/AF004627/magazinecover.asp?promo=unl&d=20001214#',\n",
       " 'investor',\n",
       " '***************************************************************',\n",
       " 'You are receiving this mail because you have registered for',\n",
       " 'Multex Investor. To unsubscribe, see bottom of this message.',\n",
       " '***************************************************************',\n",
       " '======================== Sponsored by =========================',\n",
       " 'Would you own just the energy stocks in the S&P 500?',\n",
       " 'Select Sector SPDRs divides the S&P 500 into nine sector index funds.',\n",
       " 'Pick and choose just the pieces of the S&P 500 you like best.',\n",
       " 'http://www.spdrindex.com',\n",
       " '===============================================================',\n",
       " \"Featured in today's edition of the Daily Update:\",\n",
       " \"1. SPECIAL ANNOUNCEMENT: Treat yourself to Multex Investor's NEW Personal\",\n",
       " 'Finance Channel to take advantage of top-notch content and tools  FREE.',\n",
       " '2. DAILY FREE SPONSOR REPORT: Robertson Stephens maintains a \"buy\" rating',\n",
       " 'on Divine Interventures (DVIN).',\n",
       " '3. FREE RESEARCH REPORT: Jefferies & Co. rates America Online (AOL) a',\n",
       " '\"buy,\" saying projected growth remains in place.',\n",
       " \"4. ASK THE ANALYST: Morgan Stanley Dean Witter's Lew Smith in the Analyst\",\n",
       " 'Corner',\n",
       " \"5. HOT REPORT: Oscar Gruss & Son's most recent issue of its Broadband\",\n",
       " 'Brief reports the latest developments in the broadband space.',\n",
       " \"6. EDITOR'S PICK: Bear Stearns measures the impact of broadband and the\",\n",
       " 'Internet on telecom in Latin America.',\n",
       " \"7. FREE STOCK SNAPSHOT: The current analysts' consensus rates 3M (MMM), a\",\n",
       " '\"buy/hold.\"',\n",
       " '8. JOIN THE MARKETBUZZ: where top financial industry professionals answer',\n",
       " \"your questions and offer insights every market day from noon 'til 2:00\",\n",
       " 'p.m. ET.',\n",
       " '9. TRANSCRIPTS FROM WALL STREET: Ash Rajan, senior vice president and',\n",
       " 'market analyst with Prudential Securities, answers questions about the',\n",
       " 'market.',\n",
       " '======================== Sponsored by =========================',\n",
       " 'Profit From AAII\\'s \"Cash Rich\" Stock Screen - 46% YTD Return',\n",
       " 'With so much market volatility, how did AAII\\'s \"Cash Rich\"',\n",
       " 'Stock Screen achieve such stellar returns?  Find the answer by',\n",
       " 'taking a free trial membership from the American Association',\n",
       " 'of Individual Investors and using our FREE Stock Screen service at:',\n",
       " 'http://subs.aaii.com/c/go/XAAI/MTEX1B-aaiitU1?s=S900',\n",
       " '===============================================================',\n",
       " '1. NEW ON MULTEX INVESTOR',\n",
       " 'Take charge of your personal finances',\n",
       " 'Do you have endless hours of free time to keep your financial house in',\n",
       " \"order? We didn't think so. That's why you need to treat yourself to Multex\",\n",
       " \"Investor's NEW Personal Finance Channel to take advantage of top-notch\",\n",
       " 'content and tools  FREE.',\n",
       " 'Click here for more information.',\n",
       " 'http://www.multexpf.com?mktg=sgpftx4&promo=unl&t=10&d=20001214',\n",
       " '2. DAILY FREE SPONSOR REPORT',\n",
       " 'Divine Interventures (DVIN)',\n",
       " 'Robertson Stephens maintains a \"buy\" rating on Divine Interventures, an',\n",
       " 'incubator focused on infrastructure services and business-to-business',\n",
       " \"(B2B) exchanges. Register for Robertson Stephens' free-research trial to\",\n",
       " 'access this report.',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/Download.asp?docid=5018549&sid=9&promo=unl&t=12&',\n",
       " 'd=20001214',\n",
       " '3. FREE RESEARCH REPORT',\n",
       " \"Hold 'er steady -- America Online (AOL)\",\n",
       " \"AOL's projected growth and proposed merger with Time Warner (TWX) both\",\n",
       " 'remain in place, says Jefferies & Co., which maintains a \"buy\" rating on',\n",
       " 'AOL. In the report, which is free for a limited time, analysts are',\n",
       " 'confident the deal will close soon.',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/AF004627/magazinecover.asp?promo=unl&t=11&d=2000',\n",
       " '1214',\n",
       " '4. TODAY IN THE ANALYST CORNER',\n",
       " 'Following market trends',\n",
       " \"Morgan Stanley Dean Witter's Lew Smith sees strong underlying trends\",\n",
       " 'guiding future market performance. What trends does he point to, and what',\n",
       " 'stocks and sectors does he see benefiting from his premise?',\n",
       " \"Here is your opportunity to gain free access to Morgan Stanley's research.\",\n",
       " 'Simply register and submit a question below. You will then have a free',\n",
       " \"trial membership to this top Wall Street firms' research!  Lew Smith will\",\n",
       " 'be in the Analyst Corner only until 5 p.m. ET Thurs., Dec. 14, so be sure',\n",
       " 'to ask your question now.',\n",
       " 'Ask the analyst.',\n",
       " 'http://www.multexinvestor.com/ACHome.asp?promo=unl&t=1&d=20001214',\n",
       " \"5. WHAT'S HOT? RESEARCH REPORTS FROM MULTEX INVESTOR'S HOT LIST\",\n",
       " 'Breaking the bottleneck -- An update on the broadband space',\n",
       " \"Oscar Gruss & Son's most recent issue of its Broadband Brief reports the\",\n",
       " 'latest developments in the broadband space, with coverage of Adaptive',\n",
       " 'Broadband (ADAP), Broadcom (BRCM), Efficient Networks (EFNT), and others',\n",
       " '(report for purchase - $25).',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/Download.asp?docid=5149041&promo=unl&t=4&d=20001',\n",
       " '214',\n",
       " '======================== Sponsored by =========================',\n",
       " 'Get Red Herring insight into hot IPOs, investing strategies,',\n",
       " 'stocks to watch, future technologies, and more. FREE',\n",
       " 'E-newsletters from Redherring.com provide more answers,',\n",
       " 'analysis and opinion to help you make more strategic',\n",
       " 'investing decisions. Subscribe today',\n",
       " 'http://www.redherring.com/jump/om/i/multex/email2/subscribe/47.html',\n",
       " '===============================================================',\n",
       " \"6. EDITOR'S PICK: CURRENT RESEARCH FROM THE CUTTING EDGE\",\n",
       " \"Que pasa? -- Predicting telecom's future in Latin America\",\n",
       " 'Bear Stearns measures the impact of broadband and the Internet on telecom',\n",
       " 'in Latin America, saying incumbent local-exchange carriers (ILECs) are',\n",
       " 'ideally positioned to benefit from the growth of Internet and data',\n",
       " 'services (report for purchase - $150).',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/Download.asp?docid=5140995&promo=unl&t=8&d=20001',\n",
       " '214',\n",
       " '7. FREE STOCK SNAPSHOT',\n",
       " '3M (MMM)',\n",
       " 'The current analysts\\' consensus rates 3M, a \"buy/hold.\" Analysts expect',\n",
       " 'the industrial product manufacturer to earn $4.76 per share in 2000 and',\n",
       " '$5.26 per share in 2001.',\n",
       " 'Click here.',\n",
       " 'http://www.multexinvestor.com/Download.asp?docid=1346414&promo=unl&t=3&d=20001',\n",
       " '214',\n",
       " '8. JOIN THE MARKETBUZZ!',\n",
       " 'Check out SageOnline',\n",
       " 'where top financial industry professionals answer your questions and offer',\n",
       " \"insights every market day from noon 'til 2:00 p.m. ET.\",\n",
       " 'Click here.',\n",
       " 'http://multexinvestor.sageonline.com/page2.asp?id=9512&ps=1&s=2&mktg=evn&promo',\n",
       " '=unl&t=24&d=20001214',\n",
       " \"9. TRANSCRIPTS FROM WALL STREET'S GURUS\",\n",
       " \"Prudential Securities' Ash Rajan\",\n",
       " 'In this SageOnline transcript from a chat that took place earlier this',\n",
       " 'week, Ash Rajan, senior vice president and market analyst with Prudential',\n",
       " 'Securities, answers questions about tech, retail, finance, and the outlook',\n",
       " 'for the general market.',\n",
       " 'Click here.',\n",
       " 'http://multexinvestor.sageonline.com/transcript.asp?id=10403&ps=1&s=8&mktg=trn',\n",
       " '&promo=unl&t=13&d=20001214',\n",
       " '===================================================================',\n",
       " 'Please send your questions and comments to mailto:investor.help@multex.com',\n",
       " '',\n",
       " \"If you'd like to learn more about Multex Investor, please visit:\",\n",
       " 'http://www.multexinvestor.com/welcome.asp',\n",
       " '',\n",
       " \"If you can't remember your password and/or your user name, click here:\",\n",
       " 'http://www.multexinvestor.com/lostinfo.asp',\n",
       " '',\n",
       " 'If you want to update your email address, please click on the url below:',\n",
       " 'http://www.multexinvestor.com/edituinfo.asp',\n",
       " '===================================================================',\n",
       " 'To remove yourself from the mailing list for the Daily Update, please',\n",
       " 'REPLY to THIS email message with the word UNSUBSCRIBE in the subject',\n",
       " 'line. To remove yourself from all Multex Investor mailings, including',\n",
       " 'the Daily Update and The Internet Analyst, please respond with the',\n",
       " 'words NO EMAIL in the subject line.',\n",
       " '',\n",
       " 'You may also unsubscribe on the account update page at:',\n",
       " 'http://www.multexinvestor.com/edituinfo.asp',\n",
       " '===================================================================',\n",
       " 'Please email advertising inquiries to us at mailto:advertise@multex.com.',\n",
       " '',\n",
       " 'For information on becoming an affiliate click here: ',\n",
       " 'http://www.multexinvestor.com/Affiliates/home.asp?promo=unl',\n",
       " '',\n",
       " 'Be sure to check out one of our other newsletters, The Internet Analyst by',\n",
       " 'Multex.com. The Internet Analyst informs, educates, and entertains you with',\n",
       " 'usable investment data, ideas, experts, and info about the Internet industry.',\n",
       " \"To see this week's issue, click here: http://www.theinternetanalyst.com\",\n",
       " '',\n",
       " 'If you are not 100% satisfied with a purchase you make on Multex',\n",
       " 'Investor, we will refund your money.']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = email[16:]\n",
    "for line in text:\n",
    "    if line == '':\n",
    "        text.remove(line)\n",
    "text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'America Online (AOL), Divine Interventures (DVIN),and 3M (MMM); reports on the broadband space, LatinAmerican telecom, and more.For free research, editor\\'s picks, and more come to the Daily Investor:http://www.multexinvestor.com/AF004627/magazinecover.asp?promo=unl&d=20001214#investor***************************************************************You are receiving this mail because you have registered forMultex Investor. To unsubscribe, see bottom of this message.***************************************************************======================== Sponsored by =========================Would you own just the energy stocks in the S&P 500?Select Sector SPDRs divides the S&P 500 into nine sector index funds.Pick and choose just the pieces of the S&P 500 you like best.http://www.spdrindex.com===============================================================Featured in today\\'s edition of the Daily Update:1. SPECIAL ANNOUNCEMENT: Treat yourself to Multex Investor\\'s NEW PersonalFinance Channel to take advantage of top-notch content and tools  FREE.2. DAILY FREE SPONSOR REPORT: Robertson Stephens maintains a \"buy\" ratingon Divine Interventures (DVIN).3. FREE RESEARCH REPORT: Jefferies & Co. rates America Online (AOL) a\"buy,\" saying projected growth remains in place.4. ASK THE ANALYST: Morgan Stanley Dean Witter\\'s Lew Smith in the AnalystCorner5. HOT REPORT: Oscar Gruss & Son\\'s most recent issue of its BroadbandBrief reports the latest developments in the broadband space.6. EDITOR\\'S PICK: Bear Stearns measures the impact of broadband and theInternet on telecom in Latin America.7. FREE STOCK SNAPSHOT: The current analysts\\' consensus rates 3M (MMM), a\"buy/hold.\"8. JOIN THE MARKETBUZZ: where top financial industry professionals answeryour questions and offer insights every market day from noon \\'til 2:00p.m. ET.9. TRANSCRIPTS FROM WALL STREET: Ash Rajan, senior vice president andmarket analyst with Prudential Securities, answers questions about themarket.======================== Sponsored by =========================Profit From AAII\\'s \"Cash Rich\" Stock Screen - 46% YTD ReturnWith so much market volatility, how did AAII\\'s \"Cash Rich\"Stock Screen achieve such stellar returns?  Find the answer bytaking a free trial membership from the American Associationof Individual Investors and using our FREE Stock Screen service at:http://subs.aaii.com/c/go/XAAI/MTEX1B-aaiitU1?s=S900===============================================================1. NEW ON MULTEX INVESTORTake charge of your personal financesDo you have endless hours of free time to keep your financial house inorder? We didn\\'t think so. That\\'s why you need to treat yourself to MultexInvestor\\'s NEW Personal Finance Channel to take advantage of top-notchcontent and tools  FREE.Click here for more information.http://www.multexpf.com?mktg=sgpftx4&promo=unl&t=10&d=200012142. DAILY FREE SPONSOR REPORTDivine Interventures (DVIN)Robertson Stephens maintains a \"buy\" rating on Divine Interventures, anincubator focused on infrastructure services and business-to-business(B2B) exchanges. Register for Robertson Stephens\\' free-research trial toaccess this report.Click here.http://www.multexinvestor.com/Download.asp?docid=5018549&sid=9&promo=unl&t=12&d=200012143. FREE RESEARCH REPORTHold \\'er steady -- America Online (AOL)AOL\\'s projected growth and proposed merger with Time Warner (TWX) bothremain in place, says Jefferies & Co., which maintains a \"buy\" rating onAOL. In the report, which is free for a limited time, analysts areconfident the deal will close soon.Click here.http://www.multexinvestor.com/AF004627/magazinecover.asp?promo=unl&t=11&d=200012144. TODAY IN THE ANALYST CORNERFollowing market trendsMorgan Stanley Dean Witter\\'s Lew Smith sees strong underlying trendsguiding future market performance. What trends does he point to, and whatstocks and sectors does he see benefiting from his premise?Here is your opportunity to gain free access to Morgan Stanley\\'s research.Simply register and submit a question below. You will then have a freetrial membership to this top Wall Street firms\\' research!  Lew Smith willbe in the Analyst Corner only until 5 p.m. ET Thurs., Dec. 14, so be sureto ask your question now.Ask the analyst.http://www.multexinvestor.com/ACHome.asp?promo=unl&t=1&d=200012145. WHAT\\'S HOT? RESEARCH REPORTS FROM MULTEX INVESTOR\\'S HOT LISTBreaking the bottleneck -- An update on the broadband spaceOscar Gruss & Son\\'s most recent issue of its Broadband Brief reports thelatest developments in the broadband space, with coverage of AdaptiveBroadband (ADAP), Broadcom (BRCM), Efficient Networks (EFNT), and others(report for purchase - $25).Click here.http://www.multexinvestor.com/Download.asp?docid=5149041&promo=unl&t=4&d=20001214======================== Sponsored by =========================Get Red Herring insight into hot IPOs, investing strategies,stocks to watch, future technologies, and more. FREEE-newsletters from Redherring.com provide more answers,analysis and opinion to help you make more strategicinvesting decisions. Subscribe todayhttp://www.redherring.com/jump/om/i/multex/email2/subscribe/47.html===============================================================6. EDITOR\\'S PICK: CURRENT RESEARCH FROM THE CUTTING EDGEQue pasa? -- Predicting telecom\\'s future in Latin AmericaBear Stearns measures the impact of broadband and the Internet on telecomin Latin America, saying incumbent local-exchange carriers (ILECs) areideally positioned to benefit from the growth of Internet and dataservices (report for purchase - $150).Click here.http://www.multexinvestor.com/Download.asp?docid=5140995&promo=unl&t=8&d=200012147. FREE STOCK SNAPSHOT3M (MMM)The current analysts\\' consensus rates 3M, a \"buy/hold.\" Analysts expectthe industrial product manufacturer to earn $4.76 per share in 2000 and$5.26 per share in 2001.Click here.http://www.multexinvestor.com/Download.asp?docid=1346414&promo=unl&t=3&d=200012148. JOIN THE MARKETBUZZ!Check out SageOnlinewhere top financial industry professionals answer your questions and offerinsights every market day from noon \\'til 2:00 p.m. ET.Click here.http://multexinvestor.sageonline.com/page2.asp?id=9512&ps=1&s=2&mktg=evn&promo=unl&t=24&d=200012149. TRANSCRIPTS FROM WALL STREET\\'S GURUSPrudential Securities\\' Ash RajanIn this SageOnline transcript from a chat that took place earlier thisweek, Ash Rajan, senior vice president and market analyst with PrudentialSecurities, answers questions about tech, retail, finance, and the outlookfor the general market.Click here.http://multexinvestor.sageonline.com/transcript.asp?id=10403&ps=1&s=8&mktg=trn&promo=unl&t=13&d=20001214===================================================================Please send your questions and comments to mailto:investor.help@multex.comIf you\\'d like to learn more about Multex Investor, please visit:http://www.multexinvestor.com/welcome.aspIf you can\\'t remember your password and/or your user name, click here:http://www.multexinvestor.com/lostinfo.aspIf you want to update your email address, please click on the url below:http://www.multexinvestor.com/edituinfo.asp===================================================================To remove yourself from the mailing list for the Daily Update, pleaseREPLY to THIS email message with the word UNSUBSCRIBE in the subjectline. To remove yourself from all Multex Investor mailings, includingthe Daily Update and The Internet Analyst, please respond with thewords NO EMAIL in the subject line.You may also unsubscribe on the account update page at:http://www.multexinvestor.com/edituinfo.asp===================================================================Please email advertising inquiries to us at mailto:advertise@multex.com.For information on becoming an affiliate click here: http://www.multexinvestor.com/Affiliates/home.asp?promo=unlBe sure to check out one of our other newsletters, The Internet Analyst byMultex.com. The Internet Analyst informs, educates, and entertains you withusable investment data, ideas, experts, and info about the Internet industry.To see this week\\'s issue, click here: http://www.theinternetanalyst.comIf you are not 100% satisfied with a purchase you make on MultexInvestor, we will refund your money.'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "textList = email[16:]\n",
    "textStr = \"\"\n",
    "for line in textList:\n",
    "    textStr += line\n",
    "textStr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "huh. this is so weird. for some reason, if I only run the removes on this once, it removes *some* of the whitespace at the beginning, but misses at the end. very odd.<br>\n",
    "**Problem**: So, it's going to take a lot of time and energy to wrangle this list into a nice string to put into the DF. In fact, I need to think a *lot* more about what version of this text I'll be storing in my panda. I'm not really sure what I should be parsing this text for... something to think about in the future.\n",
    "___\n",
    "at the very least, we now have a string that represents the text of the email. lets add it to the list of tuples now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "header.append([\"Text\",textStr])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Message-ID', ' <29790972.1075855665306.JavaMail.evans@thyme>']\n",
      "['Date', ' Wed, 13 Dec 2000 18', '41', '00 -0800 (PST)']\n",
      "['From', ' 1.11913372.-2@multexinvestornetwork.com']\n",
      "['To', ' pallen@enron.com']\n",
      "['Subject', \" December 14, 2000 - Bear Stearns' predictions for telecom in Latin America\", 'C', 'o', 'n', 't', 'e', 'n', 't', '-', 'T', 'y', 'p', 'e', ':', ' ', 't', 'e', 'x', 't', '/', 'p', 'l', 'a', 'i', 'n', ';', ' ', 'c', 'h', 'a', 'r', 's', 'e', 't', '=', 'u', 's', '-', 'a', 's', 'c', 'i', 'i']\n",
      "['Content-Transfer-Encoding', ' 7bit']\n",
      "['X-From', ' Multex Investor <1.11913372.-2@multexinvestornetwork.com>']\n",
      "['X-To', ' <pallen@enron.com>']\n",
      "['X-cc', ' ']\n",
      "['X-bcc', ' ']\n",
      "['X-Folder', ' \\\\Phillip_Allen_Dec2000\\\\Notes Folders\\\\All documents']\n",
      "['X-Origin', ' Allen-P']\n",
      "['X-FileName', ' pallen.nsf']\n",
      "['']\n",
      "[\"In today's Daily Update you'll find free reports on\"]\n",
      "['Text', 'America Online (AOL), Divine Interventures (DVIN),and 3M (MMM); reports on the broadband space, LatinAmerican telecom, and more.For free research, editor\\'s picks, and more come to the Daily Investor:http://www.multexinvestor.com/AF004627/magazinecover.asp?promo=unl&d=20001214#investor***************************************************************You are receiving this mail because you have registered forMultex Investor. To unsubscribe, see bottom of this message.***************************************************************======================== Sponsored by =========================Would you own just the energy stocks in the S&P 500?Select Sector SPDRs divides the S&P 500 into nine sector index funds.Pick and choose just the pieces of the S&P 500 you like best.http://www.spdrindex.com===============================================================Featured in today\\'s edition of the Daily Update:1. SPECIAL ANNOUNCEMENT: Treat yourself to Multex Investor\\'s NEW PersonalFinance Channel to take advantage of top-notch content and tools  FREE.2. DAILY FREE SPONSOR REPORT: Robertson Stephens maintains a \"buy\" ratingon Divine Interventures (DVIN).3. FREE RESEARCH REPORT: Jefferies & Co. rates America Online (AOL) a\"buy,\" saying projected growth remains in place.4. ASK THE ANALYST: Morgan Stanley Dean Witter\\'s Lew Smith in the AnalystCorner5. HOT REPORT: Oscar Gruss & Son\\'s most recent issue of its BroadbandBrief reports the latest developments in the broadband space.6. EDITOR\\'S PICK: Bear Stearns measures the impact of broadband and theInternet on telecom in Latin America.7. FREE STOCK SNAPSHOT: The current analysts\\' consensus rates 3M (MMM), a\"buy/hold.\"8. JOIN THE MARKETBUZZ: where top financial industry professionals answeryour questions and offer insights every market day from noon \\'til 2:00p.m. ET.9. TRANSCRIPTS FROM WALL STREET: Ash Rajan, senior vice president andmarket analyst with Prudential Securities, answers questions about themarket.======================== Sponsored by =========================Profit From AAII\\'s \"Cash Rich\" Stock Screen - 46% YTD ReturnWith so much market volatility, how did AAII\\'s \"Cash Rich\"Stock Screen achieve such stellar returns?  Find the answer bytaking a free trial membership from the American Associationof Individual Investors and using our FREE Stock Screen service at:http://subs.aaii.com/c/go/XAAI/MTEX1B-aaiitU1?s=S900===============================================================1. NEW ON MULTEX INVESTORTake charge of your personal financesDo you have endless hours of free time to keep your financial house inorder? We didn\\'t think so. That\\'s why you need to treat yourself to MultexInvestor\\'s NEW Personal Finance Channel to take advantage of top-notchcontent and tools  FREE.Click here for more information.http://www.multexpf.com?mktg=sgpftx4&promo=unl&t=10&d=200012142. DAILY FREE SPONSOR REPORTDivine Interventures (DVIN)Robertson Stephens maintains a \"buy\" rating on Divine Interventures, anincubator focused on infrastructure services and business-to-business(B2B) exchanges. Register for Robertson Stephens\\' free-research trial toaccess this report.Click here.http://www.multexinvestor.com/Download.asp?docid=5018549&sid=9&promo=unl&t=12&d=200012143. FREE RESEARCH REPORTHold \\'er steady -- America Online (AOL)AOL\\'s projected growth and proposed merger with Time Warner (TWX) bothremain in place, says Jefferies & Co., which maintains a \"buy\" rating onAOL. In the report, which is free for a limited time, analysts areconfident the deal will close soon.Click here.http://www.multexinvestor.com/AF004627/magazinecover.asp?promo=unl&t=11&d=200012144. TODAY IN THE ANALYST CORNERFollowing market trendsMorgan Stanley Dean Witter\\'s Lew Smith sees strong underlying trendsguiding future market performance. What trends does he point to, and whatstocks and sectors does he see benefiting from his premise?Here is your opportunity to gain free access to Morgan Stanley\\'s research.Simply register and submit a question below. You will then have a freetrial membership to this top Wall Street firms\\' research!  Lew Smith willbe in the Analyst Corner only until 5 p.m. ET Thurs., Dec. 14, so be sureto ask your question now.Ask the analyst.http://www.multexinvestor.com/ACHome.asp?promo=unl&t=1&d=200012145. WHAT\\'S HOT? RESEARCH REPORTS FROM MULTEX INVESTOR\\'S HOT LISTBreaking the bottleneck -- An update on the broadband spaceOscar Gruss & Son\\'s most recent issue of its Broadband Brief reports thelatest developments in the broadband space, with coverage of AdaptiveBroadband (ADAP), Broadcom (BRCM), Efficient Networks (EFNT), and others(report for purchase - $25).Click here.http://www.multexinvestor.com/Download.asp?docid=5149041&promo=unl&t=4&d=20001214======================== Sponsored by =========================Get Red Herring insight into hot IPOs, investing strategies,stocks to watch, future technologies, and more. FREEE-newsletters from Redherring.com provide more answers,analysis and opinion to help you make more strategicinvesting decisions. Subscribe todayhttp://www.redherring.com/jump/om/i/multex/email2/subscribe/47.html===============================================================6. EDITOR\\'S PICK: CURRENT RESEARCH FROM THE CUTTING EDGEQue pasa? -- Predicting telecom\\'s future in Latin AmericaBear Stearns measures the impact of broadband and the Internet on telecomin Latin America, saying incumbent local-exchange carriers (ILECs) areideally positioned to benefit from the growth of Internet and dataservices (report for purchase - $150).Click here.http://www.multexinvestor.com/Download.asp?docid=5140995&promo=unl&t=8&d=200012147. FREE STOCK SNAPSHOT3M (MMM)The current analysts\\' consensus rates 3M, a \"buy/hold.\" Analysts expectthe industrial product manufacturer to earn $4.76 per share in 2000 and$5.26 per share in 2001.Click here.http://www.multexinvestor.com/Download.asp?docid=1346414&promo=unl&t=3&d=200012148. JOIN THE MARKETBUZZ!Check out SageOnlinewhere top financial industry professionals answer your questions and offerinsights every market day from noon \\'til 2:00 p.m. ET.Click here.http://multexinvestor.sageonline.com/page2.asp?id=9512&ps=1&s=2&mktg=evn&promo=unl&t=24&d=200012149. TRANSCRIPTS FROM WALL STREET\\'S GURUSPrudential Securities\\' Ash RajanIn this SageOnline transcript from a chat that took place earlier thisweek, Ash Rajan, senior vice president and market analyst with PrudentialSecurities, answers questions about tech, retail, finance, and the outlookfor the general market.Click here.http://multexinvestor.sageonline.com/transcript.asp?id=10403&ps=1&s=8&mktg=trn&promo=unl&t=13&d=20001214===================================================================Please send your questions and comments to mailto:investor.help@multex.comIf you\\'d like to learn more about Multex Investor, please visit:http://www.multexinvestor.com/welcome.aspIf you can\\'t remember your password and/or your user name, click here:http://www.multexinvestor.com/lostinfo.aspIf you want to update your email address, please click on the url below:http://www.multexinvestor.com/edituinfo.asp===================================================================To remove yourself from the mailing list for the Daily Update, pleaseREPLY to THIS email message with the word UNSUBSCRIBE in the subjectline. To remove yourself from all Multex Investor mailings, includingthe Daily Update and The Internet Analyst, please respond with thewords NO EMAIL in the subject line.You may also unsubscribe on the account update page at:http://www.multexinvestor.com/edituinfo.asp===================================================================Please email advertising inquiries to us at mailto:advertise@multex.com.For information on becoming an affiliate click here: http://www.multexinvestor.com/Affiliates/home.asp?promo=unlBe sure to check out one of our other newsletters, The Internet Analyst byMultex.com. The Internet Analyst informs, educates, and entertains you withusable investment data, ideas, experts, and info about the Internet industry.To see this week\\'s issue, click here: http://www.theinternetanalyst.comIf you are not 100% satisfied with a purchase you make on MultexInvestor, we will refund your money.']\n"
     ]
    }
   ],
   "source": [
    "for cat in header:\n",
    "    print(cat)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "unruly, but we officially have a list that contains our categories and their data. lets get it into a DF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "too many values to unpack (expected 2)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-15-78fd9e8a39e0>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[0mDF\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mDataFrame\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mtup\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mheader\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m     \u001b[1;32mfor\u001b[0m \u001b[0mx\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0my\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mtup\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m         \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: too many values to unpack (expected 2)"
     ]
    }
   ],
   "source": [
    "DF = pd.DataFrame()\n",
    "for tup in header:\n",
    "    for x,y in tup:\n",
    "        print(x,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "this is going to be the error that does me in. onto the summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Project Report 1: Summary\n",
    "**What Went Well**:<br>\n",
    "1. Outside the immediate bounds of this notebook, I've definitely improved my ability to use Markdown. I definitely think my notebooks and README's look a lot better.\n",
    "2. Searching through the directory. At the beginning of this notebook, I did test the opening and reading in of mqny different files from the directory. This may not be important *this minute*, but the ability to apply my cleaning, exploration, and analysis to all parts of this corpus cannot be understated.\n",
    "3. I have a clear direction going forward. Truthfully, this notebook was a huge failure and I did not prepare for how difficult it would be to do simple tasks like reading in data. That being said, I have a much clearer idea of what I need moving forward into Spring Break.\n",
    "___\n",
    "**What Went Poorly**:\n",
    "1. I really struggled in converting data types in the wild. Reading the data in, getting the data into a tuple, and transferring this data into a dataframe were all difficult/impossible for me in this notebook. See section below.\n",
    "2. I wasn't thinking ahead. You could probably tell by 5-6 cells in that I wasn't going to make it for the long haul in this one. It's *incredibly important* to make sure when you are reading newly formatted data that you put it into the correct data type (namely, my problems with the tuples).\n",
    "___\n",
    "**What's Next**:\n",
    "1. A list of things I need to fix from this notebook:\n",
    "    1. A generalizable way to turn the heading section into a tuple with the category and the data. I was close here, but the emails with multiple lines per category (i.e. an email sent to 30 people) tripped me up here.\n",
    "    2. A better text string. This is one I need to think about more. I'm definitely going to need something to put into the dataframe, as far as the body of the email is concerned. The question is, what should I use? My current version needs fixing.\n",
    "    3. A DF. This one will come last, but I definitely need a DF that can be expanded to include N number of emails.\n",
    "___ \n",
    "**Conclusion**:<br>\n",
    "All in all, a very rough go of things. After midterms this week I'll be able to pour more time into this project, which is what I need. I definitely struggle with goal setting on individually driven projects, which is something I'll need to work on."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}