week_1_first_steps_with_python.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.3"
    },
    "colab": {
      "name": "week_1_first_steps_with_python.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/cderose/dh-courses/blob/master/week_1_first_steps_with_python.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I8i5qHjvnk8e"
      },
      "source": [
        "# Getting Started\n",
        "\n",
        "Before proceeding with the notebook below, please make sure you click \"File\" then click \"Save a copy in drive\". Doing so will make sure your work gets saved in your personal Google Drive account!\n",
        "\n",
        "To make sure you've successfully created a copy of this notebook, please click \"File\" in the upper navigation bar, then click \"Save\". Once you see a little notification that your work has been saved successfully, you're ready to proceed!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CGOstGNVsRvH"
      },
      "source": [
        "# Collecting Resources\n",
        "\n",
        "This notebook makes use of a few files that are not present in the current notebook environment. To load those resources, please hover your mouse between the [ ] braces in the cell below and click the play button that appears. Once you've done that, we're ready to start the notebook!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sq_V2nIBrdhX"
      },
      "source": [
        "!wget https://raw.githubusercontent.com/YaleDHLab/lab-workshops/master/first-steps-with-python/alice-in-wonderland.txt\n",
        "!wget https://raw.githubusercontent.com/YaleDHLab/lab-workshops/master/first-steps-with-python/alice-ocr.txt\n",
        "!wget https://raw.githubusercontent.com/YaleDHLab/lab-workshops/master/first-steps-with-python/helpers.py\n",
        "!wget https://raw.githubusercontent.com/YaleDHLab/lab-workshops/master/first-steps-with-python/jabberwocky.txt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_cnNJUCKnk8g"
      },
      "source": [
        "# Introduction to Python\n",
        "\n",
        "If you're interested in learning a programming language, Python is an excellent choice. It's an ideal language for beginners (and advanced users), as it's powerful, flexible, and equipped with packages that can help you accomplish nearly any task you might wish to solve using a computer. These are only some of the reasons Python recently became the most popular programming language in the world:\n",
        "\n",
        "<img src='https://github.com/YaleDHLab/lab-workshops/raw/master/first-steps-with-python/assets/python-popularity.jpg' style='margin: 10px auto; width: 400px'>\n",
        "\n",
        "This workshop will guide you through your first steps with Python. We will introduce some of the most important concepts in the language, and will learn how to use Python to accomplish real life research tasks. Let's dive in!\n",
        "\n",
        "# What Can You Do With Python?\n",
        "\n",
        "Before spending time studying Python, you might ask yourself: What can I do with Python? The truth is that nearly anything that can be done with a computer can be done with Python. For example, you might use it to:\n",
        "\n",
        "### Collect Data\n",
        "\n",
        "Perhaps you would like to study the influence the King James Bible had on *Moby Dick*, or you would like to study patterns in Napoleon's letter writing practices. All digital projects start with data collection, and Python can help you collect or \"scrape\" data from the internet (always check copyright status!). \n",
        "\n",
        "<img style='width: 400px' src='https://github.com/YaleDHLab/lab-workshops/raw/master/first-steps-with-python/assets/scraping-data.gif'>\n",
        "<div style='width: 600px; margin: 10px auto; color: #888; font-family: arial; font-size: 12px;'>Selenium and BeautifulSoup are two of the most popular tools for collecting data from the web, and both are written in Python.</div>\n",
        "\n",
        "### Analyze Data\n",
        "\n",
        "Python is an awesome language for data analysis. Maybe you want to study semantic shifts in periodical collections over the course of the twentieth century. Python can help you study such patterns at scale. \n",
        "\n",
        "<img style='width: 400px' src='https://github.com/YaleDHLab/lab-workshops/raw/master/first-steps-with-python/assets/rap-vocab.png'>\n",
        "<div style='width: 600px; margin: 10px auto; color: #888; font-family: arial; font-size: 12px;'>\"The Largest Vocabulary In Hip Hop\", an article on <a href='https://pudding.cool'>https://pudding.cool</a>, uses simple data analysis to identify the vocabulary size of various popular rappers, a task that one could achieve with Python.</div>\n",
        "\n",
        "### Machine Learning\n",
        "\n",
        "Python has become one of the most popular languages for conducting machine learning and artificial intelligence (AI) research in both textual and visual domains. \n",
        "\n",
        "<img style='width:400px' src='https://github.com/YaleDHLab/lab-workshops/raw/master/first-steps-with-python/assets/reinforcement-learning.gif'>\n",
        "<div style='width: 600px; margin: 10px auto; color: #888; font-family: arial; font-size: 12px;'>Reinforcement learning algorithms&mdash;the engines that allow algorithms to play games like Chess, Go, or Super Mario Brothers&mdash;are one of many areas in machine learning one can study using Python.</div>\n",
        "\n",
        "### Make Gadgets\n",
        "\n",
        "Gadget makers use the extrordinarily small and affordable Raspberry Pi microcomputer—which runs on Python—to build electronic devices like Bluetooth enabled speakers and interactive art displays.\n",
        "\n",
        "<img style='width:400px' src='https://github.com/YaleDHLab/lab-workshops/raw/master/first-steps-with-python/assets/treat-dispenser.png'>\n",
        "<div style='width: 600px; margin: 10px auto; color: #888; font-family: arial; font-size: 12px;'>Tech enthusiasts and cat lovers alike might wish to know that a <a href='https://www.adafruit.com/product/2885'>$5 Raspberry Pi computer</a> runs on Python and can be used to create homemade arcade games, security sytems, and cat treat dispensors such as the one pictured above.</div>\n",
        "\n",
        "### Create Websites\n",
        "\n",
        "The Python ecosystem is home to several popular frameworks for building websites, including Flask and Django. Major companies you know and love have use these frameworks in their websites. Flask is used by organizations including Reddit, Netflix, and AirBnB. Django is used by companies including YouTube, Instagram, and Spotify. \n",
        "\n",
        "<img style='width:400px' src='https://github.com/YaleDHLab/lab-workshops/raw/master/first-steps-with-python/assets/netflix.gif'>\n",
        "<div style='width: 600px; margin: 10px auto; color: #888; font-family: arial; font-size: 12px;'>Netflix.com is built on Flask, a popular, easy to use server framework written in Python.</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wp_7MGS4nk8h"
      },
      "source": [
        "# Getting Started\n",
        "\n",
        "### Variables\n",
        "\n",
        "Variables are the core building blocks of computer programming. A variable is a sequence of characters to which we assign a value or meaning using the = sign. Let's look at an example:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yVRcnK2Fnk8h"
      },
      "source": [
        "my_cat = 'cheshire'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m7gdPyI3nk8h"
      },
      "source": [
        "Here `my_cat` is the variable, and `'cheshire'` is the value assigned to that variable. In general in Python, the thing on the left hand side of the equals sign is the variable, and the thing on the right hand side is that variable's value.\n",
        "\n",
        "When creating a variable, keep in mind that **variable names**:\n",
        "- can only include letters, numbers, and underscores\n",
        "- must start with a letter or underscore (variables that start with underscores have a special meaning, so for now, we'll start our all of our variables with letters)\n",
        "- are case sensitive.\n",
        "\n",
        "Given the above rules, try to anticipate which cells below will return an error message:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CvzI-dS1nk8i"
      },
      "source": [
        "catepillar_question = 'Who are you?'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-yyOaJDOnk8i"
      },
      "source": [
        "_catepillar_question = 'Who are you?'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "syPCe8J6nk8i"
      },
      "source": [
        "catepillar_Question1 = 'Who are you?'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QZwuo4Gvnk8j"
      },
      "source": [
        "1catepillar_question = 'Who are you?'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TnSzf82jLWyn"
      },
      "source": [
        "<details>\n",
        "  <summary><h5>Solution</h5></summary>\n",
        "\n",
        "The final variable (`1catepillar_question`) is the problematic variable name, because it violates the second rule above (which states variables must start with a letter or an underscore).\n",
        "\n",
        "To verify this for yourself, try instantiating each variable above and you should see that only the last variable throws an error!\n",
        "</details>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oeIPjMyMnk8j"
      },
      "source": [
        "Variables can be reassigned in Python. We might assign a value to a variable, then later assign a different value to that variable:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nf0jIMa3nk8j"
      },
      "source": [
        "my_cat = 'cheshire'\n",
        "my_cat = 'grinning'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cg0wLSjOnk8j"
      },
      "source": [
        "Here `my_cat` is given an initial value `'cheshire'`, but is then given a new value `'grinning'`. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FO9yLgpank8j"
      },
      "source": [
        "### Printing\n",
        "\n",
        "To check the value assigned to a variable at any given moment, you can use the `print()` function:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R2vXPyyunk8k"
      },
      "source": [
        "my_cat = 'cheshire'\n",
        "print(my_cat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VtJDrCPGnk8k"
      },
      "source": [
        "The `print()` command is a type of function. We can recognize `print()` as a function because it has parentheses after the name. In general, **functions** are little pieces of code that take **arguments** (the text enclosed in parentheses) as input and do something with those arguments. The `print()` function, for example, takes as its argument a variable like `my_cat`, then displays the value assigned to that variable. We will explore functions in greater depth below. For now, we just want to note that `print()` is a function, which we can recognize by the presence of parentheses after the function's variable name."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tfxXZYqbnk8k"
      },
      "source": [
        "See if you can use the `print()` function to tell what value is assigned to the following variable:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oSTI6DEhnk8k"
      },
      "source": [
        "# see if you can display the value assigned to `my_motto` below\n",
        "my_motto = \"It's no use going back to yesterday\" + \", \" + \"because I was a different person then.\"\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L7iLHJbVNLFR"
      },
      "source": [
        "<details>\n",
        "  <summary><h5>Solution</h5></summary>\n",
        "  We can use the print function on the `my_motto` string as follows:\n",
        "\n",
        "  ```\n",
        "  print(my_motto)\n",
        "  ```\n",
        "\n",
        "  If you had trouble with the exercise above, try copying and pasting the code snippet in this solution block into a new code cell to see what the output looks like!\n",
        "\n",
        "</details>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i_VjnyzInk8k"
      },
      "source": [
        "### Introspection\n",
        "\n",
        "Just as we can use the `print()` function to see the value assigned to a variable, we can use the `dir()` function to see the kinds of things we can do with a variable. Brace yourself though, the `dir()` command displays lots of options!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mPmCwC3qnk8l"
      },
      "source": [
        "title = 'Alice in Wonderland'\n",
        "dir(title)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p-ihFtU-nk8l"
      },
      "source": [
        "Running the `dir()` function returns a list of methods that are defined on the provided variable. In the example above, we can see that `title` has 50 or so methods defined. **Methods** are similar to functions, except they're tied to the variable (or object) you're working with. You can call a method using dot notation. Let's look at a few examples:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HqEyf6xank8l"
      },
      "source": [
        "title.upper()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "__OgBciQnk8l"
      },
      "source": [
        "title.lower()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MJnjtmEPnk8l"
      },
      "source": [
        "We can see that the `upper()` method lets us uppercase our title, and the `lower()` method lets us lowercase our title. We will use more methods in just a moment. For now, we just want to remember that the `dir()` command tells us which methods can be called on variables."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ClWQDkz8nk8l"
      },
      "source": [
        "See if you can use the `dir()` command to identify the methods available on the variable below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BZYeLQLhnk8m"
      },
      "source": [
        "# use the `dir()` command below to identify the methods available on the variable below\n",
        "song = '''\n",
        "  How doth the little crocodile\n",
        "  Improve his shining tail,\n",
        "  And pour the waters of the Nile\n",
        "  On every golden scale!\n",
        "'''"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JptXAy0mOUQ6"
      },
      "source": [
        "<details>\n",
        "  <summary><h5>Solution</h5></summary>\n",
        "  We can run the `dir()` command on the `song` variable as follows:\n",
        "\n",
        "  ```\n",
        "  dir(song)\n",
        "  ```\n",
        "\n",
        "  The output of that command shows all of the methods one can call on `song`.\n",
        "\n",
        "</details>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ml73B4Vqnk8m"
      },
      "source": [
        "### Data Type: Strings \n",
        "\n",
        "All of the variables we have seen so far have been examples of \"strings\", another way of referring to text data. We know these variables are strings because their values are wrapped in quotation marks.\n",
        "\n",
        "Can you explain the error message we get if we try to assign text data as a value without using quotation marks?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "UKU5rJxonk8m"
      },
      "source": [
        "# the code below will result in an error\n",
        "main_character = Alice\n",
        "\n",
        "# fix the above code here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jgtL_49pOom-"
      },
      "source": [
        "<details>\n",
        "  <summary><h5>Solution</h5></summary>\n",
        "  The problem with the line above is that Python interprets Alice as a variable, as it doesn't have quotation marks around it. We can fix that by placing quotes around Alice:\n",
        "\n",
        "  ```\n",
        "  main_character = \"Alice\"\n",
        "  ```\n",
        "\n",
        "  or \n",
        "\n",
        "  ```\n",
        "  main_character = 'Alice'\n",
        "  ```\n",
        "\n",
        "(The type of quotation mark we use does not matter--both are treated identically.)\n",
        "\n",
        "</details>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0_TOVvlOnk8m"
      },
      "source": [
        "With string data, we often need to process it to find specific bits of interest, or to clean extraneous characters. Let's practice processing strings below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qv9zacuQnk8m"
      },
      "source": [
        "opening = '''fAlice was beginning to get very tired of sitting by her sister on the bank,\n",
        "and of having nothing to do. Once or twice she had peeped into the book her\n",
        "sister was reading, but it had no pictures or conversations in it, ‘and what is\n",
        "the use of a book,’ thought fAlice ‘without pictures or conversation?’'''"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZDznc5Vynk8n"
      },
      "source": [
        "The data in the string above looks good, except \"Alice\" is prefaced with an \"f\" for some reason. Let's clean that up with the `replace()` method, which replaces all instances of the first argument with the second argument:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WlYeQLZYnk8n"
      },
      "source": [
        "opening = opening.replace('fAlice', 'Alice')\n",
        "print(opening)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GYStOZEbnk8n"
      },
      "source": [
        "### Data Type: Lists\n",
        "\n",
        "A **list** is a variable that contains one or more items. Just like strings are encased in quotation marks, lists are represented by square backets, with commas separating the different items in the list. Here is an example list:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1udTm9V1nk8n"
      },
      "source": [
        "facts = ['four times five is twelve', 'four times six is thirteen']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pU1-DCncnk8n"
      },
      "source": [
        "Just like strings, lists have their own set of methods we can use. Let's practice the `append()` method, which adds an item to a list:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xOfTtDaZnk8o"
      },
      "source": [
        "facts.append('and four times seven is–oh dear!')\n",
        "print(facts)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Eo7twwE2nk8o"
      },
      "source": [
        "See if you can add a new fact to our list of \"facts\" below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "05Mn6Mpwnk8o"
      },
      "source": [
        "# add a new fact to our list of \"facts\" here\n",
        "\n",
        "# print the new list\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dJxT8hfnO_0N"
      },
      "source": [
        "<details>\n",
        "  <summary><h5>Solution</h5></summary>\n",
        "  We can add to the `facts` list as follows:\n",
        "\n",
        "  ```\n",
        "  facts.append('Here is my new fact!')\n",
        "  ```\n",
        "\n",
        "  If you then run `print(facts)` you'll see the new fact has been added to the list!\n",
        "\n",
        "</details>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PSABRVQink8o"
      },
      "source": [
        "To access an item in the list, we use that item's index value. Keep in mind, Python starts counting at 0! So, if we want to print 'four times five is twelve' (which is the first item in our `facts` list), we would type:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-0OH4G4Nnk8p"
      },
      "source": [
        "print(facts[0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bfMYuW3Qnk8p"
      },
      "source": [
        "Try to access 'and four times seven is–oh dear!' from `facts`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S-DuvFOqnk8p"
      },
      "source": [
        "# print 'and four times seven is–oh dear!' from the facts list\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lY2_8WLqPKrD"
      },
      "source": [
        "<details>\n",
        "  <summary><h5>Solution</h5></summary>\n",
        "  We can access this string as follows:\n",
        "\n",
        "  ```\n",
        "  facts[2]\n",
        "  ```\n",
        "\n",
        "  If you wanted to store that isolated fact in a new variable, you could assign the statement above to a variable of your chosing:\n",
        "\n",
        "  ```\n",
        "  my_chosen_fact = facts[2]\n",
        "  ```\n",
        "\n",
        "</details>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2RA_XuEOnk8p"
      },
      "source": [
        "Now that we know a bit about lists, let's go back to the opening paragraph of *Alice in Wonderland*, which we've saved in the variable `opening`. Suppose we only want to examine the first sentence in the paragraph. To accomplish this goal, we could use the split() function, which cuts a string into a list of strings based on a user-provided argument."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w0nFjQ1snk8p"
      },
      "source": [
        "# the code below will turn the opening paragraph-currently stored as string data-into a list\n",
        "opening_sentences = opening.split('.')\n",
        "print(opening_sentences)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Astlu0PAnk8p"
      },
      "source": [
        "# using what you know about lists and index numbers, print the first sentence to Alice in Wonderland here\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c8x_JK7_Pw6h"
      },
      "source": [
        "<details>\n",
        "  <summary><h5>Solution</h5></summary>\n",
        "  We can print the first sentence in the `opening_sentences` list as follows:\n",
        "\n",
        "  ```\n",
        "  print(opening_sentences[0])\n",
        "  ```\n",
        "\n",
        "  Remember that Python starts counting from 0!\n",
        "\n",
        "</details>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WBqq_WpNnk8p"
      },
      "source": [
        "### Reading Files\n",
        "\n",
        "Many Python programs need to read files to load data you have collected into memory. Let's practice reading a file:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vfilZnCank8q"
      },
      "source": [
        "my_file = open('alice-in-wonderland.txt').read()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8TIXk_hbnk8q"
      },
      "source": [
        "You'll notice that the line above starts with `open()`. `open()` is an example of a **built-in** function, which means it comes predefined in all Python files. There are a small handful of these built-in functions, including the `print()` and `dir()` functions we saw above. For now we should note that `open()` returns a \"file handler\" that contains a `read()` method. Calling that `read()` method lets us read the text content of a file into memory.\n",
        "\n",
        "Let's practice reading files into memory by using a new file that you select. For this task, download a text file from [Project Gutenberg](https://www.gutenberg.org/) to your personal machine. \n",
        "\n",
        "Next, click the folder icon in the left-hand navigation toolbar, then click the icon that looks like a document with an upward arrow and select your file:\n",
        "\n",
        "<img src='https://github.com/YaleDHLab/lab-workshops/raw/master/first-steps-with-python/assets/colab-file-upload.gif'>\n",
        "\n",
        "After your Project Gutenberg text file is uploaded to this Colab environment, use the technique we just practiced to read your file into memory below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R9sy9fABnk8q"
      },
      "source": [
        "# try to load a file into memory here\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-NQ_kKcMP-Fz"
      },
      "source": [
        "<details>\n",
        "  <summary><h5>Solution</h5></summary>\n",
        "  We can practice loading a new file into memory using the file entitled \"jabberwocky.txt\". (You should have loaded into this notebook when you ran the cell within the block labelled \"Collecting Resources\" above.) Let's read that file with the following code:\n",
        "\n",
        "\n",
        "  ```\n",
        "  my_file = open('jabberwocky.txt').read()\n",
        "  print(my_file)\n",
        "  ```\n",
        "\n",
        "</details>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xHnEMquQnk8q"
      },
      "source": [
        "<h3 style='color: green'>Review #1</h3>\n",
        "\n",
        "By this point, we have covered many of the most important first steps with Python. But the hardest part of learning a programming language is learning how to piece these commands together to accomplish goals. Let's start practicing this skill with the following exercise.\n",
        "\n",
        "See if you can read the file \"alice-ocr.txt\" into memory, then clean up the misspelled words in that file. You may need to investigate the file contents to determine which words are misspelled."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vtf5Wq53nk8q"
      },
      "source": [
        "# type your code here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x50Xso_cQeaw"
      },
      "source": [
        "<details>\n",
        "  <summary><h5>Solution</h5></summary>\n",
        "  We can read \"alice-ocr.txt\" into memory and clean up the misspelled words as follows:\n",
        "\n",
        "\n",
        "  ```\n",
        "  # read the file into memory\n",
        "  my_file = open(\"alice-ocr.txt\").read()\n",
        "\n",
        "  # remove the first misspelled word\n",
        "  my_file = my_file.replace('Cat333rpillar', 'Caterpillar')\n",
        "\n",
        "  # remove the second misspelled word\n",
        "  my_file = my_file.replace('Allll!ce', 'Alice')\n",
        "\n",
        "  # print the cleaned file\n",
        "  print(my_file)\n",
        "  ```\n",
        "\n",
        "</details>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j2ydAeUGnk8q"
      },
      "source": [
        "<img style='width: 100px' src='https://github.com/YaleDHLab/lab-workshops/raw/master/first-steps-with-python/assets/rabbit.jpg'>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PMuVrvzYnk8q"
      },
      "source": [
        "### Loops\n",
        "\n",
        "In our work above, we learned that lists are defined by square brackets, with commas separating each item in the list. We've learned that we can access individual list items using their index position, but oftentimes, it can be useful to automate that process of returning list items one by one(imagine if our list had 10,000 items!). We can accomplish this goal with a `for loop`. \n",
        "\n",
        "To see for loops in practice, let's first create a new list and save it in the variable `lines`. Then, we'll run a for loop to iterate through each line in our list."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uu_McBz8nk8r"
      },
      "source": [
        "lines = [\n",
        "  'How cheerfully he seems to grin',\n",
        "  'How neatly spread his claws',\n",
        "  'And welcome little fishes in',\n",
        "  'With gently smiling jaws!'\n",
        "]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YMhAZI3Unk8r"
      },
      "source": [
        "for line in lines:\n",
        "  print(line)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3l_GqDAznk8r"
      },
      "source": [
        "A **for loop** examines each item in a list from left to right (or top to bottom if you like). For loops are structured as follows:\n",
        "  \n",
        "```\n",
        "for variable_name in list_name:\n",
        "  actions_to_run_on_variable_name\n",
        "```\n",
        "\n",
        "Within the indented part of the loop (the part of the loop prefaced by whitespace), the value of `variable_name` will be equal to the current item in the list. Let's make this more concrete by discussing the loop we ran above."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l6i2gAnZnk8r"
      },
      "source": [
        "In the first pass through the loop, the variable `line` has the value `'How cheerfully he seems to grin'`.   \n",
        "In the second pass through the loop, the variable `line` has the value `'How neatly spread his claws'`.  \n",
        "And so on. Each time we move through the loop, `line` takes on the value of the next item in the list.\n",
        "\n",
        "As we move through the loop, we can perform operations on each item in the list. Let's practice with another list:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0Ckq3SkCnk8r"
      },
      "source": [
        "lines = [\n",
        "  'Beautiful Soup, so rich and green',\n",
        "  'Waiting in a hot tureen!',\n",
        "  'Who for such dainties would not stoop?',\n",
        "  'Soup of the evening, beautiful Soup',\n",
        "]\n",
        "\n",
        "for line in lines:\n",
        "  print(line.upper())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JrJKbnOink8r"
      },
      "source": [
        "### Modules\n",
        "\n",
        "One of the great things about the Python programming language is the fact that the Python ecosystem has many, many \"modules\" that will help expedite your work. A **module** is a collection of code that contains some variables or functions you can use. Some of these modules were authored by the creators and maintaners of the language itself, while others were written by other kind souls who have open sourced their code. Let's take a look at the `collections` module, a built-in module that is useful in lots of data processing tasks."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AgWE0R6tnk8r"
      },
      "source": [
        "import collections\n",
        "\n",
        "word_counter = collections.Counter()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YLnZFrpGnk8r"
      },
      "source": [
        "The lines above show the general pattern we follow to use modules in Python. First we must `import` the module we wish to use. That `import` line makes it possible for us to use the variables and functions defined in the module in our own code.\n",
        "\n",
        "After importing the module, we can access the variables and functions defined within that module. For a full list of the variables and functions defined within the `collections` module, one can use the `dir` method we discussed earlier: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sp-abi3ank8s"
      },
      "source": [
        "dir(collections)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kzXnwcsbnk8s"
      },
      "source": [
        "Now that we've imported the `collections` module and created a `Counter`, let's use that `Counter` to count some words:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "znkPlxOQnk8s"
      },
      "source": [
        "poem = '''\n",
        "  ‘The Queen of Hearts, she made some tarts,\n",
        "  All on a summer day:\n",
        "  The Knave of Hearts, he stole those tarts,\n",
        "  And took them quite away!’\n",
        "'''\n",
        "\n",
        "# split the poem into a list of words\n",
        "words = poem.split()\n",
        "\n",
        "for word in words:\n",
        "    word_counter[word] += 1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYbYkPJznk8s"
      },
      "source": [
        "The code above loops over each word in the `words` list, keeping track of how many times each word appears. The first time a word appears it is assigned a value of 1; this value is updated by 1 for each time the word reappears. The resulting counts look like this:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f1j0-PXqnk8s"
      },
      "source": [
        "print(word_counter)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O7SVh4ZCnk8s"
      },
      "source": [
        "That's a little hard to read, but by using just one more module we can visualize our word counts:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u7eouPzRnk8s"
      },
      "source": [
        "%matplotlib inline\n",
        "\n",
        "import helpers\n",
        "\n",
        "helpers.plot_counts(word_counter)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ewtTkPink8s"
      },
      "source": [
        "<h3 style='color: green'>Review #2</h3>\n",
        "\n",
        "You've now made it quite far in your Python journey! Let's review what we discussed above with the following challenge.\n",
        "\n",
        "See if you can read in `jabberwocky.txt`, count the number of times each word occurs in that file, and plot those counts. As you work to complete this challenge, you may find it helpful to subdivide the task into smaller subtasks and complete those subtasks one by one..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-vXmLXvMnk8t"
      },
      "source": [
        "# type your code here\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S9tgoSv8RQZU"
      },
      "source": [
        "<details>\n",
        "  <summary><h5>Solution</h5></summary>\n",
        "  We can read \"jabberwocky.txt\" into memory and plot the word counts as follows:\n",
        "\n",
        "\n",
        "  ```\n",
        "  # import the modules we will use\n",
        "  import helpers\n",
        "  import collections\n",
        "  \n",
        "  # read the file into memory\n",
        "  my_file = open(\"jabberwocky.txt\").read()\n",
        "\n",
        "  # create a word counter\n",
        "  word_counts = collections.Counter()\n",
        "\n",
        "  # split my_file into a list of words\n",
        "  words = my_file.split()\n",
        "\n",
        "  # loop over each word\n",
        "  for word in words:\n",
        "\n",
        "    # increment the count of the current word\n",
        "    word_counts[word] += 1\n",
        "\n",
        "  # plot the word counts\n",
        "  helpers.plot_counts(word_counts)\n",
        "  ```\n",
        "\n",
        "  As you can see, our code is getting more complicated! That's a good thing--it means we're making progress!\n",
        "\n",
        "</details>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1OF0iZC7nk8t"
      },
      "source": [
        "# Going Further\n",
        "\n",
        "Congratulations! You've covered a lot of ground with the Python programming language. From here, you should be prepared to take a deeper dive into the language. \n",
        "\n",
        "For next steps, we recommend browsing the lessons on [Library Carpentry](https://librarycarpentry.org/lc-python-intro/) and [the Programming Historian](https://programminghistorian.org/):\n",
        "\n",
        "<img src='https://github.com/YaleDHLab/lab-workshops/raw/master/first-steps-with-python/assets/programming-historian.png' style='width: 550px; margin: 10px auto'>\n",
        "\n",
        "In addition to those lessons, feel free to come to our office hours to discuss a project you might want to pursue with Python!"
      ]
    }
  ]
}