Write_Efficient_Python_Code_(Optimizing_Your Code).ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/youssefHosni/Efficient-Python-for-Data-Scientists/blob/main/Write_Efficient_Python_Code_(Optimizing_Your%C2%A0Code).ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pOhOqpLShVyF"
      },
      "source": [
        "# Write Efficient Python Code for Data Scientists: Optimizing Your Code"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VPk2KvmShVyK"
      },
      "source": [
        "As a data scientist, you should spend most of your time working on gaining insights from data not waiting for your code to finish running. Writing efficient Python code can help reduce runtime and save computational resources, ultimately freeing you up to do the things that have more impact."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bw8PZFXThVyM"
      },
      "source": [
        "### Table of content:\n",
        "\n",
        "**1. Making Your Code Efficient**\n",
        "\n",
        "    1.1. Efficiently Combining, Counting, and Iterating\n",
        "\n",
        "    1.2. Introduction to The Set Theory \n",
        "\n",
        "    1.3. Eliminating Loops\n",
        "\n",
        "    1.4. Writing Better Loops\n",
        "    \n",
        "\n",
        "**2. Introduction to Pandas Optimization**\n",
        "\n",
        "    2.1. Pandas DataFrame Iteration\n",
        "    \n",
        "    2.2. Pandas Alternative to Looping\n",
        "\n",
        "    2.3. Optimal Pandas Iterating"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OnI3SrY5hVyN"
      },
      "source": [
        "# 1. Make Your Code Efficient"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qe_MWnrjhVyP"
      },
      "source": [
        "In this section, we will cover more efficiency tips and tricks. You will learn a few useful built-in modules for writing efficient code and practice using set theory. You'll then learn about looping patterns in Python and how to make them  more efficient."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xhYJSoSihVyQ"
      },
      "source": [
        "## 1.1. Efficiently Combining, Counting, and Iterating"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AXfUeT4xhVyR"
      },
      "source": [
        "### Combining Objects"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jLEuksQDhVyT"
      },
      "source": [
        "In this subsection, we'll cover combining, counting, and iterating over objects efficiently in python. Suppose we have two lists: one of the names and the other is the age for each of them. We want to combine these lists so that each name is stored next to its age. We can iterate over the names list using enumerate and grab each name's corresponding age using the index variable."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J6BFfSwShVyU"
      },
      "outputs": [],
      "source": [
        "# combining objects \n",
        "names = ['Ahmed', 'Youssef', 'Mohammed']\n",
        "age = [25, 27, 40]\n",
        "combined = []\n",
        "\n",
        "for i,name in enumerate(names):\n",
        "    combined.append((name, age[i]))\n",
        "print(combined)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nhoNq_UphVyW"
      },
      "source": [
        "But Python's built-in function zip provides a more elegant solution. The name \"zip\" describes how this function combines objects like a zipper on a jacket (making two separate things become one). zip returns a zip object that must be unpacked into a list and printed to see the contents. Each item is a tuple of elements from the original lists."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9e7924_jhVyX"
      },
      "outputs": [],
      "source": [
        "# Combining objects with zip\n",
        "combined_zip = zip(names, age)\n",
        "print(type(combined_zip))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FrCfY3oAhVyY"
      },
      "outputs": [],
      "source": [
        "combined_zip_list = [*combined_zip]\n",
        "print(combined_zip_list)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "musB_3S2hVya"
      },
      "source": [
        "### Counting Objects"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jJnwY5L7hVyb"
      },
      "source": [
        "Let's dig deeper into the Counter object. First, we will upload the pokemon dataset and print the first five rows of it then we will count the number of pokemon per each type using for loop and then using the Counter function."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T6eK3w7PhVyc"
      },
      "source": [
        "Let's load the pokemon dataset and print the first five rows:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "F-6WO6i7hVyd"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "pokemon = pd.read_csv('pokemon.csv')\n",
        "pokemon.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YXZfxROkhVyd"
      },
      "source": [
        "Now we will count the number of pokemon per each type using loops and calculate the execution time:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g78s1mCOhVye"
      },
      "outputs": [],
      "source": [
        "%%timeit -r1 -n1\n",
        "# Counting with for loop\n",
        "poke_types = pokemon['Type 1']\n",
        "type_counts = {}\n",
        "for poke_type in poke_types:\n",
        "    if poke_type not in type_counts:\n",
        "        type_counts[poke_type] = 1\n",
        "    else:\n",
        "        type_counts[poke_type] += 1\n",
        "print(type_counts)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ziOwLrMWhVyf"
      },
      "source": [
        "Finally, we will count the number of pokemon per each type using the Counter function and compare the time:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BLYKz8vVhVyf"
      },
      "outputs": [],
      "source": [
        "%%timeit -r1 -n1\n",
        "\n",
        "# counting with collections.Counter()\n",
        "from collections import Counter\n",
        "poke_types = pokemon['Type 1']\n",
        "\n",
        "type_counts = Counter(poke_types)\n",
        "print(type_counts)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9AtLzN89hVyg"
      },
      "source": [
        "We can see that using the Counter function from the collections module is a more efficient approach. Just import Counter and provide the object to be counted. No need for a loop! Counter returns a Counter dictionary of key-value pairs. When printed, it's ordered by highest to lowest counts. If comparing runtime times, we'd see that using Counter takes less time compared to the standard dictionary approach!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fkqm0FG4hVyg"
      },
      "source": [
        "## itertools"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d6RFCsw_hVyh"
      },
      "source": [
        "Another built-in module, itertools, contains functional tools for working with iterators. A subset of these tools is:\n",
        "* Infinite iterators: count, cycle, repeat\n",
        "* Finite iterators: accumulate, chain, zip_longest, etc.\n",
        "* Combination generators: product, permutations, combinations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5TPMbr2EhVyi"
      },
      "source": [
        "We will focus on combinatoric generators. These generators efficiently yield Cartesian products, permutations, and combinations of objects. Let's explore an example."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qACgSHJ-hVyj"
      },
      "source": [
        "### Combinations "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gad4RstqhVyk"
      },
      "source": [
        "Suppose we want to gather all combination pairs of Pokémon types possible. We can do this with a nested for loop that iterates over the poke_types list twice. Notice that a conditional statement is used to skip pairs having the same type twice. For example, if x is 'Bug' and y is 'Bug', we want to skip this pair. Since we're interested in combinations (where order doesn't matter), another statement is used to ensure either order of the pair doesn't already exist within the combos list before appending it. For example, the pair ('Bug', 'Fire') is the same as the pair ('Fire', 'Bug'). We want one of these pairs, not both."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gf9Td1vyhVyl"
      },
      "outputs": [],
      "source": [
        "#Combinations with loop\n",
        "\n",
        "poke_types = ['Bug', 'Fire', 'Ghost', 'Grass', 'Water']\n",
        "combos = []\n",
        "for x in poke_types:\n",
        "    for y in poke_types:\n",
        "        if x == y:\n",
        "            continue\n",
        "        if ((x,y) not in combos) & ((y,x) not in combos):\n",
        "            combos.append((x,y))\n",
        "print(combos)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EhsBizEuhVyl"
      },
      "source": [
        "The combinations generator from itertools provides a more efficient solution. First, we import combinations and then create a combinations object by providing the poke_types list and the length of combinations we desire. Combinations return a combinations object, which we unpack into a list and print to see the result. If comparing runtimes, we'd see using combinations is significantly faster than the nested loop."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OUSxvtEohVym"
      },
      "outputs": [],
      "source": [
        "# combinations with itertools\n",
        "poke_types = ['Bug', 'Fire', 'Ghost', 'Grass', 'Water']\n",
        "from itertools import combinations\n",
        "combos_obj = combinations(poke_types, 2)\n",
        "print(type(combos_obj))\n",
        "\n",
        "combos = [*combos_obj]\n",
        "print(combos)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BL7OrnKbhVym"
      },
      "source": [
        "## 1.2. Introduction to The Set Theory"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ghGyFk0chVyn"
      },
      "source": [
        "Often, we'd like to compare two objects to observe similarities and differences between their contents. When doing this type of comparison, it's best to leverage a branch of mathematics called the set theory. As you know, Python comes with a built-in set data type. Sets come with some handy methods we can use for comparing such as:\n",
        "* intersection() : all elements that are in both sets\n",
        "* difference(): all elements in one set but not the other\n",
        "* symmetric_difference(): all elements in exactly one set\n",
        "* union(): all elements that are in either set\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FLJjgWihhVyn"
      },
      "source": [
        "When we'd like to compare objects multiple times and in different ways, we should consider storing our data in sets to use these efficient methods. Another nice feature of Python sets is their ability to quickly check if a value exists within its members. We call this membership testing using the in operator. We will go through how using  the in operator with a set is much faster than using it with a list or tuple."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y19Te0XKhVyo"
      },
      "source": [
        "Suppose we had two lists of Pokémon names: list_a and list_b and we would like to compare these lists to see which Pokémon appear in both lists. We could first use a nested for loop to compare each item in list_a to each item in list_b and collect only those items that appear in both lists. But, iterating over each item in both lists is extremely inefficient."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GIPw7FGxhVyo"
      },
      "outputs": [],
      "source": [
        "# Comparing objects with loops\n",
        "list_a = ['Bulbasaur', 'Charmander', 'Squirtle']\n",
        "list_b = ['Caterpie', 'Pidgey', 'Squirtle']\n",
        "\n",
        "in_common = []\n",
        "for pokemon_a in list_a:\n",
        "    for pokemon_b in list_b:\n",
        "        if pokemon_a == pokemon_b:\n",
        "            in_common.append(pokemon_a)\n",
        "print(in_common)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iLKgjYxIhVyq"
      },
      "source": [
        "However, a better way is to use Python's set data type to compare these lists. By converting each list into a set, we can use the dot-intersection method to collect the Pokémon shared between the two sets. One simple line of code and no need for a loop!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4qpxQXcnhVys"
      },
      "outputs": [],
      "source": [
        "# compaing objects using intersection()\n",
        "\n",
        "list_a = ['Bulbasaur', 'Charmander', 'Squirtle']\n",
        "list_b = ['Caterpie', 'Pidgey', 'Squirtle']\n",
        "set_a = set(list_a)\n",
        "set_b = set(list_b)\n",
        "set_a.intersection(set_b)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QTwEc6h1hVyt"
      },
      "source": [
        "We got the same answer with a much less number of code lines we can also compare the runtime to see how much using sets is much faster than using loops."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9f_ihjp0hVyt"
      },
      "source": [
        "Compare the runtime between using sets and using for loops"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bWzqeA8chVyu"
      },
      "outputs": [],
      "source": [
        "%%timeit\n",
        "in_common = []\n",
        "for pokemon_a in list_a:\n",
        "    for pokemon_b in list_b:\n",
        "        if pokemon_a == pokemon_b:\n",
        "            in_common.append(pokemon_a)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2d4kf44ZhVyv"
      },
      "outputs": [],
      "source": [
        "%timeit in_common = set_a.intersection(set_b)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O6oodU4ZhVyw"
      },
      "source": [
        "We can see that using sets is much faster than using for loops. We can also use a set method to see Pokémon that exist in one set but not in another. To gather Pokémon that exist in set_a but not in set_b, use set_a.difference(set_b)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_GleD8PIhVyx"
      },
      "outputs": [],
      "source": [
        "set_a = {'Bulbasaur', 'Charmander', 'Squirtle'}\n",
        "set_b = {'Caterpie', 'Pidgey', 'Squirtle'}\n",
        "set_a.difference(set_b)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vy_3BleghVyx"
      },
      "source": [
        "If we want the Pokémon in set_b, but not in set_a, we use set_b.difference(set_a)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0pEd-pXqhVyy"
      },
      "outputs": [],
      "source": [
        "set_a = {'Bulbasaur', 'Charmander', 'Squirtle'}\n",
        "set_b = {'Caterpie', 'Pidgey', 'Squirtle'}\n",
        "set_b.difference(set_a)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "402q1-m1hVyz"
      },
      "source": [
        "To collect Pokémon that exist in exactly one of the sets (but not both), we can use a method called the symmetric difference."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iqHqvH96hVy0"
      },
      "outputs": [],
      "source": [
        "set_a = {'Bulbasaur', 'Charmander', 'Squirtle'}\n",
        "set_b = {'Caterpie', 'Pidgey', 'Squirtle'}\n",
        "set_a.symmetric_difference(set_b)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9deXZpKwhVy1"
      },
      "source": [
        "Finally, we can combine these sets using the .union method. This collects all of the unique Pokémon that appear in either or both sets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VUGnW6W9hVy1"
      },
      "outputs": [],
      "source": [
        "set_a = {'Bulbasaur', 'Charmander', 'Squirtle'}\n",
        "set_b = {'Caterpie', 'Pidgey', 'Squirtle'}\n",
        "set_a.union(set_b)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ycW_W3zdhVy1"
      },
      "source": [
        "Another nice efficiency gain when using sets is the ability to quickly check if a specific item is a member of a set's elements. Consider our collection of 720 Pokémon names stored as a list, tuple, and set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x_2iyMVqhVy2"
      },
      "outputs": [],
      "source": [
        "names_list = list(pokemon['Name'])\n",
        "names_set = set(pokemon['Name'])\n",
        "names_tuple = tuple(pokemon['Name'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NaLy0_KVhVy2"
      },
      "source": [
        "We want to check whether or not the character, Zubat, is in each of these data structures and print the execution time for each data type:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mHfu2i82hVy3"
      },
      "outputs": [],
      "source": [
        "%timeit 'Zubat' in names_list\n",
        "%timeit 'Zubat' in names_tuple\n",
        "%timeit 'Zubat' in names_set"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NXW4RnvWhVy3"
      },
      "source": [
        "**When comparing runtimes, it's clear that membership testing with a set is significantly faster than a list or a tuple.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ELt8DMuzhVy3"
      },
      "source": [
        "One final efficiency gain when using sets comes from the definition of the set itself. A set is defined as a collection of distinct elements. Thus, we can use a set to collect unique items from an existing object. Let's define a primary_types list, which contains the primary types of each Pokémon. If we wanted to collect the unique Pokémon types within this list, we could write a for loop to iterate over the list, and only append the Pokémon types that haven't already been added to the unique_types list."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qUjTeNNahVy4"
      },
      "outputs": [],
      "source": [
        "primary_types = list(pokemon['Type 1'])\n",
        "unique_types = []\n",
        "for prim_type in primary_types:\n",
        "    if prim_type not in unique_types:\n",
        "        unique_types.append(prim_type)\n",
        "print(unique_types)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_3WAN-jvhVy4"
      },
      "source": [
        "Using a set makes this much easier. All we have to do is convert the primary_types list into a set, and we have our solution: a set of distinct Pokémon types."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "cMIRe4lshVy5"
      },
      "outputs": [],
      "source": [
        "unique_types_set = set(primary_types)\n",
        "print(unique_types_set)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9FctGt4qhVy5"
      },
      "source": [
        "### 1.3. Eliminating Loops"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gKVhKT9mhVy6"
      },
      "source": [
        "Although using loops when writing Python code isn't necessarily a bad design pattern, using extraneous loops can be inefficient and costly. Let's explore some tools that can help us eliminate the need to use loops in our code. Python comes with a few looping patterns that can be used when we want to iterate over an object's contents:\n",
        "\n",
        "* For loops iterate over elements of a sequence piece-by-piece. \n",
        "* While loops execute a loop repeatedly as long as some Boolean condition is met. \n",
        "* Nested loops use multiple loops inside one another."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B_oeuWZihVy7"
      },
      "source": [
        "Although all of these looping patterns are supported by Python, we should be careful when using them. Because most loops are evaluated in a piece-by-piece manner, they are often inefficient solutions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kpcki_uIhVy8"
      },
      "source": [
        "We should try to avoid looping as much as possible when writing efficient code. Eliminating loops usually results in fewer lines of code that are easier to interpret. One of the idioms of pythonic code is that  \"flat is better than nested.\" Striving to eliminate loops in our code will help us follow this idiom."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vr6704D1hVy9"
      },
      "source": [
        "Suppose we have a list of lists, called poke_stats, that contains statistical values for each Pokémon. Each row corresponds to a Pokémon, and each column corresponds to a Pokémon's specific statistical value. Here, the columns represent a Pokémon's Health Points, Attack, Defense, and Speed. We want to do a simple sum of each of these rows in order to collect the total stats for each Pokémon. If we were to use a loop to calculate row sums, we would have to iterate over each row and append the row's sum to the totals list. We can accomplish the same task, in fewer lines of code, with a list comprehension. Or, we could use the built-in map function that we've discussed previously in the previous article."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HWOjePB2hVy9"
      },
      "outputs": [],
      "source": [
        "# List of HP, Attack, Defense, Speed\n",
        "poke_stats = [\n",
        "[90, 92, 75, 60],\n",
        "[25, 20, 15, 90],\n",
        "[65, 130, 60, 75],\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "910Rx6nmhVy-"
      },
      "outputs": [],
      "source": [
        "%%timeit\n",
        "totals = []\n",
        "for row in poke_stats:\n",
        "    totals.append(sum(row))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nAABhIPKhVy_"
      },
      "outputs": [],
      "source": [
        "%timeit totals_comp = [sum(row) for row in poke_stats]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qdy8wv-ahVzA"
      },
      "outputs": [],
      "source": [
        "%timeit totals_map = [*map(sum, poke_stats)]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kwXf9rDQhVzA"
      },
      "source": [
        "Each of these approaches will return the same list, but using a list comprehension or the map function takes one line of code, and has a faster runtime."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ZfgGFR-hVzB"
      },
      "source": [
        "We've also covered a few built-in modules that can help us eliminate loops in the previous article. Instead of using the nested for loop, we can use combinations from the itertools module for a cleaner, more efficient solution."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JSBbhvx3hVzB"
      },
      "outputs": [],
      "source": [
        "poke_types = ['Bug', 'Fire', 'Ghost', 'Grass', 'Water']\n",
        "\n",
        "# Nested for loop approach\n",
        "combos = []\n",
        "for x in poke_types:\n",
        "    for y in poke_types:\n",
        "        if x == y:\n",
        "            continue\n",
        "    if ((x,y) not in combos) & ((y,x) not in combos):\n",
        "        combos.append((x,y))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aVQFtMqfhVzC"
      },
      "outputs": [],
      "source": [
        "# Built-in module approach\n",
        "from itertools import combinations\n",
        "combos2 = [*combinations(poke_types, 2)]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UzT3R14YhVzC"
      },
      "source": [
        "Another powerful technique for eliminating loops is to use the NumPy package. Suppose we had the same collection of statistics we used in a previous example but stored in a NumPy array instead of a list of lists.\n",
        "\n",
        "We'd like to collect the average stat value for each Pokémon (or row) in our array. We could use a loop to iterate over the array and collect the row averages."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BThGg0Q1hVzD"
      },
      "source": [
        "### Eliminate loops with NumPy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jqp8YTswhVzE"
      },
      "source": [
        "But, NumPy arrays allow us to perform calculations on entire arrays all at once. Here, we use the dot-mean method and specify an axis equal to 1 to calculate the mean for each row (meaning we calculate an average across the column values). This eliminates the need for a loop and is much more efficient."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vscBiP1FhVzE"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "poke_stats = np.array([\n",
        "[90, 92, 75, 60],\n",
        "[25, 20, 15, 90],\n",
        "[65, 130, 60, 75],])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "qstbdlb_hVzF"
      },
      "outputs": [],
      "source": [
        "avgs = []\n",
        "for row in poke_stats:\n",
        "    avg = np.mean(row)\n",
        "    avgs.append(avg)\n",
        "print(avgs)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_1WiPdAVhVzF"
      },
      "outputs": [],
      "source": [
        "avgs_np = poke_stats.mean(axis=1)\n",
        "print(avgs_np)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WaipC3gOhVzG"
      },
      "outputs": [],
      "source": [
        "%%timeit\n",
        "avgs = []\n",
        "for row in poke_stats:\n",
        "    avg = np.mean(row)\n",
        "    avgs.append(avg)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "ckHr0ZXahVzG"
      },
      "outputs": [],
      "source": [
        "%timeit avgs = poke_stats.mean(axis=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yKnEQ4PjhVzG"
      },
      "source": [
        "When comparing runtimes, we see that using the dot-mean method on the entire array and specifying an axis is significantly faster than using a loop."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hi5A8byOhVzH"
      },
      "source": [
        "### 1.4. Writing Better Loops"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KyxkNOXBhVzH"
      },
      "source": [
        "We've discussed how loops can be costly and inefficient. But, sometimes you can't eliminate a loop. In this section, we'll explore how to make loops more efficient when looping is unavoidable. Before diving in, some of the loops we'll discuss can be eliminated using techniques covered in previous lessons. For demonstrative purposes, we'll assume the use cases shown here are instances where a loop is unavoidable."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "60R1wzPUhVzI"
      },
      "source": [
        "The best way to make a loop more efficient is to analyze what's being done within the loop. We want to make sure that we aren't doing unnecessary work in each iteration. If a calculation is performed for each iteration of a loop, but its value doesn't change with each iteration, it's best to move this calculation outside (or above) the loop. If a loop is converting data types with each iteration, it's possible that this conversion can be done outside (or below) the loop using a map function. Anything that can be done once should be moved outside of a loop. Let's explore a few examples."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3kKGm9DDhVzI"
      },
      "source": [
        "### Moving calculations above a loop"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v4ftMcPThVzI"
      },
      "source": [
        "We have a list of Pokémon names and an array of each Pokémon's corresponding attack value. We'd like to print the names of each Pokémon with an attack value greater than the average of all attack values. To do this, we'll use a loop that iterates over each Pokémon and its attack value. For each iteration, the total attack average is calculated by finding the mean value of all attacks. Then, each Pokémon's attack value is evaluated to see if it exceeds the total average."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xW1EQnQUhVzJ"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "names = ['Absol', 'Aron', 'Jynx', 'Natu', 'Onix']\n",
        "attacks = np.array([130, 70, 50, 50, 45])\n",
        "\n",
        "for pokemon,attack in zip(names, attacks):\n",
        "    total_attack_avg = attacks.mean()\n",
        "    if attack > total_attack_avg:\n",
        "        print(\n",
        "            \"{}'s attack: {} > average: {}!\"\n",
        "            .format(pokemon, attack, total_attack_avg)\n",
        "        )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aZYMAkRfhVzJ"
      },
      "source": [
        "The inefficiency in this loop is the total_attack_avg variable being created with each iteration of the loop. But, this calculation doesn't change between iterations since it is an overall average. We only need to calculate this value once. By moving this calculation outside (or above) the loop, we calculate the total attack average only once. We get the same output, but this is a more efficient approach."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VYEAWMh3hVzK"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "names = ['Absol', 'Aron', 'Jynx', 'Natu', 'Onix']\n",
        "attacks = np.array([130, 70, 50, 50, 45])\n",
        "\n",
        "# Calculate total average once (outside the loop)\n",
        "total_attack_avg = attacks.mean()\n",
        "\n",
        "for pokemon,attack in zip(names, attacks):\n",
        "    if attack > total_attack_avg:\n",
        "        print(\n",
        "            \"{}'s attack: {} > average: {}!\"\n",
        "            .format(pokemon, attack, total_attack_avg)\n",
        "        )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UvZ1ANK-hVzK"
      },
      "source": [
        "Let's compare the runtimes of both methods:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qm1j70uVhVzL"
      },
      "outputs": [],
      "source": [
        "%%timeit\n",
        "\n",
        "for pokemon,attack in zip(names, attacks):\n",
        "    total_attack_avg = attacks.mean()\n",
        "    if attack > total_attack_avg:\n",
        "        print(\n",
        "            \"{}'s attack: {} > average: {}!\"\n",
        "            .format(pokemon, attack, total_attack_avg)\n",
        "        )    "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gStsl3PnhVzM"
      },
      "outputs": [],
      "source": [
        "%%timeit\n",
        "\n",
        "# Calculate total average once (outside the loop)\n",
        "total_attack_avg = attacks.mean()\n",
        "for pokemon,attack in zip(names, attacks):\n",
        "    if attack > total_attack_avg:\n",
        "        print(\n",
        "            \"{}'s attack: {} > average: {}!\"\n",
        "            .format(pokemon, attack, total_attack_avg)\n",
        "        )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bGRYGiqvhVzO"
      },
      "source": [
        "We see that keeping the total_attack_avg calculation within the loop takes more than 120 microseconds."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y_FwI1A8hVzO"
      },
      "source": [
        "### Using holistic conversions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XvxMGzGZhVzP"
      },
      "source": [
        "Another way to make loops more efficient is to use holistic conversions outside (or below) the loop. In the example below we have three lists from the 720 Pokémon dataset: a list of each Pokémon's name, a list corresponding to whether or not a Pokémon has a legendary status, and a list of each Pokémon's generation. We want to combine these objects so that each name, status, and generation is stored in an individual list. To do this, we'll use a loop that iterates over the output of the zip function. Remember, zip returns a collection of tuples, so we need to convert each tuple into a list since we want to create a list of lists as our output. Then, we append each individual poke_list to our poke_data output variable. By printing the result, we see our desired list of lists."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "wPwh5-VXhVzQ"
      },
      "outputs": [],
      "source": [
        "names_list = pokemon['Name']\n",
        "legend_status_list = pokemon['Legendary']\n",
        "generations_list = pokemon['Generation']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "00ApqPVyhVzQ"
      },
      "outputs": [],
      "source": [
        "%%timeit\n",
        "poke_data = []\n",
        "for poke_tuple in zip(names_list, legend_status_list, generations_list):\n",
        "    poke_list = list(poke_tuple)\n",
        "    poke_data.append(poke_list)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NH1Xl47ghVzR"
      },
      "source": [
        "However, converting each tuple to a list within the loop is not very efficient. Instead, we should collect all of our poke_tuples together, and use the map function to convert each tuple to a list. The loop no longer converts tuples to lists with each iteration. Instead, we moved this tuple to list conversion outside (or below) the loop. That way, we convert data types all at once (or holistically) rather than converting in each iteration."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g7YfxAbjhVza"
      },
      "outputs": [],
      "source": [
        "%%timeit\n",
        "poke_data_tuples = []\n",
        "for poke_tuple in zip(names_list, legend_status_list, generations_list):\n",
        "    poke_data_tuples.append(poke_tuple)\n",
        "    poke_data = [*map(list, poke_data_tuples)]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k5SzNFzvhVzb"
      },
      "source": [
        "Runtimes show that converting each tuple to a list outside of the loop is more efficient."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mK3_SFoYhVzc"
      },
      "source": [
        "## 2. Pandas Optimization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZN-4m8zZhVzd"
      },
      "source": [
        "## 2.1. Pandas DataFrame Iteration"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X9RXjFKzhVze"
      },
      "source": [
        "Pandas is a library used for data analysis. The main construct of pandas is the DataFrame, a tabular data structure with labeled rows and columns. We will focus on the best approaches for iterating over a DataFrame. Let's begin by analyzing a Major League Baseball dataset. It contains the team stats for each Major League Baseball team from the year 1962 to 2012, which are stored in a pandas DataFrame named baseball_df."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bMkyzb1VhVze"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "baseball_df = pd.read_csv('baseball_stats.csv')\n",
        "baseball_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cagw7DoChVzf"
      },
      "source": [
        "We will focus on the W column, which specifies the number of wins a team had in a season and the G column which contains the number of games a team played in a season."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mr64O23nhVzf"
      },
      "source": [
        "### Calculating & Adding win percentage"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PWY57EOYhVzg"
      },
      "source": [
        "One popular statistic used to evaluate a team's performance for a given season is the team's win percentage. This metric is calculated by dividing a team's total wins by the number of games played. Here is a simple function to perform this calculation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DpvUDNUyhVzg"
      },
      "outputs": [],
      "source": [
        "def calc_win_perc(wins, games_played):\n",
        "    win_perc = wins / games_played\n",
        "    return np.round(win_perc,2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lSNVY0FyhVzg"
      },
      "source": [
        "Now we would like to create a new column in our baseball_df DataFrame that stores each team's win percentage for a season. To do this, we'll need to iterate over the DataFrame's rows and apply our calc_win_perc function. First, we create an empty win_perc_list to store all the win percentages we'll calculate. Then, we write a loop that will iterate over each row of the DataFrame. Notice that we are using an index variable (i) that ranges from zero to the number of rows that exist within the DataFrame. We then use the .iloc method to look up each individual row within the DataFrame using the index variable."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9hUrcqq4hVzh"
      },
      "source": [
        "Then we grab each team's wins and games played by referencing the W and G columns. Next, we pass the team's wins and games played to calc_win_perc to calculate the win percentages. Finally, we append win_perc to win_perc_list and continue the loop. We create our desired column in the DataFrame, called WP, by setting the column value equal to the win_perc_list."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lXWFBK9vhVzh"
      },
      "outputs": [],
      "source": [
        "%%timeit\n",
        "win_perc_list = []\n",
        "for i in range(len(baseball_df)):\n",
        "    row = baseball_df.iloc[i]\n",
        "    wins = row['W']\n",
        "    games_played = row['G']\n",
        "    win_perc = calc_win_perc(wins, games_played)\n",
        "    win_perc_list.append(win_perc)\n",
        "baseball_df['WP'] = win_perc_list\n",
        "\n",
        "baseball_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pb-MHQZShVzh"
      },
      "source": [
        "Looping over the DataFrame with .iloc gave us our desired output, but is it inefficient. When estimating the runtime, the .iloc approach took 660 milliseconds, which is pretty inefficient."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pw_yD02hhVzi"
      },
      "source": [
        "### Iterating with .iterrows()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "geXm2DJAhVzi"
      },
      "source": [
        "pandas come with a few efficient methods for looping over a DataFrame. The first method we'll cover is the .iterrows method. This is similar to the .iloc method, but .iterrows returns each DataFrame row as a tuple of (index, pandas Series) pairs. This means each object returned from .iterrows contains the index of each row as the first element and the data in each row as a pandas Series as the second element."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZsJ_6cgvhVzj"
      },
      "source": [
        "Notice that we still create the empty win_perc_list, but now we don't have to create an index variable to look up each row within the DataFrame. .iterrows handles the indexing for us! The remainder of the for loop stays the same to create a new win percentage column within our baseball_df DataFrame."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Hn1B591ihVzj"
      },
      "outputs": [],
      "source": [
        "%%timeit\n",
        "win_perc_list = []\n",
        "for i,row in baseball_df.iterrows():\n",
        "    wins = row['W']\n",
        "    games_played = row['G']\n",
        "\n",
        "    win_perc = calc_win_perc(wins, games_played)\n",
        "    win_perc_list.append(win_perc)\n",
        "\n",
        "baseball_df['WP'] = win_perc_list"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WXACvzeehVzk"
      },
      "source": [
        "Using .iterrows takes roughly half the time .iloc takes to iterate over our DataFrame."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "whFoFurEhVzk"
      },
      "source": [
        "### Iterating with .itertuples()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fc1eyAF7hVzl"
      },
      "source": [
        "Pandas also come with another iteration method called .itertuples which is often more efficient than .iterrows. Let's continue using our baseball dataset to compare these two methods. Suppose we have a pandas DataFrame called team_wins_df that contains each team's total wins in a season.\n",
        "\n",
        "If we use .iterrows to loop over our team_wins_df DataFrame and print each row's tuple, we see that each row's values are stored as a pandas Series. Remember, .iterrows returns each DataFrame row as a tuple of (index, pandas Series) pairs, so we have to access the row's values with square bracket indexing.\n",
        "\n",
        "But, we could use .itertuples to loop over our DataFrame rows instead. The .itertuples method returns each DataFrame row as a special data type called a namedtuple. A namedtuple is one of the specialized data types that exist within the collections module we've discussed previously. These data types behave just like a Python tuple but have fields accessible using attribute lookup."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sfLJ9-dHhVzm"
      },
      "outputs": [],
      "source": [
        "team_wins_df = baseball_df[['Team','Year','W']]\n",
        "for row_namedtuple in team_wins_df.itertuples():\n",
        "    print(row_namedtuple)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hbVwNlKphVzm"
      },
      "source": [
        "Notice in the output that each printed row_namedtuple has an Index attribute and each column in our team_wins_df as an attribute. That means we can access each of these attributes with a lookup using a dot method. Here, we can print the last row_namedtuple's Index using **row_namedtuple.Index**. We can print this row_namedtuple's Team with **row_namedtuple.Team**, Year with **row_namedtuple.Year** and so on."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bQ2vX_QRhVzm"
      },
      "outputs": [],
      "source": [
        "print(row_namedtuple.Index)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yHfu3BwwhVzn"
      },
      "outputs": [],
      "source": [
        "print(row_namedtuple.Team)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_YYGfWmjhVzn"
      },
      "source": [
        "When we compare **.iterrows** to **.itertuples**, we see that there is quite a bit of improvement!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_cbj2SIzhVzo"
      },
      "outputs": [],
      "source": [
        "%%timeit\n",
        "for row_tuple in team_wins_df.iterrows():\n",
        "    print(row_tuple)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8QSibA7LhVzo"
      },
      "outputs": [],
      "source": [
        "%%timeit\n",
        "for row_namedtuple in team_wins_df.itertuples():\n",
        "    print(row_namedtuple)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nDgBa7TwhVzo"
      },
      "source": [
        "The reason **.itertuples** is more efficient than **.iterrows** is due to the way each method stores its output. Since .iterrows returns each row's values as a pandas Series, there is a bit more overhead."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "shDfQZ2WhVzp"
      },
      "source": [
        "## 2.2. Pandas Alternative to Looping"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qWT7R1o2hVzp"
      },
      "source": [
        "In order to write efficient code, we want to avoid looping as much as possible. Therefore now we will explore an alternative to using **.iterrows** and **.itertuples** to perform calculations on a DataFrame.\n",
        "\n",
        "We'll continue using the  baseball dataset. We will first create the **calc_run_diff** function. This function calculates a team's run differential for a given year by subtracting the team's total number of runs allowed from their total number of runs scored in a season."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x1IjBOIvhVzp"
      },
      "outputs": [],
      "source": [
        "def calc_run_diff(runs_scored, runs_allowed):\n",
        "    run_diff = runs_scored - runs_allowed\n",
        "    return run_diff"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_BWGIEuUhVzq"
      },
      "source": [
        "We'd like to create a new column in our baseball_df DataFrame called RD that stores each team's run differentials over the years. Previously we did this with a for loop using either .iterrows or .itertuples. Here, we'll use .iterrows as an example. Notice that we are iterating over baseball_df with a for loop, passing each row's RS and RA columns into our calc_run_diff function, and then appending each row's result to our run_diffs_iterrows list. This gives us our desired output, but it is not our most efficient option."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ij8ckfFphVzq"
      },
      "outputs": [],
      "source": [
        "%%timeit\n",
        "run_diffs_iterrows = []\n",
        "\n",
        "for i,row in baseball_df.iterrows():\n",
        "    run_diff = calc_run_diff(row['RS'], row['RA'])\n",
        "    run_diffs_iterrows.append(run_diff)\n",
        "\n",
        "baseball_df['RD'] = run_diffs_iterrows"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ENAz7xZ_hVzr"
      },
      "source": [
        "### Pandas.apply() Method"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pZBIhpWLhVzr"
      },
      "source": [
        "One alternative to using a loop to iterate over a DataFrame is to use pandas.apply method. This function acts like the map function. It takes a function as an input and applies this function to an entire DataFrame. Since we are working with tabular data, we must specify an axis that we would like our function to act on. Using a zero for the axis argument will apply our function on columns while using a one for the axis will apply our function on all rows. Just like the map function, pandas.apply method can be used with anonymous functions or lambdas. \n",
        "\n",
        "Let's walk through how'd we'd use the .apply method to calculate our run differentials. First, we call .apply on the baseball_df DataFrame. Then, we use a lambda function to iterate over the rows of the DataFrame. Notice that our argument for lambda is row (since we are applying to each row of the DataFrame). For every row, we grab the RS and RA columns and pass them to our calc_run_diff function. Lastly, we specify our axis to tell dot-apply that we want to iterate over rows instead of columns."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jNlOpFG5hVzr"
      },
      "outputs": [],
      "source": [
        "%%timeit\n",
        "run_diffs_apply = baseball_df.apply(\n",
        "    lambda row: calc_run_diff(row['RS'], row['RA']),\n",
        "    axis=1)\n",
        "\n",
        "baseball_df['RD'] = run_diffs_apply"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qmiqQj0OhVzs"
      },
      "source": [
        "When we use the .apply method to calculate our run differentials, we don't need to use a for a loop. We can collect our run differentials directly into an object called run_diffs_apply. \n",
        "\n",
        "We can see that the .iterrows approach took about 321 milliseconds to complete. But, using the dot-apply method took only 45.2 milliseconds. A definite improvement!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n37-Z7N9hVzt"
      },
      "source": [
        "### 2.3. Optimal Pandas Iterating"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dlyTIgWRhVzu"
      },
      "source": [
        "We've come a long way from our first .iloc approach for iterating over a DataFrame. Each approach we've discussed has really improved the performance. But, these approaches focus on performing calculations for each row of our DataFrame individually. Now we will explore some panda's internals that allows us to perform calculations more efficiently.\n",
        "\n",
        "Since pandas is a library that is built on NumPy. This means that each pandas DataFrame we use can take advantage of the efficient characteristics of NumPy arrays discussed in the previous article. We discussed broadcasting and how it allows the NumPy arrays to vectorize operations, so they are performed on all elements of an object at once. This allows us to efficiently perform calculations over entire arrays. Just like NumPy, pandas are designed to vectorize calculations so that they operate on entire datasets at once (not just on a row-by-row basis). Let's explore this concept with some examples.\n",
        "\n",
        "We'll continue to use the baseball_df DataFrame we have been using throughout the chapter. Since pandas are built on top of NumPy, we can grab any of these DataFrame column's values as a NumPy array using the dot-values method. Here, we are collecting the W column's values into a NumPy array called wins_np. When we print the type of wins_np, we see that it is, in fact, a NumPy array."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DssiYQVFhVzv"
      },
      "outputs": [],
      "source": [
        "wins_np = baseball_df['W'].values\n",
        "print(type(wins_np))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rBLGcIkThVzw"
      },
      "source": [
        "We can see the contents of the array by printing it and verifying that it is the same as the W column from our DataFrame."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bDKhiQ2shVzw"
      },
      "outputs": [],
      "source": [
        "print(wins_np)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZewP62BThVzx"
      },
      "source": [
        "The beauty of knowing that pandas are built on NumPy can be seen when taking advantage of a NumPy array's broadcasting abilities. Remember, this means we can vectorize our calculations, and perform them on entire arrays all at once! Instead of looping over a DataFrame, and treating each row independently, like we've done with .iterrows, .itertuples, and .apply, we can perform calculations on the underlying NumPy arrays of our baseball_df DataFrame. Here, we gather the RS and RA columns in our DataFrame as NumPy arrays, and use broadcasting to calculate run differentials all at once!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pB0LbJGdhVzx"
      },
      "outputs": [],
      "source": [
        "%%timeit\n",
        "\n",
        "run_diffs_np = baseball_df['RS'].values - baseball_df['RA'].values\n",
        "baseball_df['RD'] = run_diffs_np"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8E39Oo49hVzy"
      },
      "source": [
        "When we use NumPy arrays to perform our run differential calculations, we can see that our code becomes much more readable. Here, we can explicitly see how our run differentials are being calculated. Also when we time our NumPy arrays approach, we see that our run differential calculations take microseconds! All other approaches were reported in milliseconds. Our array approach is orders of magnitude faster than all previous approaches! It's clear that using a DataFrame's underlying NumPy arrays to perform calculations can help us gain some massive efficiencies."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.9"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}