04_more_data_visualization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='images/Practicum_AI_Logo.white_outline.svg' width=250 alt='Practicum AI logo'> <img src='https://github.com/PracticumAI/practicumai.github.io/blob/main/images/icons/practicumai_python.png?raw=true' align='right' width=50>\n",
    "\n",
    "***\n",
    "\n",
    "# Additional Data Visualization Exercise\n",
    "\n",
    "Much of the text and code in this notebook comes from the [Data Carpentry](https://datacarpentry.org/) lesson [\"Making Plots With plotnine\"](https://datacarpentry.org/python-ecology-lesson/07-visualization-ggplot-python.html). Data Carpentry and [The Carpentries](https://carpentries.org/) are a great source of learning materials.\n",
    "\n",
    "In this lesson, we're going to be exploring some USDA data on energy use in agriculture [[1](#Citations)]. **Check out [the description of the data](https://data.nal.usda.gov/dataset/data-chapter-5-energy-use-agriculture-us-agriculture-and-forestry-greenhouse-gas-inventory-1990-2018) before continuing.**\n",
    "\n",
    "It is probably also a good idea to **open a tab with the [`plotnine` documentation](https://plotnine.readthedocs.io/).**\n",
    "\n",
    "\n",
    "## Grammar of Graphics\n",
    "\n",
    "As noted on the `plotnine` website:\n",
    "\n",
    "> plotnine is an implementation of a grammar of graphics in Python based on ggplot2.\n",
    "> The grammar allows you to compose plots by explicitly mapping variables in a\n",
    "> dataframe to the visual objects that make up the plot.\n",
    "> \n",
    "> Plotting with a grammar of graphics is powerful. Custom (and otherwise complex) \n",
    "> plots are easy to think about and build incremently [sic], while the simple plots \n",
    "> remain simple to create.\n",
    "\n",
    "<img src=\"images/gramar_of_graphics_thumbnail.png\" align='right'> The Grammar of Graphics concept originated with Leland Wilkinson's [book by that name](https://link.springer.com/book/10.1007/0-387-28695-0), first published in 1999. The popular [R programming language](https://www.r-project.org/) package [`ggplot`](https://ggplot2.tidyverse.org/) was created by Hadley Wickham in 2005 as an implementation of Wilkinson's concepts.\n",
    "\n",
    "The *Towards Data Science* article [Data Visualization — Grammar of Graphics](https://medium.com/aiskunks/data-visualization-grammar-of-graphics-fccf78379b52) by Swathi Sharma provides a good explanation of the tenets and structure of the Grammar of Graphics:\n",
    "\n",
    "\n",
    "> ### Grammar vs Grammar of Graphics\n",
    ">\n",
    "> Grammar is described as a system of rules that aids in the definition and establishment\n",
    "> of language components.\n",
    ">\n",
    "> Grammar of Graphics is a framework that allows us to explain the components of any visual \n",
    "> in a straightforward manner. Instead of random trials and errors, it employs a layered technique\n",
    "> that use predetermined components to build a visualization\n",
    ">\n",
    "> Grammar of Graphics is a theory developed by Leland Wilkinson for visualizing and communicating\n",
    "> data in a structured and coherent way. The Grammar of Graphics consists of a set of principles \n",
    "> and rules that guide the design of graphics for data visualization.\n",
    ">\n",
    "> The Grammar of Graphics is based on the idea that every graphic can be broken down into a series \n",
    "> of components or layers. These components include the data, the aesthetic mapping, the geometric \n",
    "> shapes, the statistical transformation, and the scales.\n",
    "> \n",
    "> The data component refers to the raw data that is being visualized, while the aesthetic mapping \n",
    "> refers to the way that data is mapped to visual properties such as color, size, and shape. The geometric\n",
    "> shapes component refers to the basic visual elements used to represent the data, such as points, lines, \n",
    "> and bars. The statistical transformation component refers to any calculations or analyses performed on \n",
    "> the data, such as aggregations or summaries. Finally, the scales component refers to the way that data \n",
    "> is transformed to fit within the visual space, such as scaling the axis.\n",
    ">\n",
    "> By breaking down graphics into these components, the Grammar of Graphics provides a systematic approach \n",
    "> to designing and interpreting data visualizations. It allows for greater flexibility in visualizing data,\n",
    "> as well as a standardized way of communicating the design choices made in the creation of a graphic.\n",
    "> \n",
    "> A variant of this, is known as the layered grammar of graphics framework, proposed by Hadley Wickham, \n",
    "> reputed Data Scientist and the creator of the famous R visualization package ggplot2.\n",
    "\n",
    "\n",
    "\n",
    "## `plotnine` provides an API for plotting\n",
    "\n",
    "An API, or Application Programming Interface, defines how two or more programs can communicate with eachother. `plotnine` is an API that communicates with `matplotlib`, the primary Python plotting library. Essentially, `plotnine` implements the Grammar of Graphics syntax and translates your `plotnine` commands into `matplotlib` code to create your graphics.\n",
    "\n",
    "This may be our first encounter with APIs, but it will certainly not be our last--much of the code used in AI applications is using one or more APIs to communicate with other libraries to implement the tasks we want to accomplish.\n",
    "\n",
    "\n",
    "## Let's get plotting!\n",
    "\n",
    "Just as with the other packages, `plotnine` needs to be imported. It is good practice to not just load an entire package, such as `from plotnine import *`, but to use an abbreviation as we used `pd` for Pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import plotnine as p9"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and examine the data\n",
    "\n",
    "Now, let's load the data. We'll be loading in \"Data for Figure 5-2: Energy use in agriculture, by source, 1965–2018 in QBTU (quadrillion British thermal units)\". We've included a copy of this CSV file in the data folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "file_path = 'data/Figure5_2.csv'\n",
    "energy_data = pd.read_csv(file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Take a look at the data\n",
    "energy_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Get some summary statistics\n",
    "energy_data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting with `plotnine`\n",
    "\n",
    "The `plotnine` package supports the creation of complex plots from data in a\n",
    "DataFrame. It uses default settings, which help create publication-quality\n",
    "plots with a minimal amount of settings and tweaking.\n",
    "\n",
    "`plotnine` graphics are built step by step by adding different elements on \n",
    "top of each other using the `+` operator. Putting the individual steps \n",
    "together in parentheses `()` provides Python-compatible syntax.\n",
    "\n",
    "To build a `plotnine` graphic, we need to:\n",
    "\n",
    "- Bind the plot to a specific DataFrame using the `data` argument:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(p9.ggplot(data=energy_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have not defined anything else, just an empty figure is available and\n",
    "presented.\n",
    "\n",
    "- Define aesthetics (`aes`) by **selecting variables** used in the plot and\n",
    "`mapping` them to a presentation, such as plotting size, shape, color, etc. You\n",
    "can interpret this as **which** of the variables will influence the plotted\n",
    "objects/geometries:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=energy_data,\n",
    "           mapping=p9.aes(x='Year', y='Gasoline')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most important aes mappings are: `x`, `y`, `alpha`, `color`, `fill`, \n",
    "`linetype`, `shape`, `size` and `stroke`.\n",
    "\n",
    "- Still no specific data is plotted, as we have to define what kind of geometry\n",
    "will be used for the plot. The most straightforward is probably using points.\n",
    "Points is one of the `geom` options, the graphical representation of the data\n",
    "in the plot. Others are lines, bars, and many more. To add a geom to the plot use the `+`\n",
    "operator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=energy_data,\n",
    "           mapping=p9.aes(x='Year', y='Gasoline'))\n",
    "    + p9.geom_point()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `+` in the `plotnine` package is particularly useful because it allows you\n",
    "to modify existing `plotnine` objects. This means you can easily set up plot\n",
    "*templates* and conveniently explore different types of plots, so the above\n",
    "plot can also be generated with code like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the base plot\n",
    "gasoline_plot = p9.ggplot(data=energy_data,\n",
    "                   mapping=p9.aes(x='Year', y='Gasoline'))\n",
    "\n",
    "# Draw points on the plot\n",
    "gasoline_plot + p9.geom_point()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <img src='images/note_icon.svg' alt=\"Note icon\" width=40 align=center> Note\n",
    "\n",
    "- Anything you put in the `ggplot()` function can be seen by any geom layers\n",
    "  that you add (i.e., these are universal plot settings). This includes the `x`\n",
    "  and `y` axes you set up in `aes()`.\n",
    "- You can also specify aesthetics for a given `geom` independently of the\n",
    "  aesthetics defined globally in the `ggplot()` function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <img src='images/exercise_icon.svg' alt=\"Exercise icon\" width=40 align=center> Exercise 1\n",
    "\n",
    "> Make a plot similar to the Gasoline plot above but for one of the other energy sources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building your plots iteratively\n",
    "\n",
    "Building plots with `plotnine` is typically an iterative process. We start by\n",
    "defining the dataset we'll use, lay the axes, and choose a geom. Hence, the\n",
    "`data`, `aes` and `geom_*` are the elementary elements of any graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=energy_data,\n",
    "           mapping=p9.aes(x='Year', y='Gasoline'))\n",
    "    + p9.geom_point()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we start modifying this plot to add more information to it or improve its appearance. For\n",
    "instance, we can add color to the points on our plot!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=energy_data,\n",
    "           mapping=p9.aes(x='Year', y='Gasoline'))\n",
    "    + p9.geom_point(color='blue')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Maybe we want to try a different `shape` of point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=energy_data,\n",
    "           mapping=p9.aes(x='Year', y='Gasoline'))\n",
    "    + p9.geom_point(color='blue', shape='*')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Those stars are a bit small, let's change their `size`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=energy_data,\n",
    "           mapping=p9.aes(x='Year', y='Gasoline'))\n",
    "    + p9.geom_point(color='blue', shape='*', size=4)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting Multiple Columns of Data\n",
    "\n",
    "For what we might want to do next (plot not just Gasoline or Diesel, but all of the different energy sources), the design of `plotnine` doesn't work particularly well with how our data are currently formatted.\n",
    "\n",
    "`plotnine` cannot currently plot multiple `y` values or columns of data (there is some discussion of adding that ability in a future release). That doesn't mean we can't do it; it just means we need to transform our data a bit.\n",
    "\n",
    "`pandas` has a helpful function that will get our DataFrame in a more suitable format for `plotnine`: the `melt` function. Here's the documentation for the melt function ([full documentation here](https://pandas.pydata.org/docs/reference/api/pandas.melt.html)):\n",
    "\n",
    "> Unpivot a DataFrame from wide to long format, optionally leaving identifiers set.\n",
    ">\n",
    "> This function is useful to massage a DataFrame into a format where one or more columns\n",
    "> are identifier variables (id_vars), while all other columns, considered measured variables\n",
    "> (value_vars), are \"unpivoted\" to the row axis, leaving just two non-identifier columns,\n",
    "> 'variable 'va’\n",
    "\n",
    "It's easier to see this in action....\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Current energy_data DataFrame\n",
    "energy_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the melt function and make a long format DataFrame\n",
    "# The columns become text values in the var_name column\n",
    "# The corresponding values are placed in the value_name column\n",
    "\n",
    "energy_data_long = pd.melt(energy_data, id_vars='Year', var_name='Energy Source', value_name='Consumption')\n",
    "energy_data_long.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can go back to iterative plot development.\n",
    "\n",
    "Let's start more or less where we ended, but update the command to use `data=energy_data_long` and `y='Consumption'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=energy_data_long,\n",
    "           mapping=p9.aes(x='Year', y='Consumption'))\n",
    "    + p9.geom_point(color='blue', shape='*', size=4)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That worked, but it is a bit of a mess because everything is in the same blue color!\n",
    "\n",
    "Luckily, `plotnine` is set to tackle that! We can move the `color` and `shape` parameters into `aes` to have different colors and shapes for each category in the Energy Source column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=energy_data_long,\n",
    "           mapping=p9.aes(x='Year', y='Consumption', \n",
    "                          color='Energy Source', \n",
    "                          shape='Energy Source'))\n",
    "    + p9.geom_point()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking better!! `plotnine` even made a nice legend for us!\n",
    "\n",
    "But, while points were a reasonable starting place, this plot might look better using lines.\n",
    "\n",
    "That's a simple change: `geom_line` rather than `geom_point`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(p9.ggplot(data=energy_data_long,\n",
    "           mapping=p9.aes(x='Year', y='Consumption', color='Energy Source'))\n",
    "    + p9.geom_line()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <img src='images/exercise_icon.svg' alt=\"Exercise icon\" width=40 align=center> Exercise 2\n",
    "\n",
    "This is looking good! \n",
    "\n",
    "There are a few more things we could do. Use the [`plotnine` documentation](https://plotnine.readthedocs.io) to:\n",
    "\n",
    "* Add a plot title.\n",
    "* Make the lines a bit thicker.\n",
    "* Change the y-axis label to read \"Energy Consumption (QBTU)\"\n",
    "* Anything else you want to explore!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And that's it! We encourage you to keep playing with code, and look out for more Practicum AI lessons!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "## Push changes to GitHub <img src=\"images/push_to_github.png\" alt=\"Push to GitHub icon\" align=\"right\" width=150>\n",
    "\n",
    " Remember to **add**, **commit**, and **push** the changes you have made to this notebook to GitHub to keep your repository in sync.\n",
    "\n",
    "In Jupyter, those are done in the git tab on the left. In Google Colab, use File > Save a copy in GitHub.\n",
    "\n",
    "----\n",
    "\n",
    "## Citations\n",
    "\n",
    "[1]\n",
    "(original article) Xiarchos, I.M. (2022). Chapter 5: Energy Use in Agriculture. In U.S. Agriculture and Forestry Greenhouse Gas Inventory: 1990–2018. Technical Bulletin No. 1957, United States Department of Agriculture, Office of the Chief Economist, Washington, DC. p. 177-181. January 2022. Hanson, W.L., S.J. Del Grosso, L. Gallagher, Eds.\n",
    "\n",
    "(dataset) Xiarchos, Irene M. (2021). Data from: Chapter 5: Energy Use in Agriculture. U.S. Agriculture and Forestry Greenhouse Gas Inventory: 1990-2018. Ag Data Commons. https://doi.org/10.15482/USDA.ADC/1524410. Accessed 2023-11-30."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}