workflow-basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(workflow-basics)=\n",
    "# Workflow Basics\n",
    "\n",
    "This chapter will take you through some of the essential parts of a Python workflow.\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "You'll need an installation of Python and Visual Studio Code with the Python extensions to get to grips with this chapter. If you haven't installed those yet, head back to {ref}`code-preliminaries` and follow the instructions there."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Python scripts and the interactive window\n",
    "\n",
    "As a reminder, the figure below shows the typical layout of Visual Studio Code.\n",
    "\n",
    "![A typical user view in Visual Studio Code](https://github.com/aeturrell/coding-for-economists/blob/main/img/vscode_layout.png?raw=true)\n",
    "\n",
    "When you create a new script (File->New File->Save as 'your_script_name.py), it will appear in the part of the screen labelled as 3.\n",
    "\n",
    "To run a script, select the code you want to run, right click, and select \"Run Selection/Line in Interactive Window\". You can also hit shift + enter if you set this shortcut up. Using the \"Run Selection/Line in Interactive Window\" option (or doing the same via the shortcut) will cause panel 5 (the interactive window) to appear, where you will see the code run and the outputs of your script appear.\n",
    "\n",
    "When you are first writing a script, it's useful to be able to move back and forth between the script and the interactive window. You might execute a line of code (put the cursor on the relevant line and hit shift and enter) in the interactive window, then manually write out some code in the interactive window's execution box (seen at the bottom of panel 5 saying \"Type code here...\"), and then explore some of the variables you've created with the variable explorer (using the button \"Variables\") at the top of the interactive window.\n",
    "\n",
    "But, once you've honed the code in your script, it's good to make the script a complete analytical process that you are happy running end-to-end and that—for production or 'final' work—you would use the \"Run Current File in Interactive Window\" option to run all the way through. This is good practice because *what is in your script is reproducible* but what you've entered manually in the interactive window is not. And you want the outputs from your code to be reproducible and understandable by others (including future you!), but this is hard if there are undocumented extra lines of code that you only did on the fly via the interactive window's execution box."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0154c032",
   "metadata": {},
   "source": [
    "## Using installed packages and modules\n",
    "\n",
    "We already saw how to install packages in the previous chapter. If you forgot, look back at the Installing Packages section of {ref}`code-preliminaries` now. In short, packages are installed using the command line or, on Windows, the Anaconda prompt. With either of these open, first try typing `conda install packagename` and hitting enter. If the package isn't available on conda, you can use `pip install packagename` as a back-up.\n",
    "\n",
    "What about using a package that you've installed? That's what we'll look at now.\n",
    "\n",
    "Let's see an example of using the powerful numerical library **numpy**. There are different ways to import packages to use within a script or notebook; you can import the entire package in one go or just import the functions you need (if you know their names). When an entire package is imported, you can give it any name you like and the convention for **numpy** is to import it as the shortened 'np'. All of the functions and methods of the package can be accessed by typing `np` followed by `.` and then typing the function name. This convention of importing packages with a given name makes your code easier to read, because you know exactly which package is doing the heavy lifting, and avoids any conflicts when functions from different packages have the same name.\n",
    "\n",
    "As well as demonstrating importing the whole package for **numpy**, the example below shows importing just one specific function, for matrix inversion and called `inv`, from within **numpy**. Note that because `inv` was imported separately it can be used without an `np` prefix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f86893cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from numpy.linalg import inv\n",
    "\n",
    "matrix = np.array([[4.0, 2.0, 4.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]])\n",
    "\n",
    "print(\"Matrix:\")\n",
    "print(matrix)\n",
    "\n",
    "inv_mat = inv(matrix)\n",
    "print(\"Inverse:\")\n",
    "print(inv_mat)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb7ac525",
   "metadata": {},
   "source": [
    "We could have imported all of **numpy** using `from numpy import *` but this is considered bad practice as it fills our 'namespace' with function names that might clash with other packages and it's less easy to parse which package's function is doing what. R also relies heavily on imported libraries but uses a different convention for namespaces: in that case, everything is imported but the *order* of package imports matters for what functions are default (and clashing function names are over-written).\n",
    "\n",
    "```{note}\n",
    "If you want to check what packages you have installed in your Python environment, run `conda list` on your computer's command line (aka the *terminal* or *command prompt*).\n",
    "```\n",
    "\n",
    "There are times when you might forget what a function you have imported does! In Visual Studio Code, you can just hover your cursor over the name of the function and a box will come up that tells you everything you need to know about it. This box is auto-generated by the doc-strings we saw earlier in this Chapter (another reason why it's good practice to write doc-strings!).\n",
    "\n",
    "An alternative way to see what a function does is to use a wonderful package called [**rich**](https://github.com/willmcgugan/rich) that does many things including providing an `inspect` function. You will need to use `pip` to install rich by running `pip install rich` on the command line. Here's an example of using **rich**'s inpsect method on the `inv` function we imported above (`methods=True` reports all of the functionality of `inv`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fd14fd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from rich import inspect\n",
    "\n",
    "inspect(inv, help=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "653880d8",
   "metadata": {},
   "source": [
    "```{admonition} Exercise\n",
    "Write a code block that imports the **numpy** function `numpy.linalg.det` as `det`. Run `inspect` on it. Find the determinant of `[[4, 3], [1, 7]]`.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38d912c5",
   "metadata": {},
   "source": [
    "### Modules\n",
    "\n",
    "Sometimes, you will want to call in some code from a different script that you wrote rather than a different package. Imagine you have several code scripts (a, b, and c), all of which use the same underlying function that *you* have written. A central tenet of good coding is that you *do not repeat yourself*. Therefore, a bad solution to this problem would be to copy and paste the same code into all three of the scripts. A *good* solution is to write the code that's need just once in a separate 'utility' script and have the other scripts import that one function. This also adheres to another important programming principle: that of writing modular code.\n",
    "\n",
    "This schematic shows the kind of situation we're talking about:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0efeaee9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import networkx as nx\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib_inline.backend_inline\n",
    "\n",
    "# Plot settings\n",
    "plt.style.use(\n",
    "    \"https://github.com/aeturrell/coding-for-economists/raw/main/plot_style.txt\"\n",
    ")\n",
    "matplotlib_inline.backend_inline.set_matplotlib_formats(\"svg\")\n",
    "\n",
    "\n",
    "graph = nx.DiGraph()\n",
    "graph.add_edges_from(\n",
    "    [\n",
    "        (\"Utility script\", \"code file a\"),\n",
    "        (\"Utility script\", \"code file b\"),\n",
    "        (\"code file a\", \"code file c\"),\n",
    "        (\"code file b\", \"code file c\"),\n",
    "        (\"Utility script\", \"code file c\"),\n",
    "    ]\n",
    ")\n",
    "colour_node = \"#AFCBFF\"\n",
    "fixed_pos = nx.spring_layout(graph, seed=100)\n",
    "nx.draw(graph, pos=fixed_pos, with_labels=True, node_size=6000, node_color=colour_node)\n",
    "extent = 1.4\n",
    "plt.xlim(-extent, extent)\n",
    "plt.ylim(-extent, extent)\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5df7665d",
   "metadata": {},
   "source": [
    "How can we give code files a, b, and c access to the functions etc in the \"Utility script\"? We would define a file 'utilities.py' that had the following function in that we would like to use in the other code files:\n",
    "\n",
    "```python\n",
    "# Contents of utilities.py file\n",
    "def really_useful_func(number):\n",
    "    return number*10\n",
    "```\n",
    "\n",
    "Then, in 'code_script_a.py', we would write:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2e3422c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import utilities as utils\n",
    "\n",
    "print(utils.really_useful_func(20))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb775643",
   "metadata": {},
   "source": [
    "An alternative is to *just* import the function we want, with the name we want:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb14c455",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utilities import really_useful_func as ru_fn\n",
    "\n",
    "print(ru_fn(30))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15365d79",
   "metadata": {},
   "source": [
    "Another important example is the case where you want to run 'utilities.py' as a script, but still want to borrow functions from it to run in other scripts. There's a way to do this. Let's change utilities.py to\n",
    "\n",
    "```python\n",
    "# Contents of utilities.py file\n",
    "def really_useful_func(number):\n",
    "    return number*10\n",
    "\n",
    "\n",
    "def default_func():\n",
    "    print('Script has run')\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    default_func()\n",
    "```\n",
    "\n",
    "What this says is that if we call 'utilities.py' from the command line, eg\n",
    "\n",
    "```bash\n",
    "python utilities.py\n",
    "```\n",
    "\n",
    "It will return `Script has run` because, by executing the script alone, we are asking for anything in the `main` block defined at the end of the file to be run. But we can still import anything from utilities into other scripts as before--and in that case it is not the main script, but an import, and so the `main` block will *not* be executed.\n",
    "\n",
    "You can important several functions at once from a module (aka another script file) like this:\n",
    "\n",
    "```python\n",
    "from utilities import really_useful_func, default_func\n",
    "```\n",
    "\n",
    "```{admonition} Exercise\n",
    "Write your own `utilities.py` that has a `super_useful_func` that accepts a number and returns the number divided by 10. In another script, `main.py`, try a) importing all of utilities and running `super_useful_func` on a number and, b), importing just `super_useful_func` from utilities and running it on a number.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45bfe197",
   "metadata": {},
   "source": [
    "## Reading and writing files\n",
    "\n",
    "Although most applications in economics will use the **pandas** package to read and write tabular data, it's sometimes useful to know how to read and write arbitrary files using the built-in Python libraries too. To open a file\n",
    "\n",
    "```python\n",
    "open('filename', mode)\n",
    "```\n",
    "\n",
    "where `mode` could be `r` for read, `a` for append, `w` for write, and `x` to create a file. Create a file called `text_example.txt` and write a single line in it, 'hello world'. To open the file and print the text, use:\n",
    "\n",
    "```python\n",
    "with open('text_example.txt') as f:\n",
    "    text_in = f.read()\n",
    "\n",
    "print(text_in)\n",
    "```\n",
    "\n",
    "```python\n",
    "'hello world!\\n'\n",
    "```\n",
    "\n",
    "`\\n` is the new line character. Now let's try adding a line to the file:\n",
    "\n",
    "```python\n",
    "with open('text_example.txt', 'a') as f:\n",
    "    f.write('this is another line\\n')\n",
    "```\n",
    "\n",
    "Writing and reading files using the `with` command is a quick and convenient shorthand for the less concise open, action, close pattern. For example, the above example can also be written as:\n",
    "\n",
    "```python\n",
    "f = open('text_example.txt', 'a')\n",
    "f.write('this is another line\\n')\n",
    "f.close()\n",
    "```\n",
    "\n",
    "Although this short example shows opening and writing a text file, this approach can be used to edit a wide range of file extensions including .json, .xml, .csv, .tsv, and many more, including binary files in addition to plain text files."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.12 ('codeforecon')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "c4570b151692b3082981c89d172815ada9960dee4eb0bedb37dc10c95601d3bd"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}