lessons/python_advanced/pickle_saving_objects_for_later.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "print(\"Python Version:\", sys.version, '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pickle: Saving Objects for Later"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often in data science, we'll create some model or some version of our data and want to use it later. We have many options - we can save the coefficients, or save the data to csv, or...\n",
    "\n",
    "Actually, we don't have that many options. \n",
    "\n",
    "One way to overcome that is to save the python object to a file as a serialized object. That means we convert the entire object to a bunch of bytes, save those bytes into a file, and then have the ability to unpack those bytes back into their original format later. \n",
    "\n",
    "This is done by a module called `pickle`. Let's see it in action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "import random\n",
    "\n",
    "lots_of_noise = {\n",
    "    'CA': [random.randint(0,65) for _ in range(100)],\n",
    "    'IL': [random.randint(0,65) for _ in range(50)],\n",
    "    'NY': [random.randint(0,65) for _ in range(90)],\n",
    "    'WA': [random.randint(0,65) for _ in range(33)]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(lots_of_noise)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "whos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see in this `whos` command that the object `lots_of_noise` exists and is a `dict` with 4 keys. Nice. Now let's look at our file system and verify that there isn't a file called `noise.pickle`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, now we're ready to create a file and write the bytes to it. To do this with `pickle`, we use python's read-write streamer `open` and create a writable-binary (`wb`) file. We'll then use `pickle.dump` to put an object into that file as a string of bytes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('noise.pickle', 'wb') as to_write:\n",
    "    pickle.dump(lots_of_noise, to_write)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's delete `lots_of_noise` and prove to ourselves it doesn't exist in Python's memory anymore."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del lots_of_noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "whos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(lots_of_noise)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lovely. It's dead forever. Or is it?\n",
    "\n",
    "Let's open that `noise.pickle` file with read-binary (`rb`) mode. Then we'll ask pickle to retrieve the file with `pickle.load` and store it back in a variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('noise.pickle','rb') as read_file:\n",
    "    new_noise = pickle.load(read_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(new_noise)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "whos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random noise lives! We retrieved the entire structure from file. Nice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Okay, but I don't use dictionaries... I use pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "df = pd.DataFrame(np.random.uniform(-10,10, size=(100,4)), columns=['Yay','specific','column','names'])\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('my_dataframe.pickle', 'wb') as to_write:\n",
    "    pickle.dump(df, to_write)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del df\n",
    "\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('my_dataframe.pickle','rb') as read_file:\n",
    "    new_df = pickle.load(read_file)\n",
    "    \n",
    "new_df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pickle is a great tool. One recommended way of using it is to make it an end point of every step in your process. Example:\n",
    "\n",
    "* I got my data! Nice. Pickle it and stop your \"getting the data\" notebook.\n",
    "* Load your data from pickle. Clean it. Save your clean data to a new pickle.\n",
    "* Load your cleaned_data pickle. Do analysis and visualize it.\n",
    "\n",
    "This can provide natural \"pick-up-where-I-left-off-but-before-I-broke-my-data\" points. It's a great way to control the flow of your data.\n",
    "\n",
    "#### Resources\n",
    "\n",
    "https://docs.python.org/3.7/library/pickle.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}