notebooks/REST_APIs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using REST APIs as data sources\n",
    "\n",
    "* Data is everywhere and it is generated constantly\n",
    "* The number of datasources is amazingly huge\n",
    "* Datasets are huge and can be used in many ways\n",
    "\n",
    "* We may do amazing things using data made available by third-party:\n",
    "    - [https://developer.walmartlabs.com/docs]\n",
    "    - [https://developer.spotify.com/documentation/web-api/]\n",
    "    - [https://earthquake.usgs.gov/fdsnws/event/1/]\n",
    "    \n",
    "    \n",
    "We will have a nice and brief overview about how to consume data from REST APIs, mainly focusing on **JSON**.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is an API?\n",
    "\n",
    "**Application Programming Interface** defines the methods for one software program to interact with the other. \n",
    "\n",
    "In the case of this lecture, we are dealing with a REST API, which sends data over a network: one type of Web service.\n",
    "\n",
    "When we want to receive data from an Web service, we need to make a `request` to this service. When the server receives this request, it sends a `response`.\n",
    "\n",
    "![request.png](request.png)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Requests\n",
    "\n",
    "Knowing that, we will not have to learn about making requests in Python\n",
    "\n",
    "We do it by importing the module requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are different types of requests. \n",
    "\n",
    "In our case we will use a `GET`, which is used to retrieve data. This is the type of request we use to collect data.\n",
    "\n",
    "A response from the API contains 2 things (among others): \n",
    "* response code\n",
    "* response data\n",
    "\n",
    "To make a request, we use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "requests.models.Response"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = requests.get('http://twitter.com/igorsteinmacher')\n",
    "type(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `request.get(URL)` returns an object Response, which provides, among other things, the response code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "THe most common codes are:\n",
    "* 200: Everything went okay, and the result has been returned (if any).\n",
    "* 301: The server is redirecting you to a different endpoint. This can happen when a company switches domain names, or an endpoint name is changed.\n",
    "* 400: The server thinks you made a bad request. This can happen when you don’t send along the right data, among other things.\n",
    "* 401: The server thinks you’re not authenticated. Many APIs require login ccredentials, so this happens when you don’t send the right credentials to access an API.\n",
    "* 403: The resource you’re trying to access is forbidden: you don’t have the right permissions to see it.\n",
    "* 404: The resource you tried to access wasn’t found on the server.\n",
    "* 503: The server is not ready to handle the request.\n",
    "\n",
    "More details about status codes list can be found [here](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What about getting the data?\n",
    "\n",
    "First, read the documentation! Everytime you use an API, please read the documentation to understand how to use, the structure, etc.\n",
    "\n",
    "We will use the [Open Notify API](http://api.open-notify.org/), which gives access to data about the international space station.\n",
    "\n",
    "These APIs usually provide multiple endpoints, which are the ways we can interact with that service.\n",
    "\n",
    "Let's try a request and see how it goes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\n"
     ]
    }
   ],
   "source": [
    "response = requests.get(\"http://api.open-notify.org/astros.json\")\n",
    "print(response.status_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can see the data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'{\"people\": [{\"name\": \"Christina Koch\", \"craft\": \"ISS\"}, {\"name\": \"Alexander Skvortsov\", \"craft\": \"ISS\"}, {\"name\": \"Luca Parmitano\", \"craft\": \"ISS\"}, {\"name\": \"Andrew Morgan\", \"craft\": \"ISS\"}, {\"name\": \"Oleg Skripochka\", \"craft\": \"ISS\"}, {\"name\": \"Jessica Meir\", \"craft\": \"ISS\"}], \"number\": 6, \"message\": \"success\"}'"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"people\": [{\"name\": \"Christina Koch\", \"craft\": \"ISS\"}, {\"name\": \"Alexander Skvortsov\", \"craft\": \"ISS\"}, {\"name\": \"Luca Parmitano\", \"craft\": \"ISS\"}, {\"name\": \"Andrew Morgan\", \"craft\": \"ISS\"}, {\"name\": \"Oleg Skripochka\", \"craft\": \"ISS\"}, {\"name\": \"Jessica Meir\", \"craft\": \"ISS\"}], \"number\": 6, \"message\": \"success\"}'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'people': [{'name': 'Christina Koch', 'craft': 'ISS'},\n",
       "  {'name': 'Alexander Skvortsov', 'craft': 'ISS'},\n",
       "  {'name': 'Luca Parmitano', 'craft': 'ISS'},\n",
       "  {'name': 'Andrew Morgan', 'craft': 'ISS'},\n",
       "  {'name': 'Oleg Skripochka', 'craft': 'ISS'},\n",
       "  {'name': 'Jessica Meir', 'craft': 'ISS'}],\n",
       " 'number': 6,\n",
       " 'message': 'success'}"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with JSON \n",
    "JSON stands for JavaScript Object Notation. It is a way to encode data structures that ensures that they are easily readable. \n",
    "\n",
    "JSON output look like Python something with *dictionaries, lists, strings* and *integers*. And it is...\n",
    "\n",
    "But, how to use it? Well, we used it in the last command.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "json has two main functions:\n",
    "\n",
    "* `json.dumps()` — Takes in a Python object and converts (dumps) to a string.\n",
    "* `json.loads()` — Takes a JSON string and converts (loads) to a Python object.\n",
    "\n",
    "The `dumps()` is particularly useful as we can use it to format the json, making it easier to understand the output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"people\": [\n",
      "    {\n",
      "      \"name\": \"Christina Koch\",\n",
      "      \"craft\": \"ISS\"\n",
      "    },\n",
      "    {\n",
      "      \"name\": \"Alexander Skvortsov\",\n",
      "      \"craft\": \"ISS\"\n",
      "    },\n",
      "    {\n",
      "      \"name\": \"Luca Parmitano\",\n",
      "      \"craft\": \"ISS\"\n",
      "    },\n",
      "    {\n",
      "      \"name\": \"Andrew Morgan\",\n",
      "      \"craft\": \"ISS\"\n",
      "    },\n",
      "    {\n",
      "      \"name\": \"Oleg Skripochka\",\n",
      "      \"craft\": \"ISS\"\n",
      "    },\n",
      "    {\n",
      "      \"name\": \"Jessica Meir\",\n",
      "      \"craft\": \"ISS\"\n",
      "    }\n",
      "  ],\n",
      "  \"number\": 6,\n",
      "  \"message\": \"success\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "json_response = response.json()\n",
    "formatted_json = json.dumps(json_response, sort_keys=False, indent=2)\n",
    "\n",
    "print(formatted_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### REST API with Query Parameters\n",
    "\n",
    "In some cases, it is possible to pass parameters to filter the output of the API. \n",
    "\n",
    "The http://api.open-notify.org/iss-pass.json endpoint tells the next times that the international space station will pass over a given location on the earth.\n",
    "\n",
    "It requires parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RESPONSE CODE:400\n",
      "{'message': 'failure', 'reason': 'Latitude must be specified'}\n"
     ]
    }
   ],
   "source": [
    "response = requests.get(\"http://api.open-notify.org/iss-pass.json\")\n",
    "print(\"RESPONSE CODE:\" + str(response.status_code))\n",
    "print(response.json())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's read the docs: \n",
    "* http://open-notify.org/Open-Notify-API/ISS-Pass-Times/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RESPONSE CODE:200\n",
      "{'message': 'success', 'request': {'altitude': 100, 'datetime': 1572331818, 'latitude': 51.1983, 'longitude': 111.6513, 'passes': 5}, 'response': [{'duration': 452, 'risetime': 1572377032}, {'duration': 630, 'risetime': 1572382697}, {'duration': 656, 'risetime': 1572388476}, {'duration': 655, 'risetime': 1572394283}, {'duration': 633, 'risetime': 1572400087}]}\n"
     ]
    }
   ],
   "source": [
    "response = requests.get(\"http://api.open-notify.org/iss-pass.json?lat=51.1983&lon=111.6513\")\n",
    "print(\"RESPONSE CODE:\" + str(response.status_code))\n",
    "print(response.json())\n",
    "#35.1983, 111.6513"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"message\": \"success\",\n",
      "  \"request\": {\n",
      "    \"altitude\": 100,\n",
      "    \"datetime\": 1572331818,\n",
      "    \"latitude\": 51.1983,\n",
      "    \"longitude\": 111.6513,\n",
      "    \"passes\": 5\n",
      "  },\n",
      "  \"response\": [\n",
      "    {\n",
      "      \"duration\": 452,\n",
      "      \"risetime\": 1572377032\n",
      "    },\n",
      "    {\n",
      "      \"duration\": 630,\n",
      "      \"risetime\": 1572382697\n",
      "    },\n",
      "    {\n",
      "      \"duration\": 656,\n",
      "      \"risetime\": 1572388476\n",
      "    },\n",
      "    {\n",
      "      \"duration\": 655,\n",
      "      \"risetime\": 1572394283\n",
      "    },\n",
      "    {\n",
      "      \"duration\": 633,\n",
      "      \"risetime\": 1572400087\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "formatted_json = json.dumps(response.json(), sort_keys=False, indent=2)\n",
    "print(formatted_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let’s deal with the pass times from our JSON object\n",
    "\n",
    "Reading the docs (and looking at our JSON), we can see what we need to do"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1572377032, 1572382697, 1572388476, 1572394283, 1572400087]\n"
     ]
    }
   ],
   "source": [
    "times = []\n",
    "\n",
    "for item in response.json()['response']:\n",
    "    times.append(item['risetime'])\n",
    "    \n",
    "print(times)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just to finish, what are these number?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's play: your turn\n",
    "\n",
    "Look at this API:\n",
    "* https://earthquake.usgs.gov/fdsnws/event/1/\n",
    "\n",
    "I want you to \n",
    "1. use filters to get the earthquakes from the previous 60 days, with magnitude between 5.8 and 7.\n",
    "2. print the name, date, and magnitude of each of them\n",
    "3. find the highest magnitude\n",
    "4. using the ISSS API, show when the satelite will go through the place where the earthquake with the highest magnitude happened\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}