data-ingestion-and-preparation/parquet-read-write.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading and Writing Parquet Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Overview](#overview)\n",
    "- [Setup](#setup)\n",
    "- [Write Data to a Parquet Table in the Platform](#write-data-to-a-parquet-platform-table)\n",
    "- [Advanced Usage](#advanced-usage)\n",
    "- [Cleanup](#cleanup)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"overview\"></a>\n",
    "## Overview\n",
    "\n",
    "[Parquet](https://parquet.apache.org/) is a columnar storage format that provides high-density high-performance file organization.\n",
    "This tutorial demonstrates how to create and write data to a Parquet table in the Iguazio Data Science Platform (**\"the platform\"**) and read data from the table.\n",
    "For information about reading an writing Parquet files from Python applications, see https://arrow.apache.org/docs/python/parquet.html."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"setup\"></a>\n",
    "## Setup\n",
    "\n",
    "Run the following code to import required libraries and ingest CSV data into a pandas DataFrame, which will be converted to a Parquet table in the next step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyarrow.parquet as pq\n",
    "import pandas as pd\n",
    "import pyarrow as pa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>radius_mean</th>\n",
       "      <th>texture_mean</th>\n",
       "      <th>perimeter_mean</th>\n",
       "      <th>area_mean</th>\n",
       "      <th>smoothness_mean</th>\n",
       "      <th>compactness_mean</th>\n",
       "      <th>concavity_mean</th>\n",
       "      <th>concave points_mean</th>\n",
       "      <th>...</th>\n",
       "      <th>radius_worst</th>\n",
       "      <th>texture_worst</th>\n",
       "      <th>perimeter_worst</th>\n",
       "      <th>area_worst</th>\n",
       "      <th>smoothness_worst</th>\n",
       "      <th>compactness_worst</th>\n",
       "      <th>concavity_worst</th>\n",
       "      <th>concave points_worst</th>\n",
       "      <th>symmetry_worst</th>\n",
       "      <th>fractal_dimension_worst</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>842302</td>\n",
       "      <td>M</td>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>...</td>\n",
       "      <td>25.38</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>842517</td>\n",
       "      <td>M</td>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>...</td>\n",
       "      <td>24.99</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>84300903</td>\n",
       "      <td>M</td>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>...</td>\n",
       "      <td>23.57</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>84348301</td>\n",
       "      <td>M</td>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>...</td>\n",
       "      <td>14.91</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>84358402</td>\n",
       "      <td>M</td>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>...</td>\n",
       "      <td>22.54</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         id diagnosis  radius_mean  texture_mean  perimeter_mean  area_mean  \\\n",
       "0    842302         M        17.99         10.38          122.80     1001.0   \n",
       "1    842517         M        20.57         17.77          132.90     1326.0   \n",
       "2  84300903         M        19.69         21.25          130.00     1203.0   \n",
       "3  84348301         M        11.42         20.38           77.58      386.1   \n",
       "4  84358402         M        20.29         14.34          135.10     1297.0   \n",
       "\n",
       "   smoothness_mean  compactness_mean  concavity_mean  concave points_mean  \\\n",
       "0          0.11840           0.27760          0.3001              0.14710   \n",
       "1          0.08474           0.07864          0.0869              0.07017   \n",
       "2          0.10960           0.15990          0.1974              0.12790   \n",
       "3          0.14250           0.28390          0.2414              0.10520   \n",
       "4          0.10030           0.13280          0.1980              0.10430   \n",
       "\n",
       "   ...  radius_worst  texture_worst  perimeter_worst  area_worst  \\\n",
       "0  ...         25.38          17.33           184.60      2019.0   \n",
       "1  ...         24.99          23.41           158.80      1956.0   \n",
       "2  ...         23.57          25.53           152.50      1709.0   \n",
       "3  ...         14.91          26.50            98.87       567.7   \n",
       "4  ...         22.54          16.67           152.20      1575.0   \n",
       "\n",
       "   smoothness_worst  compactness_worst  concavity_worst  concave points_worst  \\\n",
       "0            0.1622             0.6656           0.7119                0.2654   \n",
       "1            0.1238             0.1866           0.2416                0.1860   \n",
       "2            0.1444             0.4245           0.4504                0.2430   \n",
       "3            0.2098             0.8663           0.6869                0.2575   \n",
       "4            0.1374             0.2050           0.4000                0.1625   \n",
       "\n",
       "   symmetry_worst  fractal_dimension_worst  \n",
       "0          0.4601                  0.11890  \n",
       "1          0.2750                  0.08902  \n",
       "2          0.3613                  0.08758  \n",
       "3          0.6638                  0.17300  \n",
       "4          0.2364                  0.07678  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read a CSV file into a pandas DataFrame and display the data and metadata that was read\n",
    "data = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data', header = None)\n",
    "\n",
    "# Specify columns extracted from wbdc.names\n",
    "data.columns = [\"id\",\"diagnosis\",\"radius_mean\",\"texture_mean\",\"perimeter_mean\",\"area_mean\",\"smoothness_mean\",\n",
    "                \"compactness_mean\",\"concavity_mean\",\"concave points_mean\",\"symmetry_mean\",\"fractal_dimension_mean\",\n",
    "                \"radius_se\",\"texture_se\",\"perimeter_se\",\"area_se\",\"smoothness_se\",\"compactness_se\",\"concavity_se\",\n",
    "                \"concave points_se\",\"symmetry_se\",\"fractal_dimension_se\",\"radius_worst\",\"texture_worst\",\n",
    "                \"perimeter_worst\",\"area_worst\",\"smoothness_worst\",\"compactness_worst\",\"concavity_worst\",\n",
    "                \"concave points_worst\",\"symmetry_worst\",\"fractal_dimension_worst\"] \n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"write-data-to-a-parquet-platform-table\"></a>\n",
    "## Write Data to a Parquet Table in the Platform\n",
    "\n",
    "Write the CSV data that was read into the pandas DataFrame to a Parquet table in a platform data container (i.e., in the distributed file system of the Iguazio Data Science Platform).\n",
    "\n",
    "> **Note:** For information about using the `v3io` or `User` data mounts to reference data in the platform's data containers, see [Platform Data Containers](README.ipynb#platform-data-containers) in the **data-ingestion-and-preparation/basic-data-ingestion-and-preparation.ipynb** notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write the pandas DataFrame to a new Parquet table in the platform's file system\n",
    "import os\n",
    "filepath = os.path.join('/User/examples/parquet_example')\n",
    "\n",
    "table = pa.Table.from_pandas(data)\n",
    "pq.write_table(table, filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>radius_mean</th>\n",
       "      <th>texture_mean</th>\n",
       "      <th>perimeter_mean</th>\n",
       "      <th>area_mean</th>\n",
       "      <th>smoothness_mean</th>\n",
       "      <th>compactness_mean</th>\n",
       "      <th>concavity_mean</th>\n",
       "      <th>concave points_mean</th>\n",
       "      <th>...</th>\n",
       "      <th>radius_worst</th>\n",
       "      <th>texture_worst</th>\n",
       "      <th>perimeter_worst</th>\n",
       "      <th>area_worst</th>\n",
       "      <th>smoothness_worst</th>\n",
       "      <th>compactness_worst</th>\n",
       "      <th>concavity_worst</th>\n",
       "      <th>concave points_worst</th>\n",
       "      <th>symmetry_worst</th>\n",
       "      <th>fractal_dimension_worst</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>842302</td>\n",
       "      <td>M</td>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>...</td>\n",
       "      <td>25.38</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>842517</td>\n",
       "      <td>M</td>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>...</td>\n",
       "      <td>24.99</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>84300903</td>\n",
       "      <td>M</td>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>...</td>\n",
       "      <td>23.57</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>84348301</td>\n",
       "      <td>M</td>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>...</td>\n",
       "      <td>14.91</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>84358402</td>\n",
       "      <td>M</td>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>...</td>\n",
       "      <td>22.54</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         id diagnosis  radius_mean  texture_mean  perimeter_mean  area_mean  \\\n",
       "0    842302         M        17.99         10.38          122.80     1001.0   \n",
       "1    842517         M        20.57         17.77          132.90     1326.0   \n",
       "2  84300903         M        19.69         21.25          130.00     1203.0   \n",
       "3  84348301         M        11.42         20.38           77.58      386.1   \n",
       "4  84358402         M        20.29         14.34          135.10     1297.0   \n",
       "\n",
       "   smoothness_mean  compactness_mean  concavity_mean  concave points_mean  \\\n",
       "0          0.11840           0.27760          0.3001              0.14710   \n",
       "1          0.08474           0.07864          0.0869              0.07017   \n",
       "2          0.10960           0.15990          0.1974              0.12790   \n",
       "3          0.14250           0.28390          0.2414              0.10520   \n",
       "4          0.10030           0.13280          0.1980              0.10430   \n",
       "\n",
       "   ...  radius_worst  texture_worst  perimeter_worst  area_worst  \\\n",
       "0  ...         25.38          17.33           184.60      2019.0   \n",
       "1  ...         24.99          23.41           158.80      1956.0   \n",
       "2  ...         23.57          25.53           152.50      1709.0   \n",
       "3  ...         14.91          26.50            98.87       567.7   \n",
       "4  ...         22.54          16.67           152.20      1575.0   \n",
       "\n",
       "   smoothness_worst  compactness_worst  concavity_worst  concave points_worst  \\\n",
       "0            0.1622             0.6656           0.7119                0.2654   \n",
       "1            0.1238             0.1866           0.2416                0.1860   \n",
       "2            0.1444             0.4245           0.4504                0.2430   \n",
       "3            0.2098             0.8663           0.6869                0.2575   \n",
       "4            0.1374             0.2050           0.4000                0.1625   \n",
       "\n",
       "   symmetry_worst  fractal_dimension_worst  \n",
       "0          0.4601                  0.11890  \n",
       "1          0.2750                  0.08902  \n",
       "2          0.3613                  0.08758  \n",
       "3          0.6638                  0.17300  \n",
       "4          0.2364                  0.07678  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read the Parquet table to local memory and print the head \n",
    "table2 = pq.read_table(filepath)\n",
    "table2.to_pandas().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"advanced-usage\"></a>\n",
    "## Advanced Usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>area_mean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>842302</td>\n",
       "      <td>1001.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>842517</td>\n",
       "      <td>1326.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>84300903</td>\n",
       "      <td>1203.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>84348301</td>\n",
       "      <td>386.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>84358402</td>\n",
       "      <td>1297.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id  area_mean\n",
       "0    842302     1001.0\n",
       "1    842517     1326.0\n",
       "2  84300903     1203.0\n",
       "3  84348301      386.1\n",
       "4  84358402     1297.0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read specific columns (attributes) from the Parquet table to save bandwidth/memory and accelerate load\n",
    "pq.read_table(filepath, columns=['id', 'area_mean']).to_pandas().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>area_mean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>842302</td>\n",
       "      <td>1001.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>842517</td>\n",
       "      <td>1326.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>84300903</td>\n",
       "      <td>1203.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>84348301</td>\n",
       "      <td>386.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>84358402</td>\n",
       "      <td>1297.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id  area_mean\n",
       "0    842302     1001.0\n",
       "1    842517     1326.0\n",
       "2  84300903     1203.0\n",
       "3  84348301      386.1\n",
       "4  84358402     1297.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For comparison, read the same columns from the pandas DataFrame that was used to create the Parequet table\n",
    "pq.read_pandas(filepath, columns=['id', 'area_mean']).to_pandas().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pyarrow._parquet.ParquetSchema object at 0x7f0b95648f98>\n",
       "id: INT64\n",
       "diagnosis: BYTE_ARRAY String\n",
       "radius_mean: DOUBLE\n",
       "texture_mean: DOUBLE\n",
       "perimeter_mean: DOUBLE\n",
       "area_mean: DOUBLE\n",
       "smoothness_mean: DOUBLE\n",
       "compactness_mean: DOUBLE\n",
       "concavity_mean: DOUBLE\n",
       "concave points_mean: DOUBLE\n",
       "symmetry_mean: DOUBLE\n",
       "fractal_dimension_mean: DOUBLE\n",
       "radius_se: DOUBLE\n",
       "texture_se: DOUBLE\n",
       "perimeter_se: DOUBLE\n",
       "area_se: DOUBLE\n",
       "smoothness_se: DOUBLE\n",
       "compactness_se: DOUBLE\n",
       "concavity_se: DOUBLE\n",
       "concave points_se: DOUBLE\n",
       "symmetry_se: DOUBLE\n",
       "fractal_dimension_se: DOUBLE\n",
       "radius_worst: DOUBLE\n",
       "texture_worst: DOUBLE\n",
       "perimeter_worst: DOUBLE\n",
       "area_worst: DOUBLE\n",
       "smoothness_worst: DOUBLE\n",
       "compactness_worst: DOUBLE\n",
       "concavity_worst: DOUBLE\n",
       "concave points_worst: DOUBLE\n",
       "symmetry_worst: DOUBLE\n",
       "fractal_dimension_worst: DOUBLE\n",
       " "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read the table as a Parquet file and display its schema\n",
    "parquet_file = pq.ParquetFile(filepath)\n",
    "parquet_file.schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pyarrow._parquet.FileMetaData object at 0x7f0b95655728>\n",
       "  created_by: parquet-cpp version 1.5.1-SNAPSHOT\n",
       "  num_columns: 32\n",
       "  num_rows: 569\n",
       "  num_row_groups: 1\n",
       "  format_version: 1.0\n",
       "  serialized_size: 16745"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display the table metadata stored in the Parquet file\n",
    "parquet_file.metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write data to multiple table partitions \n",
    "pq.write_to_dataset(table, root_path='my_prqt', partition_cols=['radius_mean'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>texture_mean</th>\n",
       "      <th>perimeter_mean</th>\n",
       "      <th>area_mean</th>\n",
       "      <th>smoothness_mean</th>\n",
       "      <th>compactness_mean</th>\n",
       "      <th>concavity_mean</th>\n",
       "      <th>concave points_mean</th>\n",
       "      <th>symmetry_mean</th>\n",
       "      <th>...</th>\n",
       "      <th>texture_worst</th>\n",
       "      <th>perimeter_worst</th>\n",
       "      <th>area_worst</th>\n",
       "      <th>smoothness_worst</th>\n",
       "      <th>compactness_worst</th>\n",
       "      <th>concavity_worst</th>\n",
       "      <th>concave points_worst</th>\n",
       "      <th>symmetry_worst</th>\n",
       "      <th>fractal_dimension_worst</th>\n",
       "      <th>radius_mean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>907367</td>\n",
       "      <td>B</td>\n",
       "      <td>21.28</td>\n",
       "      <td>63.19</td>\n",
       "      <td>307.3</td>\n",
       "      <td>0.08117</td>\n",
       "      <td>0.03912</td>\n",
       "      <td>0.002470</td>\n",
       "      <td>0.005159</td>\n",
       "      <td>0.1630</td>\n",
       "      <td>...</td>\n",
       "      <td>28.94</td>\n",
       "      <td>69.92</td>\n",
       "      <td>376.3</td>\n",
       "      <td>0.1126</td>\n",
       "      <td>0.07094</td>\n",
       "      <td>0.01235</td>\n",
       "      <td>0.02579</td>\n",
       "      <td>0.2349</td>\n",
       "      <td>0.08061</td>\n",
       "      <td>10.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>897880</td>\n",
       "      <td>B</td>\n",
       "      <td>17.53</td>\n",
       "      <td>64.41</td>\n",
       "      <td>310.8</td>\n",
       "      <td>0.10070</td>\n",
       "      <td>0.07326</td>\n",
       "      <td>0.025110</td>\n",
       "      <td>0.017750</td>\n",
       "      <td>0.1890</td>\n",
       "      <td>...</td>\n",
       "      <td>26.84</td>\n",
       "      <td>71.98</td>\n",
       "      <td>384.0</td>\n",
       "      <td>0.1402</td>\n",
       "      <td>0.14020</td>\n",
       "      <td>0.10550</td>\n",
       "      <td>0.06499</td>\n",
       "      <td>0.2894</td>\n",
       "      <td>0.07664</td>\n",
       "      <td>10.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>874158</td>\n",
       "      <td>B</td>\n",
       "      <td>15.11</td>\n",
       "      <td>63.76</td>\n",
       "      <td>317.5</td>\n",
       "      <td>0.09267</td>\n",
       "      <td>0.04695</td>\n",
       "      <td>0.001597</td>\n",
       "      <td>0.002404</td>\n",
       "      <td>0.1703</td>\n",
       "      <td>...</td>\n",
       "      <td>21.18</td>\n",
       "      <td>75.39</td>\n",
       "      <td>437.0</td>\n",
       "      <td>0.1521</td>\n",
       "      <td>0.10190</td>\n",
       "      <td>0.00692</td>\n",
       "      <td>0.01042</td>\n",
       "      <td>0.2933</td>\n",
       "      <td>0.07697</td>\n",
       "      <td>10.08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>924964</td>\n",
       "      <td>B</td>\n",
       "      <td>19.59</td>\n",
       "      <td>64.73</td>\n",
       "      <td>311.7</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.07504</td>\n",
       "      <td>0.005025</td>\n",
       "      <td>0.011160</td>\n",
       "      <td>0.1791</td>\n",
       "      <td>...</td>\n",
       "      <td>22.88</td>\n",
       "      <td>67.88</td>\n",
       "      <td>347.3</td>\n",
       "      <td>0.1265</td>\n",
       "      <td>0.12000</td>\n",
       "      <td>0.01005</td>\n",
       "      <td>0.02232</td>\n",
       "      <td>0.2262</td>\n",
       "      <td>0.06742</td>\n",
       "      <td>10.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>858970</td>\n",
       "      <td>B</td>\n",
       "      <td>14.88</td>\n",
       "      <td>64.55</td>\n",
       "      <td>311.9</td>\n",
       "      <td>0.11340</td>\n",
       "      <td>0.08061</td>\n",
       "      <td>0.010840</td>\n",
       "      <td>0.012900</td>\n",
       "      <td>0.2743</td>\n",
       "      <td>...</td>\n",
       "      <td>17.45</td>\n",
       "      <td>69.86</td>\n",
       "      <td>368.6</td>\n",
       "      <td>0.1275</td>\n",
       "      <td>0.09866</td>\n",
       "      <td>0.02168</td>\n",
       "      <td>0.02579</td>\n",
       "      <td>0.3557</td>\n",
       "      <td>0.08020</td>\n",
       "      <td>10.17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       id diagnosis  texture_mean  perimeter_mean  area_mean  smoothness_mean  \\\n",
       "0  907367         B         21.28           63.19      307.3          0.08117   \n",
       "1  897880         B         17.53           64.41      310.8          0.10070   \n",
       "2  874158         B         15.11           63.76      317.5          0.09267   \n",
       "3  924964         B         19.59           64.73      311.7          0.10030   \n",
       "4  858970         B         14.88           64.55      311.9          0.11340   \n",
       "\n",
       "   compactness_mean  concavity_mean  concave points_mean  symmetry_mean  ...  \\\n",
       "0           0.03912        0.002470             0.005159         0.1630  ...   \n",
       "1           0.07326        0.025110             0.017750         0.1890  ...   \n",
       "2           0.04695        0.001597             0.002404         0.1703  ...   \n",
       "3           0.07504        0.005025             0.011160         0.1791  ...   \n",
       "4           0.08061        0.010840             0.012900         0.2743  ...   \n",
       "\n",
       "   texture_worst  perimeter_worst  area_worst  smoothness_worst  \\\n",
       "0          28.94            69.92       376.3            0.1126   \n",
       "1          26.84            71.98       384.0            0.1402   \n",
       "2          21.18            75.39       437.0            0.1521   \n",
       "3          22.88            67.88       347.3            0.1265   \n",
       "4          17.45            69.86       368.6            0.1275   \n",
       "\n",
       "   compactness_worst  concavity_worst  concave points_worst  symmetry_worst  \\\n",
       "0            0.07094          0.01235               0.02579          0.2349   \n",
       "1            0.14020          0.10550               0.06499          0.2894   \n",
       "2            0.10190          0.00692               0.01042          0.2933   \n",
       "3            0.12000          0.01005               0.02232          0.2262   \n",
       "4            0.09866          0.02168               0.02579          0.3557   \n",
       "\n",
       "   fractal_dimension_worst  radius_mean  \n",
       "0                  0.08061        10.03  \n",
       "1                  0.07664        10.05  \n",
       "2                  0.07697        10.08  \n",
       "3                  0.06742        10.16  \n",
       "4                  0.08020        10.17  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read the partitioned table\n",
    "table3 = pq.read_table('my_prqt')\n",
    "table3.to_pandas().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>texture_mean</th>\n",
       "      <th>perimeter_mean</th>\n",
       "      <th>area_mean</th>\n",
       "      <th>smoothness_mean</th>\n",
       "      <th>compactness_mean</th>\n",
       "      <th>concavity_mean</th>\n",
       "      <th>concave points_mean</th>\n",
       "      <th>symmetry_mean</th>\n",
       "      <th>...</th>\n",
       "      <th>texture_worst</th>\n",
       "      <th>perimeter_worst</th>\n",
       "      <th>area_worst</th>\n",
       "      <th>smoothness_worst</th>\n",
       "      <th>compactness_worst</th>\n",
       "      <th>concavity_worst</th>\n",
       "      <th>concave points_worst</th>\n",
       "      <th>symmetry_worst</th>\n",
       "      <th>fractal_dimension_worst</th>\n",
       "      <th>radius_mean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>907367</td>\n",
       "      <td>B</td>\n",
       "      <td>21.28</td>\n",
       "      <td>63.19</td>\n",
       "      <td>307.3</td>\n",
       "      <td>0.08117</td>\n",
       "      <td>0.03912</td>\n",
       "      <td>0.002470</td>\n",
       "      <td>0.005159</td>\n",
       "      <td>0.1630</td>\n",
       "      <td>...</td>\n",
       "      <td>28.94</td>\n",
       "      <td>69.92</td>\n",
       "      <td>376.3</td>\n",
       "      <td>0.1126</td>\n",
       "      <td>0.07094</td>\n",
       "      <td>0.01235</td>\n",
       "      <td>0.02579</td>\n",
       "      <td>0.2349</td>\n",
       "      <td>0.08061</td>\n",
       "      <td>10.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>897880</td>\n",
       "      <td>B</td>\n",
       "      <td>17.53</td>\n",
       "      <td>64.41</td>\n",
       "      <td>310.8</td>\n",
       "      <td>0.10070</td>\n",
       "      <td>0.07326</td>\n",
       "      <td>0.025110</td>\n",
       "      <td>0.017750</td>\n",
       "      <td>0.1890</td>\n",
       "      <td>...</td>\n",
       "      <td>26.84</td>\n",
       "      <td>71.98</td>\n",
       "      <td>384.0</td>\n",
       "      <td>0.1402</td>\n",
       "      <td>0.14020</td>\n",
       "      <td>0.10550</td>\n",
       "      <td>0.06499</td>\n",
       "      <td>0.2894</td>\n",
       "      <td>0.07664</td>\n",
       "      <td>10.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>874158</td>\n",
       "      <td>B</td>\n",
       "      <td>15.11</td>\n",
       "      <td>63.76</td>\n",
       "      <td>317.5</td>\n",
       "      <td>0.09267</td>\n",
       "      <td>0.04695</td>\n",
       "      <td>0.001597</td>\n",
       "      <td>0.002404</td>\n",
       "      <td>0.1703</td>\n",
       "      <td>...</td>\n",
       "      <td>21.18</td>\n",
       "      <td>75.39</td>\n",
       "      <td>437.0</td>\n",
       "      <td>0.1521</td>\n",
       "      <td>0.10190</td>\n",
       "      <td>0.00692</td>\n",
       "      <td>0.01042</td>\n",
       "      <td>0.2933</td>\n",
       "      <td>0.07697</td>\n",
       "      <td>10.08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>924964</td>\n",
       "      <td>B</td>\n",
       "      <td>19.59</td>\n",
       "      <td>64.73</td>\n",
       "      <td>311.7</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.07504</td>\n",
       "      <td>0.005025</td>\n",
       "      <td>0.011160</td>\n",
       "      <td>0.1791</td>\n",
       "      <td>...</td>\n",
       "      <td>22.88</td>\n",
       "      <td>67.88</td>\n",
       "      <td>347.3</td>\n",
       "      <td>0.1265</td>\n",
       "      <td>0.12000</td>\n",
       "      <td>0.01005</td>\n",
       "      <td>0.02232</td>\n",
       "      <td>0.2262</td>\n",
       "      <td>0.06742</td>\n",
       "      <td>10.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>858970</td>\n",
       "      <td>B</td>\n",
       "      <td>14.88</td>\n",
       "      <td>64.55</td>\n",
       "      <td>311.9</td>\n",
       "      <td>0.11340</td>\n",
       "      <td>0.08061</td>\n",
       "      <td>0.010840</td>\n",
       "      <td>0.012900</td>\n",
       "      <td>0.2743</td>\n",
       "      <td>...</td>\n",
       "      <td>17.45</td>\n",
       "      <td>69.86</td>\n",
       "      <td>368.6</td>\n",
       "      <td>0.1275</td>\n",
       "      <td>0.09866</td>\n",
       "      <td>0.02168</td>\n",
       "      <td>0.02579</td>\n",
       "      <td>0.3557</td>\n",
       "      <td>0.08020</td>\n",
       "      <td>10.17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       id diagnosis  texture_mean  perimeter_mean  area_mean  smoothness_mean  \\\n",
       "0  907367         B         21.28           63.19      307.3          0.08117   \n",
       "1  897880         B         17.53           64.41      310.8          0.10070   \n",
       "2  874158         B         15.11           63.76      317.5          0.09267   \n",
       "3  924964         B         19.59           64.73      311.7          0.10030   \n",
       "4  858970         B         14.88           64.55      311.9          0.11340   \n",
       "\n",
       "   compactness_mean  concavity_mean  concave points_mean  symmetry_mean  ...  \\\n",
       "0           0.03912        0.002470             0.005159         0.1630  ...   \n",
       "1           0.07326        0.025110             0.017750         0.1890  ...   \n",
       "2           0.04695        0.001597             0.002404         0.1703  ...   \n",
       "3           0.07504        0.005025             0.011160         0.1791  ...   \n",
       "4           0.08061        0.010840             0.012900         0.2743  ...   \n",
       "\n",
       "   texture_worst  perimeter_worst  area_worst  smoothness_worst  \\\n",
       "0          28.94            69.92       376.3            0.1126   \n",
       "1          26.84            71.98       384.0            0.1402   \n",
       "2          21.18            75.39       437.0            0.1521   \n",
       "3          22.88            67.88       347.3            0.1265   \n",
       "4          17.45            69.86       368.6            0.1275   \n",
       "\n",
       "   compactness_worst  concavity_worst  concave points_worst  symmetry_worst  \\\n",
       "0            0.07094          0.01235               0.02579          0.2349   \n",
       "1            0.14020          0.10550               0.06499          0.2894   \n",
       "2            0.10190          0.00692               0.01042          0.2933   \n",
       "3            0.12000          0.01005               0.02232          0.2262   \n",
       "4            0.09866          0.02168               0.02579          0.3557   \n",
       "\n",
       "   fractal_dimension_worst  radius_mean  \n",
       "0                  0.08061        10.03  \n",
       "1                  0.07664        10.05  \n",
       "2                  0.07697        10.08  \n",
       "3                  0.06742        10.16  \n",
       "4                  0.08020        10.17  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read data from the table using multiple threads to accelerate performance\n",
    "pq.read_table('my_prqt', use_threads=4).to_pandas().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"cleanup\"></a>\n",
    "## Cleanup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncommemt the following line and run the code to delete the example directory\n",
    "#!rm -rf $HOME/examples/parquet_example/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}