data-ingestion-and-preparation/basic-data-ingestion-and-preparation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started with Data Ingestion and Preparation\n",
    "\n",
    "Learn how to quickly start using the Iguazio Data Science Platform to collect, ingest, and explore data.\n",
    "\n",
    "- [Overview](#gs-overview)\n",
    "- [Collecting and Ingesting Data](#gs-data-collection-and-ingestion)\n",
    "  - [Ingesting Data From an External Database to a NoSQL Table Using V3IO Frames](#ingest-from-external-db-to-no-sql-using-frames)\n",
    "  - [Ingesting Files from Amazon S3](#ingest-from-amazon-s3)\n",
    "  - [Streaming Data From an External Streaming Engine Using Nuclio](#streaming-data-from-an-external-streaming-engine-using-nuclio)\n",
    "- [Exploring and Processing Data](#gs-data-exploration-and-processing)\n",
    "  - [Exploring Data Using Spark DataFrames](#data-exploration-spark)\n",
    "  - [Exploring Data Using V3IO Frames and pandas DataFrames](#data-exploration-v3io-frames-n-pandas)\n",
    "  - [Exploring Data Using SQL](#data-exploration-sql)\n",
    "- [Data Collection and Exploration Getting-Started Example](#getting-started-example)\n",
    "  - [Step 1: Ingest a Sample CSV File from Amazon S3](#getting-started-example-step-ingest-csv)\n",
    "  - [Step 2: Convert the Sample CSV File to a NoSQL Table](#getting-started-example-step-convert-csv-to-nosql-table)\n",
    "  - [Step 3: Run Interactive SQL Queries](#getting-started-example-step-run-sql-queries)\n",
    "  - [Step 4: Convert the Data to a Parquet Table](#getting-started-example-step-convert-data-to-parquet)\n",
    "  - [Step 5: Browse the Example Container Directory](#getting-started-example-step-browse-the-examples-dir)\n",
    "- [Cleanup](#cleanup)\n",
    "  - [Delete Data](#delete-data)\n",
    "  - [Release Spark Resources](#release-spark-resources)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-overview\"></a>\n",
    "## Overview\n",
    "\n",
    "This tutorial explains and demonstrates how to collect, ingest, and explore data with the Iguazio Data Science Platform (**\"the platform\"**).<br>\n",
    "For an overview of working with data in the platform's data store and the various available methods for ingesting, storing, and manipulating data in the platform, see the data ingestion and preparation **README** ([notebook](README.ipynb) / [Markdown](README.md))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-data-collection-and-ingestion\"></a>\n",
    "## Collecting and Ingesting Data\n",
    "\n",
    "The platform supports various alternative methods for collecting and ingesting data into its data containers (i.e., its data store).\n",
    "For more information, see the [**platform-overview.ipynb**](../platform-overview.ipynb.ipynb#data-collection-and-ingestion) tutorial notebook\n",
    "The data collection and ingestion can be done as a one-time operation, using different platform APIs &mdash; which can be run from your preferred programming interface, such as an interactive web-based Jupyter or Zeppelin notebook &mdash; or as an ongoing ingestion stream, using Nuclio serverless functions.\n",
    "This section explains and demonstrates how to collect and ingest (import) data into the platform using code that's run from a Jupyter notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ingest-from-external-db-to-no-sql-using-frames\"></a>\n",
    "### Ingesting Data From an External Database to a NoSQL Table Using V3IO Frames\n",
    "\n",
    "For an example of how to collect data from an external database &mdash; such as MySQL, Oracle, and PostgreSQL &mdash; and ingest (write) it into a NoSQL table in the platform, using the V3IO Frames API, see the [read-external-db](read-external-db.ipynb) tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ingest-from-amazon-s3\"></a>\n",
    "### Ingesting Files from Amazon S3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ingest-from-amazon-s3-using-curl\"></a>\n",
    "#### Ingesting Files from Amazon S3 to the Platform File System Using curl\n",
    "\n",
    "You can use a simple [curl](https://curl.haxx.se/) command to ingest a file (object) from an external web data source, such as an Amazon S3 bucket, to the platform's distributed file system (i.e., into the platform's data store).\n",
    "This is demonstrated in the following code example and in the [getting-started example](#getting-started-example) in this notebook.\n",
    "The [spark-sql-analytics](spark-sql-analytics.ipynb) tutorial notebook demonstrates a similar ingestion using [Botocore](https://github.com/boto/botocore).\n",
    "\n",
    "The example in the following cells uses curl to read a CSV file from the Iguazio sample data-sets (`https://s3.wasabisys.com/iguazio/`) public S3 bucket and save it to an **examples/stocks** directory in the running-user directory of the predefined \"users\" data container (`/v3io/users/$V3IO_USERNAME` = `v3io/$V3IO_HOME` = `/User`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p /User/examples/stocks # <=> /v3io/${V3IO_HOME}/examples/stocks or /v3io/users/${V3IO_USERNAME}/examples/stocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100  861k  100  861k    0     0  3431k      0 --:--:-- --:--:-- --:--:-- 3431k\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "CSV_PATH=\"/User/examples/stocks/stocks.csv\"\n",
    "curl -L \"https://s3.wasabisys.com/iguazio/data/stocks/2018-03-26_BINS_XETR08.csv\" > ${CSV_PATH}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ingest-from-amazon-s3-to-nosql-table-using-v3io-frames-n-pandas\"></a>\n",
    "#### Ingesting Data from Amazon S3 to a NoSQL Table Using V3IO Frames and pandas\n",
    "\n",
    "For an example of how to import data from Amazon S3 and save it into a NoSQL table in the platform's data store by using V3IO Frames and pandas DataFrames, see the [frames](frames.ipynb) tutorial notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"streaming-data-from-an-external-streaming-engine-using-nuclio\"></a>\n",
    "### Streaming Data From an External Streaming Engine Using Nuclio\n",
    "\n",
    "To read data from an external streaming engine &mdash; such as Kafka, Kinesis, or RabbitMQ &mdash; create a Nuclio function that listens on the stream, and write the stream data to a NoSQL or time-series database (TSDB) table:\n",
    "\n",
    "1. In the dashboard's side navigation menu, select **Projects** to display the projects dashboard.\n",
    "2. Create a new Nuclio project or select an existing project.\n",
    "3. In the action toolbar, select **Create Function**.\n",
    "4. Enter a function name, select an appropriate template, such as **kafka-to-tsdb**, configure the required template parameters, and apply your changes.\n",
    "5. Select **Deploy** from the action toolbar to deploy your function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"gs-data-exploration-and-processing\"></a>\n",
    "## Exploring and Processing Data\n",
    "\n",
    "After you have ingested data into the platform's data containers, you can use various alternative methods and tools to explore and analyze the data.\n",
    "Data scientists typically use Jupyter Notebook to run the exploration phase.\n",
    "As outlined in the [**welcome**](../welcome.ipynb#data-exploration-and-processing) tutorial notebook, the platform's Jupyter Notebook service has a wide range of pre-deployed popular data science tools (such as Spark and Presto) and allows installation of additional tools and packages, enabling you to use different APIs to access the same data from a single Jupyter notebook.\n",
    "This section explains and demonstrates how to explore data in the platform from a Jupyter notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"data-exploration-spark\"></a>\n",
    "### Exploring Data using Spark DataFrames\n",
    "\n",
    "Spark is a distributed computation framework for data analytics.\n",
    "You can easily run distributed Spark jobs on you platform cluster that use Spark DataFrames to access data files (objects), tables, or streams in the platform's data store.\n",
    "For more information and examples, see the [spark-sql-analytics](spark-sql-analytics.ipynb) tutorial notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"data-exploration-v3io-frames-n-pandas\"></a>\n",
    "### Exploring Data Using V3IO Frames and pandas DataFrames\n",
    "\n",
    "Iguazio's V3IO Frames open-source data-access library provides a unified high-performance DataFrames API for accessing NoSQL, stream, and time-series data in the platform's data store.\n",
    "These DataFrames can also be used to analyze the data with pandas. \n",
    "For details and examples, see the [frames](frames.ipynb) tutorial notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"data-exploration-sql\"></a>\n",
    "### Exploring Data Using SQL\n",
    "\n",
    "You can run SQL statements (`SELECT` only) on top of NoSQL tables in the platform's data store.\n",
    "To do this, you need to use the Jupyter `%sql` or `%%sql` IPython Jupyter magic followed by an SQL statement.\n",
    "The platform supports standard ANSI SQL semantics.\n",
    "Under the hood, the SQL statements are executed via [Presto](https://trino.io/), which is a distributed SQL engine designed from the ground up for fast analytics queries.\n",
    "\n",
    "In the example in the following cell, as a preparation for the SQL query, the **stocks.csv** file that was ingested to the **users/&lt;running user&gt;/examples/stocks** platform data-container directory in the previous [Ingesting Files from Amazon S3 to the Platform](#ingest-from-amazon-s3) example is written to a **stocks_example_tab** NoSQL table in the same directory.\n",
    "Then, an SQL `SELECT` query is run on this table.\n",
    "You can also find a similar example in the [getting-started example](#getting-started-example) in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use V3IO Frames to convert the CSV file that was ingested in the AWS S3 data-collection example to a NoSQL table.\n",
    "# NOTE: Make sure to first create a V3IO Frames service from the \"Services\" page of the platform dashboard, and run the\n",
    "# \"Ingesting Files from Amazon S3 to the Platform File System Using curl\" example to create users/$V3IO_USERNAME/examples/stocks/stocks.csv.\n",
    "import pandas as pd\n",
    "import v3io_frames as v3f\n",
    "import os\n",
    "\n",
    "# Create a V3IO Frames client for the \"users\" data container\n",
    "client = v3f.Client(\"framesd:8081\", container=\"users\")\n",
    "\n",
    "# Full CSV file path\n",
    "csv_path = os.path.join(\"/User\", \"examples\", \"stocks\", \"stocks.csv\")\n",
    "# Relative NoSQL table path within the \"users\" container\n",
    "rel_nosql_table_path = os.path.join(os.getenv('V3IO_USERNAME'), \"examples\", \"stocks\", \"stocks_example_tab\")\n",
    "\n",
    "# Read the CSV file into a Pandas DataFrame\n",
    "df = pd.read_csv(csv_path, header=\"infer\")\n",
    "\n",
    "# Convert the CSV file to a NoSQL table\n",
    "client.write(\"kv\", rel_nosql_table_path, df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "    <tr>\n",
       "        <th>securitydesc</th>\n",
       "        <th>securitytype</th>\n",
       "        <th>time</th>\n",
       "        <th>isin</th>\n",
       "        <th>minprice</th>\n",
       "        <th>date</th>\n",
       "        <th>endprice</th>\n",
       "        <th>numberoftrades</th>\n",
       "        <th>mnemonic</th>\n",
       "        <th>currency</th>\n",
       "        <th>securityid</th>\n",
       "        <th>idx</th>\n",
       "        <th>maxprice</th>\n",
       "        <th>tradedvolume</th>\n",
       "        <th>startprice</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>ZALANDO SE</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:46</td>\n",
       "        <td>DE000ZAL1111</td>\n",
       "        <td>45.23</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>45.23</td>\n",
       "        <td>3</td>\n",
       "        <td>ZAL</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504863</td>\n",
       "        <td>5669</td>\n",
       "        <td>45.23</td>\n",
       "        <td>152</td>\n",
       "        <td>45.23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>PUBLITY AG NA O.N.</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:05</td>\n",
       "        <td>DE0006972508</td>\n",
       "        <td>12.96</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>13.1</td>\n",
       "        <td>4</td>\n",
       "        <td>PBY</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505068</td>\n",
       "        <td>746</td>\n",
       "        <td>13.1</td>\n",
       "        <td>537</td>\n",
       "        <td>12.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>CELGENE CORP.      DL-,01</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:47</td>\n",
       "        <td>US1510201049</td>\n",
       "        <td>69.3</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>69.3</td>\n",
       "        <td>1</td>\n",
       "        <td>CG3</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2506454</td>\n",
       "        <td>5852</td>\n",
       "        <td>69.3</td>\n",
       "        <td>20</td>\n",
       "        <td>69.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>DUERR AG O.N.</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:49</td>\n",
       "        <td>DE0005565204</td>\n",
       "        <td>90.34</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>90.34</td>\n",
       "        <td>2</td>\n",
       "        <td>DUE</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504956</td>\n",
       "        <td>6036</td>\n",
       "        <td>90.36</td>\n",
       "        <td>30</td>\n",
       "        <td>90.36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>HENKEL AG+CO.KGAA VZO</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:08</td>\n",
       "        <td>DE0006048432</td>\n",
       "        <td>104.2</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>104.2</td>\n",
       "        <td>4</td>\n",
       "        <td>HEN3</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505004</td>\n",
       "        <td>1131</td>\n",
       "        <td>104.2</td>\n",
       "        <td>648</td>\n",
       "        <td>104.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>LPKF LASER+ELECTRON.</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:45</td>\n",
       "        <td>DE0006450000</td>\n",
       "        <td>8.43</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>8.43</td>\n",
       "        <td>1</td>\n",
       "        <td>LPK</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505039</td>\n",
       "        <td>5578</td>\n",
       "        <td>8.43</td>\n",
       "        <td>225</td>\n",
       "        <td>8.43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>BAYWA AG VINK.NA. O.N.</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:41</td>\n",
       "        <td>DE0005194062</td>\n",
       "        <td>28.35</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>28.35</td>\n",
       "        <td>1</td>\n",
       "        <td>BYW6</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504903</td>\n",
       "        <td>5143</td>\n",
       "        <td>28.35</td>\n",
       "        <td>22</td>\n",
       "        <td>28.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>ISHS IV-HEALTHC.INNOV.ETF</td>\n",
       "        <td>ETF</td>\n",
       "        <td>08:28</td>\n",
       "        <td>IE00BYZK4776</td>\n",
       "        <td>5.444</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>5.444</td>\n",
       "        <td>1</td>\n",
       "        <td>2B78</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505553</td>\n",
       "        <td>3598</td>\n",
       "        <td>5.444</td>\n",
       "        <td>2100</td>\n",
       "        <td>5.444</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>VONOVIA SE NA O.N.</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:14</td>\n",
       "        <td>DE000A1ML7J1</td>\n",
       "        <td>39.18</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>39.2</td>\n",
       "        <td>9</td>\n",
       "        <td>VNA</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504501</td>\n",
       "        <td>1851</td>\n",
       "        <td>39.21</td>\n",
       "        <td>3202</td>\n",
       "        <td>39.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>HELLA GMBH+CO. KGAA O.N.</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:04</td>\n",
       "        <td>DE000A13SX22</td>\n",
       "        <td>52.0</td>\n",
       "        <td>2018-03-26</td>\n",
       "        <td>52.0</td>\n",
       "        <td>14</td>\n",
       "        <td>HLE</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504580</td>\n",
       "        <td>559</td>\n",
       "        <td>52.0</td>\n",
       "        <td>2381</td>\n",
       "        <td>52.0</td>\n",
       "    </tr>\n",
       "</table>"
      ],
      "text/plain": [
       "[('ZALANDO SE', 'Common stock', '08:46', 'DE000ZAL1111', 45.23, '2018-03-26', 45.23, 3, 'ZAL', 'EUR', 2504863, 5669, 45.23, 152, 45.23),\n",
       " ('PUBLITY AG NA O.N.', 'Common stock', '08:05', 'DE0006972508', 12.96, '2018-03-26', 13.1, 4, 'PBY', 'EUR', 2505068, 746, 13.1, 537, 12.96),\n",
       " ('CELGENE CORP.      DL-,01', 'Common stock', '08:47', 'US1510201049', 69.3, '2018-03-26', 69.3, 1, 'CG3', 'EUR', 2506454, 5852, 69.3, 20, 69.3),\n",
       " ('DUERR AG O.N.', 'Common stock', '08:49', 'DE0005565204', 90.34, '2018-03-26', 90.34, 2, 'DUE', 'EUR', 2504956, 6036, 90.36, 30, 90.36),\n",
       " ('HENKEL AG+CO.KGAA VZO', 'Common stock', '08:08', 'DE0006048432', 104.2, '2018-03-26', 104.2, 4, 'HEN3', 'EUR', 2505004, 1131, 104.2, 648, 104.2),\n",
       " ('LPKF LASER+ELECTRON.', 'Common stock', '08:45', 'DE0006450000', 8.43, '2018-03-26', 8.43, 1, 'LPK', 'EUR', 2505039, 5578, 8.43, 225, 8.43),\n",
       " ('BAYWA AG VINK.NA. O.N.', 'Common stock', '08:41', 'DE0005194062', 28.35, '2018-03-26', 28.35, 1, 'BYW6', 'EUR', 2504903, 5143, 28.35, 22, 28.35),\n",
       " ('ISHS IV-HEALTHC.INNOV.ETF', 'ETF', '08:28', 'IE00BYZK4776', 5.444, '2018-03-26', 5.444, 1, '2B78', 'EUR', 2505553, 3598, 5.444, 2100, 5.444),\n",
       " ('VONOVIA SE NA O.N.', 'Common stock', '08:14', 'DE000A1ML7J1', 39.18, '2018-03-26', 39.2, 9, 'VNA', 'EUR', 2504501, 1851, 39.21, 3202, 39.18),\n",
       " ('HELLA GMBH+CO. KGAA O.N.', 'Common stock', '08:04', 'DE000A13SX22', 52.0, '2018-03-26', 52.0, 14, 'HLE', 'EUR', 2504580, 559, 52.0, 2381, 52.0)]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use Presto to query the NoSQL table that was created in the previous step\n",
    "presto_nosql_table_path = os.path.join('v3io.users.\"' + os.getenv('V3IO_USERNAME'), 'examples', \"stocks\", 'stocks_example_tab\"')\n",
    "%sql select * from $presto_nosql_table_path limit 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"getting-started-example\"></a>\n",
    "## Data Collection and Exploration Getting-Started Example\n",
    "\n",
    "This section demonstrates a data collection, ingestion, and exploration flow.\n",
    "Follow the tutorial by running the code cells in order of appearance:\n",
    "- [Step #1](#getting-started-example-step-ingest-csv) &mdash; a CSV file is read from an Amazon S3 bucket and saved into an examples data-container directory using curl.\n",
    "  The examples directory is first created by using a file-system command.\n",
    "- [Step #2](#getting-started-example-step-convert-csv-to-nosql-table) &mdash; the ingested file is converted into a NoSQL table by using Spark DataFrames.\n",
    "- [Step #3](#getting-started-example-step-run-sql-queries) &mdash; a Presto SQL query is run on the NoSQL table.\n",
    "- [Step #4](#getting-started-example-step-convert-data-to-parquet) &mdash; the ingested CSV file is converted into a Parquet table by using Spark DataFrames.\n",
    "- [Step #5](#getting-started-example-step-browse-the-examples-dir) &mdash; the examples container directory is browsed by using local and Hadoop file-system commands.\n",
    "- At the end of the flow, you can optionally [delete](#getting-started-example-deleting-data) the examples directory using a file-system command.\n",
    "\n",
    "> **Tip:** You can also browse the files and directories that you write to the \"users\" container in this tutorial from the platform dashboard: in the side navigation menu, select **Data**, and then select the **users** container from the table. On the container data page, select the **Browse** tab, and then use the side directory-navigation tree to browse the directories. Selecting a file or directory in the browse table displays its metadata."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"getting-started-example-step-ingest-csv\"></a>\n",
    "### Step 1: Ingest a Sample CSV File from Amazon S3\n",
    "\n",
    "Use `curl` to download a sample stocks-data CSV file from the Iguazio sample data-sets (`https://s3.wasabisys.com/iguazio/`) public S3 bucket.\n",
    "For additional public data sets, check out [Registry of Open Data on AWS](https://registry.opendata.aws/).\n",
    "\n",
    "> **NOTE:** All the platform tutorial notebook examples store the data in an **examples/stocks** directory in the running-user directory of the predefined \"users\" container &mdash; **users/&lt;running user&gt;/examples/stocks**.\n",
    "> The running-user directory is automatically created by the Jupyter Notebook service.\n",
    "> The `V3IO_HOME` environment variable is used to reference the **users/&lt;running user&gt;** directory.\n",
    "> To save the data to a different root container directory or to a different container, you need to specify the data path in the local file-system commands as `/v3io/<container name>/<data path>`, and in Spark DataFrames or Hadoop FS commands as a fully qualified path of the format `v3io://<container name>/<table path>`.\n",
    "> For more information, see the [v3io-mount](#v3io-mount) and [running-user directory](#running-user-dir) information [Jupyter Notebook Basics](#jupyter-notebook-basics) section of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100  861k  100  861k    0     0  3794k      0 --:--:-- --:--:-- --:--:-- 3794k\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "DIR_PATH=\"/User/examples/stocks/\" # <=> \"/v3io/${V3IO_HOME}/examples/stocks/\" or \"/v3io/users/${V3IO_USERNAME}/examples/stocks/\"\n",
    "CSV_PATH=\"${DIR_PATH}stocks.csv\"\n",
    "\n",
    "# Create the examples directory\n",
    "mkdir -p ${DIR_PATH}\n",
    "\n",
    "# Download a sample stocks CSV file from the Iguazio sample data-set Amazon S3 bucket to the examples/stocks directory\n",
    "curl -L \"https://s3.wasabisys.com/iguazio/data/stocks/2018-03-26_BINS_XETR08.csv\" > ${CSV_PATH}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"getting-started-example-step-convert-csv-to-nosql-table\"></a>\n",
    "### Step 2: Convert the Sample CSV File to a NoSQL Table\n",
    "\n",
    "Read the sample **stocks.csv** file that you downloaded and ingested in the previous step into a Spark DataFrame, and write the data in NoSQL format to a new \"stocks_tab\" table in the same container directory (**users/&lt;running user&gt;/examples/stocks/stocks_tab**). \n",
    "\n",
    "> **Note**\n",
    "> - To use the Iguazio Spark Connector, set the DataFrame's data-source format to `io.iguaz.v3io.spark.sql.kv`.\n",
    "> - The data path in the Spark DataFrame is specified by using the `V3IO_HOME_URL` environment variable, which is set to `v3io://users/<running user>`.\n",
    ">   See the [running-user directory](#running-user-dir) information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+--------+--------------------+------------+--------+----------+-------------------+-----+----------+--------+--------+--------+------------+--------------+\n",
      "|        ISIN|Mnemonic|        SecurityDesc|SecurityType|Currency|SecurityID|               Date| Time|StartPrice|MaxPrice|MinPrice|EndPrice|TradedVolume|NumberOfTrades|\n",
      "+------------+--------+--------------------+------------+--------+----------+-------------------+-----+----------+--------+--------+--------+------------+--------------+\n",
      "|CH0038389992|    BBZA|BB BIOTECH NAM.  ...|Common stock|     EUR|   2504244|2018-03-26 00:00:00|08:00|      56.4|    56.4|    56.4|    56.4|         320|             4|\n",
      "|CH0038863350|    NESR|NESTLE NAM.      ...|Common stock|     EUR|   2504245|2018-03-26 00:00:00|08:00|     63.04|   63.06|    63.0|   63.06|         314|             3|\n",
      "|LU0378438732|    C001|COMSTAGE-DAX UCIT...|         ETF|     EUR|   2504271|2018-03-26 00:00:00|08:00|    113.42|  113.42|  113.42|  113.42|         100|             1|\n",
      "|LU0411075020|    DBPD|XTR.SHORTDAX X2 D...|         ETF|     EUR|   2504272|2018-03-26 00:00:00|08:00|    4.1335|  4.1335|  4.1295|    4.13|      102993|             8|\n",
      "|LU0838782315|    XDDX|   XTR.DAX INCOME 1D|         ETF|     EUR|   2504277|2018-03-26 00:00:00|08:00|    105.14|   105.2|  105.14|   105.2|         239|             3|\n",
      "|DE000A0DJ6J9|     S92|SMA SOLAR TECHNOL.AG|Common stock|     EUR|   2504287|2018-03-26 00:00:00|08:00|     55.65|   55.65|   55.65|   55.65|         543|             3|\n",
      "|DE000A0D6554|    NDX1|      NORDEX SE O.N.|Common stock|     EUR|   2504290|2018-03-26 00:00:00|08:00|     8.182|    8.21|   8.174|    8.21|       10941|             8|\n",
      "|DE000A0F5UE8|    EXXU|IS.DJ CHINA OFFS....|         ETF|     EUR|   2504302|2018-03-26 00:00:00|08:00|     47.52|   47.52|   47.52|   47.52|         420|             1|\n",
      "|DE000A0HN5C6|    DWNI|DEUTSCHE WOHNEN S...|Common stock|     EUR|   2504314|2018-03-26 00:00:00|08:00|      36.2|   36.24|    36.2|   36.24|         580|             5|\n",
      "|DE000A0LD2U1|     AOX|ALSTRIA OFFICE RE...|Common stock|     EUR|   2504379|2018-03-26 00:00:00|08:00|     12.25|   12.25|   12.25|   12.25|        1728|             3|\n",
      "|DE000A0LR936|     ST5|           STEICO SE|Common stock|     EUR|   2504382|2018-03-26 00:00:00|08:00|     22.35|   22.35|   22.35|   22.35|         334|             1|\n",
      "|DE000A0MZ4B0|     DLX|DELIGNIT AG      ...|Common stock|     EUR|   2504390|2018-03-26 00:00:00|08:00|      10.3|    10.3|    10.3|    10.3|         850|             1|\n",
      "|DE000A0Q8NC8|    ETLX|ETFS DAXGL.G.MIN....|         ETF|     EUR|   2504397|2018-03-26 00:00:00|08:00|    17.844|  17.844|  17.838|  17.838|        3085|             5|\n",
      "|DE000A0V9YU8|    4RT3|ETFS COM.SEC.DZ08...|         ETC|     EUR|   2504421|2018-03-26 00:00:00|08:00|    5.8895|  5.8895|  5.8895|  5.8895|           0|             1|\n",
      "|DE000A0WMPJ6|    AIXA|  AIXTRON SE NA O.N.|Common stock|     EUR|   2504428|2018-03-26 00:00:00|08:00|      16.8|    16.8|   16.75|  16.755|        3329|             8|\n",
      "|DE000A0Z2XN6|     RIB|RIB SOFTWARE SE  ...|Common stock|     EUR|   2504436|2018-03-26 00:00:00|08:00|     24.66|   24.66|   24.52|   24.52|       11741|            29|\n",
      "|DE000A0Z2ZZ5|    FNTN|  FREENET AG NA O.N.|Common stock|     EUR|   2504438|2018-03-26 00:00:00|08:00|     24.41|   24.42|   24.41|   24.42|         695|             6|\n",
      "|DE000A1A6V48|     KSC|      KPS AG NA O.N.|Common stock|     EUR|   2504441|2018-03-26 00:00:00|08:00|      9.15|    9.15|    9.15|    9.15|          73|             1|\n",
      "|DE000A1DAHH0|     BNR| BRENNTAG AG NA O.N.|Common stock|     EUR|   2504453|2018-03-26 00:00:00|08:00|     48.14|   48.14|   48.14|   48.14|         185|             2|\n",
      "|DE000A1EWWW0|     ADS|   ADIDAS AG NA O.N.|Common stock|     EUR|   2504471|2018-03-26 00:00:00|08:00|     196.3|  196.35|   196.3|  196.35|         591|            12|\n",
      "+------------+--------+--------------------+------------+--------+----------+-------------------+-----+----------+--------+--------+--------+------------+--------------+\n",
      "only showing top 20 rows\n",
      "\n",
      "root\n",
      " |-- ISIN: string (nullable = true)\n",
      " |-- Mnemonic: string (nullable = true)\n",
      " |-- SecurityDesc: string (nullable = true)\n",
      " |-- SecurityType: string (nullable = true)\n",
      " |-- Currency: string (nullable = true)\n",
      " |-- SecurityID: integer (nullable = true)\n",
      " |-- Date: timestamp (nullable = true)\n",
      " |-- Time: string (nullable = true)\n",
      " |-- StartPrice: double (nullable = true)\n",
      " |-- MaxPrice: double (nullable = true)\n",
      " |-- MinPrice: double (nullable = true)\n",
      " |-- EndPrice: double (nullable = true)\n",
      " |-- TradedVolume: integer (nullable = true)\n",
      " |-- NumberOfTrades: integer (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from pyspark.sql import SparkSession\n",
    "\n",
    "# Example diretory path - a <running user>/examples/stocks directory in the \"users\" container\n",
    "dir_path = os.path.join(os.getenv(\"V3IO_HOME_URL\"), \"examples\", \"stocks\")\n",
    "# CSV file path\n",
    "csv_path = os.path.join(dir_path, \"stocks.csv\")\n",
    "# NoSQL table path\n",
    "nosql_table_path = os.path.join(dir_path, \"stocks_tab\")\n",
    "\n",
    "# Create a new Spark session\n",
    "spark = SparkSession.builder.appName(\"Iguazio data ingestion and preparation getting-started example\").getOrCreate()\n",
    "\n",
    "# Read the sample CSV file into a Spark DataFrame, and let Spark infer the schema of the data\n",
    "df = spark.read.option(\"header\", \"true\").csv(csv_path, inferSchema=\"true\")\n",
    "\n",
    "# Show the DataFrame data\n",
    "df.show()\n",
    "\n",
    "# Write the DataFrame data to a NoSQL table in a platform data container.\n",
    "# Define the \"ISIN\" column (attribute) as the table's primary key.\n",
    "df.write.format(\"io.iguaz.v3io.spark.sql.kv\").mode(\"append\") \\\n",
    "    .option(\"key\", \"ISIN\").option(\"allow-overwrite-schema\", \"true\") \\\n",
    "    .save(nosql_table_path)\n",
    "\n",
    "# Display the table schema:\n",
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"getting-started-example-step-run-sql-queries\"></a>\n",
    "### Step 3: Run Interactive SQL Queries\n",
    "\n",
    "Use the `%sql` Jupyter magic to run an SQL queries on the \"stocks_tab\" table that was created in the previous step.\n",
    "(The queries is processed using Presto.)\n",
    "The example runs a `SELECT` query that reads the first ten table items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "presto_nosql_table_path = os.path.join('v3io.users.\"' + os.getenv('V3IO_USERNAME'), 'examples', 'stocks', 'stocks_tab\"')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " * presto://iguazio:***@presto-api-presto.default-tenant.app.aopaqdknqiot.iguazio-cd2.com:443/v3io?protocol=https\n",
      "Done.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "    <tr>\n",
       "        <th>securitydesc</th>\n",
       "        <th>securitytype</th>\n",
       "        <th>time</th>\n",
       "        <th>isin</th>\n",
       "        <th>minprice</th>\n",
       "        <th>date</th>\n",
       "        <th>endprice</th>\n",
       "        <th>numberoftrades</th>\n",
       "        <th>mnemonic</th>\n",
       "        <th>currency</th>\n",
       "        <th>securityid</th>\n",
       "        <th>maxprice</th>\n",
       "        <th>tradedvolume</th>\n",
       "        <th>startprice</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>FIELMANN AG O.N.</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:00</td>\n",
       "        <td>DE0005772206</td>\n",
       "        <td>65.7</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>65.7</td>\n",
       "        <td>1</td>\n",
       "        <td>FIE</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504969</td>\n",
       "        <td>65.7</td>\n",
       "        <td>199</td>\n",
       "        <td>65.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>BIOTEST AG VZ O.N.</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:00</td>\n",
       "        <td>DE0005227235</td>\n",
       "        <td>26.5</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>26.5</td>\n",
       "        <td>1</td>\n",
       "        <td>BIO3</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504915</td>\n",
       "        <td>26.5</td>\n",
       "        <td>50</td>\n",
       "        <td>26.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>ASMALLWORLD AG SF 1</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:23</td>\n",
       "        <td>CH0404880129</td>\n",
       "        <td>12.7</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>12.7</td>\n",
       "        <td>1</td>\n",
       "        <td>1Q7</td>\n",
       "        <td>EUR</td>\n",
       "        <td>3089122</td>\n",
       "        <td>12.7</td>\n",
       "        <td>400</td>\n",
       "        <td>12.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>ISHSIV-MSCI CHINA A DL A</td>\n",
       "        <td>ETF</td>\n",
       "        <td>08:33</td>\n",
       "        <td>IE00BQT3WG13</td>\n",
       "        <td>3.5625</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>3.5625</td>\n",
       "        <td>2</td>\n",
       "        <td>36BZ</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505483</td>\n",
       "        <td>3.572</td>\n",
       "        <td>2800</td>\n",
       "        <td>3.572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>IS EURO DIVID.U.ETF EOD</td>\n",
       "        <td>ETF</td>\n",
       "        <td>08:16</td>\n",
       "        <td>IE00B0M62S72</td>\n",
       "        <td>21.44</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>21.44</td>\n",
       "        <td>1</td>\n",
       "        <td>IQQA</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2506395</td>\n",
       "        <td>21.44</td>\n",
       "        <td>105</td>\n",
       "        <td>21.44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>ISHS ESTXX BNKS.30-15 UC.</td>\n",
       "        <td>ETF</td>\n",
       "        <td>08:03</td>\n",
       "        <td>DE0006289309</td>\n",
       "        <td>12.066</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>12.07</td>\n",
       "        <td>4</td>\n",
       "        <td>EXX1</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505027</td>\n",
       "        <td>12.07</td>\n",
       "        <td>16274</td>\n",
       "        <td>12.068</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>ISHSVII-MSCI EM AS.DL ACC</td>\n",
       "        <td>ETF</td>\n",
       "        <td>08:22</td>\n",
       "        <td>IE00B5L8K969</td>\n",
       "        <td>128.68</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>128.68</td>\n",
       "        <td>1</td>\n",
       "        <td>CEBL</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2505690</td>\n",
       "        <td>128.68</td>\n",
       "        <td>7</td>\n",
       "        <td>128.68</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>ARCELORMITTAL S.A. NOUV.</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:01</td>\n",
       "        <td>LU1598757687</td>\n",
       "        <td>25.395</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>25.395</td>\n",
       "        <td>1</td>\n",
       "        <td>ARRD</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2506190</td>\n",
       "        <td>25.395</td>\n",
       "        <td>171</td>\n",
       "        <td>25.395</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>ETFS ISE CYBER SEC.GO DZ</td>\n",
       "        <td>ETF</td>\n",
       "        <td>08:15</td>\n",
       "        <td>DE000A14ZT85</td>\n",
       "        <td>10.73</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>10.73</td>\n",
       "        <td>1</td>\n",
       "        <td>USPY</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504597</td>\n",
       "        <td>10.73</td>\n",
       "        <td>500</td>\n",
       "        <td>10.73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>EINHELL GERMANY VZO O.N.</td>\n",
       "        <td>Common stock</td>\n",
       "        <td>08:05</td>\n",
       "        <td>DE0005654933</td>\n",
       "        <td>90.0</td>\n",
       "        <td>2018-03-26 00:00:00.000</td>\n",
       "        <td>90.0</td>\n",
       "        <td>1</td>\n",
       "        <td>EIN3</td>\n",
       "        <td>EUR</td>\n",
       "        <td>2504961</td>\n",
       "        <td>90.0</td>\n",
       "        <td>20</td>\n",
       "        <td>90.0</td>\n",
       "    </tr>\n",
       "</table>"
      ],
      "text/plain": [
       "[('FIELMANN AG O.N.', 'Common stock', '08:00', 'DE0005772206', 65.7, '2018-03-26 00:00:00.000', 65.7, 1, 'FIE', 'EUR', 2504969, 65.7, 199, 65.7),\n",
       " ('BIOTEST AG VZ O.N.', 'Common stock', '08:00', 'DE0005227235', 26.5, '2018-03-26 00:00:00.000', 26.5, 1, 'BIO3', 'EUR', 2504915, 26.5, 50, 26.5),\n",
       " ('ASMALLWORLD AG SF 1', 'Common stock', '08:23', 'CH0404880129', 12.7, '2018-03-26 00:00:00.000', 12.7, 1, '1Q7', 'EUR', 3089122, 12.7, 400, 12.7),\n",
       " ('ISHSIV-MSCI CHINA A DL A', 'ETF', '08:33', 'IE00BQT3WG13', 3.5625, '2018-03-26 00:00:00.000', 3.5625, 2, '36BZ', 'EUR', 2505483, 3.572, 2800, 3.572),\n",
       " ('IS EURO DIVID.U.ETF EOD', 'ETF', '08:16', 'IE00B0M62S72', 21.44, '2018-03-26 00:00:00.000', 21.44, 1, 'IQQA', 'EUR', 2506395, 21.44, 105, 21.44),\n",
       " ('ISHS ESTXX BNKS.30-15 UC.', 'ETF', '08:03', 'DE0006289309', 12.066, '2018-03-26 00:00:00.000', 12.07, 4, 'EXX1', 'EUR', 2505027, 12.07, 16274, 12.068),\n",
       " ('ISHSVII-MSCI EM AS.DL ACC', 'ETF', '08:22', 'IE00B5L8K969', 128.68, '2018-03-26 00:00:00.000', 128.68, 1, 'CEBL', 'EUR', 2505690, 128.68, 7, 128.68),\n",
       " ('ARCELORMITTAL S.A. NOUV.', 'Common stock', '08:01', 'LU1598757687', 25.395, '2018-03-26 00:00:00.000', 25.395, 1, 'ARRD', 'EUR', 2506190, 25.395, 171, 25.395),\n",
       " ('ETFS ISE CYBER SEC.GO DZ', 'ETF', '08:15', 'DE000A14ZT85', 10.73, '2018-03-26 00:00:00.000', 10.73, 1, 'USPY', 'EUR', 2504597, 10.73, 500, 10.73),\n",
       " ('EINHELL GERMANY VZO O.N.', 'Common stock', '08:05', 'DE0005654933', 90.0, '2018-03-26 00:00:00.000', 90.0, 1, 'EIN3', 'EUR', 2504961, 90.0, 20, 90.0)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%sql select * from $presto_nosql_table_path limit 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " * presto://iguazio:***@presto-api-presto.default-tenant.app.aopaqdknqiot.iguazio-cd2.com:443/v3io?protocol=https\n",
      "Done.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "    <tr>\n",
       "        <th>_col0</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "        <td>737</td>\n",
       "    </tr>\n",
       "</table>"
      ],
      "text/plain": [
       "[(737,)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%sql select count(*) from $presto_nosql_table_path "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"getting-started-example-step-convert-data-to-parquet\"></a>\n",
    "### Step 4: Convert the Data to a Parquet Table\n",
    "\n",
    "Use a Spark DataFrame `write` command to write the data in the Spark DaraFrame &mdash; which was created from the CSV file and used to create the NoSQL table in [Step 2](#getting-started-example-step-convert-csv-to-nosql-table) &mdash; to a new **users/&lt;running user&gt;/examples/stocks/stocks_prqt** Parquet table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write the DataFrame data that was read from the CSV file in Step 2 to a Parquet table in a platform data container\n",
    "prqt_table_path = os.path.join(dir_path, \"stocks_prqt\")\n",
    "df.write.mode('overwrite').parquet(prqt_table_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"getting-started-example-step-browse-the-examples-dir\"></a>\n",
    "### Step 5: Browse the Example Container Directory\n",
    "\n",
    "Use a file-system bash-shell command to list the contents of the **users/&lt;running user&gt;/examples/stocks** data-container directory to which all the ingested data in the previous steps were saved.\n",
    "You should see in this directory the **stocks.csv** file, **stocks_tab** NoSQL table directory, and **stocks_prqt** Parquet table directory that you created in the previous steps.\n",
    "The following cells demonstrate how to issue the same command using the local file system and using Hadoop FS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 862\n",
      "drwxrwxr-x 2 51 nogroup      0 Oct 18 09:18 stocks_example_tab\n",
      "-rw-r--r-- 1 51 nogroup 882055 Oct 18 09:18 stocks.csv\n",
      "drwxrwxrwx 2 51 nogroup      0 Oct 18 09:18 stocks_tab\n",
      "drwxr-xr-x 2 51 nogroup      0 Oct 18 09:18 stocks_prqt\n"
     ]
    }
   ],
   "source": [
    "# List the contents of the users/<running user>/examples/stocks directory using a local file-system command\n",
    "!ls -lrt /User/examples/stocks\n",
    "# The following are equivalent commands that demonstrate different ways to reference your user home directory:\n",
    "#!ls -lrt /v3io/${V3IO_HOME}/examples/stocks\n",
    "#!ls -lrt /v3io/users/${V3IO_USERNAME}/examples/stocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 4 items\n",
      "-rw-r--r--   1 51 nogroup     882055 2020-10-18 09:18 v3io://users/iguazio/examples/stocks/stocks.csv\n",
      "drwxrwxr-x   - 51 nogroup          0 2020-10-18 09:18 v3io://users/iguazio/examples/stocks/stocks_example_tab\n",
      "drwxr-xr-x   - 51 nogroup          0 2020-10-18 09:18 v3io://users/iguazio/examples/stocks/stocks_prqt\n",
      "drwxrwxrwx   - 51 nogroup          0 2020-10-18 09:18 v3io://users/iguazio/examples/stocks/stocks_tab\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "20/10/18 09:18:50 INFO slf_4j.Slf4jLogger: Slf4jLogger started\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "# List the contents of the users/<running user>/examples/stocks directory using an Hadoop FS command\n",
    "hadoop fs -ls ${V3IO_HOME_URL}/examples/stocks\n",
    "# The following are equivalent commands that demonstrate different ways to reference your user home directory:\n",
    "#hadoop fs -ls v3io://${V3IO_HOME}/examples/stocks\n",
    "#hadoop fs -ls v3io://users/${V3IO_USERNAME}/examples/stocks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"cleanup\"></a>\n",
    "## Cleanup\n",
    "\n",
    "Prior to exiting, release disk space, computation, and memory resources consumed by the active session:\n",
    "\n",
    "- [Delete Data](#delete-data)\n",
    "- [Release Spark Resources](#release-spark-resources)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"delete-data\"></a>\n",
    "### Delete Data\n",
    "\n",
    "Optionally delete  any of the directories or files that you created.\n",
    "See the instructions in the [Creating and Deleting Container Directories](https://www.iguazio.com/docs/v3.0/data-layer/containers/working-with-containers/#create-delete-container-dirs) tutorial.\n",
    "The following example uses a local file-system command to delete the entire contents of the **users/&lt;running user&gt;/examples/stocks** directory that was created in this example, but not the directory itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the contents of the stocks examples directory:\n",
    "#!rm -rf /User/examples/stocks/*\n",
    "# You can also delete the stocks examples directory iteself (and all its contents):\n",
    "#!rm -rf /User/examples/stocks/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"release-spark-resources\"></a>\n",
    "### Release Spark Resources\n",
    "\n",
    "When you're done, run the following command to stop your Spark session and release its computation and memory resources:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}