Merge pull request #127 from Sharon-iguazio/doc-gpu-demos

[DOC] GPU demos: replace gpu-prerequisites.ipynb with README.ipynb/.md; minor edits to RAPIDS NBs

demos/gpu/README.ipynb

@@ -0,0 +1,109 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# GPU Demos\n",
+    "\n",
+    "- [Overview](#gpu-demos-overview)\n",
+    "- [Horovod Demos Prerequisites](#horovod-prereq)\n",
+    "- [RAPIDS Demos Prerequisites](#rapids-prereq)\n",
+    "- [GPU Resources Note](#gpu-resources-note)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id=\"gpu-demos-overview\"></a>\n",
+    "## Overview\n",
+    "\n",
+    "The GPU demos are aimed at demonstrating how to run code over graphics processing units (GPUs) in the Iguazio Data Science Platform (**the platform**).\n",
+    "The **demos/gpu** directory includes the following:\n",
+    "\n",
+    "- A **horovod** directory with applications that use Uber's [Horovod](https://eng.uber.com/horovod/) distributed deep-learning framework, which can be used to convert a single-GPU TensorFlow, Keras, or PyTorch model-training program to a distributed program that trains the model simultaneously over multiple GPUs.\n",
+    "    The objective is to speed up your model training with minimal changes to your existing single-GPU code and without complicating the execution.\n",
+    "    Horovod code can also run over CPUs with only minor modifications.\n",
+    "    The Horovod tutorials include the following:\n",
+    "\n",
+    "    - An image-recognition demo application for execution over GPUs (**image-classification**).\n",
+    "    - A slightly modified version of the GPU image-classification demo application for execution over CPUs (**cpu/image-classification**).\n",
+    "    - Benchmark tests (**benchmark-tf.ipynb**, which executes **tf_cnn_benchmarks.py**).\n",
+    "\n",
+    "- A **rapids** directory with applications that use NVIDIA's [RAPIDS](https://rapids.ai/) open-source libraries suite for executing end-to-end data science and analytics pipelines entirely on GPUs.\n",
+    "  The RAPIDS tutorials include the following:\n",
+    "\n",
+    "    - Demo applications that use the [cuDF](https://rapidsai.github.io/projects/cudf/en/latest/index.html) RAPIDS GPU DataFrame library to perform batching and aggregation of data that's read from a Kafaka stream, and then write the results to a Parquet file.<br>\n",
+    "      The **nuclio-cudf-agg.ipynb** demo implements this by using a Nuclio serverless function while the **python-agg.ipynb** demo implements this by using a standalone Python function.\n",
+    "    - Benchmark tests that compare the performance of RAPIDS cuDF to pandas DataFrames (**benchmark-cudf-vs-pd.ipynb**)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id=\"horovod-prereq\"></a>\n",
+    "## Horovod Demos Prerequisites\n",
+    "\n",
+    "The following prerequisites must be met to successfully run the Horovod demo tutorials (**demos/horovod**):\n",
+    "\n",
+    "- The parent platform tenant has an enabled Horovod service.\n",
+    "- To run the GPU demos &mdash;\n",
+    "  - The platform environment has one or more [NVIDIA](https://www.nvidia.com/en-us/) GPUs that are available for consumption by Horovod.\n",
+    "    See also the [GPU Resources Note](#gpu-resources-note).\n",
+    "  - The Jupyter Notebook service from which the code is executed uses the **\"Jupyter Full Stack with GPU\"** flavor."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id=\"rapids-prereq\"></a>\n",
+    "## RAPIDS Demos Prerequisites\n",
+    "\n",
+    "The following prerequisites must be met to successfully run the RAPIDS demo tutorials (**demos/rapids**):\n",
+    "\n",
+    "- Your environment has one or more GPUs with the [NVIDIA Pascal](https://www.nvidia.com/en-us/geforce/products/10series/architecture/) architecture or better and [compute capability](https://developer.nvidia.com/cuda-gpus) 6.0+.\n",
+    "- At least one GPU resource has been configured for the parent Jupyter Notebook service (**Common Parameters > Resources > GPU > Limit**).\n",
+    "  See also the [GPU Resources Note](#gpu-resources-note).\n",
+    "- The Jupyter Notebook service from which the code is executed uses the **\"Jupyter Full Stack with GPU\"** flavor."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<a id=\"gpu-resources-note\"></a>\n",
+    "## GPU Resources Note\n",
+    "\n",
+    "In environments with GPUs, the platform's Jupyter Notebook service has an optional GPU-limit parameter (**Common Parameters > Resources > GPU > Limit**), which guarantees the configured number of GPUs for use by each replica.\n",
+    "While the Jupyter Notebook service is enabled, it monopolizes the configured amount of GPUs even when the GPUs aren't in use.\n",
+    "The RAPIDS demos use the GPUs that were allocated for the Jupyter Notebook service from which the code is executed.\n",
+    "However, the Horovod demos don't use the GPUs of the Jupyter Notebook service; they use GPUs that are allocated dynamically by Horovod.\n",
+    "Therefore, note that on systems with limited GPU resources you might need to reduce the amount of GPU resources allocated to the Jupyter Notebook service or set it to 0 to successfully run the Horovod code over GPUs."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

demos/gpu/README.md

@@ -0,0 +1,59 @@
+
+# GPU Demos
+
+- [Overview](#gpu-demos-overview)
+- [Horovod Demos Prerequisites](#horovod-prereq)
+- [RAPIDS Demos Prerequisites](#rapids-prereq)
+- [GPU Resources Note](#gpu-resources-note)
+
+<a id="gpu-demos-overview"></a>
+## Overview
+
+The GPU demos are aimed at demonstrating how to run code over graphics processing units (GPUs) in the Iguazio Data Science Platform (**the platform**).
+The **demos/gpu** directory includes the following:
+
+- A **horovod** directory with applications that use Uber's [Horovod](https://eng.uber.com/horovod/) distributed deep-learning framework, which can be used to convert a single-GPU TensorFlow, Keras, or PyTorch model-training program to a distributed program that trains the model simultaneously over multiple GPUs.
+    The objective is to speed up your model training with minimal changes to your existing single-GPU code and without complicating the execution.
+    Horovod code can also run over CPUs with only minor modifications.
+    The Horovod tutorials include the following:
+
+    - An image-recognition demo application for execution over GPUs (**image-classification**).
+    - A slightly modified version of the GPU image-classification demo application for execution over CPUs (**cpu/image-classification**).
+    - Benchmark tests (**benchmark-tf.ipynb**, which executes **tf_cnn_benchmarks.py**).
+
+- A **rapids** directory with applications that use NVIDIA's [RAPIDS](https://rapids.ai/) open-source libraries suite for executing end-to-end data science and analytics pipelines entirely on GPUs.
+  The RAPIDS tutorials include the following:
+
+    - Demo applications that use the [cuDF](https://rapidsai.github.io/projects/cudf/en/latest/index.html) RAPIDS GPU DataFrame library to perform batching and aggregation of data that's read from a Kafaka stream, and then write the results to a Parquet file.<br>
+      The **nuclio-cudf-agg.ipynb** demo implements this by using a Nuclio serverless function while the **python-agg.ipynb** demo implements this by using a standalone Python function.
+    - Benchmark tests that compare the performance of RAPIDS cuDF to pandas DataFrames (**benchmark-cudf-vs-pd.ipynb**).
+
+<a id="horovod-prereq"></a>
+## Horovod Demos Prerequisites
+
+The following prerequisites must be met to successfully run the Horovod demo tutorials (**demos/horovod**):
+
+- The parent platform tenant has an enabled Horovod service.
+- To run the GPU demos &mdash;
+  - The platform environment has one or more [NVIDIA](https://www.nvidia.com/en-us/) GPUs that are available for consumption by Horovod.
+    See also the [GPU Resources Note](#gpu-resources-note).
+  - The Jupyter Notebook service from which the code is executed uses the **"Jupyter Full Stack with GPU"** flavor.
+
+<a id="rapids-prereq"></a>
+## RAPIDS Demos Prerequisites
+
+The following prerequisites must be met to successfully run the RAPIDS demo tutorials (**demos/rapids**):
+
+- Your environment has one or more GPUs with the [NVIDIA Pascal](https://www.nvidia.com/en-us/geforce/products/10series/architecture/) architecture or better and [compute capability](https://developer.nvidia.com/cuda-gpus) 6.0+.
+- At least one GPU resource has been configured for the parent Jupyter Notebook service (**Common Parameters > Resources > GPU > Limit**).
+  See also the [GPU Resources Note](#gpu-resources-note).
+- The Jupyter Notebook service from which the code is executed uses the **"Jupyter Full Stack with GPU"** flavor.
+
+<a id="gpu-resources-note"></a>
+## GPU Resources Note
+
+In environments with GPUs, the platform's Jupyter Notebook service has an optional GPU-limit parameter (**Common Parameters > Resources > GPU > Limit**), which guarantees the configured number of GPUs for use by each replica.
+While the Jupyter Notebook service is enabled, it monopolizes the configured amount of GPUs even when the GPUs aren't in use.
+The RAPIDS demos use the GPUs that were allocated for the Jupyter Notebook service from which the code is executed.
+However, the Horovod demos don't use the GPUs of the Jupyter Notebook service; they use GPUs that are allocated dynamically by Horovod.
+Therefore, note that on systems with limited GPU resources you might need to reduce the amount of GPU resources allocated to the Jupyter Notebook service or set it to 0 to successfully run the Horovod code over GPUs.

demos/gpu/rapids/benchmark-cudf-vs-pd.ipynb

@@ -4,9 +4,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Benchmark pandas Versus cuDF\n",
+    "# Benchmarks Comparison — pandas Versus RAPIDS cuDF\n",
     "\n",
-    "Using `timeit` to compare benchmarks pandas and RAPIDS cuDF."
+    "This tutorial uses `timeit` to compare performance benchmarks with pandas and RAPIDS cuDF."
    ]
   },
   {

demos/gpu/rapids/nuclio-cudf-agg.ipynb

@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Nuclio Function for Unified Data Batching and Aggregation"
+    "# Nuclio Function for Unified Data Batching and Aggregation with RAPIDS cuDF"
    ]
   },
   {

demos/gpu/rapids/python-agg.ipynb

@@ -4,14 +4,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Standalone Python Function for Unified Data Batching and Aggregation"
+    "# Standalone Python Function for Unified Data Batching and Aggregation with RAPIDS cuDF"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Installations"
+    "## Installation"
    ]
   },
   {