code/16-input-output.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "📝 **Author:** Amirhossein Heydari - 📧 **Email:** <amirhosseinheydari78@gmail.com> - 📍 **Origin:** [mr-pylin/numpy-workshop](https://github.com/mr-pylin/numpy-workshop)\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Table of contents**<a id='toc0_'></a>    \n",
    "- [Dependencies](#toc1_)    \n",
    "- [NumPy - Input and output](#toc2_)    \n",
    "  - [NumPy binary files (npy, npz)](#toc2_1_)    \n",
    "  - [Text files](#toc2_2_)    \n",
    "  - [String formatting](#toc2_3_)    \n",
    "  - [Memory mapping files](#toc2_4_)    \n",
    "  - [Text formatting options](#toc2_5_)    \n",
    "\n",
    "<!-- vscode-jupyter-toc-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=1\n",
    "\tmaxLevel=6\n",
    "\t/vscode-jupyter-toc-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc1_'></a>[Dependencies](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "rng = np.random.default_rng(seed=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc2_'></a>[NumPy - Input and output](#toc0_)\n",
    "\n",
    "📝 Doc:\n",
    "\n",
    "- Input and output: [numpy.org/doc/stable/reference/routines.io.html](https://numpy.org/doc/stable/reference/routines.io.html)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_1_'></a>[NumPy binary files (npy, npz)](#toc0_)\n",
    "\n",
    "- npy\n",
    "  - Stores a single NumPy array in binary format\n",
    "  - Efficient for loading and saving large arrays\n",
    "  - Use `np.save` to save, `np.load` to load\n",
    "- npz\n",
    "  - Stores multiple NumPy arrays in a compressed archive\n",
    "  - Use `np.savez` or `np.savez_compressed` to save, `np.load` to load.\n",
    "  - Offers compression for smaller file size\n",
    "\n",
    "<table style=\"margin: 0 auto;\">\n",
    "  <thead>\n",
    "    <tr>\n",
    "      <th style=\"text-align: center;\">Function</th>\n",
    "      <th style=\"text-align: center;\">Description</th>\n",
    "      <th style=\"text-align: center;\">Details</th>\n",
    "    </tr>\n",
    "  </thead>\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <td><code>np.load</code></td>\n",
    "      <td>Load arrays or pickled objects from <code>.npy</code>, <code>.npz</code> or pickled files</td>\n",
    "      <td style=\"text-align: center;\"><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.load.html\">link</a></td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td><code>np.save</code></td>\n",
    "      <td>Save an array to a binary file in NumPy <code>.npy</code> format</td>\n",
    "      <td style=\"text-align: center;\"><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.save.html\">link</a></td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td><code>np.savez</code></td>\n",
    "      <td>Save several arrays into a single file in uncompressed <code>.npz</code> format</td>\n",
    "      <td style=\"text-align: center;\"><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.savez.html\">link</a></td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td><code>np.savez_compressed</code></td>\n",
    "      <td>Save several arrays into a single file in compressed <code>.npz</code> format</td>\n",
    "      <td style=\"text-align: center;\"><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.savez_compressed.html\">link</a></td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "arr_1d_1 = np.array([1, 2, 3, 4, 5])\n",
    "\n",
    "# save\n",
    "np.save(\"../assets/binaries/binary_1.npy\", arr_1d_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "arr_1d_2 = np.array([1, 2, 3, 4, 5])\n",
    "arr_1d_3 = np.array([5, 4, 3, 2, 1])\n",
    "\n",
    "# savez\n",
    "np.savez(\"../assets/binaries/binary_2.npz\", arr_1d_2, arr_1d_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "arr_1d_4 = np.array([4, 5, 6, 7, 8])\n",
    "arr_1d_5 = np.array([8, 7, 6, 5, 4])\n",
    "\n",
    "# savez_compressed\n",
    "np.savez_compressed(\"../assets/binaries/binary_3.npz\", arr_1d_4, arr_1d_5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arr_1d_6 : [1 2 3 4 5]\n",
      "arr_1d_7 : [1 2 3 4 5]\n",
      "arr_1d_8 : [5 4 3 2 1]\n",
      "arr_1d_9 : [4 5 6 7 8]\n",
      "arr_1d_10: [8 7 6 5 4]\n"
     ]
    }
   ],
   "source": [
    "# load .npy\n",
    "arr_1d_6 = np.load(\"../assets/binaries/binary_1.npy\")\n",
    "\n",
    "# load .npz\n",
    "load_1 = np.load(\"../assets/binaries/binary_2.npz\")\n",
    "arr_1d_7, arr_1d_8 = load_1[\"arr_0\"], load_1[\"arr_1\"]\n",
    "\n",
    "# load .npz [compressed file]\n",
    "load_2 = np.load(\"../assets/binaries/binary_3.npz\")\n",
    "arr_1d_9, arr_1d_10 = load_2[\"arr_0\"], load_2[\"arr_1\"]\n",
    "\n",
    "# log\n",
    "print(f\"arr_1d_6 : {arr_1d_6}\")\n",
    "print(f\"arr_1d_7 : {arr_1d_7}\")\n",
    "print(f\"arr_1d_8 : {arr_1d_8}\")\n",
    "print(f\"arr_1d_9 : {arr_1d_9}\")\n",
    "print(f\"arr_1d_10: {arr_1d_10}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_2_'></a>[Text files](#toc0_)\n",
    "\n",
    "<table style=\"margin: 0 auto;\">\n",
    "  <thead>\n",
    "    <tr>\n",
    "      <th style=\"text-align: center;\">Function</th>\n",
    "      <th style=\"text-align: center;\">Description</th>\n",
    "      <th style=\"text-align: center;\">Details</th>\n",
    "    </tr>\n",
    "  </thead>\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <td><code>np.loadtxt</code></td>\n",
    "      <td>Load data from a text file</td>\n",
    "      <td style=\"text-align: center;\"><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.loadtxt.html\">link</a></td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td><code>np.savetxt</code></td>\n",
    "      <td>Save an array to a text file</td>\n",
    "      <td style=\"text-align: center;\"><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.savetxt.html\">link</a></td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td><code>np.genfromtxt</code></td>\n",
    "      <td>Load data from a text file, with missing values handled as specified</td>\n",
    "      <td style=\"text-align: center;\"><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.genfromtxt.html\">link</a></td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td><code>np.fromstring</code></td>\n",
    "      <td>A new 1-D array initialized from text data in a string</td>\n",
    "      <td style=\"text-align: center;\"><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.fromstring.html\">link</a></td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td><code>np.ndarray.tofile</code></td>\n",
    "      <td>Write array to a file as text or binary (default)</td>\n",
    "      <td style=\"text-align: center;\"><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.ndarray.tofile.html\">link</a></td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td><code>np.ndarray.tolist</code></td>\n",
    "      <td>Return the array as an <code>a.ndim</code>-levels deep nested list of Python scalars</td>\n",
    "      <td style=\"text-align: center;\"><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.ndarray.tolist.html\">link</a></td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arr_2d_2:\n",
      "[[ 1  2  3]\n",
      " [ 4  5  6]\n",
      " [ 7  8  9]\n",
      " [10 11 12]]\n"
     ]
    }
   ],
   "source": [
    "arr_2d_1 = np.array([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0], [10.0, 11.0, 12.0]])\n",
    "\n",
    "# savetxt\n",
    "np.savetxt(\"../assets/txtfiles/file_1.csv\", X=arr_2d_1, fmt=\"%i\", delimiter=\",\", header=\"A, B, C\", comments=\"\")\n",
    "\n",
    "# load txt file\n",
    "arr_2d_2 = np.loadtxt(\n",
    "    fname=\"../assets/txtfiles/file_1.csv\",\n",
    "    dtype=np.int64,\n",
    "    delimiter=\",\",\n",
    "    skiprows=1,\n",
    ")\n",
    "\n",
    "# log\n",
    "print(f\"arr_2d_2:\\n{arr_2d_2}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arr_2d_3       : [( 1.,  2.,  3.) ( 4., nan,  6.) ( 7.,  8.,  9.) (10., 11., 12.)]\n",
      "arr_2d_3.dtype : [('A', '<f8'), ('B', '<f8'), ('C', '<f8')]\n"
     ]
    }
   ],
   "source": [
    "arr_2d_2 = np.array([[1.0, 2.0, 3.0], [4.0, None, 6.0], [7.0, 8.0, 9.0], [10.0, 11.0, 12.0]])\n",
    "arr_2d_2 = np.where(arr_2d_2 == None, np.nan, arr_2d_2)\n",
    "\n",
    "# savetxt [contains null values]\n",
    "np.savetxt(\"../assets/txtfiles/file_2.csv\", X=arr_2d_2, fmt=\"%.1f\", delimiter=\",\", header=\"A, B, C\", comments=\"\")\n",
    "\n",
    "# load txt file [advancced]\n",
    "arr_2d_3 = np.genfromtxt(\n",
    "    fname=\"../assets/txtfiles/file_2.csv\",\n",
    "    dtype=np.float64,\n",
    "    delimiter=\",\",\n",
    "    names=True,\n",
    ")\n",
    "\n",
    "# log\n",
    "print(f\"arr_2d_3       : {arr_2d_3}\")\n",
    "print(f\"arr_2d_3.dtype : {arr_2d_3.dtype}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fromstring_1: [1. 2. 3. 4. 5.]\n"
     ]
    }
   ],
   "source": [
    "str_1 = \"1.0,2.0,3.0,4.0,5.0\"\n",
    "\n",
    "# fromstring\n",
    "fromstring_1 = np.fromstring(str_1, sep=\",\")\n",
    "\n",
    "# log\n",
    "print(f\"fromstring_1: {fromstring_1}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_3_'></a>[String formatting](#toc0_)\n",
    "\n",
    "<table style=\"margin: 0 auto;\">\n",
    "  <thead>\n",
    "    <tr>\n",
    "      <th style=\"text-align: center;\">Function</th>\n",
    "      <th style=\"text-align: center;\">Description</th>\n",
    "      <th style=\"text-align: center;\">Details</th>\n",
    "    </tr>\n",
    "  </thead>\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <td><code>np.array2string</code></td>\n",
    "      <td>Return a string representation of an array</td>\n",
    "      <td style=\"text-align: center;\"><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.array2string.html\">link</a></td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type(array2string_1) : <class 'str'>\n",
      "array2string_1:\n",
      "[[1 2]\n",
      " [3 4]\n",
      " [5 6]]\n"
     ]
    }
   ],
   "source": [
    "arr_2d_4 = np.array([[1, 2], [3, 4], [5, 6]])\n",
    "\n",
    "# array2string\n",
    "array2string_1 = np.array2string(arr_2d_4)\n",
    "\n",
    "# log\n",
    "print(f\"type(array2string_1) : {type(array2string_1)}\")\n",
    "print(f\"array2string_1:\\n{array2string_1}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_4_'></a>[Memory mapping files](#toc0_)\n",
    "\n",
    "- It's used when datasets are too large to fit into RAM\n",
    "- `np.memmap` allows you to work with them as if they were in memory, but without loading the entire dataset into RAM\n",
    "- You can efficiently read and process only the parts of the data you need\n",
    "\n",
    "<table style=\"margin: 0 auto;\">\n",
    "  <thead>\n",
    "    <tr>\n",
    "      <th style=\"text-align: center;\">Function</th>\n",
    "      <th style=\"text-align: center;\">Description</th>\n",
    "      <th style=\"text-align: center;\">Details</th>\n",
    "    </tr>\n",
    "  </thead>\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <td><code>np.memmap</code></td>\n",
    "      <td>Create a memory-map to an array stored in a binary file on disk</td>\n",
    "      <td style=\"text-align: center;\"><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.memmap.html\">link</a></td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "memmap_arr_1  size : 160 bytes\n",
      "memmap_arr_2  size : 160 bytes\n",
      "random_data_1 size : 928 bytes\n"
     ]
    }
   ],
   "source": [
    "# create an empty memmap array\n",
    "memmap_arr_1 = np.memmap(\"../assets/memmaps/memmap_1.dat\", dtype=np.float32, mode=\"w+\", shape=(10, 10))\n",
    "\n",
    "# fill the array with data\n",
    "random_data_1 = rng.random((10, 10))\n",
    "memmap_arr_1[:] = random_data_1\n",
    "\n",
    "# flush changes to disk\n",
    "memmap_arr_1.flush()\n",
    "\n",
    "# open the memory-mapped file in read mode\n",
    "memmap_arr_2 = np.memmap(\"../assets/memmaps/memmap_1.dat\", dtype=np.float32, mode=\"r\", shape=(10, 10))\n",
    "\n",
    "# log\n",
    "print(f\"memmap_arr_1  size : {sys.getsizeof(memmap_arr_1)} bytes\")\n",
    "print(f\"memmap_arr_2  size : {sys.getsizeof(memmap_arr_2)} bytes\")\n",
    "print(f\"random_data_1 size : {sys.getsizeof(random_data_1)} bytes\")\n",
    "\n",
    "# when done, delete the reference to the memmap array\n",
    "del memmap_arr_1\n",
    "del memmap_arr_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_5_'></a>[Text formatting options](#toc0_)\n",
    "\n",
    "<table style=\"margin: 0 auto;\">\n",
    "  <thead>\n",
    "    <tr>\n",
    "      <th style=\"text-align: center;\">Function</th>\n",
    "      <th style=\"text-align: center;\">Description</th>\n",
    "      <th style=\"text-align: center;\">Details</th>\n",
    "    </tr>\n",
    "  </thead>\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <td><code>np.set_printoptions</code></td>\n",
    "      <td>Set printing options</td>\n",
    "      <td style=\"text-align: center;\"><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.set_printoptions.html\">link</a></td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td><code>np.get_printoptions</code></td>\n",
    "      <td>Return the current print options</td>\n",
    "      <td style=\"text-align: center;\"><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.set_printoptions.html\">link</a></td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td><code>np.printoptions</code></td>\n",
    "      <td>Context manager for setting print options</td>\n",
    "      <td style=\"text-align: center;\"><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.printoptions.html\">link</a></td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arr_2d_5:\n",
      "[[1.12345679 2.98765432 3.23532788]\n",
      " [3.14159265 4.56789012 1.24612875]]\n"
     ]
    }
   ],
   "source": [
    "arr_2d_5 = np.array([[1.123456789, 2.987654321, 3.2353278765], [3.141592653, 4.567890123, 1.2461287468175]])\n",
    "\n",
    "# log\n",
    "print(f\"arr_2d_5:\\n{arr_2d_5}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "print_options_1:\n",
      "{'edgeitems': 3, 'threshold': 1000, 'floatmode': 'maxprec', 'precision': 8, 'suppress': False, 'linewidth': 75, 'nanstr': 'nan', 'infstr': 'inf', 'sign': '-', 'formatter': None, 'legacy': False, 'override_repr': None}\n"
     ]
    }
   ],
   "source": [
    "# retrieve the current print options\n",
    "print_options_1 = np.get_printoptions()\n",
    "\n",
    "# log\n",
    "print(f\"print_options_1:\\n{print_options_1}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arr_2d_5:\n",
      "[[1.12 2.99 3.24]\n",
      " [3.14 4.57 1.25]]\n",
      "arr_2d_5:\n",
      "[[1.12345679 2.98765432 3.23532788]\n",
      " [3.14159265 4.56789012 1.24612875]]\n"
     ]
    }
   ],
   "source": [
    "# use as a context manager to temporarily set print options within a specific block of code\n",
    "with np.printoptions(precision=2, suppress=True, linewidth=20):\n",
    "    print(f\"arr_2d_5:\\n{arr_2d_5}\")\n",
    "\n",
    "# log\n",
    "print(f\"arr_2d_5:\\n{arr_2d_5}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arr_2d_5:\n",
      "[[1.12 2.99 3.24]\n",
      " [3.14 4.57 1.25]]\n"
     ]
    }
   ],
   "source": [
    "# set global print options for NumPy arrays\n",
    "np.set_printoptions(precision=2, suppress=True, linewidth=20)\n",
    "\n",
    "# log\n",
    "print(f\"arr_2d_5:\\n{arr_2d_5}\")\n",
    "\n",
    "# reset to default settings according to the docs [but it's not!!] [tested on v2.0]\n",
    "np.set_printoptions()"
   ]
  }
 ],
 "metadata": {
  "author_email": "AmirhosseinHeydari78@gmail.com",
  "author_github": "https://github.com/mr-pylin",
  "author_name": "Amirhossein Heydari",
  "kernelspec": {
   "display_name": "numpy-workshop-0psJcWas-py3.12",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  },
  "origin_repo": "https://github.com/mr-pylin/numpy-workshop"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}