url_to_google_drive.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# **Download file from given url to Google Drive**\n"
      ],
      "metadata": {
        "id": "ycRR1MlogzaG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Mount google drive to notebook"
      ],
      "metadata": {
        "id": "O1PuhV3Vhq_A"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PEdc8SYonvmT"
      },
      "outputs": [],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Specify File path\n",
        "- Change data_path to the required folder where you want to save the downloaded\n",
        "  file\n",
        "  - Eg. Default named \"dataset\" change to it point to other folder location\n",
        "  or remove to save to google drive base path"
      ],
      "metadata": {
        "id": "_zq1KU9HhKLD"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "LhYONhkesEB6"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "base_path = '/content/drive/My Drive/'\n",
        "data_path = base_path + \"datasets\"\n",
        "if not os.path.exists(data_path):\n",
        "  !mkdir -p {data_path}"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Downloading file\n",
        "- Usage request to get the file\n",
        "- Uses tqdm to show download progress"
      ],
      "metadata": {
        "id": "_E2J66cZhvmu"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "kcfwNMkM3z7m"
      },
      "outputs": [],
      "source": [
        "import requests\n",
        "from tqdm.notebook import tqdm\n",
        "\n",
        "def save_to_drive(url, file_path):\n",
        "  \"\"\"\n",
        "  Downloads a file from a given URL and saves it to Google Drive.\n",
        "\n",
        "    Args:\n",
        "        url (str): The URL of the file to download.\n",
        "        file_path (str): The path within Google Drive to save the file,\n",
        "                         including the filename and extension\n",
        "                         (e.g., \"datasets/indian_food_dataset.zip\").\n",
        "\n",
        "    Returns:\n",
        "        None\n",
        "  \"\"\"\n",
        "\n",
        "  try:\n",
        "    g_file_path = f\"/content/drive/MyDrive/{file_path}\".split(\"/\")\n",
        "    g_file_path = \"/\".join(g_file_path[:-1])\n",
        "    if not os.path.exists(g_file_path):\n",
        "      !mkdir -p {g_file_path}\n",
        "\n",
        "    file_url = url\n",
        "\n",
        "    r = requests.get(file_url, stream = True)\n",
        "    total_size = int(r.headers.get('content-length', 0))\n",
        "    if total_size == 0:\n",
        "      print(\"Content-Length not available, progress bar might not be accurate.\")\n",
        "\n",
        "    block_size = 1024\n",
        "    t=tqdm(total=total_size, unit='iB', unit_scale=True,\n",
        "           position=0,\n",
        "           leave=True)\n",
        "\n",
        "    with open(f\"/content/drive/My Drive/{file_path}\", \"wb\") as file:\n",
        "        for data in r.iter_content(1024):\n",
        "          t.update(len(data))\n",
        "          file.write(data)\n",
        "    t.close()\n",
        "    if total_size != 0 and t.n != total_size:\n",
        "      print(\"ERROR, something went wrong\")\n",
        "  except FileNotFoundError:\n",
        "    print(\"\\nFolder doesn't exist! Create filepath in gdrive\")\n",
        "  except Exception as e:\n",
        "    print(e)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Use\n",
        "- **url**: Specify the url to download (must be public url)\n",
        "- call **save_to_drive** function with params (url: string, file_path:string)\n",
        "  - file_path: must be pointed to folder location with filename (specify with extension) inside My Drive in gDrive\n",
        "    (Eg. dataset/food.zip)"
      ],
      "metadata": {
        "id": "fJLhW3bZiHAV"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ct-pN5n1qMoH"
      },
      "outputs": [],
      "source": [
        "url = \"https://storage.googleapis.com/kaggle-data-sets/1864/33884/bundle/archive.zip?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=gcp-kaggle-com%40kaggle-161607.iam.gserviceaccount.com%2F20240703%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20240703T171808Z&X-Goog-Expires=259200&X-Goog-SignedHeaders=host&X-Goog-Signature=b7d13b4432d0af7f442c96dc351b4a24938a009a1117c4af8288ec9f1c9f5724563f8d6bec2e9d3b40b030690d23658282ee293b2759faba18c5c02ae2a3888129a33f4b537e782de8441e1fc6ea4fba0ddebf8de7af0d36f7c3247aeb03345051a0a7bfb6b41aa4e8e0a6e7a097876654715d4fdb3e951916aead3394a2cb7c89ffc4e1a8a4da57308e46b8ea9e0b8018717d796b22b9e18435e1df3b1df2e8a6a606afd4cecadfdc1fb7d101092ea3922284934a40caea0c88136b52976cfe93fc11153f935cd8a4538bb9426aefcfbef4401ab3da11668f9e8828bd9678326acc7e3245e3a282e4d7425b62b2ca46feeae535bada70441f04fb8e41948dbe\"\n",
        "save_to_drive(url, \"datasets/food_data_6903.zip\")\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}