Add nda notebook (#13)

* add nda download notebook

* add nda download notebook

---------

Co-authored-by: J Montgomery Maxwell <[email protected]>

notebooks/NDA_data_package_download.ipynb

@@ -0,0 +1,330 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "95392afb-c92b-442a-b769-1eebd9b6b928",
+   "metadata": {},
+   "source": [
+    "## NIMH Data Archive Data Access\n",
+    "\n",
+    "In this notebook a HEAL user can access a NIMH Data Archive (NDA) data package using a miNDAR data package ID, a NDA username and password for the data package, and the NDA API.\n",
+    "\n",
+    "Before using this notebook, the user must have been granted access to the dataset they wish to download, and have created a miNDAR for the data package. (A miNDAR is a cloud-based Oracle database that contains a copy of a data package.)  \n",
+    "\n",
+    "* You can request access to a NDA dataset from the [Data Permissions tab on the NDA site](https://nda.nih.gov/user/dashboard/data_permissions.html). If you do not have an NDA account, you can create one by first logging in using your NIH login credentials.\n",
+    "* Once you have access, you can create a miNDAR from the data package by [following the instructions here](https://docs.google.com/document/d/1-DuyUke3I7CK_riRvVDIesgtnjhJBQVlDu9tQpe2JYo/edit?usp=sharing). \n",
+    "\n",
+    "Once you have a miNDAR for your dataset, the only inputs required from the user of this notebook is to create a nimh_credentials.json file in the second cell of this notebok using your NDA NIMH username and password and to define the miNDAR data package ID in the third cell of this notebook.  \n",
+    "\n",
+    "* The **miNDAR ID** can be found in the `ID` column on the [Data Packages tab of the NDA portal](https://nda.nih.gov/user/dashboard/packages.html)  \n",
+    "* Data packages and their corresponding **miNDAR IDs** expire after 60 days and must be generated again in order to download the data.\n",
+    "\n",
+    "Please note: Users are responsible for complying with all aspects of their Data Use Agreement, including deleting any accessed data in accordance with the parameters specified in their DUA."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fadcd3c8-fbb4-4cc6-b5fb-5951775cf4d4",
+   "metadata": {},
+   "source": [
+    "### Import Necessary Packages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "724c0b85-e666-40bc-89a3-b01353165195",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "import base64\n",
+    "import requests\n",
+    "import json\n",
+    "import urllib.request\n",
+    "import shutil\n",
+    "from pathlib import Path\n",
+    "import os\n",
+    "import getpass"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "be23668d-578c-43c2-a18a-0825503b0f61",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "### Create Your NDA Credentials JSON File\n",
+    "\n",
+    "Input your NDA username and password in the 'nda_username' and 'nda_password' fields in the nda_credentials dictionary object. Note these are NOT your RAS credentials; this username and password are set when you create an account at NDA. You can view your NDA username and reset your NDA password if needed by visiting your Profile at the NIMH Data Archive. \n",
+    "\n",
+    "If you have already created your nda_credentials.json file, then there is no need to run the cell below.\n",
+    "\n",
+    "After you create the nda_credentials.json file we suggest that you delete the input of your NDA username and password in the cell below to minimize the risk of potentially sharing these credentials. Your newly created nda_credentials.json file will persist between workspace sessions under the /pd directory."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "603297cf-a6b0-40d2-ae4e-67c3fedc722f",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "nda_username = getpass.getpass()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c76cb3d1-ca38-4308-807a-6bd4e8b7b96c",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "nda_password = getpass.getpass()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f80ad469-dc43-498b-b517-3bef2f01fd7f",
+   "metadata": {},
+   "source": [
+    "### Define NIMH Data Package ID\n",
+    "\n",
+    "Enter your miNDAR ID (replace '123')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "02222358-2b9f-4320-aeea-90cf94d58fcf",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "packageId = 123 # miNDAR package ID as integer "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4cacf612-b6cd-4fb2-a6c8-a7578a04597a",
+   "metadata": {},
+   "source": [
+    "### Encode A Credentials Object Using The Credentials JSON File"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "73aafcdb-7526-43fc-93ab-939dc01555d8",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "credentials = nda_username + ':' + nda_password\n",
+    "credentials = base64.b64encode(credentials.encode('ascii')).decode('utf-8')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7b6ddcce-bda0-4840-bed3-4d95c9591527",
+   "metadata": {},
+   "source": [
+    "### Create Header And Test Request\n",
+    "Create the headers to be used for all requests and send initial HTTP request to test connection"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "85c730d3-190a-4ac4-bf4c-9e46f554b699",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "headers = {\n",
+    "    'Authorization': 'Basic ' + credentials,\n",
+    "    'User-Agent': 'Example Client',\n",
+    "    'Accept': 'application/json'\n",
+    "}\n",
+    "\n",
+    "response = requests.get('https://nda.nih.gov/api/package/auth', headers=headers)\n",
+    "\n",
+    "if response.status_code != requests.codes.ok:\n",
+    "    print('failed to authenticate')\n",
+    "    response.raise_for_status()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "293c664c-c1d9-4dc9-8941-2c07cbddca38",
+   "metadata": {},
+   "source": [
+    "### Request File IDs & Names\n",
+    "Return all file names and associated file IDs as a nested json/dictionary object"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "989f934c-3ef6-4e0b-a4ad-edad0dd2f06e",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "response = requests.get('https://nda.nih.gov/api/package/' + str(packageId) + '/files', headers=headers)\n",
+    "results = response.json()['results']\n",
+    "\n",
+    "api_files = {}\n",
+    "for f in results:\n",
+    "    api_files[f['package_file_id']] = {'name': f['download_alias']}\n",
+    "api_files"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "15cce5ee-80d1-4b7a-adc0-aa76b7b2f0db",
+   "metadata": {},
+   "source": [
+    "### Request Files' Presigned URLs\n",
+    "Use the data package file IDs to return the presigned URLs for each file in the data package. Manually selecting the presigned download URL can be used to download the associated file to your local machine.\n",
+    "\n",
+    "Define the file ID (file_id) by either copying and pasting one of the file IDs listed above or index a file ID from the list of file IDs as can be seen below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9cb9d661-1495-450f-9c25-f00d958bdbaa",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "response = requests.post('https://nda.nih.gov/api/package/' + str(packageId) + '/files/batchGeneratePresignedUrls', json=list(api_files.keys()), headers=headers)\n",
+    "results = response.json()['presignedUrls']\n",
+    "\n",
+    "for url in results:\n",
+    "    api_files[url['package_file_id']]['download'] = url['downloadURL']\n",
+    "\n",
+    "file_ids = list(api_files.keys())\n",
+    "api_files[file_ids[0]]['download']                       "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b4a35ad7-ad2d-4a36-94e9-a573d736aa79",
+   "metadata": {},
+   "source": [
+    "### Download All Files In The Data Package\n",
+    "Use the complete list of data package file IDs to download all of the files in the data package."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "181053d6-eb37-403f-aff1-c48f9d1aac02",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "for key in list(api_files.keys()):\n",
+    "\n",
+    "    name = api_files[key]['name']\n",
+    "    downloadUrl = api_files[key]['download']\n",
+    "    \n",
+    "    # Create a downloads directory - you may change the download directory path here\n",
+    "    file = 'downloads/' + name\n",
+    "    # Strip out the file's name for creating non-existent directories\n",
+    "    directory = file[:file.rfind('/')]\n",
+    "\n",
+    "    # Create non-existent directories, package files have their\n",
+    "    # own directory structure, and this will ensure that it is\n",
+    "    # kept in tact when downloading.\n",
+    "    Path(directory).mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "    # Initiate the download.\n",
+    "    with urllib.request.urlopen(downloadUrl) as dl, open(file, 'wb') as out_file:\n",
+    "        try:\n",
+    "            shutil.copytree(dl, out_file)\n",
+    "        except:\n",
+    "            pass\n",
+    "        "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ec724c2c-e2a2-4fc6-9869-e647fd287640",
+   "metadata": {},
+   "source": [
+    "### Download Select Files By File ID\n",
+    "Download files associated with select data package file IDs."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1976154a-fd08-4675-a0d7-f45c20cff02d",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "select_file_ids = [file_ids[0], file_ids[1], file_ids[2]]\n",
+    "for key in select_file_ids:\n",
+    "\n",
+    "    name = api_files[key]['name']\n",
+    "    downloadUrl = api_files[key]['download']\n",
+    "    \n",
+    "    # Create a downloads directory - you may change the download directory path here\n",
+    "    file = 'downloads/' + name\n",
+    "    # Strip out the file's name for creating non-existent directories\n",
+    "    directory = file[:file.rfind('/')]\n",
+    "    \n",
+    "    # Create non-existent directories; package files have their\n",
+    "    # own directory structure, and this will ensure that the structure is\n",
+    "    # kept intact when downloading.\n",
+    "    Path(directory).mkdir(parents=True, exist_ok=True)\n",
+    "    \n",
+    "    # Initiate the download.\n",
+    "    with urllib.request.urlopen(downloadUrl) as dl, open(file, 'wb') as out_file:\n",
+    "        try:\n",
+    "            shutil.copytree(dl, out_file)\n",
+    "        except:\n",
+    "            pass\n",
+    "        "
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}