notebooks/data-acquisition/03-download-sliiders-input-data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9c487926-08b8-49b2-928d-c95730c36d44",
   "metadata": {},
   "source": [
    "## Notebook for downloading inputs to create SLIIDERS\n",
    "\n",
    "This notebook contains directions for downloading various input datasets to create the final product for this directory, the **SLIIDERS** dataset.\n",
    "\n",
    "In general, we will keep the format, file name, and data unaltered, but apply changes when\n",
    "- file name is not human-readable, too long, or is not much informative about the dataset (assign appropriate file names)\n",
    "- file format causes errors (save in a similar file format that is not error-prone)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "28c697c0-f372-4bf7-b528-d8ffb9f8f6b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "517784ca-badd-41fe-a88c-7b4370260e5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "import subprocess\n",
    "import tarfile\n",
    "import tempfile\n",
    "from io import BytesIO\n",
    "from pathlib import Path\n",
    "from tempfile import TemporaryDirectory\n",
    "from zipfile import ZipFile\n",
    "\n",
    "import cartopy.io.shapereader as shpreader\n",
    "import geopandas as gpd\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import requests\n",
    "from pandas_datareader import wb as dr_wb\n",
    "from pint import UnitRegistry\n",
    "from sliiders import settings as sset\n",
    "from sliiders.io import save, save_geoparquet\n",
    "from tqdm import tqdm\n",
    "\n",
    "ureg = UnitRegistry()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e1d4cce-5cc8-4c4d-b3cd-1aaf56e90ce9",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Fetching raw data from various sources"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b95e5706-6257-42d5-8497-20018af5e1e5",
   "metadata": {},
   "source": [
    "## NLDB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "65737527-ae3e-43d3-ad03-7316f059794d",
   "metadata": {},
   "outputs": [],
   "source": [
    "USACE_URL = \"https://levees.sec.usace.army.mil/api-local/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c54006fb-bd40-477d-a9fa-42021320e0c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get list of all\n",
    "ID_PARAMS = {\n",
    "    \"in\": \"@nation:USA\",\n",
    "    \"md\": \"false\",\n",
    "    \"syarray\": \"@PURPOSE_IDS:(7)\",\n",
    "    \"return\": \"id,name\",\n",
    "}\n",
    "system_ids = pd.read_json(\n",
    "    requests.get(USACE_URL + \"systems/query\", params=ID_PARAMS).url\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ba3f5c27-5924-48f5-a6f6-cc67a061389b",
   "metadata": {},
   "outputs": [],
   "source": [
    "SHP_PARAMS = {\"type\": \"leveed-area\", \"format\": \"geo\", \"props\": \"false\", \"coll\": \"false\"}\n",
    "\n",
    "\n",
    "def return_shp(sid):\n",
    "    out = requests.get(\n",
    "        USACE_URL + \"geometries/query\", params={**SHP_PARAMS, \"systemId\": sid}\n",
    "    ).json()[0]\n",
    "    # drop the 3d z-coord\n",
    "    out[\"coordinates\"] = [\n",
    "        [np.array(i)[..., :2].tolist() for i in out[\"coordinates\"][0]]\n",
    "    ]\n",
    "    out = str(out).replace(\" \", \"\").replace(\"'\", '\"')\n",
    "    return gpd.read_file(out, driver=\"GeoJSON\").iloc[0, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "07a4e986-d7a7-4d48-87f3-abb5a5ed5720",
   "metadata": {},
   "outputs": [],
   "source": [
    "system_ids[\"geometry\"] = gpd.GeoSeries(\n",
    "    system_ids.id.map(return_shp), index=system_ids.index, crs=4326\n",
    ")\n",
    "system_ids = gpd.GeoDataFrame(system_ids, geometry=\"geometry\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "abc25d7f-258e-4616-a324-1c305333b654",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_height(sid):\n",
    "    out = requests.get(USACE_URL + \"segments\", params={\"system_id\": sid}).json()\n",
    "    min_heights = [i[\"minHeight\"] for i in out if \"minHeight\" in i.keys()]\n",
    "    max_heights = [i[\"maxHeight\"] for i in out if \"maxHeight\" in i.keys()]\n",
    "    heights = min_heights + max_heights\n",
    "    if not len(heights):\n",
    "        return np.nan\n",
    "    return min(heights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9b0a7fac-28e5-4b04-b5df-852fa8e35623",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/srv/conda/envs/notebook/lib/python3.11/site-packages/pandas/core/construction.py:631: UnitStrippedWarning: The unit of the quantity is stripped when downcasting to ndarray.\n",
      "  data = np.array(data, copy=copy)\n"
     ]
    }
   ],
   "source": [
    "system_ids[\"min_height\"] = system_ids.id.map(get_height)\n",
    "system_ids = system_ids.set_index(\"id\")\n",
    "system_ids[\"min_height\"] = (system_ids.min_height.values * ureg.feet).to(ureg.meter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "7cbf9125-fcb4-4930-9aba-ad4628cd8b1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "save_geoparquet(system_ids, sset.PATH_NLDB)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fc6ec0b-b6d9-4bc6-bac3-3f9c5fc0b929",
   "metadata": {},
   "source": [
    "### Natural Earth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b044a8e1-2251-4b91-8bc2-1d0b04eb0b1c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/srv/conda/envs/notebook/lib/python3.11/site-packages/cartopy/io/__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/10m_physical/ne_10m_land.zip\n",
      "  warnings.warn(f'Downloading: {url}', DownloadWarning)\n",
      "/srv/conda/envs/notebook/lib/python3.11/site-packages/cartopy/io/__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/10m_physical/ne_10m_ocean.zip\n",
      "  warnings.warn(f'Downloading: {url}', DownloadWarning)\n",
      "/srv/conda/envs/notebook/lib/python3.11/site-packages/cartopy/io/__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/10m_physical/ne_10m_lakes.zip\n",
      "  warnings.warn(f'Downloading: {url}', DownloadWarning)\n",
      "/srv/conda/envs/notebook/lib/python3.11/site-packages/cartopy/io/__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/10m_physical/ne_10m_lakes_australia.zip\n",
      "  warnings.warn(f'Downloading: {url}', DownloadWarning)\n",
      "/srv/conda/envs/notebook/lib/python3.11/site-packages/cartopy/io/__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/10m_physical/ne_10m_lakes_europe.zip\n",
      "  warnings.warn(f'Downloading: {url}', DownloadWarning)\n",
      "/srv/conda/envs/notebook/lib/python3.11/site-packages/cartopy/io/__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/10m_physical/ne_10m_lakes_north_america.zip\n",
      "  warnings.warn(f'Downloading: {url}', DownloadWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GSPath('gs://rhg-data/impactlab-rhg/coastal/sliiders/raw/geography/natural_earth/20240222')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# first, clear cache to download newest version\n",
    "shpfilename = shpreader.natural_earth(\"10m\", \"physical\", \"land\")\n",
    "shutil.rmtree(Path(shpfilename).parent)\n",
    "\n",
    "# now download\n",
    "for kind in [\n",
    "    \"land\",\n",
    "    \"ocean\",\n",
    "    \"lakes\",\n",
    "    \"lakes_australia\",\n",
    "    \"lakes_europe\",\n",
    "    \"lakes_north_america\",\n",
    "]:\n",
    "    shpfilename = shpreader.natural_earth(\"10m\", \"physical\", kind)\n",
    "\n",
    "sset.DIR_NATEARTH_RAW.upload_from(Path(shpfilename).parent)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83c92a29-bd27-4b1d-838c-7989d7561757",
   "metadata": {},
   "source": [
    "### Penn World Tables 10.01 (PWT 10.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "029359fb-e17a-44c8-a536-4696f7d8c00c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# PWT10.01\n",
    "pwt_data = pd.read_excel(\n",
    "    \"https://dataverse.nl/api/access/datafile/354095\", sheet_name=2\n",
    ")\n",
    "\n",
    "# PWT10.01 capital details\n",
    "pwt_data_K = pd.read_excel(\n",
    "    \"https://dataverse.nl/api/access/datafile/354109\", sheet_name=2\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a2d0fc66-6d26-4d61-bc77-5d5b1a8562ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "pwt_filenames = [\"pwt.xlsx\", \"pwt_K_detail.xlsx\"]\n",
    "for i, data in enumerate([pwt_data, pwt_data_K]):\n",
    "    data.to_excel(\n",
    "        (sset.PATH_PWT_RAW.parent / pwt_filenames[i]).open(\"wb\"),\n",
    "        sheet_name=\"Sheet1\",\n",
    "        index=False,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d08579cb-0c3e-4722-8ae3-944b46297b68",
   "metadata": {},
   "source": [
    "### Maddison Project Dataset (MPD, Maddison Project Database 2020)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "eb664e52-bdbe-4e2e-b30e-3102fdf8ae83",
   "metadata": {},
   "outputs": [],
   "source": [
    "madd = pd.read_excel(\n",
    "    \"https://www.rug.nl/ggdc/historicaldevelopment/maddison/data/mpd2020.xlsx\",\n",
    "    sheet_name=2,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "bc8d4bb9-d483-4633-8851-9d950e173ff5",
   "metadata": {},
   "outputs": [],
   "source": [
    "madd.to_excel(\n",
    "    excel_writer=sset.PATH_MPD_RAW.open(\"wb\"),\n",
    "    index=False,\n",
    "    sheet_name=\"Sheet1\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b3c486f-f4b0-4ad1-849a-542e2a875a11",
   "metadata": {},
   "source": [
    "### World Bank WDI (WB WDI)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "02a17959-e43a-4db0-969b-4f2bb26bada7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# country name and iso3 country code information\n",
    "country_info = (\n",
    "    dr_wb.get_countries()[[\"name\", \"iso3c\"]]\n",
    "    .rename(columns={\"name\": \"country\", \"iso3c\": \"ccode\"})\n",
    "    .set_index(\"country\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2883eb00-f580-4a5f-9df8-1419aac6e90c",
   "metadata": {},
   "source": [
    "#### Investment-to-GDP ratio, GDP and GDPpc (nominal and PPP), and Population"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b33996e7-fa36-4496-bda9-1f8a459cde1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# relevant indicator information for the `dr_wb` module to fetch the variables\n",
    "wbwdi_indicators = [\n",
    "    \"SP.POP.TOTL\",  # population\n",
    "    \"NE.GDI.FTOT.ZS\",  # investment-to-GDP ratio\n",
    "    \"NY.GDP.MKTP.PP.KD\",  # GDP PPP\n",
    "    \"NY.GDP.PCAP.PP.KD\",  # GDP per capita PPP\n",
    "    \"NY.GDP.MKTP.KD\",  # GDP nominal\n",
    "    \"NY.GDP.PCAP.KD\",  # GDP per capita nominal\n",
    "]\n",
    "\n",
    "j = 0\n",
    "for indi in wbwdi_indicators:\n",
    "    indi_info = dr_wb.download(indicator=indi, country=\"all\", start=1950, end=2023)\n",
    "\n",
    "    if j == 0:\n",
    "        j += 1\n",
    "        wbwdi_info = indi_info.copy()\n",
    "    else:\n",
    "        wbwdi_info = wbwdi_info.join(\n",
    "            indi_info,\n",
    "            how=\"outer\",\n",
    "        )\n",
    "\n",
    "# convert year to integer\n",
    "wbwdi_info.index = wbwdi_info.index.set_levels(\n",
    "    wbwdi_info.index.levels[-1].astype(\"uint16\"), level=\"year\"\n",
    ")\n",
    "\n",
    "# excluding those that have no information and saving the data\n",
    "wbwdi_info = wbwdi_info[~pd.isnull(wbwdi_info).all(axis=1)].join(\n",
    "    country_info, how=\"left\", on=\"country\"\n",
    ")\n",
    "assert wbwdi_info.index.is_unique"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "07c35edf-aa86-4630-83bc-63c8ab158498",
   "metadata": {},
   "outputs": [],
   "source": [
    "save(wbwdi_info, sset.DIR_WB_WDI_RAW / \"wdi_pop_iy_gdp.parquet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdec086b-c676-4690-9f2f-93585172e4d7",
   "metadata": {},
   "source": [
    "#### WB WDI: exchange rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "e766dc90-ef52-4f10-ba4c-0fdae8c768c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# country name and iso3 country code information\n",
    "xr_code = \"PA.NUS.FCRF\"\n",
    "xr_wb = (\n",
    "    dr_wb.download(indicator=xr_code, country=\"all\", start=1950, end=2023)\n",
    "    .join(country_info, on=[\"country\"], how=\"left\")\n",
    "    .rename(columns={xr_code: \"xrate\"})\n",
    ")\n",
    "xr_wb = xr_wb.reset_index().astype({\"year\": \"uint16\"}).set_index([\"ccode\", \"year\"])\n",
    "xr_wb = xr_wb[xr_wb.xrate.notnull()].sort_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "eea055d3-78f5-45fd-a123-ee29c5fb4720",
   "metadata": {},
   "outputs": [],
   "source": [
    "save(xr_wb, sset.DIR_WB_WDI_RAW / \"wdi_xr.parquet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df339e99-0571-4c23-8c7f-6e691b7c39c6",
   "metadata": {},
   "source": [
    "### UN WPP populations (overall and by-population-group data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c5c47a28-484c-4d3e-a2ae-9072b969a706",
   "metadata": {},
   "outputs": [],
   "source": [
    "for ix, sex in enumerate([\"MALE\", \"FEMALE\"]):\n",
    "    fname = f\"WPP2022_POP_F02_{ix+2}_POPULATION_5-YEAR_AGE_GROUPS_{sex}\"\n",
    "    r = requests.get(\n",
    "        \"https://population.un.org/wpp/Download/Files/1_Indicators%20(Standard)\"\n",
    "        f\"/EXCEL_FILES/2_Population/{fname}.xlsx\"\n",
    "    )\n",
    "    with (sset.DIR_UN_WPP_RAW / f\"{fname}.xlsx\").open(\"wb\") as f:\n",
    "        f.write(r.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cdb4d29-7c84-49e0-ba3e-4fb7663ecd32",
   "metadata": {},
   "source": [
    "### Åland Island GDP and population (from Statistics and Research Åland or ÅSUB)\n",
    "\n",
    "Note when newer versions are available, old links from ÅSUB will become deprecated; the below links in `ALA_GDP_LINK` and `ALA_POP_LINK` are valid as of 2022-09-19."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "792b59d8-44fa-4d85-93db-8682b1857e24",
   "metadata": {},
   "outputs": [],
   "source": [
    "for name, save_name in [\n",
    "    (\n",
    "        \"NR005en_GDP per capita in euro PPS 1995-2021%2C current prices.xls\",\n",
    "        \"ala_gdp.xls\",\n",
    "    ),\n",
    "    (\n",
    "        \"%C3%85LV01_%C3%85land%2C%20Faroe%20Islands%20and%20Greenland%20-%20an%20overview%20with%20comparable%20data%202022.xlsx\",\n",
    "        \"ala_pop.xlsx\",\n",
    "    ),\n",
    "]:\n",
    "    with (sset.DIR_ALAND_STATISTICS_RAW / save_name).open(\"wb\") as f:\n",
    "        f.write(\n",
    "            requests.get(\n",
    "                f\"https://www.asub.ax/sites/default/files/media/document/{name}\"\n",
    "            ).content\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fe8a00c-5822-467a-80a8-b4b6722eea3c",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Global Wealth Databook (from Credit Suisse)\n",
    "\n",
    "We download the 2021 vintage (latest as of 2022-09-19)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "id": "174f26cc-0355-42e2-bfa7-80a9fe52ed23",
   "metadata": {},
   "outputs": [],
   "source": [
    "with sset.PATH_GWDB_RAW.open(\"wb\") as f:\n",
    "    f.write(\n",
    "        requests.get(\n",
    "            \"https://www.ubs.com/global/en/family-office-uhnw/reports/\"\n",
    "            \"global-wealth-report-2023/_jcr_content/mainpar/toplevelgrid_5684475/col2/\"\n",
    "            \"linklistnewlook/link_copy.0357374027.file/\"\n",
    "            \"PS9jb250ZW50L2RhbS9hc3NldHMvd20vZ2xvYmFsL2ltZy9nbG9iYWwtZmFtaWx5LW9mZmljZS\"\n",
    "            \"9kb2NzL2RhdGFib29rLWdsb2JhbC13ZWFsdGgtcmVwb3J0LTIwMjMtZW4ucGRm/\"\n",
    "            \"databook-global-wealth-report-2023-en.pdf\"\n",
    "        ).content\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9f0b8fa-7c93-4735-9caa-e07777d150a2",
   "metadata": {},
   "source": [
    "### LitPop (Eberenz et al. 2020, Earth Syst. Sci. Data)\n",
    "\n",
    "#### Download Data from the Internet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3ed65443-7818-406e-a23a-1f62aba91a5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# link for downloading the LitPop files\n",
    "link_base = (\n",
    "    \"https://www.research-collection.ethz.ch/bitstream/handle/20.500.11850/331316\"\n",
    ")\n",
    "\n",
    "# readme, data, normalized data, and metadata\n",
    "links = [\n",
    "    link_base + \"/_readme_v1_2.txt?sequence=18&isAllowed=y\",\n",
    "    link_base + \"/LitPop_v1_2.tar?sequence=16&isAllowed=y\",\n",
    "    link_base + \"/Lit_Pop_norm_v1.tar?sequence=4&isAllowed=y\",\n",
    "    link_base + \"/_metadata_countries_v1_2.csv?sequence=12&isAllowed=y\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d74ef086-b44e-489a-b3cc-0de60dd8551c",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Input \u001b[0;32mIn [15]\u001b[0m, in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m name \u001b[38;5;241m=\u001b[39m link\u001b[38;5;241m.\u001b[39msplit(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m/\u001b[39m\u001b[38;5;124m\"\u001b[39m)[\u001b[38;5;241m-\u001b[39m\u001b[38;5;241m1\u001b[39m]\u001b[38;5;241m.\u001b[39msplit(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m?\u001b[39m\u001b[38;5;124m\"\u001b[39m)[\u001b[38;5;241m0\u001b[39m]\n\u001b[1;32m      3\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m (sset\u001b[38;5;241m.\u001b[39mDIR_LITPOP_RAW \u001b[38;5;241m/\u001b[39m name)\u001b[38;5;241m.\u001b[39mopen(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mwb\u001b[39m\u001b[38;5;124m\"\u001b[39m) \u001b[38;5;28;01mas\u001b[39;00m f:\n\u001b[0;32m----> 4\u001b[0m     f\u001b[38;5;241m.\u001b[39mwrite(\u001b[43mrequests\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget\u001b[49m\u001b[43m(\u001b[49m\u001b[43mlink\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241m.\u001b[39mcontent)\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.9/site-packages/requests/api.py:75\u001b[0m, in \u001b[0;36mget\u001b[0;34m(url, params, **kwargs)\u001b[0m\n\u001b[1;32m     64\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mget\u001b[39m(url, params\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[1;32m     65\u001b[0m     \u001b[38;5;124mr\u001b[39m\u001b[38;5;124;03m\"\"\"Sends a GET request.\u001b[39;00m\n\u001b[1;32m     66\u001b[0m \n\u001b[1;32m     67\u001b[0m \u001b[38;5;124;03m    :param url: URL for the new :class:`Request` object.\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     72\u001b[0m \u001b[38;5;124;03m    :rtype: requests.Response\u001b[39;00m\n\u001b[1;32m     73\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m---> 75\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mrequest\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mget\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43murl\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mparams\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mparams\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.9/site-packages/requests/api.py:61\u001b[0m, in \u001b[0;36mrequest\u001b[0;34m(method, url, **kwargs)\u001b[0m\n\u001b[1;32m     57\u001b[0m \u001b[38;5;66;03m# By using the 'with' statement we are sure the session is closed, thus we\u001b[39;00m\n\u001b[1;32m     58\u001b[0m \u001b[38;5;66;03m# avoid leaving sockets open which can trigger a ResourceWarning in some\u001b[39;00m\n\u001b[1;32m     59\u001b[0m \u001b[38;5;66;03m# cases, and look like a memory leak in others.\u001b[39;00m\n\u001b[1;32m     60\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m sessions\u001b[38;5;241m.\u001b[39mSession() \u001b[38;5;28;01mas\u001b[39;00m session:\n\u001b[0;32m---> 61\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43msession\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrequest\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmethod\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmethod\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43murl\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43murl\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.9/site-packages/requests/sessions.py:529\u001b[0m, in \u001b[0;36mSession.request\u001b[0;34m(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)\u001b[0m\n\u001b[1;32m    524\u001b[0m send_kwargs \u001b[38;5;241m=\u001b[39m {\n\u001b[1;32m    525\u001b[0m     \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mtimeout\u001b[39m\u001b[38;5;124m'\u001b[39m: timeout,\n\u001b[1;32m    526\u001b[0m     \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mallow_redirects\u001b[39m\u001b[38;5;124m'\u001b[39m: allow_redirects,\n\u001b[1;32m    527\u001b[0m }\n\u001b[1;32m    528\u001b[0m send_kwargs\u001b[38;5;241m.\u001b[39mupdate(settings)\n\u001b[0;32m--> 529\u001b[0m resp \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msend\u001b[49m\u001b[43m(\u001b[49m\u001b[43mprep\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43msend_kwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    531\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m resp\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.9/site-packages/requests/sessions.py:687\u001b[0m, in \u001b[0;36mSession.send\u001b[0;34m(self, request, **kwargs)\u001b[0m\n\u001b[1;32m    684\u001b[0m         \u001b[38;5;28;01mpass\u001b[39;00m\n\u001b[1;32m    686\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m stream:\n\u001b[0;32m--> 687\u001b[0m     \u001b[43mr\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcontent\u001b[49m\n\u001b[1;32m    689\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m r\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.9/site-packages/requests/models.py:838\u001b[0m, in \u001b[0;36mResponse.content\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    836\u001b[0m         \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_content \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m    837\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m--> 838\u001b[0m         \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_content \u001b[38;5;241m=\u001b[39m \u001b[38;5;124;43mb\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mjoin\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43miter_content\u001b[49m\u001b[43m(\u001b[49m\u001b[43mCONTENT_CHUNK_SIZE\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;124mb\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m'\u001b[39m\n\u001b[1;32m    840\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_content_consumed \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[1;32m    841\u001b[0m \u001b[38;5;66;03m# don't need to release the connection; that's been handled by urllib3\u001b[39;00m\n\u001b[1;32m    842\u001b[0m \u001b[38;5;66;03m# since we exhausted the data.\u001b[39;00m\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.9/site-packages/requests/models.py:760\u001b[0m, in \u001b[0;36mResponse.iter_content.<locals>.generate\u001b[0;34m()\u001b[0m\n\u001b[1;32m    758\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mraw, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mstream\u001b[39m\u001b[38;5;124m'\u001b[39m):\n\u001b[1;32m    759\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 760\u001b[0m         \u001b[38;5;28;01mfor\u001b[39;00m chunk \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mraw\u001b[38;5;241m.\u001b[39mstream(chunk_size, decode_content\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m):\n\u001b[1;32m    761\u001b[0m             \u001b[38;5;28;01myield\u001b[39;00m chunk\n\u001b[1;32m    762\u001b[0m     \u001b[38;5;28;01mexcept\u001b[39;00m ProtocolError \u001b[38;5;28;01mas\u001b[39;00m e:\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.9/site-packages/urllib3/response.py:579\u001b[0m, in \u001b[0;36mHTTPResponse.stream\u001b[0;34m(self, amt, decode_content)\u001b[0m\n\u001b[1;32m    577\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    578\u001b[0m     \u001b[38;5;28;01mwhile\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m is_fp_closed(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_fp):\n\u001b[0;32m--> 579\u001b[0m         data \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mread\u001b[49m\u001b[43m(\u001b[49m\u001b[43mamt\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mamt\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdecode_content\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdecode_content\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    581\u001b[0m         \u001b[38;5;28;01mif\u001b[39;00m data:\n\u001b[1;32m    582\u001b[0m             \u001b[38;5;28;01myield\u001b[39;00m data\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.9/site-packages/urllib3/response.py:522\u001b[0m, in \u001b[0;36mHTTPResponse.read\u001b[0;34m(self, amt, decode_content, cache_content)\u001b[0m\n\u001b[1;32m    520\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    521\u001b[0m     cache_content \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mFalse\u001b[39;00m\n\u001b[0;32m--> 522\u001b[0m     data \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_fp\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mread\u001b[49m\u001b[43m(\u001b[49m\u001b[43mamt\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m fp_closed \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;124mb\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    523\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m (\n\u001b[1;32m    524\u001b[0m         amt \u001b[38;5;241m!=\u001b[39m \u001b[38;5;241m0\u001b[39m \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m data\n\u001b[1;32m    525\u001b[0m     ):  \u001b[38;5;66;03m# Platform-specific: Buggy versions of Python.\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    531\u001b[0m         \u001b[38;5;66;03m# not properly close the connection in all cases. There is\u001b[39;00m\n\u001b[1;32m    532\u001b[0m         \u001b[38;5;66;03m# no harm in redundantly calling close.\u001b[39;00m\n\u001b[1;32m    533\u001b[0m         \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_fp\u001b[38;5;241m.\u001b[39mclose()\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.9/http/client.py:463\u001b[0m, in \u001b[0;36mHTTPResponse.read\u001b[0;34m(self, amt)\u001b[0m\n\u001b[1;32m    460\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m amt \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    461\u001b[0m     \u001b[38;5;66;03m# Amount is given, implement using readinto\u001b[39;00m\n\u001b[1;32m    462\u001b[0m     b \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mbytearray\u001b[39m(amt)\n\u001b[0;32m--> 463\u001b[0m     n \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mreadinto\u001b[49m\u001b[43m(\u001b[49m\u001b[43mb\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    464\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mmemoryview\u001b[39m(b)[:n]\u001b[38;5;241m.\u001b[39mtobytes()\n\u001b[1;32m    465\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m    466\u001b[0m     \u001b[38;5;66;03m# Amount is not given (unbounded read) so we must check self.length\u001b[39;00m\n\u001b[1;32m    467\u001b[0m     \u001b[38;5;66;03m# and self.chunked\u001b[39;00m\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.9/http/client.py:507\u001b[0m, in \u001b[0;36mHTTPResponse.readinto\u001b[0;34m(self, b)\u001b[0m\n\u001b[1;32m    502\u001b[0m         b \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mmemoryview\u001b[39m(b)[\u001b[38;5;241m0\u001b[39m:\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mlength]\n\u001b[1;32m    504\u001b[0m \u001b[38;5;66;03m# we do not use _safe_read() here because this may be a .will_close\u001b[39;00m\n\u001b[1;32m    505\u001b[0m \u001b[38;5;66;03m# connection, and the user is reading more bytes than will be provided\u001b[39;00m\n\u001b[1;32m    506\u001b[0m \u001b[38;5;66;03m# (for example, reading in 1k chunks)\u001b[39;00m\n\u001b[0;32m--> 507\u001b[0m n \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfp\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mreadinto\u001b[49m\u001b[43m(\u001b[49m\u001b[43mb\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    508\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m n \u001b[38;5;129;01mand\u001b[39;00m b:\n\u001b[1;32m    509\u001b[0m     \u001b[38;5;66;03m# Ideally, we would raise IncompleteRead if the content-length\u001b[39;00m\n\u001b[1;32m    510\u001b[0m     \u001b[38;5;66;03m# wasn't satisfied, but it might break compatibility.\u001b[39;00m\n\u001b[1;32m    511\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_close_conn()\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.9/socket.py:704\u001b[0m, in \u001b[0;36mSocketIO.readinto\u001b[0;34m(self, b)\u001b[0m\n\u001b[1;32m    702\u001b[0m \u001b[38;5;28;01mwhile\u001b[39;00m \u001b[38;5;28;01mTrue\u001b[39;00m:\n\u001b[1;32m    703\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 704\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_sock\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrecv_into\u001b[49m\u001b[43m(\u001b[49m\u001b[43mb\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    705\u001b[0m     \u001b[38;5;28;01mexcept\u001b[39;00m timeout:\n\u001b[1;32m    706\u001b[0m         \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_timeout_occurred \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.9/ssl.py:1241\u001b[0m, in \u001b[0;36mSSLSocket.recv_into\u001b[0;34m(self, buffer, nbytes, flags)\u001b[0m\n\u001b[1;32m   1237\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m flags \u001b[38;5;241m!=\u001b[39m \u001b[38;5;241m0\u001b[39m:\n\u001b[1;32m   1238\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m   1239\u001b[0m           \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnon-zero flags not allowed in calls to recv_into() on \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;241m%\u001b[39m\n\u001b[1;32m   1240\u001b[0m           \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__class__\u001b[39m)\n\u001b[0;32m-> 1241\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mread\u001b[49m\u001b[43m(\u001b[49m\u001b[43mnbytes\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbuffer\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1242\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m   1243\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28msuper\u001b[39m()\u001b[38;5;241m.\u001b[39mrecv_into(buffer, nbytes, flags)\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.9/ssl.py:1099\u001b[0m, in \u001b[0;36mSSLSocket.read\u001b[0;34m(self, len, buffer)\u001b[0m\n\u001b[1;32m   1097\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m   1098\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m buffer \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m-> 1099\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_sslobj\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mread\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mlen\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbuffer\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1100\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m   1101\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_sslobj\u001b[38;5;241m.\u001b[39mread(\u001b[38;5;28mlen\u001b[39m)\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "for link in links:\n",
    "    name = link.split(\"/\")[-1].split(\"?\")[0]\n",
    "    with (sset.DIR_LITPOP_RAW / name).open(\"wb\") as f:\n",
    "        f.write(requests.get(link).content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae5d6120-d2f8-474a-891e-5b59838d3b11",
   "metadata": {},
   "source": [
    "#### Un-tar and clear storage\n",
    "\n",
    "We only un-tar the regular (not normalized) LitPop data here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d685d7d9-0c7a-4c99-af5f-639cdd3c618d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# un-tar\n",
    "regular_litpop = sset.DIR_LITPOP_RAW / \"LitPop_v1_2.tar\"\n",
    "out_path = regular_litpop.parents[1] / \"test\" / regular_litpop.stem\n",
    "with tarfile.open(regular_litpop) as file:\n",
    "    with tempfile.TemporaryDirectory() as d:\n",
    "        file.extractall(d)\n",
    "        for file in Path(d).glob(\"*\"):\n",
    "            (sset.DIR_LITPOP_RAW / file.name).upload_from(file)\n",
    "\n",
    "# clear storage for the existing tar file\n",
    "regular_litpop.unlink()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4702790c-75dd-484a-b8e3-0acdf36d5c35",
   "metadata": {},
   "source": [
    "### GEG-15\n",
    "\n",
    "We download 2'30\" GEG15 and unzip."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d3109c6-fa04-406d-9def-21b745f6d83c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# downloading\n",
    "zip_url = (\n",
    "    \"https://data.humdata.org/dataset/1c9cf1eb-c20a-4a06-8309-9416464af746/\"\n",
    "    \"resource/e321d56d-022e-4070-80ac-f7860646408d/download/gar-exp.zip\"\n",
    ")\n",
    "\n",
    "tmppath = Path(\"/tmp/gar-exp.zip\")\n",
    "outpath = Path(\"/tmp/gar-exp\")\n",
    "with tmppath.open(\"wb\") as f:\n",
    "    f.write(requests.get(zip_url).content)\n",
    "subprocess.run([\"unzip\", str(tmppath), \"-d\", outpath])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c917e070-2946-4b9c-9d3d-b680cee3b2ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# upload\n",
    "for f in outpath.glob(\"*\"):\n",
    "    (sset.DIR_GEG15_RAW / f.name).upload_from(f, force_overwrite_to_cloud=True)\n",
    "\n",
    "# remove local\n",
    "shutil.rmtree(outpath)\n",
    "tmppath.unlink()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96bd9ff7-2b4f-4082-9d4c-0042c2b3ee9f",
   "metadata": {},
   "source": [
    "### Country-level Construction Cost Index from [Lincke and Hinkel (2021, *Earth's Future*)](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2020EF001965?campaign=woletoc)\n",
    "\n",
    "The accompanying GitHub repository to Lincke and Hinkel (2021) is at [this link](https://github.com/daniellincke/DIVA_paper_migration)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cb05b93-c8cc-4f9e-9e99-ce5b7d511d16",
   "metadata": {},
   "outputs": [],
   "source": [
    "with sset.PATH_EXPOSURE_LINCKE.open(\"w\") as f:\n",
    "    f.write(\n",
    "        requests.get(\n",
    "            \"https://raw.githubusercontent.com/daniellincke/\"\n",
    "            \"DIVA_paper_migration/master/data/csv/country_input.csv\"\n",
    "        ).content\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "342a0f73-06b7-4ed9-8ae8-113775265c15",
   "metadata": {},
   "source": [
    "### GEBCO 2023"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "7f1c9a38-79f9-40f2-b47a-9a933d824bf0",
   "metadata": {},
   "outputs": [],
   "source": [
    "URL_GEBCO = \"https://www.bodc.ac.uk/data/open_download/gebco/gebco_2023/zip/\"\n",
    "with sset.PATH_GEBCO_RAW.open(\"wb\") as f:\n",
    "    f.write(requests.get(URL_GEBCO).content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3aa9254-f6bb-4f14-bbb3-8e90c15c3d40",
   "metadata": {},
   "source": [
    "### Asian Development Bank (ADB) Key Indicators 2022 Economy Tables\n",
    "\n",
    "Link is available [here](https://kidb.adb.org/economies)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "68116e24-dbce-423f-b315-dbb2f71c1e3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "adb_link_excel = (\n",
    "    \"https://www.adb.org/sites/default/files/publication/900716/\"\n",
    "    f\"{sset.PATH_ADB_RAW.name}\"\n",
    ")\n",
    "with sset.PATH_ADB_RAW.open(\"wb\") as f:\n",
    "    f.write(requests.get(adb_link_excel).content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40116a24-ac34-4a9a-884d-90d2738dda07",
   "metadata": {},
   "source": [
    "### GADM v4.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "93bb86d9-d337-44ad-9a37-d5531f9f6beb",
   "metadata": {},
   "outputs": [],
   "source": [
    "GADM_NAME = sset.PATH_GADM.stem\n",
    "URL_GADM = f\"https://geodata.ucdavis.edu/gadm/gadm{sset.GADM_VERS}/{GADM_NAME}.zip\"\n",
    "\n",
    "\n",
    "with tempfile.NamedTemporaryFile() as tf:\n",
    "    tf.file.write(requests.get(URL_GADM).content)\n",
    "    with tempfile.TemporaryDirectory as d:\n",
    "        shutil.unpack_archive(tf.name, d, format=\"zip\")\n",
    "        sset.PATH_GADM.upload_file(d / f\"{GADM_NAME}.gpkg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb0c81a5-b53b-47f4-88fa-f793f8fd5c8f",
   "metadata": {},
   "source": [
    "### Gleditsch and Ward table of independent states"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cb8d303-58a4-44d3-b4a7-e5da758b1a52",
   "metadata": {},
   "outputs": [],
   "source": [
    "URL_GW_TABLE = (\n",
    "    f\"https://github.com/andybega/states/raw/master/data/{sset.PATH_GW_TABLE.name}\"\n",
    ")\n",
    "with sset.PATH_GW_TABLE.open(\"wb\") as f:\n",
    "    f.write(requests.get(URL_GW_TABLE).content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4911b7a0-b215-49cf-aa74-1bec51680efe",
   "metadata": {},
   "source": [
    "### CIA World Factbook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5292fc14-5114-45a3-ab09-0aef48459f2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# CIA World Factbook, versions 2000 to 2020\n",
    "print(f\"Downloading CIA World Factbooks to {sset.DIR_CIA_RAW.as_uri()}...\")\n",
    "with TemporaryDirectory() as tmp:\n",
    "    for f in tqdm([f\"factbook-{x}.zip\" for x in range(2000, 2021)]):\n",
    "        this_dir = sset.DIR_CIA_RAW / f[:-4]\n",
    "        if sset.FS.isdir(this_dir.as_uri()):\n",
    "            continue\n",
    "        print(f\"...downloading {f}...\")\n",
    "        with ZipFile(\n",
    "            BytesIO(\n",
    "                requests.get(\n",
    "                    \"https://www.cia.gov/the-world-factbook/about/archives/\"\n",
    "                    f\"download/{f}\"\n",
    "                ).content\n",
    "            )\n",
    "        ) as zipfile:\n",
    "            zipfile.extractall(tmp)\n",
    "        sset.FS.upload(\n",
    "            str(Path(tmp) / this_dir.name), this_dir.as_uri(), recursive=True\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93196f2e-7610-4cb8-a843-b641cb5f215c",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Various manual input disaggregated sources"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e664bfac-337d-4d2a-af4d-2b7c6ba63120",
   "metadata": {},
   "source": [
    "### Aland (`ALA`) information from Aland Statistics\n",
    "\n",
    "#### GDPpc (current PPP in euro terms)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "9f8d6f05-7668-46da-8f14-a7b04fbdba12",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Aland Islands current PPP GDPpc; we also extract Finnish GDPpc for ratio comparisons\n",
    "aland_gdp = (\n",
    "    pd.read_excel(sset.DIR_ALAND_STATISTICS_RAW / \"ala_gdp.xls\", skiprows=3)\n",
    "    .rename(columns={\"Unnamed: 0\": \"country\"})\n",
    "    .set_index([\"country\"])\n",
    ")\n",
    "aland_years = aland_gdp.columns.values\n",
    "aland_cgdpo = aland_gdp.loc[\"Åland\", :]\n",
    "fin_cgdpo = aland_gdp.loc[\"Finland\", :]\n",
    "\n",
    "# exchange rate; EMU only has down to 1999, so for convenience's sake\n",
    "# for 1995-1998, we will use 1999 rates\n",
    "wdi_xrate = (\n",
    "    pd.read_parquet(sset.DIR_WB_WDI_RAW / \"wdi_xr.parquet\")\n",
    "    .loc[(\"EMU\", list(range(1999, aland_years.max() + 1))), \"xrate\"]\n",
    "    .values\n",
    ")\n",
    "aland_xrate = np.hstack([[wdi_xrate[0]] * (1999 - aland_years.min()), wdi_xrate])\n",
    "aland_cgdpo, fin_cgdpo = aland_cgdpo * aland_xrate, fin_cgdpo * aland_xrate\n",
    "aland_df = pd.DataFrame(data={\"aland_cgdpo_pc\": aland_cgdpo, \"year\": aland_years})\n",
    "aland_df[\"ccode\"] = \"ALA\"\n",
    "fin_df = pd.DataFrame(data={\"aland_cgdpo_pc\": fin_cgdpo, \"year\": aland_years})\n",
    "fin_df[\"ccode\"] = \"FIN\"\n",
    "aland_df = pd.concat([aland_df, fin_df]).set_index([\"ccode\", \"year\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "606ac620-a293-4f78-8d3e-6174a1ce4a29",
   "metadata": {},
   "source": [
    "#### Population"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "e96b4807-5149-425b-ae88-2716c2a82e32",
   "metadata": {},
   "outputs": [],
   "source": [
    "aland_pop_df_all = pd.read_excel(\n",
    "    sset.DIR_ALAND_STATISTICS_RAW / \"ala_pop.xlsx\",\n",
    "    \"Population development\",\n",
    "    skiprows=2,\n",
    "    index_col=0,\n",
    ")\n",
    "\n",
    "aland_pop_df = aland_pop_df_all.loc[\"Population 1.1.\"].copy()\n",
    "aland_pop_df.index = aland_pop_df_all[\n",
    "    aland_pop_df_all.index.isin([\"Åland\", \"Faroe Islands\", \"Greenland\"])\n",
    "].index\n",
    "\n",
    "aland_pop_df[\"ccode\"] = aland_pop_df.index.map(\n",
    "    {\"Åland\": \"ALA\", \"Faroe Islands\": \"FRO\", \"Greenland\": \"GRL\"}\n",
    ")\n",
    "aland_pop_df = aland_pop_df.set_index(\"ccode\")\n",
    "aland_pop_df.columns = aland_pop_df.columns.astype(int)\n",
    "aland_pop_df = aland_pop_df.rename_axis(columns=\"year\").stack().rename(\"aland_pop\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "4f3d36b8-b6d8-4f12-a4b9-bef435a9ed9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "aland_df = aland_df.join(aland_pop_df, how=\"outer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c07aef1-9f96-476e-8385-508bc5fd6767",
   "metadata": {},
   "source": [
    "### `BES`: Bonaire, St Eustatius, and Saba current and non-PPP GDP from CBS (Statistics Netherlands)\n",
    "\n",
    "**Trends in the Caribbean Netherlands 2021** (link [here](https://longreads.cbs.nl/ticn2021/)) contains 2012-2018 current, non-PPP GDP for Bonaire, St Eustatius, and Saba separately. We will have them added up to represent `BES`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "e191cd5d-5381-4a42-bdaf-020739e5241d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# millions of USD (assumed to be current, non-PPP)\n",
    "bonaire = np.array([417, 434, 452, 466, 487, 480, 505]) * 1e6\n",
    "eustatius = np.array([133, 137, 131, 134, 131, 142, 128]) * 1e6\n",
    "saba = np.array([43, 46, 47, 47, 48, 48, 48]) * 1e6\n",
    "bes = pd.DataFrame(\n",
    "    data={\"year\": range(2012, 2019), \"bes_gov_gdp_nom\": bonaire + eustatius + saba}\n",
    ")\n",
    "bes[\"ccode\"] = \"BES\"\n",
    "various_df = aland_df.join(\n",
    "    bes.set_index([\"ccode\", \"year\"]).bes_gov_gdp_nom, how=\"outer\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43833846-d811-4768-bc39-85a122cab05a",
   "metadata": {},
   "source": [
    "### Saint Barthelmy (`BLM`), from CEROM\n",
    "\n",
    "Link to the document is provided [here](https://www.cerom-outremer.fr/IMG/pdf/note_cerom_pib_saint-barthelemy_-_octobre_2014.pdf). Nominal GDPpc of Saint Barthelemy is 26000 euros (1999) and 35700 euros (2010)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "ac939c3b-079f-477f-8cf1-cb44b2ecab4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# exchange rate for the years 1999 and 2010 are retrieved and multiplied\n",
    "cerom_years = [1999, 2010]\n",
    "wdi_xrate = xr_wb.loc[(\"EMU\", cerom_years), \"xrate\"].values\n",
    "cerom_gdppc_nom = np.array([26000, 35700]) / wdi_xrate\n",
    "blm = pd.DataFrame(data={\"year\": cerom_years, \"cerom_gdppc_nom\": cerom_gdppc_nom})\n",
    "blm[\"ccode\"] = \"BLM\"\n",
    "various_df = various_df.join(\n",
    "    blm.set_index([\"ccode\", \"year\"]).cerom_gdppc_nom, how=\"outer\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c6cc525-572e-4590-85bc-95a2777a56ed",
   "metadata": {},
   "source": [
    "### 2010 nominal GDP of Cocos (Keeling) Island (`CCK`) and Christmas Island (`CXR`) from House of Representative Committees of Parliament of Australia\n",
    "\n",
    "Link to the document is provided [here](https://www.aph.gov.au/parliamentary_business/committees/House_of_Representatives_Committees?url=ncet/economicenvironment/report/index.htm); please refer to Chapter 3. Only 2010 nominal GDP (in Australian dollars) are provided for these regions (`CXR`: 71 million AUS dollars, `CCK`: 15 million AUS dollars)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "715722ec-6595-43ec-b367-d084d6b55706",
   "metadata": {},
   "outputs": [],
   "source": [
    "# multiplying the AUS exchange rate of 2010 (GDP values in millions of AUS dollars)\n",
    "wdi_xrate = xr_wb.loc[(\"AUS\", 2010), \"xrate\"]\n",
    "cxr_cck = np.array([71, 15]) * 1e6 / wdi_xrate\n",
    "cxr_cck = pd.DataFrame(data={\"aus_parl_gdp_nom\": cxr_cck, \"ccode\": [\"CXR\", \"CCK\"]})\n",
    "cxr_cck[\"year\"] = 2010\n",
    "various_df = various_df.join(\n",
    "    cxr_cck.set_index([\"ccode\", \"year\"]).aus_parl_gdp_nom, how=\"outer\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c74a8002-3175-4a49-8078-3a91badaf08b",
   "metadata": {},
   "source": [
    "### Australian Government population information for Cocos (Keeling) Island (`CCK`), Christmas Island (`CXR`), and Norfolk Island (`NFK`)\n",
    "\n",
    "These links are from the Australian Bureau of Statistics (ABS):\n",
    "- `CCK`: years [2001](https://www.abs.gov.au/census/find-census-data/quickstats/2001/910053009), [2006](https://www.abs.gov.au/census/find-census-data/quickstats/2006/910053009), [2011](https://www.abs.gov.au/census/find-census-data/quickstats/2011/90102), and [2016](https://quickstats.censusdata.abs.gov.au/census_services/getproduct/census/2016/quickstat/90102)\n",
    "- `CXR`: years [2001](https://www.abs.gov.au/census/find-census-data/quickstats/2001/910052009), [2006](https://www.abs.gov.au/census/find-census-data/quickstats/2006/910052009), [2011](https://www.abs.gov.au/census/find-census-data/quickstats/2011/910052009), and [2016](https://www.abs.gov.au/census/find-census-data/quickstats/2016/90101)\n",
    "\n",
    "The following links are for `NFK`, partially from the ABS but also from a separate report from the Government of Norfolk Island:\n",
    "- 2011 and 2016 population values are provided [here](https://www.infrastructure.gov.au/territories-regions-cities/territories/norfolk-island)\n",
    "- 1986, 1991, 1996, 2001, and 2006 population values are provided [here](http://www.norfolkisland.gov.nf/sites/default/files/public/documents/ANIReports/Census/Census_2006.pdf) (see page 8, Table A2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "5294fe5a-6a72-4b57-b603-c2f9da4a9a3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# CCK and CXR, for the years 2001, 2006, 2011, 2016\n",
    "cck = [621, 572, 550, 544]\n",
    "cxr = [1446, 1349, 2072, 1843]\n",
    "aus_gov_pop = pd.DataFrame(\n",
    "    data={\n",
    "        \"aus_gov_pop\": cck + cxr,\n",
    "        \"year\": [2001, 2006, 2011, 2016] * 2,\n",
    "        \"ccode\": [\"CCK\"] * 4 + [\"CXR\"] * 4,\n",
    "    }\n",
    ")\n",
    "\n",
    "# NFK, for the years 1986, 1991, 1996, 2001, 2006, 2011, 2016\n",
    "nfk = [2367, 2285, 2181, 2601, 2523, 1796, 1748]\n",
    "nfk = pd.DataFrame(\n",
    "    data={\n",
    "        \"aus_gov_pop\": nfk,\n",
    "        \"year\": list(range(1986, 2017))[0::5],\n",
    "        \"ccode\": [\"NFK\"] * len(nfk),\n",
    "    }\n",
    ")\n",
    "aus_gov_pop = pd.concat([aus_gov_pop, nfk], axis=0).set_index([\"ccode\", \"year\"])\n",
    "\n",
    "various_df = various_df.join(aus_gov_pop.aus_gov_pop, how=\"outer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "497ef704-c4b2-41ec-9f2e-11fac71d7ffa",
   "metadata": {},
   "source": [
    "### Falkland Government GDP (current GBP) for Falkland (`FLK`)\n",
    "\n",
    "The report is provided [here](http://www.falklands.gov.fk/policy/jdownloads/Reports%20&%20Publications/Economy%20and%20Economic%20Development/State%20of%20the%20Economy%20Reports/State%20of%20the%20Falkland%20Islands%20Economy%202020.pdf) on page 18, and we will turn the values in current British Pound to current USD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "87c20e13-6ffc-473b-b3ba-c1e74ca54394",
   "metadata": {},
   "outputs": [],
   "source": [
    "# GBR xrate\n",
    "gbr_xrate = xr_wb.loc[\"GBR\", \"xrate\"]\n",
    "\n",
    "# Falkland\n",
    "flk = (\n",
    "    np.array(\n",
    "        [\n",
    "            106.0,\n",
    "            120.1,\n",
    "            97.7,\n",
    "            167.5,\n",
    "            184.7,\n",
    "            204.3,\n",
    "            160.3,\n",
    "            175.6,\n",
    "            209.0,\n",
    "            282.3,\n",
    "            220.1,\n",
    "            254.7,\n",
    "        ]\n",
    "    )\n",
    "    * 1e6\n",
    ")\n",
    "flk = pd.DataFrame(data={\"year\": range(2007, 2019), \"flk_gov_gdp_curr\": flk})\n",
    "flk[\"ccode\"] = \"FLK\"\n",
    "flk = flk.set_index([\"ccode\", \"year\"]).join(gbr_xrate, how=\"left\")\n",
    "flk[\"flk_gov_gdp_curr\"] = flk[\"flk_gov_gdp_curr\"].div(flk[\"xrate\"])\n",
    "\n",
    "various_df = various_df.join(flk.flk_gov_gdp_curr, how=\"outer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfe89224-de71-4cb8-93ed-61e3fae889a2",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Gibraltar Government GDP per capita (2006-2020) for Gibraltar (`GIB`)\n",
    "\n",
    "We will assume that the original data (link [here](https://www.gibraltar.gov.gi/uploads/statistics/2021/National%20Income/GDP%20Estimates.pdf)) is in current British Pound and turn them into current USD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "948582d8-1a64-44c2-b5fd-1b78c6315963",
   "metadata": {},
   "outputs": [],
   "source": [
    "gib = [\n",
    "    24859,\n",
    "    26714,\n",
    "    29357,\n",
    "    32570,\n",
    "    34247,\n",
    "    37369,\n",
    "    40381,\n",
    "    45032,\n",
    "    48522,\n",
    "    53433,\n",
    "    59403,\n",
    "    66691,\n",
    "    72228,\n",
    "    75467,\n",
    "    71787,\n",
    "]\n",
    "gib = pd.DataFrame(data={\"year\": range(2006, 2021), \"gib_gov_gdppc_curr\": gib})\n",
    "gib[\"ccode\"] = \"GIB\"\n",
    "gib.set_index([\"ccode\", \"year\"], inplace=True)\n",
    "gib = gib.join(gbr_xrate, how=\"left\")\n",
    "gib.loc[(\"GIB\", 2020), \"xrate\"] = gib.loc[(\"GIB\", 2019), \"xrate\"]\n",
    "gib[\"gib_gov_gdppc_curr\"] = gib[\"gib_gov_gdppc_curr\"].div(gib[\"xrate\"])\n",
    "\n",
    "various_df = various_df.join(gib.gib_gov_gdppc_curr, how=\"outer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ea62a36-b765-41e0-b61d-5118703f77fa",
   "metadata": {},
   "source": [
    "### Guernsey Government information of Guernsey (`GGY`)\n",
    "\n",
    "- Population (1986, 1991, 1996, 2001, and 2009-2021): see [March 2021 Report](https://www.gov.gg/CHttpHandler.ashx?id=149564&p=0) for 2011-2021 on p.3; [March 2019 Report](https://www.gov.gg/CHttpHandler.ashx?id=123171&p=0) for 2009-2019 on p.3; [traditional census report](https://www.gov.gg/census) for the years 1986, 1991, 1996, and 2001 (click on the download file \"Historic Population and Employment Data\" to see the `.xls` file)\n",
    "- GDP (2004-2020): reports [here for 2010-2020](https://www.gov.gg/CHttpHandler.ashx?id=147608&p=0), [here for 2009](https://gov.gg/CHttpHandler.ashx?id=111088&p=0), and [here for 2004-2008](https://gov.gg/CHttpHandler.ashx?id=90671&p=0); note that we will later use `ypk_fn.smooth_fill` for 2004-2008 values due to there being discrepancies between pre-2015 reports and the later ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "e2ae0c21-1017-48e8-a8e6-6e4c7c94b848",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2011-2021 information from March 2021 Census\n",
    "ggy_2021_census = [\n",
    "    62915,\n",
    "    63085,\n",
    "    62732,\n",
    "    62341,\n",
    "    62234,\n",
    "    62208,\n",
    "    62106,\n",
    "    62290,\n",
    "    62681,\n",
    "    63083,\n",
    "    63448,\n",
    "]\n",
    "\n",
    "# 2009-2010 information from March 2019 Census\n",
    "ggy_2019_census = [62274, 62431]\n",
    "\n",
    "# 1986, 1991, 1996, 2001 information from 'Historic Population and Employment Data'\n",
    "ggy_traditional = [55482, 58867, 58681, 59807]\n",
    "\n",
    "# merging them into a dataset\n",
    "ggy = pd.DataFrame(\n",
    "    data={\n",
    "        \"ggy_gov_pop\": np.hstack([ggy_traditional, ggy_2019_census, ggy_2021_census]),\n",
    "        \"year\": np.hstack([[1986, 1991, 1996, 2001], range(2009, 2022)]),\n",
    "    }\n",
    ")\n",
    "ggy[\"ccode\"] = \"GGY\"\n",
    "ggy.set_index([\"ccode\", \"year\"], inplace=True)\n",
    "ggy[\"ggy_gov_gdp_curr\"] = np.nan\n",
    "ggy.loc[(\"GGY\", list(range(2009, 2021))), \"ggy_gov_gdp_curr\"] = (\n",
    "    np.array(\n",
    "        [\n",
    "            2458,\n",
    "            2423,\n",
    "            2629,\n",
    "            2615,\n",
    "            2715,\n",
    "            2779,\n",
    "            2816,\n",
    "            2934,\n",
    "            3101,\n",
    "            3170,\n",
    "            3244,\n",
    "            3178,\n",
    "        ]\n",
    "    )\n",
    "    * 1e6\n",
    ")\n",
    "ggy_alt = np.array([1453, 1465, 1584, 1774, 1841, 1832, 1909, 2033, 2117, 2186]) * 1e6\n",
    "ggy = ggy.join(\n",
    "    pd.DataFrame(\n",
    "        data={\n",
    "            \"year\": range(2004, 2014),\n",
    "            \"ggy_gov_gdp_alt\": ggy_alt,\n",
    "            \"ccode\": [\"GGY\"] * len(ggy_alt),\n",
    "        }\n",
    "    ).set_index([\"ccode\", \"year\"]),\n",
    "    how=\"outer\",\n",
    ").join(gbr_xrate, how=\"left\")\n",
    "ggy.loc[(\"GGY\", 2020), \"xrate\"] = ggy.loc[(\"GGY\", 2019), \"xrate\"]\n",
    "for i in [\"ggy_gov_gdp_curr\", \"ggy_gov_gdp_alt\"]:\n",
    "    ggy[i] = ggy[i].div(ggy[\"xrate\"])\n",
    "various_df = various_df.join(ggy.drop([\"xrate\"], axis=1), how=\"outer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "514922dd-9157-4622-9cec-c8dc57a81b81",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Jersey Government information for `JEY`\n",
    "\n",
    "- Population (2000-2019): link to the Jersey Resident Population from Statistics Jersey is [here](https://www.gov.je/SiteCollectionDocuments/Government%20and%20administration/R%20Population%20Estimate%20Current%2020180620%20SU.pdf); see page 2.\n",
    "- GDP (2012-2020): report [here](https://opendata.gov.je/dataset/national-accounts/resource/ae620bf3-41be-4461-adb8-2220ab7cb000?inner_span=True); these values are in constant 2020, non-PPP GBP. We turn these into constant 2017, non-PPP USD using official exchange rate of 2019 (since 2020 value is unavailable) and deflate 2019 USD to 2017 USD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "3b95545f-e748-4964-84a6-8a3074870a8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# USD deflator\n",
    "defla = (\n",
    "    pd.read_excel(sset.PATH_PWT_RAW)\n",
    "    .set_index([\"countrycode\", \"year\"])\n",
    "    .loc[\"USA\", \"pl_gdpo\"]\n",
    ")\n",
    "\n",
    "# population data 2000-2019, in millions\n",
    "jey = (\n",
    "    np.array(\n",
    "        [\n",
    "            0.088400,\n",
    "            0.088900,\n",
    "            0.089300,\n",
    "            0.089600,\n",
    "            0.090100,\n",
    "            0.091000,\n",
    "            0.092300,\n",
    "            0.094000,\n",
    "            0.095400,\n",
    "            0.096200,\n",
    "            0.097100,\n",
    "            0.098100,\n",
    "            0.099000,\n",
    "            0.100000,\n",
    "            0.101000,\n",
    "            0.102700,\n",
    "            0.104200,\n",
    "            0.105600,\n",
    "            0.106700,\n",
    "            0.107800,\n",
    "            np.nan,\n",
    "        ]\n",
    "    )\n",
    "    * 1e6\n",
    ")\n",
    "jey = pd.DataFrame(data={\"jey_gov_pop\": jey, \"year\": list(range(2000, 2021))})\n",
    "jey[\"ccode\"], jey[\"jey_gov_gdp_const\"] = \"JEY\", np.nan\n",
    "jey.set_index([\"ccode\", \"year\"], inplace=True)\n",
    "jey.loc[(\"JEY\", list(range(2012, 2021))), \"jey_gov_gdp_const\"] = (\n",
    "    np.array(\n",
    "        [\n",
    "            4495,\n",
    "            4504,\n",
    "            4678,\n",
    "            4737,\n",
    "            4745,\n",
    "            4787,\n",
    "            4881,\n",
    "            4988,\n",
    "            4528,\n",
    "        ]\n",
    "    )\n",
    "    * 1e6\n",
    ")\n",
    "jey[\"jey_gov_gdp_const\"] /= gbr_xrate.loc[2019] * defla.loc[2019]\n",
    "various_df = various_df.join(jey, how=\"outer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dac16ba7-d304-4c7f-aa5d-e39c329b31ae",
   "metadata": {},
   "source": [
    "### Norfolk Island (`NFK`) GDP per capita information (as a percentage of Australia) from Treadgold (1999) and Treadgold (1998)\n",
    "\n",
    "GDPpc as a percentage of the Australian level for the years 1951-52 are shown in [Treadgold (Asia Pacific Viewpoint, 1999)](https://doi.org/10.1111/1467-8373.00095) and similar percentage for 1995-96 are shown in [Treadgold (Pacific Economic Bulletin, 1998)](https://openresearch-repository.anu.edu.au/handle/1885/157535). Let us multiply these with the Australian GDP per capita from Penn World Tables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "d22fcd86-a336-4b07-bf97-c4db7a47ea2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "nfk_yrs = [1951, 1952, 1995, 1996]\n",
    "nfk_ratios = np.array([0.39, 0.39, 1.12, 1.12])\n",
    "aus_info = (\n",
    "    pd.read_excel(sset.PATH_PWT_RAW)\n",
    "    .set_index([\"countrycode\", \"year\"])\n",
    "    .loc[(\"AUS\", nfk_yrs), [\"cgdpo\", \"rgdpna\", \"pop\"]]\n",
    ")\n",
    "aus_info[\"treadgold_cgdpo_pc\"] = aus_info[\"cgdpo\"].div(aus_info[\"pop\"]) * nfk_ratios\n",
    "aus_info[\"treadgold_rgdpna_pc\"] = aus_info[\"rgdpna\"].div(aus_info[\"pop\"]) * nfk_ratios\n",
    "nfk_info = aus_info.reset_index().rename(columns={\"countrycode\": \"ccode\"})[\n",
    "    [\"ccode\", \"year\", \"treadgold_rgdpna_pc\", \"treadgold_cgdpo_pc\"]\n",
    "]\n",
    "nfk_info[\"ccode\"] = \"NFK\"\n",
    "nfk_info.set_index([\"ccode\", \"year\"], inplace=True)\n",
    "various_df = various_df.join(\n",
    "    [nfk_info.treadgold_cgdpo_pc, nfk_info.treadgold_rgdpna_pc], how=\"outer\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64ea16b3-a0d5-4ed6-a713-e5fc3a0bf23a",
   "metadata": {},
   "source": [
    "### Pitcairn Island (`PCN`) information from the Government of Pitcairn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d381ea9-367f-4967-8235-794685f199e3",
   "metadata": {},
   "source": [
    "#### Nominal GDP of 2006\n",
    "\n",
    "`PCN` has an estimate of approximately 217,000 New Zealand dollars in 2006 (from [this link](https://web.archive.org/web/20150705134639/http://www.government.pn/policies/Pitcairn%20Island%20SDP%202012-2016.pdf#page=4) for a WayBackMachine Archive of the Government of Pitcairn's \"Pitcairn Islands Strategic Development Plan\").\n",
    "\n",
    "#### Population in 1937\n",
    "\n",
    "1937 population is 237, and this information is from the Pitcairn Island [government website](http://www.immigration.gov.pn/community/the_people/index.html). Population information for 2000-2016 and 2020-2021 is provided in CIA WFB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "f250817c-32f9-4220-a60c-37129c131217",
   "metadata": {},
   "outputs": [],
   "source": [
    "# GDP in millions of USD and population in millions of people\n",
    "pcn_gdp_nom_2006 = 0.217 * 1e6 / xr_wb.loc[(\"NZL\", 2006), \"xrate\"]\n",
    "pcn = pd.DataFrame(\n",
    "    [[\"PCN\", 1937, np.nan, 237], [\"PCN\", 2006, pcn_gdp_nom_2006, np.nan]],\n",
    "    columns=[\"ccode\", \"year\", \"pcn_gov_gdp_nom\", \"pcn_gov_pop\"],\n",
    ").set_index([\"ccode\", \"year\"])\n",
    "various_df = various_df.join([pcn.pcn_gov_gdp_nom, pcn.pcn_gov_pop], how=\"outer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cef51f2-9b83-43ab-83b0-7e6cea4e9685",
   "metadata": {},
   "source": [
    "### North Korea (`PRK`) GDP growth estimates from the Bank of Korea\n",
    "\n",
    "The link to the information is [here](https://www.bok.or.kr/portal/main/contents.do?menuNo=200091); we use the real GDP (in South Korean won) to calculate the real GDP growth for 2018-2020, and use these rates later with MPD GDP values (which only have GDPpc values up to 2018)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "b4f8d292-e6a8-419c-aaa3-5b58faa87bac",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2018-2020 real GDP in billions of South Korean won (approx. millions of USD)\n",
    "prk_2018_2020_real_gdp = np.array([328030, 329189, 314269])\n",
    "prk_2019_2020_gr = prk_2018_2020_real_gdp[1:] / prk_2018_2020_real_gdp[0:-1]\n",
    "prk = pd.DataFrame(\n",
    "    data={\n",
    "        \"ccode\": [\"PRK\"] * 2,\n",
    "        \"year\": [2019, 2020],\n",
    "        \"bok_prk_real_gdp_gr\": prk_2019_2020_gr,\n",
    "    }\n",
    ").set_index([\"ccode\", \"year\"])\n",
    "various_df = various_df.join(prk.bok_prk_real_gdp_gr, how=\"outer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2dda177-0a09-46db-b9f9-ec72ce4fbf0b",
   "metadata": {},
   "source": [
    "### 2018-2019 nominal GDP per capita from St. Helena Government\n",
    "\n",
    "Link to **St Helena's Sustainable Economic Development Plan** is [here](https://www.sainthelena.gov.sh/wp-content/uploads/2020/07/SEDP-EOY-Progress-Report-Final-160720.pdf); see page 5 for the nominal GDP per capita values. Note that this is **not** the GDP per capita value for St. Helena, Ascension, and Tristan de Cunha (represented by `SHN`) but **just St. Helena**. We will use this GDP per capita with the entire `SHN` population to get the GDP values. Note also that for convenience, these values are recorded under the country code `SHN`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "a62edb39-e61f-4a0a-9918-c2dca5c6ab0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# values are in British pounds\n",
    "st_helena = pd.DataFrame(\n",
    "    [[\"SHN\", 2018, 8490], [\"SHN\", 2019, 8320]],\n",
    "    columns=[\"ccode\", \"year\", \"st_helena_gov_gdppc_nom\"],\n",
    ").set_index([\"ccode\", \"year\"])\n",
    "st_helena[\"st_helena_gov_gdppc_nom\"] = (\n",
    "    st_helena[\"st_helena_gov_gdppc_nom\"]\n",
    "    / xr_wb.loc[(\"GBR\", [2018, 2019]), \"xrate\"].values\n",
    ")\n",
    "various_df = various_df.join(st_helena.st_helena_gov_gdppc_nom, how=\"outer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fc4d7f3-7734-4b94-a62d-82933895ade6",
   "metadata": {},
   "source": [
    "### Svalbard and Jan Mayen (`SJM`) 2009-2021 population from Statistics Norway\n",
    "\n",
    "Link to the relevant page from **Statistics Norway** is [here](https://www.ssb.no/en/statbank/table/07429); the values below are half-year populations (`h1` having the first half, `h2` having the second half), and we will average them to get the yearly populations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "32da1fb8-1ac2-4833-b048-3e6bbddba265",
   "metadata": {},
   "outputs": [],
   "source": [
    "# data from 2009-2021\n",
    "h1 = [2085, 2052, 2017, 2115, 2158, 2100, 2185, 2152, 2145, 2214, 2258, 2428, 2459]\n",
    "h2 = [2140, 2071, 2140, 2195, 2195, 2118, 2189, 2162, 2210, 2310, 2379, 2417, 2552]\n",
    "sjm = np.round(0.5 * (np.array(h1) + np.array(h2)), 0)\n",
    "sjm = pd.DataFrame(data={\"stat_nor_pop\": sjm, \"year\": list(range(2009, 2022))})\n",
    "sjm[\"ccode\"] = \"SJM\"\n",
    "various_df = various_df.join(sjm.set_index([\"ccode\", \"year\"]).stat_nor_pop, how=\"outer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc3bffd4-67d5-42c5-ab78-7bb1c5a19130",
   "metadata": {},
   "source": [
    "### United States Minor Outlying Islands (`UMI`) 1980, 1990, 2000 population from the U.S. Census\n",
    "\n",
    "Link to the Census report is [here](https://www.census.gov/history/pdf/2000-minoroutlyingislands.pdf). \n",
    "\n",
    "As the report says, `UMI` is composed of Baker, Howland, and Jarvis Islands, Johnston Atoll, Kingman Reef, Midway Islands, Navassa Island, Palmyra Atoll, and Wake Island. According to the 2022 CIA World Factbook (links [here for Wake Island](https://www.cia.gov/the-world-factbook/countries/wake-island/#people-and-society) and [here for others](https://www.cia.gov/the-world-factbook/countries/united-states-pacific-island-wildlife-refuges/#people-and-society)), all of these locations are either closed to public or used as wildlife refuges; therefore, in our projections (2010-2100), we will assume that  population, GDP, and capital stock are 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "4ec5faf7-60c0-4c4e-ba5c-746680a4b7af",
   "metadata": {},
   "outputs": [],
   "source": [
    "umi = np.array([1082, 193, 316])\n",
    "umi = pd.DataFrame(data={\"us_census_pop\": umi, \"year\": [1980, 1990, 2000]})\n",
    "umi[\"ccode\"] = \"UMI\"\n",
    "various_df = various_df.join(\n",
    "    umi.set_index([\"ccode\", \"year\"]).us_census_pop, how=\"outer\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "191b620b-2e7a-4bfb-a6f1-2dfbab774b67",
   "metadata": {},
   "source": [
    "### Exporting cleaned, various-source dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "ff111b2a-3a26-4ee4-9583-fa5bde960d37",
   "metadata": {},
   "outputs": [],
   "source": [
    "save(various_df, sset.PATH_INC_POP_AUX)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "959f9f18-431c-4f66-b246-896ae2639f6a",
   "metadata": {},
   "source": [
    "## CoDEC GTSM Historical Surge Values (Muis et al. 2020)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50ae84ac-2e70-49d0-a3b4-a3f54309c9e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "r = requests.get(\n",
    "    f\"https://zenodo.org/record/3660927/files/{sset.PATH_GEOG_GTSM_SURGE.name}\"\n",
    ")\n",
    "with sset.PATH_GEOG_GTSM_SURGE.open(\"wb\") as f:\n",
    "    f.write(r.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5efcb44-1f60-4377-9d01-e970b918ca5e",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Further data requiring separate manual instructions\n",
    "\n",
    "### UN Statistics National Accounts (Analysis of Main Aggregates; abbreviated as UN AMA)\n",
    "\n",
    "#### UN AMA nominal (current prices) GDP per capita information\n",
    "\n",
    "1. Travel to this [link](https://unstats.un.org/unsd/snaama/Basic) to get to the UN Statistics National Accounts search page.\n",
    "2. Select all countries and all years available, and select \"GDP, Per Capita GDP - US Dollars\".\n",
    "3. Select \"Export to CSV\", and you will download the file `Results.csv`. Rename this file as `un_snaama_nom_gdppc.csv`. We save this in `sset.DIR_UN_AMA_RAW`.\n",
    "\n",
    "#### UN AMA nominal (current prices) GDP information\n",
    "\n",
    "1. Similar to the nominal GDP per capita information, travel to this [link](https://unstats.un.org/unsd/snaama/Basic) to get to the UN Statistics National Accounts search page.\n",
    "2. Select all countries and all years available, and select \"GDP, at current prices - US Dollars\".\n",
    "3. Select \"Export to CSV\", and you will download the file `Results.csv`. Rename this file as `un_snaama_nom_gdp.csv`. We save this in `sset.DIR_UN_AMA_RAW`.\n",
    "\n",
    "#### Mapping to region and subregion\n",
    "\n",
    "1. Go to [https://unstats.un.org/unsd/methodology/m49/overview/](https://unstats.un.org/unsd/methodology/m49/overview/) and click on the `CSV` download button.\n",
    "2. Save this to `sset.PATH_UN_REGION_DATA`\n",
    "\n",
    "### OECD region-level information\n",
    "\n",
    "#### OECD: population (region-level)\n",
    "1. Go to the following OECD Stat website: link [here](https://stats.oecd.org/)\n",
    "2. On the left, find the header \"Regions and Cities\" and click the \"+\" button.\n",
    "3. From the drop down menu, click on \"Regional Statistics\".\n",
    "4. Again from the drop down menu, click on \"Regional Demography.\"\n",
    "5. Finally, select \"Population by 5-year age groups, small regions TL3.\" Make sure that \"Indicator\" is selected as \"Population, All ages\".\n",
    "6. Download the file by selecting \"Export,\" then \"Text File (CSV).\"\n",
    "7. When a pop-up appears, select \"Default format\" then \"Download.\" Rename the file as `REGION_DEMOGR.csv`.\n",
    "8. Place this file in `sset.DIR_OECD_REGIONS_RAW`.\n",
    "\n",
    "#### OECD: GDP (region-level, in millions of constant 2015 PPP USD)\n",
    "1. Similar to the population information, go to the following OECD Stat website: link [here](https://stats.oecd.org/)\n",
    "2. On the left, find the header \"Regions and Cities\" and click the \"+\" button.\n",
    "3. From the drop down menu, click on \"Regional Statistics\".\n",
    "4. Again from the drop down menu, click on \"Regional Economy.\"\n",
    "5. Finally, select \"Gross Domestic Product, Small regions TL3.\" Make sure that \"Measure\" is selected as \"Millions USD, constant prices, constant PPP, base year 2015\".\n",
    "6. Download the file by selecting \"Export,\" then \"Text File (CSV).\"\n",
    "7. When a pop-up appears, select \"Default format\" then \"Download.\" Rename the file as `REGION_ECONOM.csv`\n",
    "8. Place this file in `sset.DIR_OECD_REGIONS_RAW`.\n",
    "\n",
    "### IMF investment-to-GDP ratio, population, and GDP\n",
    "\n",
    "1. Travel to this [link](https://www.imf.org/en/Publications/SPROLLs/world-economic-outlook-databases#sort=%40imfdate%20descending) to get to the World Economic Outlook Databases page.\n",
    "2. Click on the latest \"World Economic Outlook Database\" link on the page; for our purposes, we have used the latest available one, which was \"World Economic Outlook Database, April 2022\" (may be updated in the future).\n",
    "3. Click \"By Countries\", then click \"ALL COUNTRIES\", then click \"CONTINUE\" on the page that says \"Select Countries.\"\n",
    "4. Under the \"NATIONAL ACCOUNTS\" tab, check the following categories:\n",
    "   - Gross domestic product, current prices (U.S. DOLLARS)\n",
    "   - Gross domestic product per capita, current prices (U.S. DOLLARS)\n",
    "   - Gross domestic product per capita, constant prices (PURCHASING POWER PARITY; 2017 INTERNATIONAL DOLLARS)\n",
    "   - Implied PPP conversion rate (NATIONAL CURRENCY PER CURRENT INTERNATIONAL DOLLAR)\n",
    "   - Total investment (PERCENT OF GDP)\n",
    "5. Under the \"PEOPLE\" tab, check the category \"Population,\" then click on \"CONTINUE.\"\n",
    "6. Under the tab \"DATE RANGE,\" use the earliest year for \"Start Year\" (1980, in our case), and the latest non-future year for \"End Year\" (2020, in our case).\n",
    "7. Under the tab \"ADVANCED SETTINGS\", click on \"ISO Alpha-3 Code\" for getting country codes. \n",
    "8. Click on \"PREPARE REPORT.\" Then, click on \"DOWNLOAD REPORT.\" Saved data should be in Excel format.\n",
    "10. Save this file as `sset.PATH_IMF_WEO_RAW`.\n",
    "\n",
    "### World Bank Intercomparison Project 2017 (WB ICP 2017): Construction Cost Index\n",
    "\n",
    "While most World Bank data can be downloaded by using `pandas_datareader.wb`, it seems that variables in WB ICP 2017 - including `1501200:CONSTRUCTION`, which is necessary for SLIIDERS - cannot be downloaded using the said module (despite being searchable in the module using `pandas_datareader.wb.search`). Therefore, we follow the below manual process for downloading the WB ICP 2017 dataset.\n",
    "1. Use [this link](https://databank.worldbank.org/embed/ICP-2017-Cycle/id/4add74e?inf=n) to access WB ICP 2017 in table format.\n",
    "2. Upon entering the webpage, look to the upper right corner and click on the icon with downward arrow with an underline. This should prompt the download.\n",
    "3. When the download finishes, there should be a `.zip` file called `ICP 2017 Cycle.zip`. Access the `.csv` file whose name ends in `_Data.csv` (there should be two files in the `.zip` file, the other being a file whose name ends in `_Series - Metadata.csv`).\n",
    "4. Save that `.csv` file as `sset.PATH_EXPOSURE_WB_ICP`.\n",
    "\n",
    "### IIASA and OECD models' GDP and population projections (2010-2100, every 5 years)\n",
    "\n",
    "1. Go to the following IIASA SSP Database website: link [here](https://tntcat.iiasa.ac.at/SspDb); you may need to register and create your log-in.\n",
    "2. In the above tabs, there is a tab called \"Download\"; click on it.\n",
    "3. Under \"SSP Database Version 2 Downloads (2018)\" and under the sub-header \"Basic Elements\", there is a download link for `SspDb_country_data_2013-06-12.csv.zip`. Click and download the said `.zip` file.\n",
    "4. Extract and save the `SspDb_country_data_2013-06-12.csv`. Again, for our purposes, we save this as `sset.PATH_IIASA_PROJECTIONS_RAW`.\n",
    "\n",
    "### LandScan 2022\n",
    "\n",
    "1. Go to https://landscan.ornl.gov/\n",
    "2. Click Download in the upper right. Provide your information and select the Landscan Global product for 2022.\n",
    "3. Extract and save all extracted files to `sset.PATH_LANDSCAN_RAW`.\n",
    "\n",
    "### Global geoids, based on select Earth Gravitational Models (EGMs)\n",
    "1. Go to the following International Centre for Global Earth Models (ICGEM) website (link [here](http://icgem.gfz-potsdam.de/calcgrid)) to reach the page \"Calculation of Gravity Field Functionals on Ellipsoidal Grids\".\n",
    "2. Under **Model Selection**, select `XGM2019e_2159`.\n",
    "3. Under **Functional Selection**, select `geoid`.\n",
    "4. Under **Grid selection**, there's a **Grid Step [°]** option. Change the value to **0.05**. Also, make sure that the **Reference System** is `WGS84`.\n",
    "5. Due to download size constraints, we need to download this data in 4 chunks. Do the following:\n",
    "   - Split the full range of latitudes and longitudes in half, which yields the following 4 combinations of longitude-latitude ranges: $([-180, 0], [-90, 0]), ([-180, 0], [0, 90]), ([0, 180], [-90, 0])$, and $([0, 180], [0, 90])$.\n",
    "   - Under **Grid selection** again, one can select the range of longitudes and latitudes. Select one of the above combinations and press `start computation`.\n",
    "   - This will open up a new tab for calculations, which may take some time to complete. Once this is done, press **Download Grid**.\n",
    "   - Once the download is complete, go back to the previous page with **Model selection**, **Functional selection**, and more. Make sure the selections you made are intact, select another longitude-latitude combination, and repeat the process until there are no combinations left.\n",
    "6. Once the above steps are done, go back to Step 2 above; but instead of selecting `XGM2019e_2159` for **Model selection**, select `EGM96`. Go through the Steps 3 to 5 again with this new selection.\n",
    "7. Once the downloads for `XGM2019e_2159` and `EGM96` are complete, you should have 4 files for each model (8 in total, in `.gdf` format). Save the `XGM2019e_2159` files in `sset.DIR_GEOG_DATUMS_XGM2019e_WGS84` and `EGM96` files in `sset.DIR_GEOG_DATUMS_EGM96_WGS84`.\n",
    "\n",
    "### Global Mean Dynamic Ocean Topography (MDT) from AVISO+\n",
    "**Note**: While this dataset has a relatively open license, you will first need to obtain a MY AVISO+ account, which requires verification from the AVISO+ team and may take several days or weeks.\n",
    "1. Go to the following AVISO+ website for **MDT CNES-CLS22**: link [here](https://www.aviso.altimetry.fr/en/data/products/auxiliary-products/mdt/mdt-global-cnes-cls.html).\n",
    "2. Once on the page, download the dataset through your MY AVISO+ account (click on `access via MY AVISO+` link and follow the instructions).\n",
    "3. After following the instructions, you will acquire the file `mdt_cnes_cls18_global.nc.gz`. Extract the file `mdt_cnes_cls18_global.nc` from the `.gz` file and save it as `sset.PATH_GEOG_MDT_RAW`.\n",
    "\n",
    "### Aggregate Capital Stock Estimations from 122 Countries from Berlemann and Jan-Erik Wesselhöft (2017, Review of Economics)\n",
    "\n",
    "Link to the paper is [here](https://www.degruyter.com/document/doi/10.1515/roe-2017-0004/html?lang=en#j_roe-2017-0004_tab_001_w2aab2b8c20b1b7b1ab1b1c10Aa). This dataset can be acquired by contacting Michael Berlemann (one of the authors). Original dataset file was named `Capital Stock Data Update 2017.xlsx`; we save it as `Berlemann_Wesselhoft_2017.xlsx` and place it in `sset.DIR_YPK_RAW`.\n",
    "\n",
    "### Fariss, Anders, Markowitz, and Barnum (2022, Journal of Conflict Resolutions) data for GDP, GDP per capita, and population\n",
    "\n",
    "1. Go to the following [Dataverse link](https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/FALCGS).\n",
    "2. Click on **Access Dataset**, and a drop-down menu will appear. Click on **Download ZIP**.\n",
    "3. Once the download is complete, you will have a `.zip` file named `dataverse_files.zip`. We save this as `Fariss_JCR_2022.zip` and place it in `sset.DIR_YPK_RAW`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "438f5274-f65c-41a3-8deb-7e62069f6138",
   "metadata": {},
   "source": [
    "### Other SLIIDERS input datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b048d8c-ef4e-4119-9de5-9ec341f0d5a5",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "There are some datasets that were manually constructed for use in `SLIIDERS`. They are available for download on Zenodo. Please download each file from the Zenodo deposit [here](https://doi.org/10.5281/zenodo.6449231) and copy to the paths designated for each dataset.\n",
    "\n",
    "#### 1. `gtsm_stations_eur_tothin.parquet`\n",
    "Path: `sset.PATH_GTSM_STATIONS_TOTHIN`\n",
    "\n",
    "These 5,637 station points are a subset of the full CoDEC dataset (n=14,110) representing sites along European coastlines that are roughly five times more densely-spaced compared to the rest of the globe, as described in Muis et al. 2020. This subset of points are those that will be thinned by 5x to approximately match the density of CoDEC coast stations globally. Some manual inclusion criteria for this subset was applied in GIS due to the fact that simply seeking to select dense European stations based on the “station_name” field in the dataset, which contains the substring “eur” for all European locations, results in an over-selection of desired points (n=6,132), with many North African coastal points that are not densely-spaced containing this substring in their “station_name” as well. Therefore, European points were manually identified, with small islands, such as in the Mediterranean, included if their land mass contained 5 or more station points, which guarantees that they will be represented by at least one station point following the 5x thinning process. The resultant subset of points is used as a data input for the coastal segment construction in the preprocessing of the SLIIDERS dataset.\n",
    "\n",
    "#### 2. `gtsm_stations_ciam_ne_coastline_snapped.parquet`\n",
    "Path: `sset.PATH_GEOG_GTSM_SNAPPED`\n",
    "\n",
    "This contains the locations of all of the CoDEC (Muis et al. 2020) nodes, snapped to the Natural Earth coastlines dataset.\n",
    "\n",
    "#### 3. `ciam_segment_pts_manual_adds`\n",
    "Path: `sset.PATH_SEG_PTS_MANUAL`\n",
    "\n",
    "This contains a list of additional segment centroids to create in order to ensure that each coastal admin1 has at least one segment assigned to it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0323ad0-afc1-43bf-9e5b-82e37e5455ce",
   "metadata": {},
   "source": [
    "### CoastalDEM v2.1\n",
    "1. Acquire the global 1 arc-second CoastalDEM dataset from Climate Central (https://go.climatecentral.org/coastaldem/).\n",
    "2. Save all 1-degree GeoTIFF files in `sset.DIR_COASTALDEM`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}