initial commit

.gitignore

@@ -0,0 +1,92 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# DotEnv configuration
+.env
+
+# Database
+*.db
+*.rdb
+
+# Pycharm
+.idea
+
+# VS Code
+.vscode/
+
+# Spyder
+.spyproject/
+
+# Jupyter NB Checkpoints
+.ipynb_checkpoints/
+
+# exclude data from source control by default
+/data/
+
+# Mac OS-specific storage files
+.DS_Store
+
+# vim
+*.swp
+*.swo
+
+# Mypy cache
+.mypy_cache/
+
+# dask
+dask-worker-space

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "LocalizeSL"]
+	path = LocalizeSL
+	url = [email protected]:bobkopp/LocalizeSL.git

HISTORY.rst

@@ -0,0 +1,11 @@
+History
+=======
+
+.. v1.0.0:
+
+v1.0.0
+------
+
+Released on April 8, 2022
+
+Initial commit. Version associated with initial manuscript submission.

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 Climate Impact Lab
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,92 @@
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.6010452.svg)](https://doi.org/10.5281/zenodo.6010452)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.6012027.svg)](https://doi.org/10.5281/zenodo.6012027)
+
+# Sea Level Impacts Input Dataset by Elevation, Region, and Scenario (SLIIDERS)
+
+This repository hosts the code used to create the [SLIIDERS-ECON](https://doi.org/10.5281/zenodo.6010452) and [SLIIDERS-SLR](https://doi.org/10.5281/zenodo.6012027) datasets. The SLIIDERS datasets contain current and forecasted physical and socioeconomic metrics from 2000-2100 - organized by coastal segment, elevation slice, and scenario - for use as inputs to global coastal climate impacts research.
+
+**SLIIDERS-ECON** contains socioeconomic variables, varying horizontally and vertically over space. **SLIIDERS-SLR** contains Monte Carlo projections of Local Sea Level Rise under different emissions and ice sheet dynamics assumptions, based on the outputs of [LocalizeSL](https://github.com/bobkopp/LocalizeSL). Coastal segments in SLIIDERS-ECON can be matched to gridded LSLR projections in SLIIDERS-SLR via the `SLR_site` key.
+
+All work utilizing this code or the resulting SLIIDERS datasets should cite Depsky, Bolliger et al. 2022 (in prep.). See [License](#license) for details.
+
+## Installation
+Most users will want to just use the datasets directly, accessible at the DOIs linked above. If you wish to recreate and/or modify the datasets, which we encourage, you will need to run the Jupyter notebooks in this repository. A collection of helper functions, organized into a Python package, is necessary to run the notebooks and can be found within the `sliiders` directory. A simple pip install will install this package
+
+```bash
+pip install -e sliiders
+```
+
+In addition, you will need to have [Dask Gateway](https://gateway.dask.org) installed and configured to execute the parallel, Dask-backed workflows contained in this repo. Advanced users can use other Dask Cluster backends (including simply running [Dask Distributed](https://distributed.dask.org) locally), but doing so will require modifying the cluster setup portion of notebooks that employ dask.
+
+A Conda environment file better specifying a full environment needed to execute all of the workflows in this repo is in development and will be posted when complete.
+
+## Filepaths and other settings
+All filepaths and settings for the notebooks can be found within `settings.py`. Before moving onto executing different parts of this repository, please adjust these settings to match your directory structure and desired values. Most values will not need to be updated unless you change a particular dataset. However, at minimum you should:
+
+1. Update the `DIR_DATA` filepath within this file to point to the root directory within which all of the data consumed and generated by this workflow will live.
+2. Update `DASK_IMAGE` to point to a Docker Image that you will use for Dask workers (advanced users not using Dask Gateway may not need this parameter).
+
+## Package Structure
+* `sliiders`: Contains `.py` files with essential settings and functions for the SLIIDERS workflow
+  - `settings.py`: Contains essential settings, including various parameters and data storage directories
+  - `gcs.py`: Contains functions related to the use of Google Cloud Storage (GCS). Users running workflows locally or on a different cloud provider are encouraged to contribute similar modules for other contexts.
+  - `io.py`: Contains various I/O-related functions
+  - `spatial.py`: Contains functions for executing spatial and geographic operations including those related to shapefiles, grid-cell level operations, and more.
+  - `dask.py`: Contains utility functions for working with dask clusters
+  - `country_level_ypk.py`: Contains functions for cleaning and working with country-level socioeconomic data, especially for the workflow in `notebooks/country_level_ypk`
+
+* `notebooks`: contains the workflows to create SLIIDERS-ECON and SLIIDERS-SLR.
+
+## Instructions (Generating SLIIDERS-ECON and SLIIDERS-SLR)
+
+To generate **SLIIDERS-ECON** and **SLIIDERS-SLR**, please follow the directions in `notebooks/README.md` and other readme files in the subdirectories within `notebooks` to learn about how to execute the workflows.  
+\
+The list and order of notebooks to run is reproduced in full here, along with any necessary manual steps. Click the `docs` link for each workflow to navigate to the relevant directory's page.
+
+1. `create-SLIIDERS-SLR` ([docs](notebooks/create-SLIIDERS-SLR)): Workflow to generate **SLIIDERS-SLR**
+   1. `download-ifile-to-gcs.ipynb`
+   2. `convert-mat-version.ipynb`
+   3. `generate-projected-lsl.ipynb`
+   4. `retrieve-num-gcms.ipynb`
+   5. `process-localizesl-output.ipynb`
+2. `create-SLIIDERS-ECON` ([docs](notebooks/create-SLIIDERS-ECON)): Workflow to generate **SLIIDERS-ECON**
+   1. `download-sliiders-econ-input-data.ipynb`
+   2. `country_level_ypk` ([docs](notebooks/create-SLIIDERS-ECON/country_level_ypk)): Workflow for organizing and projecting GDP (Y), population (P), capital stock (K), and related variables for historical (1950-2020) and future (2010-2100) timelines.
+      1. `ypk1_prep_clean.ipynb`
+      2. `ypk2_reorg_and_impute_ypk.ipynb`
+      3. `ypk3_demo_ratios_historical_reg.ipynb`
+      4. `ypk4_impute_hist_capital.ipynb`
+      5. `ypk5_projected_yp.ipynb`
+      6. `ypk6_projected_capital.ipynb`
+   3. `exposure` ([docs](notebooks/create-SLIIDERS-ECON/exposure)): Workflow to generate present-day exposure grid.
+      1. `1-create-coastline-segments.ipynb`
+      2. `2-create-segment-regions.ipynb`
+      3. `3-fill_missing_litpop_with_geg.ipynb`
+      4. `4-vectorize-wetlands.ipynb`
+      5. `5-get_positive_elev_tiles.ipynb`
+      6. `6-generate_datum_conversion_grid.ipynb`
+      7. `7-create_dem_mss.ipynb`
+      8. `8-generate_protected_areas.ipynb`
+      9. `9-generate_exposure_tiles.ipynb`
+      10. `10-combine_exposure_tiles.ipynb`
+   4. `create-SLIIDERS-ECON.ipynb`
+
+The resulting datasets can be found at these paths, defined in `settings.py`:  
+**SLIIDERS-ECON**: `PATH_SLIIDERS_ECON`  
+**SLIIDERS-SLR**: `PATH_SLIIDERS_SLR`
+
+## Support
+Please file an issue for any problems you encounter.
+
+## Contributing
+We encourage community contributions. At the moment, we have no contribution template. Please fork the project and file a Merge Request to propose your addition. Clearly define the contribution that the Merge Request is making and, when any issues have been resolved, we will merge the new code.
+
+## Authors
+The original authors of this code include:
+- Daniel Allen
+- Ian Bolliger
+- Junho Choi
+- Nicholas Depsky
+
+## License
+This code is licensed under the [MIT License](./LICENSE). However, we request that wherever this code or the SLIIDERS datasets are used, that the underlying manuscript (Depsky et al. 2022) is cited. A citation guide will be posted once the manuscript preprint is available.

notebooks/README.md

@@ -0,0 +1,6 @@
+# Instructions
+
+This directory contains sub-directories to produce final SLIIDERS outputs. The order of execution is as follows. For further instructions, we refer to the `README.md` files in the respective sub-directories.
+
+1. `create-SLIIDERS-SLR`: Workflow to generate **SLIIDERS-SLR**, a dataset of gridded local sea-level Monte Carlo samples for each RCP scenario, year (decadal), and site ID (defined by LocalizeSL).
+2. `create-SLIIDERS-ECON`: Workflow to generate **SLIIDERS-ECON**, a dataset containing socioeconomic variables by coastal segment, elevation, Shared Socioeconomic Pathway scenario. Note that this workflow uses the SLIIDERS-SLR dataset to find nearest grid cells to match to coastal segments.

notebooks/create-SLIIDERS-ECON/README.md

@@ -0,0 +1,10 @@
+# Workflow for generating the SLIIDERS-ECON dataset
+
+This directory contains notebooks to generate the **SLIIDERS-ECON** dataset. The final output for future projections is a Zarr store containing socioeconomic variables binned by coastal segment, elevation slice, and Shared Socioeconomic Pathway.
+
+The steps to produce the final output are as follows.
+
+1. Use `download-sliiders-econ-input-data.ipynb` to download necessary datasets, including various country-level datasets and datasets such as including World Bank Intercomparison Project 2017 and construction cost index by Lincke and Hinkel (2021, *Earth's Future*).
+2. Go to the directory `country_level_ypk` and follow the instructions in the `README.md` in that directory. The workflow in `country_level_ypk` cleans (and when necessary, imputes) various country-level socioeconomic variables.
+3. Go to the directory `exposure` and follow the instructions in the `README.md` in that directory. The workflow in `exposure` generates current-day global exposure data by coastal segment, elevation, and other variables.
+4. Use `create-SLIIDERS-ECON.ipynb` to combine disparate data sources to generate the final output.

notebooks/create-SLIIDERS-ECON/country_level_ypk/README.md

@@ -0,0 +1,67 @@
+# Workflow for organizing and projecting GDP (`Y`), population (`P`), capital stock (`K`), and related variables for historical (1950-2020) and future (2010-2100) timelines
+
+**This version: last updated on March 30, 2022**
+
+## 1. Overview
+
+This directory contains the data acquistion, clean-up, and projection notebook files to organize and project variables including GDP, GDP per capita (GDPpc), population, and capital stock for both historical (1950-2020) and future or projected (2010-2100) timelines. Many of the data sources used to generate historical and future panels have missing data, and therefore efforts were made to impute these missing data through either some form of extrapolation or other established methods. Also, efforts were made to keep the PPP and USD units consistent (e.g., constant 2019 PPP USD) across different sources having different vintages of PPP and USD units.
+
+Below is a quick summary of what each file seeks to accomplish (where the header `ypk` stands for "GDP, population, and capital stock").
+1. `ypk1_prep_clean.ipynb`: cleans up selected raw datasets requiring more attention than others to be consistent and workable with other datasets.
+2. `ypk2_reorg_and_impute.ipynb`: reorganizes the raw and previously-cleaned historical datasets so that each variable considered has a single, consistent stream of values for each country. After this process, imputes missing GDPpc, GDP, and population values that might still be missing from the cleaned historical dataset.
+3. `ypk3_demo_ratios_historical_reg.ipynb`: contains code to clean and extrapolate demographic (age-group) ratios and create the "demographic variables" necessary to conduct the "historical regression" (According to Higgins, 1998) of finding the relationship between investment-to-GDP ratio (I/Y ratio) and demographic variables, (relative) GDPpc, and GDPpc growth rate. Furthermore, the said historical regression is conducted to acquire estimates of investment-to-GDP ratios for missing country-years.
+4. `ypk4_impute_hist_capital.ipynb`: contains code to use the historical and estimated investment-to-GDP ratios to create current-PPP investment values. These are used to replicate the initial-year capital stock estimation (country-by-country) as described in Inklaar, Woltjer, and Albarrán (2019). Also, the investment values are used in conjunction with GEG-15 and LitPop data sources to fill in missing values for the latter parts of the historical capital stock data. The end product is a filled (1950-2020) capital stock data for all relevant countries.
+5. `ypk5_projected_yp.ipynb`: contains code to clean up GDP, GDPpc, and population for the future timeline, with some basic extrapolation conducted for countries with missing projections.
+6. `ypk6_projected_capital.ipynb`: generates projections of capital stocks based on the Dellink et al. (2017) methodology.
+
+For running these files, note that they have to be **run consecutively** (i.e., from `ypk1~` to `ypk7~`). Each notebook file contains basic descriptions on what each step does; in all cases, the cells must be run consecutively from top to bottom.
+
+## 2. Basic description of key variables
+
+We describe below some key variables produced by the above process. Note that our naming conventions largely follow Penn World Table 10.0.
+- `cgdpo_19`: Current PPP (purchasing power parity) GDP in millions of 2017 and 2019 USD 
+- `cgdpo_pc_19`: Current PPP GDP per capita in ones of 2017 and 2019 USD
+- `rgdpna_19`: (National account-based) GDP in millions of constant 2019 PPP USD
+- `rgdpna_pc_19`: (National account-based) GDP per capita in ones of constant 2019 PPP USD
+- `cn_19`: Current PPP capital stock in millions of 2019 USD
+- `rnna_19`: Capital stock in millions of constant PPP 2019 USD
+- `pop`: Population in millions of people
+- `k_movable_ratio`: ratio movable capital out of total physical capital (values in <img src="https://render.githubusercontent.com/render/math?math=[0, 1]">)
+- `iy_ratio`: Investment-to-GDP ratio
+- `delta`: Physical capital depreciation rate
+
+Note that for GDP, GDP per capita, and capital stock variables, there are also versions with `_17` at the end instead of `_19`. For current PPP variables, this means using 2017 USD; for constant PPP variables, this means using constant 2017 PPP USD (i.e., constant PPP of 2017 and 2017 USD).
+
+## 3. Output storage
+
+We import the SLIIDERS `settings.py` as `sset`, which can be done as follows:
+```
+from sliiders import as settings as sset
+```
+For the aggregate long-panel format historical and future timeline variables, you may refer to the following:
+1. Historical: `sset.DIR_YPK_FINAL / "gdp_gdppc_pop_capital_1950_2020.parquet"`
+2. Future: `sset.DIR_YPK_FINAL / "gdp_gdppc_pop_capital_proj_2010_2100.parquet"`
+
+where the metadata (e.g., units and sources) are also attached to the respective files.
+
+## 4. Regression results for imputing missing historical investment-to-GDP ratios
+
+We elaborate on the regression involving investment-to-GDP ratios mentioned in Section A3.2 in the notebook `ypk4_demo_ratios_historical_reg.ipynb`. The said notebook also contains information on how to derive each variable involved. We present the results below, where the dependent variable is investment-to-GDP ratio (denoted as <img src="https://render.githubusercontent.com/render/math?math=\frac{I}{Y}"> in the notebook).
+
+| Variables | (1) | (2) | (3) | (4) |
+| ------ | :------: | :------: | :------: | :------: |
+| <img src="https://render.githubusercontent.com/render/math?math=\hat{g}"> | 0.405 <br/> (0.161) | 0.346<br/>(0.076) | 0.502<br/>(0.201) | 0.480<br/>(0.129) |
+| <img src="https://render.githubusercontent.com/render/math?math=\hat{g}^2"> | 0.864<br/>(0.742) | 0.515<br/>(0.611) | 0.506<br/>(0.879) | 0.493<br/>(0.915) |
+| <img src="https://render.githubusercontent.com/render/math?math=\hat{yhr}"> | -0.021<br/>(0.052) | -0.027<br/>(0.052) | 0.076<br/>(0.022) | 0.108<br/>(0.016) |
+| <img src="https://render.githubusercontent.com/render/math?math=\hat{yhr}^2"> | 0.004<br/>(0.007) | 0.003<br/>(0.006) | -0.011<br/>(0.005) | -0.015<br/>(0.005) |
+| <img src="https://render.githubusercontent.com/render/math?math=D_1"> | 0.184<br/>(0.186) |  | 0.348<br/>(0.190) |  |
+| <img src="https://render.githubusercontent.com/render/math?math=D_2"> | -0.008<br/>(0.035) |  | -0.038<br/>(0.030) |  |
+| <img src="https://render.githubusercontent.com/render/math?math=D_3"> | -0.000<br/>(0.002) |  | 0.001<br/>(0.001) |  |
+| <img src="https://render.githubusercontent.com/render/math?math=D_1\times\hat{g}"> | 3.988<br/>(3.149) |  | 1.784<br/>(3.945) |  |
+| <img src="https://render.githubusercontent.com/render/math?math=D_2\times\hat{g}"> | -0.797<br/>(0.570) |  | -0.465<br/>(0.597) |  |
+| <img src="https://render.githubusercontent.com/render/math?math=D_3\times\hat{g}"> | 0.040<br/>(0.028) |  | 0.026<br/>(0.026) |  |
+|  <img src="https://render.githubusercontent.com/render/math?math=N">  | 11145 | 11145 | 11145 | 11145 |
+| Country fixed effects | Yes | Yes | No | No |
+| Adjusted  <img src="https://render.githubusercontent.com/render/math?math=R^2">  | 0.325 | 0.315 | 0.068 | 0.054 |
+| AIC | -12712 | -12557 | -9317 | -9157 |
+| BIC | -11153 | -11042 | -9236 | -9120 |