Full spatial calibration on 14 parameters with tutorial Notebook (#286)

* Executions (no changes)

* Execute Notebooks to verify whether they still work

Note that the json files needed to run these files had to be added manually!

* Create Notebook to explore the localised vaccinations in a model

* Add vaccination and VOC parameters to spatial model

* Update Proximus mobility data

* Explicitly add cp1252 encoding to read_csv

Check encoding in Python: https://stackoverflow.com/questions/37177069/how-to-check-encoding-of-a-csv-file/52648848

* Add steepness parameter k to VOC function for delta variant

* Update raw Sciensano data

* Add more documentation and partially add framework for VOC and vaccination

* Copy all parameters for VOC and vaccination simulation from national model

* Add comment on four lines that don't do jack

* Add some lines in the spatial vaccination

* Rename spatial model class to COVID19_SEIRD_spatial_vacc

* Solve confusion with scalar e_i versus vector e_i

* Ignore results from second-wave calibration with manually tweaked init cond

* Make vaccination parameters vectors rather than scalars

* Add output figures from calibration with manually tweaked init cond

* Start changing the ODEs for doubled spatial vacc model, and comment-out previous spatial vacc model

* Change all e_i scalars to e_i_eff etc.

* Final changes to the ODEs in the spatial vacc model

* Delete previous spatial vacc model

* Execute Notebook up until error occurs

* Execute Notebook

* Add (again) the default spatial model

* Creater corner plot of latest wave-2 calibration (poor fit result though ...)

* Adjust Notebook

* Add extra note to Notebook

* Add explanatory comment for optional argument

* Fix problem with truth value of optional data argument

* Change __init__ of vaccination_function to also handle spatial data

* Add try-except procedure in __init__ and replace all df_sciensano --> df

* Make sure that the start_week values are read as Timestamp objects

* Add read_coordinates_nis import

* Update all dataframes when update=True, and read from separate CSVs (faster!)

* Delete superfluous CSV (split in three new CSVs)

* Add three separate CSVs with vaccination data for faster read-in times

* Change description of interim vaccination data file

* Update raw vaccination data from the internet

* Fix bug with placement in try-except statement of vaccination function

* Create plot of vaccination per age and per place

* Rename and order Notebook

* Fix problem with wrong age extrapolation in vaccination function

* Fix bug in time determination of vaccination function

* Add new paragraph for spatial case in vaccination function future scenarios

* Clean up the notebook and start initialising the model

* Make age and space stratification number more elegant in code

* Make visual representation of plot a bit nicer and more compact

* Fix correct parameter names and change e_h to e_h_eff

* Change hard-coded 9 to overarching age class number

* Delete print thingie

* Make a preliminary simulation and show vaccinated people from this simulation

* Execute Quick-update for mobility

* Add simulation plot and complete simulation (from start of pandemic)

* Update national Sciensano data

* Subtract number of recovered people that get vaccinated (forgotten before!)

* Save figures locally for Overleaf presentation

* Simplify dS_v formula

* Add reason for keeping the D_v category (in comment)

* Add some more comments in the spatial vaccination model

* Nothing much, must simple execution

* Update Google Mobility Data

* Update mobility data graph

* Execute Notebooks with newest data

* Update Proximus mobility visualisation between Antwerp and Brussels

* Add seasonality between 2020-01 and 2021-09

* Add visualisations of vaccination degree and VOC presence

* Create new hospitalisation and vaccination plots

* Fix typo

* Slightly alter hospitalisation image

* Slightly alter vaccination image

* Add seasonality per transmission coefficient (bad idea I think)

* Add seasonality factor to spatial vaccination model

* Add calibration over all dates without seasonality

* Minor changes

* Update wrong figure

* Create better seasonality figure

* Show smoothened function with data (better than previous one)

* Add line to save fig

* Add function to handle dates before Sept 2020 in policies_WAVE2_full_relaxation

* Add line to deal with social contact behaviour before Sept 2020

* Add policies_all to contact function

* Save bad PSO output

* Add comments on how to include spatially stratified contact matrices

* Delete local fictitious vaccination strategy functions

* Create Notebook for full-pandemic calibration of spatial model

* Rename Notebook to the core of what it is used for (show graphs of VOC/vacc)

* id update (not sure, minor stuff)

* Delete policies_WAVE1 (outside class) and create class make_seasonality_function

* Fix typo

* Finalise Notebook for spatial calibration on full data set

* Create script to run full spatial calibration on HPC

* Manually overwrite nwalkers

* Plot new timeline (for analysis paper)

* Add image demonstrating Proximus coverage (with one arr missing)

* Create small Notebook to plot Proximus coverage

* Adapt Notebook showing full calibration procedure (copied to .py script)

* Adapt Notebook for social contact visualisation

* Executions only

* Comment out "if __main__" statement

* Change label subscript to superscript

* Create corner plot of test calibration

* Add full-pandemic calibration post-processing

* Add test output (can be deleted, in fact)

Co-authored-by: Michiel Rollier <[email protected]>

.gitignore

@@ -163,3 +163,4 @@ data/interim/model_parameters/COVID19_SEIRD/calibrations/arr/arr_WAVE1-FULL_job-
 data/interim/model_parameters/COVID19_SEIRD/calibrations/arr/arr_WAVE1-FULL_job-FULL_1000xPSO_20000xMCMC_arr_3-index-in-frac_2021-06-10.json
 data/interim/model_parameters/COVID19_SEIRD/calibrations/arr/arr_WAVE1-R0_job-R0_1000xPSO_20000xMCMC_arr_3-index-in-frac_2021-06-10.json
 data/interim/model_parameters/COVID19_SEIRD/calibrations/arr/arr_WAVE2-FULL_job-FULL_1000xPSO_20000xMCMC_arr_2021-06-08.json
+data/interim/model_parameters/COVID19_SEIRD/calibrations/arr/arr_WAVE2-FULL_job-FULL_1000xPSO_20000xMCMC_arr_2021-08-24.json

data/README.md

@@ -176,7 +176,7 @@ The dataset contained, for each Belgian province, a column of 'unknowns', indica
 
 + `clusters.csv` contains the number of clusters traced back to 1) families, 2) WZCs, 3) schools, 4) workplaces and 5) others over the period 2020-12-28 until 2021-02-21. Data extracted from the weekly Sciensano reports, available at https://covid-19.sciensano.be/nl/covid-19-epidemiologische-situatie. Last visited 2021-03-31.
 + `sciensano_detailed_mortality.csv` contains the number of deaths (incidence and cumulative number) per age group, in total, in hospitals and in nursing homes. Since our model does not predict nursing home deaths, model output must be compared to deaths in hospitals. Data conversion was done using the script` ~/notebooks/preprocessing/sciensano-mortality-data-analysis.py`. You must place the super secret detailed hospitalization dataset `COVID19BE_MORT_RAW.csv` in the same folder as this script in order to run it. Permission of Sciensano is needed to obtain the raw dataset.
-+ `COVID19BE_VACC_MUNI_format.csv` contains the formatted number of (first) vaccine doses per week, per municipality (NUTS5) and per age group in a pandas dataframe with three-level multiindex.
++ `COVID19BE_VACC_MUNI_format_{agg}.csv` contains the formatted number of (first) vaccine doses per week, per geographical aggregation and per age group in a pandas dataframe with three-level multiindex.
 
 #### Demographic data