diff --git a/sessions/forecast-evaluation-of-multiple-models.qmd b/sessions/forecast-evaluation-of-multiple-models.qmd index e99469e..6b34bc3 100644 --- a/sessions/forecast-evaluation-of-multiple-models.qmd +++ b/sessions/forecast-evaluation-of-multiple-models.qmd @@ -353,7 +353,7 @@ As in the [forecasting concepts session](forecasting-concepts), we will start by ::: {.callout-tip} ## Reminder: Key things to note about the CRPS - Small values are better - - As it is an absolute scoring rule it can be difficult to use to compare forecasts across scales. + - When scoring absolute values (e.g. number of cases) it can be difficult to use to compare forecasts across scales (i.e., when case numbers are different, for example between locations or at different times). ::: First by forecast horizon. @@ -376,19 +376,19 @@ sc_scores |> ::: {.callout-tip} ## Take 5 minutes -How do the CRPS scores change based on forecast date? -How do the CRPS scores change with forecast horizon? +How do the CRPS values change based on forecast date? +How do the CRPS values change with forecast horizon? ::: ::: {.callout-note collapse="true"} ## Solution -How do the CRPS scores change based on forecast horizon? +How do the CRPS values change based on forecast horizon? - All models have increasing CRPS with forecast horizon. - The more mechanistic model has the lowest CRPS at all forecast horizon. - The more stastical model starts to outperform the random walk model at longer time horizons. -How do the CRPS scores change with forecast date? +How do the CRPS values change with forecast date? - The more statistical model does particularly poorly around the peak of the outbreak but outperforms the random walk model. - The more mechanistic model does particularly well around the peak of the outbreak versus all other models @@ -477,7 +477,7 @@ log_sc_scores <- log_sc_forecasts |> ::: {.callout-tip} -Reminder: For more on scoring on the log scale see [this paper on scoring forecasts on transformed scales](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1011393). +Reminder: For more on scoring on the log scale see the paper by @bosse2023scorin. ::: ### At a glance diff --git a/sessions/forecast-evaluation.qmd b/sessions/forecast-evaluation.qmd index f305e44..07bb221 100644 --- a/sessions/forecast-evaluation.qmd +++ b/sessions/forecast-evaluation.qmd @@ -366,7 +366,7 @@ log_scores <- log_sc_forecasts |> score() ``` -For more on scoring on the log scale see [this paper on scoring forecasts on transformed scales](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1011393). +For more on scoring on the log scale see the paper by @bosse2023scorin. ### At a glance diff --git a/sessions/ueifid.bib b/sessions/ueifid.bib index d897462..84c8194 100644 --- a/sessions/ueifid.bib +++ b/sessions/ueifid.bib @@ -42,5 +42,23 @@ @Article{gneiting2007 issn = {1537-274X}, doi = {10.1198/016214506000001437}, url = {http://dx.doi.org/10.1198/016214506000001437}, - publisher = {Informa UK Limited} + publisher = {Informa UK Limited}, + note = {PDF available at https://sites.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf} +} + +@Article{bosse2023scorin, + author = {Bosse, Nikos I. and Abbott, Sam and Cori, Anne and van + Leeuwen, Edwin and Bracher, Johannes and Funk, Sebastian}, + title = {Scoring epidemiological forecasts on transformed scales}, + journal = {PLOS Computational Biology}, + year = 2023, + editor = {McCaw, James M}, + volume = 19, + number = 8, + month = aug, + pages = {e1011393}, + issn = {1553-7358}, + doi = {10.1371/journal.pcbi.1011393}, + url = {http://dx.doi.org/10.1371/journal.pcbi.1011393}, + publisher = {Public Library of Science (PLoS)} }