Energy Score Functions for Model Evaluation #289
Conversation
trobacker
changed the title
Double braket. Co-authored-by: Zhian N. Kamvar <[email protected]>
There was a problem hiding this comment.
I believe the scoring procedure used here is slightly different from what is discussed here:
https://github.com/reichlab/variant-nowcast-hub?tab=readme-ov-file#background
Happy to meet and discuss on Monday! Overall great coding though!
… * 100 samples of multinomial counts, on each day/date.
|
Hi @rogersbw, I modified the ES calculation to include 10000 samples, 100 (posterior draws) * 100 (samples of multinomial counts) = 10000 total on each day/date. Do you mind reviewing that piece (last commit) and confirming that's correct? It certainly takes longer to run now and I'll plan on optimizing the code next - i.e. replacing for loops with say functional programming techniques. I want to be sure the ES algorithm is correct first. Thanks! |
src/model_scoring_functions.R
Outdated
|
|
||
| ## Generate 100 multinomial observations for samp_props | ||
| # Need the N for each loc/day from the validation data | ||
| N <- sum(subset(targets, |
There was a problem hiding this comment.
This can be moved outside of the loop, as for each combination of loc and day, the N will stay constant through this loop.
There was a problem hiding this comment.
Also, how are day/loc combinations with 0 observations handled? As I'm running through this, it seems to be keeping in the 0 observation days and giving them an energy score of 0 (perfect score). I believe we will want these to be NA values instead.
There was a problem hiding this comment.
Thanks for the comments, Ben! I modified the energy scores to have NA when the observed counts sum to 0. I also moved the N assignment. I'll focus on code optimization next.
src/model_scoring_functions.R
Outdated
| df_scores <- rbind(df_scores, df_temp) | ||
| next | ||
| } | ||
|
|
There was a problem hiding this comment.
Looks good, when we're doing code optimization, could initialize an array of the correct size filled with NA, and then replace only for positive counts.
|
As discussed in our modeling meeting today, I'm going to have the energy scores available for all dates/locs and build another function that will filter for ones we've generally agreed to use for final model scoring according to the hub evaluation scheme. |
Co-authored-by: Ben Rogers <[email protected]>
Co-authored-by: Ben Rogers <[email protected]>
Anti-join by date, location. Co-authored-by: Ben Rogers <[email protected]>
|
FWIW, I wrote a different version that I was hoping to be computationally faster which uses |
|
Looks good, merge away! |
R script containing functions to get energy scores from model output and produce basic summaries of scores.