Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue 553: interval_coverage_ testing improvements #554

Merged
merged 8 commits into from
Jan 5, 2024
1 change: 1 addition & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ The update introduces breaking changes. If you want to keep using the older vers
- Files ending in ".Rda" were renamed to ".rds" where appropriate when used together with `saveRDS()` or `readRDS()`.
- `score()` now calls `na.omit()` on the data, instead of only removing rows with missing values in the columns `observed` and `predicted`. This is because `NA` values in other columns can also mess up e.g. grouping of forecasts according to the unit of a single forecast.
- added documentation for the return value of `summarise_scores()`.
- Added unit tests for `interval_coverage_quantile()` and `interval_coverage_dev_quantile()` in order to make sure that the functions provide the correct warnings when insufficient quantiles are provided.

# scoringutils 1.2.2

Expand Down
17 changes: 17 additions & 0 deletions tests/testthat/test-metrics-quantile.R
Original file line number Diff line number Diff line change
Expand Up @@ -601,6 +601,17 @@ test_that("interval_coverage_quantile rejects wrong inputs", {
)
})

test_that("interval_coverage_quantile throws a warning when a required quantile is not available", {
dropped_quantile_pred <- predicted[, -4]
dropped_quantiles <- quantile[-4]
expect_warning(
interval_coverage_quantile(
observed, dropped_quantile_pred, dropped_quantiles, range = 50
),
"To compute the interval coverage for a range of 50%, the quantiles `0.25, 0.75` are required. Returning `NA`"
)
})


# ============================================================================ #
# `interval_coverage_dev_quantile` ===================================== #
Expand All @@ -617,6 +628,12 @@ test_that("interval_coverage_dev_quantile works", {
interval_coverage_dev_quantile(observed, predicted, quantile),
manual
)
expect_warning(
interval_coverage_dev_quantile(
observed, predicted, c(quantile[-4], 0.76)
),
"To compute inteval coverage deviation, all quantiles must form central symmetric prediction intervals. Missing quantiles: 0.24, 0.75. Returning `NA`."
)
})


Expand Down
Loading