Adding some notes on existing verification systems/packages for reference:

• scoringrules: https://github.com/frazane/scoringrules
  • Python, uses numba implementations from properscoring
  • Includes more metrics than properscoring
  • Seems like a good fit for implementation in TEEHR
• properscoring: https://github.com/properscoring/properscoring
  • CRPS and Brier Score only
  • Python, numba implementations with guvectorize
• METplus: https://dtcenter.org/community-code/model-evaluation-tools-met/documentation
  • Through NCAR's Developmental Testbed Center
  • METplus verification system
  • Seems like a framework consisting of statistical engines, visualization, data management
  • Focuses on gridded NWP models (originally focused on WRF)
  • ~25 ensemble verification stats (ENCT - Ensemble Continuous Statistics)
  • Includes python wrappers
• evalhyd: https://hydrogr.github.io/evalhyd/
  • Deterministic and Probabilistic metrics
  • Bootstrapping
  • Includes other functionalities like memoization, missing data handling, masking, transformation, diagnostics)
  • C++ core with python and R bindings
• hydrostats: https://github.com/BYU-Hydroinformatics/Hydrostats
  • Developed by BYU
  • Mostly a collection of lots of metrics, with some added functionality (viz, preprocessing)
  • Includes sample data
  • Goodness-of-fit metrics from HydroErr module: https://hydroerr.readthedocs.io/en/stable/list_of_metrics.html
  • Includes ensemble metrics
  • I found ens_crps to be ~1e5 times slower than scoringrules implementation (same results)
• hydrotools.metrics: https://noaa-owp.github.io/hydrotools/hydrotools.metrics.html
  • Does not include ensemble metrics
• NWM Ensemble Verification System: https://www.weather.gov/media/owp/oh/rfcdev/docs/EVS_MANUAL_V1.0.pdf
  • Doesn't look like the code is publicly available (java?)
  • GUI-based
• scores: https://scores.readthedocs.io/en/stable/included.html
  • Includes many continuous, probabilistic, and categorical metrics
  • Built around dask and xarray
  • Handles gridded and point-based input data

Some additional notes on the NWM Ensemble Verification System (EVS) (Note: EVS is no longer available or maintained -- it has been/is being replaced by WRES)

reference: https://www.sciencedirect.com/science/article/pii/S1364815210000204?via%3Dihub

EVS is a GUI-based ensemble verification system built specifically for NWS written in java.

It contains three main components:

• Verification Units (VU): Paired forecast/obs timeseries for a single variable at a single geographic location (sort of like TEEHR's configuration?). Metrics are calculated per VU.
• Aggregation Units: Allows users to aggregate performance metrics across VU's (optional).
• Output: Provides tabular and graphical results.

Included metrics:

• Mean error
• RMSE
• Correlation coef
• Brier Score (binary classification)
• Brier Skill Score
• Mean CRPS
• Mean CRPS reliability
• CRPSS
• ROC Score (Relative Opearting Characteristic) - discriminate between binary events (flooding/non-flooding). ROC curves show prob of detection vs. prob of false detection
• Skill scores are relative to some reference forecast. Can be user-defined. Typically climatology.

Visualizations:

• Mean error in prob diagram (MEPD)
• Mean capture rate diagram (MCRD)
• Spread-bias diagram
• Reliability diagram
• ROC diagram
• Modified box plots

Also just making a note that an experimental HEFS API data service has recently been released by NWS.

Description: https://www.weather.gov/media/notification/pdf_2023_24/pns24-77_exp_hefs_api_data_services.pdf

Notebooks: https://github.com/NOAA-OWP/data-service-notebooks/blob/master/HEFS/2_ensemble_plotting_demo.ipynb