use pydantic to define data structure

[sunt05]

Use a pydantic-based class to define all data structures, ensuring consistency and improving sustainability for maintenance.

[dayantur]

Hi all,
I have created a branch from sunt05/issue299 called dayantur/adding_pydantic_rules and add commits there today.
I mainly tested the def_config_suews.py manually changing values in config-suews.yml, and then extended some classes to validate range consistency for parameters (i.e. parameters with max and min values). Then, I implemented a way to see all the ValueError messages at once (instead of only the first one occurring in the code) by using ExceptionGroup.

More specifically:

• I have extended a bit some of the classes with model_validator:
  -- added @model_validator (validate_porosity_range) in class DectrPropertie
  -- added @model_validator (validate_cap_range) in class DectrProperties
  -- added @model_validator (validate_albedo_range) in class VegetatedSurfaceProperties
  -- added @model_validator (validate_albedo_range) in class VegetatedSurfaceProperties
  -- added @model_validator (validate_crw_range and validate_snowalb_range) in SnowParams
• introduced ExceptionGroup to handle multiple validation errors
  -- see screenshot
  

[dayantur]

I've created two new issues related to my last comment here:

• one for keeping track of what validation rule/constrain we have already added/we need to add next:
  pydantic todo list - validators #300 (comment)

• one for discussing the use of ExceptionGroup in def_config_suews.py:
  pydantic handling multiple errors #301 (comment)

[sunt05]

Just to follow up on this - I've completed the framework of to_df_state for converting the new YAML-based input into df_state.

The remaining task is to address the missing columns/variables in both the YAML file and Pydantic class definitions.

To continue with this work, run python def_config_suews.py under SUEWS/schema/dev, and the missing columns will be printed out.

[dayantur]

After making sure that my branch was up to date with @sunt05, I worked on a new @model_validator (validate_rsl_zd_range) in class BuildingProperties and pushed it in dayantur/adding_pydantic_rules.

The new @model_validator is related to RSL scheme as described in #302 by @vitorlavor and represents a first example of how we can avoid some dangerous regimes for the model run by controlling the inputs upfront.

This is the warning we get if we do not avoid the regime:

I've added this new @model_validator to the list at #300.

[dayantur]

@sunt05, I've tried to run def_config_suews.py today, and I got a (weird to me) kernel error when entering storageheatmethod: 1 in YAML file config-suews.yml and trying a run with sp.run_supy(df_forcing, df_state_test):

2024-11-19 16:48:08,382 - SuPy - INFO - All cache cleared.
2024-11-19 16:48:09,554 - SuPy - INFO - ====================
2024-11-19 16:48:09,554 - SuPy - INFO - Simulation period:
2024-11-19 16:48:09,554 - SuPy - INFO -   Start: 2012-01-01 00:05:00
2024-11-19 16:48:09,554 - SuPy - INFO -   End: 2013-01-01 00:00:00
2024-11-19 16:48:09,554 - SuPy - INFO - 
2024-11-19 16:48:09,554 - SuPy - INFO - No. of grids: 1
2024-11-19 16:48:09,554 - SuPy - INFO - SuPy is running in serial mode
 >>> See problems.txt for serious issues in the run <<<
 ERROR! SUEWS run stopped.
 problem: In SUEWS_OHM.f95: bad value for qn1 found during qs calculation.
 See problems.txt for more info.
STOP Fatal error in SUEWS!

The error is not present when trying sp.run_supy(df_forcing, df_state):

2024-11-19 16:51:06,779 - SuPy - INFO - All cache cleared.
2024-11-19 16:51:07,931 - SuPy - INFO - ====================
2024-11-19 16:51:07,931 - SuPy - INFO - Simulation period:
2024-11-19 16:51:07,931 - SuPy - INFO -   Start: 2012-01-01 00:05:00
2024-11-19 16:51:07,931 - SuPy - INFO -   End: 2013-01-01 00:00:00
2024-11-19 16:51:07,931 - SuPy - INFO - 
2024-11-19 16:51:07,931 - SuPy - INFO - No. of grids: 1
2024-11-19 16:51:07,931 - SuPy - INFO - SuPy is running in serial mode
2024-11-19 16:51:16,304 - SuPy - INFO - Execution time: 8.4 s
2024-11-19 16:51:16,304 - SuPy - INFO - ====================

I tried changing storageheatmethod: 2 and supy can run with both df_state_test and df_state (and give outputs).

Any idea of what can this be related to?

[sunt05]

No worries—that's expected and can be reproduced here.

The YAML input was somewhat randomly created, so such errors are anticipated.

Let's continue adding rules and hope the input YAML errors can be properly detected.

[dayantur]

No worries—that's expected and can be reproduced here.

The YAML input was somewhat randomly created, so such errors are anticipated.

Let's continue adding rules and hope the input YAML errors can be properly detected.

Thanks @sunt05! I also tried to plot outputs from df_state_test, and this is what I got (again, storageheatmethod = 2):

I suppose this might be again related to the YAML being randomly populated.

[sunt05]

#301 has been separated from this issue as an ongoing task to conclude this stage.