pydantic handling multiple errors

[dayantur]

In def_config_suews.py in branch dayantur/adding_pydantic_rules (from sunt05/issue29), I've implemented a way to see all the exceptions together instead of having the script stopping after raising the first ValueError. This might be useful for the user, so they can see all the consistency errors together/save time in understanding the problem they are having with the configuration of the .yml file (see screenshot below).

But this might not be enough. For how it is now structured the code, we also have cases like:

If we change the alb_min in the config-suews.yml to be alb_min < 0 AND we also change pormin_dec and pormax_dec to be inconsistent:

we only have printed:

so we are not actually able to see that there is also another problem in another set of parameters.

So, my suggestion is -if we agree that it might be actually useful for the user to see all the exceptions and not only the first one - to avoid using ge and le in Field() and convert them into @model_validator cases (leaving in Field() only the description of the variable).

Let me know what you think!

[sunt05]

Hi @dayantur, I've done some quick research on different validation approaches in Pydantic. Please see below:

• https://www.perplexity.ai/page/pydantic-field-validation-prin-eNZzvJJ3Qyy3wgr8HECYUQ

• https://www.perplexity.ai/page/downstream-impacts-of-approach-EOZojrXYT7qLy9U_oARg4g

Please let me know your thoughts.
Or, we could try both and see which works better for us.

[dayantur]

Hi @sunt05, I will have a look at this and let you know.

I've converted some Field ranges into model_validator (as comments) today, and committed in my branch dayantur/adding_pydantic_rules.

But before merging these commits to sunt05/issue299 (or continuing with the conversion), I think it might be probably better for me to read first this documentation and have a clearer idea of these validation approaches

[dayantur]

Hi @sunt05 -- as discussed, I found a way to use Field(), @model_validator, and still get all the errors from the program (from both Field() and @model_validator).

This is a test_script.py that mimic the structure we have now in def_config_suews.py, and has one validation made in a Field(), and one validation made in a @model_validator (n.b.: the rules here are nonsense and made up just for testing):

from pydantic import BaseModel, Field, ValidationError, model_validator
import yaml
from typing import List

class SnowAlb(BaseModel):
    snowalb: float = Field(ge=0, le=1, description="Snow albedo")

class BuildingProperties(BaseModel):
    faibldg: float 
    bldgh: float

    @model_validator(mode="after")
    def test_validator(self) -> "BuildingProperties":
        if self.faibldg < self.bldgh:
            raise ValueError(
                f"Frontal area index of buildings (faibldg) should not be less than building height (bldgh). "
            )
        return self

class SUEWSConfig(BaseModel):
    snowalb: SnowAlb
    bldgprop: BuildingProperties

if __name__ == "__main__":
    exceptions: List[str] = []

    try:
        with open("./config_test.yml", "r") as file:
            yaml_config = yaml.safe_load(file)
        suews_config = SUEWSConfig(**yaml_config[0])

    except ValidationError as e:
        exceptions.append(str(e))

    if exceptions:
        print("Validation Errors Detected:\n")
        for exc in exceptions:
            print(exc)
    else:
        print("All validations passed!")

If fed with this config_test.yml:

- snowalb:
    snowalb: -0.5  
  bldgprop:
    faibldg: 5  
    bldgh: 10   

It correctly runs until the end AND prints together errors from Field() and @model_validator:

Validation Errors Detected:

2 validation errors for SUEWSConfig
snowalb.snowalb
  Input should be greater than or equal to 0 [type=greater_than_equal, input_value=-0.5, input_type=float]
    For further information visit https://errors.pydantic.dev/2.9/v/greater_than_equal
bldgprop
  Value error, Frontal area index of buildings (faibldg) should not be less than building height (bldgh).  [type=value_error, input_value={'faibldg': 5, 'bldgh': 10}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/value_error

If we agree that this might be useful/a good strategy to use, I can edit the code accordingly in the next few days.

[sunt05]

Looks brilliant - go ahead and get these in. Thanks, @dayantur!

[dayantur]

Bad news - I've tried to convert the def_config_suews.py today to handle the errors as above and failed.
Here is an example (similar to yesterday, but more complex) that shows where the problem is:

from pydantic import BaseModel, Field, ValidationError, model_validator
from typing import List
import yaml

class SnowAlb(BaseModel):
    snowalb: float = Field(ge=0, le=1, description="Snow albedo")


class BuildingProperties(BaseModel):
    faibldg: float

    @model_validator(mode="after")
    def validate_first(self) -> "BuildingProperties":
        if self.faibldg == 4:
            raise ValueError(
                "Error 1!"
            )
        return self

    @model_validator(mode="after")
    def validate_second(self) -> "BuildingProperties":
        if self.faibldg <= 5:
            raise ValueError(
                "Error 2!"
            )
        return self


class SUEWSConfig(BaseModel):
    snowalb: SnowAlb
    bldgprop: BuildingProperties

    @model_validator(mode="after")
    def validate_third(self) -> "SUEWSConfig":
        if self.bldgprop.faibldg <= 6 and self.snowalb.snowalb < 0:
            raise ValueError(
                "Error 3!"
            )
        return self


if __name__ == "__main__":
    exceptions: List[str] = []

    try:
        with open("./config_test.yml", "r") as file:
            yaml_config = yaml.safe_load(file)
        suews_config = SUEWSConfig(**yaml_config)

    except ValidationError as e:
        print("Validation Errors Detected:\n")
        print(e)  

If we set faibldg: 4 and snowalb: -0.5 in the test.yml:

Validation Errors Detected:

2 validation errors for SUEWSConfig
snowalb.snowalb
  Input should be greater than or equal to 0 [type=greater_than_equal, input_value=-0.5, input_type=float]
    For further information visit https://errors.pydantic.dev/2.9/v/greater_than_equal
bldgprop
  Value error, Error 1! [type=value_error, input_value={'faibldg': 4}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/value_error

The program halts before validate_second AND validate_third, despite faibldg being <=5 and <=6, which errors are not printed.
If we change faibldg to be 5:

Validation Errors Detected:

2 validation errors for SUEWSConfig
snowalb.snowalb
  Input should be greater than or equal to 0 [type=greater_than_equal, input_value=-0.5, input_type=float]
    For further information visit https://errors.pydantic.dev/2.9/v/greater_than_equal
bldgprop
  Value error, Error 2! [type=value_error, input_value={'faibldg': 5}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/value_error

We get error from validate_second, but not from validate_third.
And probably the worst part is that if we change faibldg to be 6, i.e. making true the if in validate_third (but not in validate_first and validate_second), the Field() halts the program before validating the model_validator in SUEWSConfig:

Validation Errors Detected:

1 validation error for SUEWSConfig
snowalb.snowalb
  Input should be greater than or equal to 0 [type=greater_than_equal, input_value=-0.5, input_type=float]
    For further information visit https://errors.pydantic.dev/2.9/v/greater_than_equal

[dayantur]

Just for completeness, this approach:

from pydantic import BaseModel, Field, ValidationError, model_validator
from typing import List
import yaml

class SnowAlb(BaseModel):
    snowalb: float = Field(description="Snow albedo")

    @model_validator(mode="after")
    def validate_zero(self) -> "SnowAlb":
        if self.snowalb < 0 or self.snowalb > 1:
            e_message = ValueError(
                "Error 0!"
            )
            exc.append(e_message)
        return self

class BuildingProperties(BaseModel):
    faibldg: float

    @model_validator(mode="after")
    def validate_first(self) -> "BuildingProperties":
        if self.faibldg == 4:
            e_message = ValueError(
                "Error 1!"
            )
            exc.append(e_message)
        return self

    @model_validator(mode="after")
    def validate_second(self) -> "BuildingProperties": 
        if self.faibldg <= 5:
            e_message = ValueError(
                "Error 2!"
            )
            exc.append(e_message)
        return self


class SUEWSConfig(BaseModel):
    snowalb: SnowAlb
    bldgprop: BuildingProperties

    @model_validator(mode="after")
    def validate_third(self) -> "SUEWSConfig":
        if self.bldgprop.faibldg <= 6 and self.snowalb.snowalb < 0:
            e_message = ValueError(
                "Error 3!"
            )
            exc.append(e_message)
        return self


if __name__ == "__main__":
    exc: List[str] = []


    with open("./config_test.yml", "r") as file:
        yaml_config = yaml.safe_load(file)
    suews_config = SUEWSConfig(**yaml_config)

if exc:
    raise ExceptionGroup("Validation errors occurred", exc)

with

bldgprop:
  faibldg: 4   
snowalb:
  snowalb: -0.5  

Can collect all the errors without halting:

  + Exception Group Traceback (most recent call last):
  |   File "/Users/silviarognone/Documents/UrbanClimate/PlottingAndTests/pydantic_test_multerrors.py", line 62, in <module>
  |     raise ExceptionGroup("Validation errors occurred", exc)
  | ExceptionGroup: Validation errors occurred (4 sub-exceptions)
  +-+---------------- 1 ----------------
    | ValueError: Error 0!
    +---------------- 2 ----------------
    | ValueError: Error 1!
    +---------------- 3 ----------------
    | ValueError: Error 2!
    +---------------- 4 ----------------
    | ValueError: Error 3!
    +------------------------------------