Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collect simulation errors in results json #437

Merged
merged 5 commits into from
Mar 26, 2024

Conversation

nweires
Copy link
Collaborator

@nweires nweires commented Feb 26, 2024

Pull Request Description

Copy OpenStudio errors into each task's results json file, so they can more easily be summarized and surfaced after the job is complete. (We are viewing them from a python notebook that shows other summary statistics about a job.)

Example output for one simulation (error caused by a mismatch in package version numbers):

{
  "started_at": "20240108T153906Z",
  "completed_at": "20240108T153921Z",
  "completed_status": "Fail",
  "building_id": 14,
  "step_failures": [
    {
      "measure_dir_name": "HPXMLtoOpenStudio",
      "step_errors": ["undefined method `setTranslateFileWithRelativePath' for #<OpenStudio::Model::ScheduleFile:0x000058fd7f7f1108>\n/var/simdata/openstudio/resources/hpxml-measures/HPXMLtoOpenStudio/resources/schedules.rb:1499:in `create_schedule_file'\n/var/simdata/openstudio/resources/hpxml-measures/HPXMLtoOpenStudio/resources/geometry.rb:533:in `apply_occupants'\n/var/simdata/openstudio/resources/hpxml-measures/HPXMLtoOpenStudio/measure.rb:567:in `add_num_occupants'\n/var/simdata/openstudio/resources/hpxml-measures/HPXMLtoOpenStudio/measure.rb:445:in `create_unit_model'\n/var/simdata/openstudio/resources/hpxml-measures/HPXMLtoOpenStudio/measure.rb:182:in `block in run'\n/var/simdata/openstudio/resources/hpxml-measures/HPXMLtoOpenStudio/measure.rb:174:in `each'\n/var/simdata/openstudio/resources/hpxml-measures/HPXMLtoOpenStudio/measure.rb:174:in `each_with_index'\n/var/simdata/openstudio/resources/hpxml-measures/HPXMLtoOpenStudio/measure.rb:174:in `run'\n:/ruby/2.7.0/gems/openstudio-workflow-2.3.1/lib/openstudio/workflow/util/measure.rb:517:in `apply_measure'\n:/ruby/2.7.0/gems/openstudio-workflow-2.3.1/lib/openstudio/workflow/util/measure.rb:114:in `block in apply_measures'\n:/ruby/2.7.0/gems/openstudio-workflow-2.3.1/lib/openstudio/workflow/util/measure.rb:67:in `each_index'\n:/ruby/2.7.0/gems/openstudio-workflow-2.3.1/lib/openstudio/workflow/util/measure.rb:67:in `apply_measures'\n:/ruby/2.7.0/gems/openstudio-workflow-2.3.1/lib/openstudio/workflow/jobs/run_os_measures.rb:70:in `perform'\n:/ruby/2.7.0/gems/openstudio-workflow-2.3.1/lib/openstudio/workflow/run.rb:291:in `step'\n:/ruby/2.7.0/gems/openstudio-workflow-2.3.1/lib/openstudio/workflow/run.rb:233:in `run'\n:/openstudio_cli.rb:1190:in `execute'\n:/openstudio_cli.rb:820:in `execute'\n:/openstudio_cli.rb:1987:in `<main>'\neval:193:in `eval'\neval:193:in `require_embedded_absolute'\neval:178:in `block in require_embedded'\neval:172:in `each'\neval:172:in `require_embedded'\neval:131:in `require'\neval:3:in `<main>'"]
    }
  ],
  "upgrade": 0
}

Also, unzip the test file results_job1.json.gz to make it human-readable. (It's un-zipped before being used in the test anyway.)

Checklist

Not all may apply

  • Code changes (must work)
  • Tests exercising your feature/bug fix (check coverage report on Checks -> BuildStockBatch Tests -> Artifacts)
  • Coverage has increased or at least not decreased. Update minimum_coverage in .github/workflows/coverage.yml as necessary.
  • All other unit and integration tests passing
  • Update validation for project config yaml file changes
  • Update existing documentation
  • Run a small batch run on Kestrel/Eagle to make sure it all works if you made changes that will affect Kestrel/Eagle
  • Add to the changelog_dev.rst file and propose migration text in the pull request

Comment on lines +1052 to +1058
"step_failures": [
{
"measure_dir_name": "BuildExistingModel",
"step_errors": [
"Cannot find weather file at /var/simdata/openstudio/lib/resources/measures/HPXMLtoOpenStudio/resources/var/simdata/openstudio/weather/USA_TX_Houston-Bush.Intercontinental.AP.722430.epw."
]
}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This step_failures entry is the only change to this file, other than unzipping it.

Copy link

github-actions bot commented Feb 26, 2024

File Coverage
All files 87%
base.py 92%
exc.py 57%
hpc.py 78%
local.py 70%
postprocessing.py 85%
utils.py 92%
cloud/docker_base.py 88%
sampler/base.py 79%
sampler/downselect.py 33%
sampler/precomputed.py 93%
sampler/residential_quota.py 61%
test/shared_testing_stuff.py 85%
test/test_docker.py 33%
test/test_local.py 97%
test/test_validation.py 97%
workflow_generator/base.py 90%
workflow_generator/commercial.py 53%
workflow_generator/residential_hpxml.py 86%

Minimum allowed coverage is 33%

Generated by 🐒 cobertura-action against f4a0503

@nweires nweires requested a review from nmerket February 27, 2024 14:43
@nweires nweires mentioned this pull request Mar 7, 2024
8 tasks
Copy link
Member

@nmerket nmerket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. It's a good first step for doing #379.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants