Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

469 update retrofit dataset #470

Merged
merged 23 commits into from
Feb 21, 2024
Merged

469 update retrofit dataset #470

merged 23 commits into from
Feb 21, 2024

Conversation

longshuicy
Copy link
Member

@longshuicy longshuicy commented Dec 18, 2023

add additional function to update the existing inventory dataset with additional retrofit and mapping entry key information. https://github.com/IN-CORE/pyincore/pull/470/files#diff-d1b1cb98d647ccb4ebd9f558b2894472d298769fdd3c2bde8d9d4e00205e4de0R73

Since we use fiona object and looping through fiona is too slow, I used geopandas dataframe to merge and udpate; but then save to shapefile so we can resume using fiona to carry out normal fragility matching and calculation


To test:

  • Use this notebook to test SLC retrofit strategy
    SLC_retrofit_test.zip
  • Use this notebook to tests Galveston
    Galveston_retrofit_test.zip
  • RUN tests/pyincore/analyses/buildingdamage/test_buildingdamage_retrofit.py and see results in the retrofit folder

Note:

  • Joplin takes 31 seconds;
  • Lumberton takes 61 seconds.
  • SLC if only retrofit 5000+ building with inner join total runtime: 158 seconds;
  • Galveston retrofitting 1500 buildings combined damage takes 115 seconds

@longshuicy longshuicy linked an issue Dec 18, 2023 that may be closed by this pull request
@longshuicy longshuicy marked this pull request as ready for review January 30, 2024 23:55
@longshuicy longshuicy marked this pull request as draft February 1, 2024 14:18
@longshuicy longshuicy marked this pull request as ready for review February 5, 2024 17:21
@longshuicy
Copy link
Member Author

Made one update to use inner join when joining with retrofit strategies and mapping; this will make sure only the building that has retrofit strategy will get computed again. Hence help with the performance.

@ylyangtw
Copy link
Contributor

ylyangtw commented Feb 19, 2024

Encountered FutureWarning in Galveston notebook:

/pyincore/utils/dataprocessutil.py:281: FutureWarning: The behavior of DataFrame.idxmax with all-NA values, or any-NA and skipna=False, is deprecated. In a future version this will raise ValueError
  max_key = dmg_result[dmg_states].idxmax(axis=1)

Just committed a fix if that's ok. Feel free to revert it if it affects anything.
Other than this, both notebooks and test work.

Some test records from my end:
Joplin Tornado Retrofit execution time: 11.40898 seconds
Lumberton Flood Retrofit execution time: 44.84685 seconds
Galveston notebook runtime: 71 & 20
SLC notebook: 162 & 52

Good job on the hardwork!

@longshuicy
Copy link
Member Author

Encountered FutureWarning in Galveston notebook:

/pyincore/utils/dataprocessutil.py:281: FutureWarning: The behavior of DataFrame.idxmax with all-NA values, or any-NA and skipna=False, is deprecated. In a future version this will raise ValueError
  max_key = dmg_result[dmg_states].idxmax(axis=1)

Just committed a fix if that's ok. Feel free to revert it if it affects anything. Other than this, both notebooks and test work.

Some test records from my end: Joplin Tornado Retrofit execution time: 11.40898 seconds Lumberton Flood Retrofit execution time: 44.84685 seconds Galveston notebook runtime: 71 & 20 SLC notebook: 162 & 52

Good job on the hardwork!

Thanks for the fix and thorough test!

@longshuicy longshuicy merged commit 40d8e8b into develop Feb 21, 2024
7 checks passed
@longshuicy longshuicy deleted the 469-update-retrofit-dataset branch February 21, 2024 17:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Refactoring retrofit calculation
4 participants