Reduce memory usage during GWDO stats estimation #1235
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Tested on Eris (GNU) with 24km grid, and Derecho (GNU) with 15km and 24km. Derecho NVHPC build and run is successful for 15km grid. |
|
Tested on 5km grid as well. |
mgduda
requested changes
Oct 19, 2024
mgduda
requested changes
Oct 23, 2024
|
Thanks for the additional review @mgduda! I have addressed with the last two commits. |
mgduda
requested changes
Dec 30, 2024
There was a problem hiding this comment.
Other than an update to a routine name in a comment block (as noted), I think we're in good shape. After addressing the routine name issue, feel free to rework the commit history for this PR so that it has one or a few commits with a clear and distinct purpose.
Additionally -- though it can be put off to a future PR -- it could be worth looking for simple ways of optimizing the get_box and/or get_tile_from_box_point routines.
…on of GWDO statistics This commit introduces code changes in order to improve the memory footprint of the computation of Gravity Wave Drag Orography statistics in the init_atmosphere. The previous approach involved each MPI rank reading in the entirety of the topography and land use tiles, and this may prevent fully-subscribing to all cores in a node. The new approach only reads in one tile at a time, when it encounters a pixel whose data is not already available in a linked list. The new algorithm is able to fully subscribe to all ranks in a node, leading to better parallel performance. The get_box subroutine is modified to call get_tile_from_box_point to check if the current pixel in the box is present in the linked list, and if not, it appends this tile (after reading both topo and land use data) to the head of the list. This commit also changes box, box_landuse, dxm, nx and ny to be local variables, instead of module variables. This provides better readability, along with advantages of thread safety. This commit also removes the extraneous conversion of the cell latitude from radians to degrees and back to radians, prior to the estimation of zonal box size. The result is a numerically more correct code, but it results in marginal differences with the previous approach
…last matching tile This commit introduces an optimization for the lookup of tiles in subroutine get_tile_from_box_point by passing the most recent successful tile lookup to the next iteration of the search. These changes substantially improves the single-core performance of the compute_gwd_fields subroutine, and also improve the parallel performance of this computation.
abishekg7
force-pushed
the
atmosphere/reduce_gwd_mem_usage
branch
from
650a60c to
7ad8bf6
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A copy of #1228 with branch renamed. This PR attempts to improve the memory footprint during the computation of Gravity Wave Drag Orography statistics in the pre-processing step.
The previous approach relied on each MPI rank reading all of the topography and land use tiles, and hence would run out of memory before we could fully subscribe to all cores in a node. In the new approach, we only read in one tile at a time and when we encounter a pixel whose data is not already available in a linked list. The
get_boxsubroutine is modified to callget_tile_from_box_pointto check if the current pixel in the box is present in the linked list, and if not, it appends this tile (after reading both topo and land use data) to the head of the list.This PR also changes
box, box_landuse, dxm, nx and nyto be local variables, instead of module variables. This provides a little better readability, along with advantages of thread safety, etc.