Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Umpire logs refactor - table output + adding percentages #3052

Merged
merged 128 commits into from
Jul 2, 2024

Conversation

arng40
Copy link
Contributor

@arng40 arng40 commented Mar 21, 2024

Update for memory statistics table :

What we currently have :

Umpire          DEVICE sum across ranks: 15293.4 GB
Umpire          DEVICE         rank max:   31.3 GB
Umpire       DEVICE::0 sum across ranks: 15293.4 G
Umpire       DEVICE::0         rank max:   31.3 GB
Umpire            HOST sum across ranks: 7272.2 GB
Umpire            HOST         rank max:   14.7 GB
Umpire HYPRE_DEVICE_POOL sum across ranks: 7178.4 GB
Umpire HYPRE_DEVICE_POOL         rank max:   14.8 GB
Umpire          PINNED sum across ranks:  256.3 GB
Umpire          PINNED         rank max:  589.6 MB

This is what I propose inspired by the display proposed by @victorapm

+-----------------------------+---------------------+---------------------+---------------------+--------------------+
|         Umpire Memory Pool  |     Min over ranks  |    Max  over ranks  |    Avg  over ranks  |    Sum over ranks  |
|  (reserved / % over total)  |                     |                     |                     |                    |
+-----------------------------+---------------------+---------------------+---------------------+--------------------+
|                     DEVICE  |    1.4 MB   (0.0%)  |    1.7 MB   (0.0%)  |    1.5 MB   (0.0%)  |   6.1 MB   (0.0%)  |
|                  DEVICE::0  |    1.4 MB   (0.0%)  |    1.7 MB   (0.0%)  |    1.5 MB   (0.0%)  |   6.1 MB   (0.0%)  |
|                       HOST  |    2.3 MB   (0.0%)  |    2.5 MB   (0.0%)  |    2.4 MB   (0.0%)  |   9.8 MB   (0.0%)  |
|              HYPRE_UM_POOL  |  504.7 KB   (0.0%)  |  645.8 KB   (0.0%)  |  542.4 KB   (0.0%)  |   2.1 MB   (0.0%)  |
|                     PINNED  |  488.1 KB   (0.0%)  |  963.0 KB   (0.0%)  |  725.5 KB   (0.0%)  |   2.8 MB   (0.0%)  |
|                         UM  |    4.0 GB  (25.3%)  |    4.0 GB  (25.3%)  |    4.0 GB  (25.3%)  |  16.0 GB  (25.3%)  |
+-----------------------------+---------------------+---------------------+---------------------+--------------------+

Code side

  • I've also refactored a bit the addUmpireHighWaterMarks() function to modern C++

arng40 added 30 commits February 9, 2024 16:11
…feature/dudes/table-layout"

This reverts commit 8f74cfa.
@paveltomin
Copy link
Contributor

@arng40 @MelReyCG any update?

@arng40 arng40 closed this Jun 10, 2024
@arng40 arng40 reopened this Jun 10, 2024
@arng40
Copy link
Contributor Author

arng40 commented Jun 10, 2024

@paveltomin
This PR is being tested by a user on a simulation to see whether the results are consistent or not

@arng40
Copy link
Contributor Author

arng40 commented Jun 24, 2024

@paveltomin umpire values are consistent, you can have a look to the structure and let me know if it's good

@paveltomin
Copy link
Contributor

@paveltomin umpire values are consistent, you can have a look to the structure and let me know if it's good

looks good, thanks

@paveltomin paveltomin requested a review from victorapm June 24, 2024 14:34
Copy link
Contributor

@victorapm victorapm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Thanks for implementing this

@arng40 arng40 added the ci: run CUDA builds Allows to triggers (costly) CUDA jobs label Jun 26, 2024
@arng40 arng40 added the ci: run integrated tests Allows to run the integrated tests in GEOS CI label Jun 27, 2024
@rrsettgast
Copy link
Member

@arng40 Is there something inside of umpire to do this? It might make sense to do this sort of change within umpire for all to use.

@rrsettgast rrsettgast merged commit c74702a into develop Jul 2, 2024
26 checks passed
@rrsettgast rrsettgast deleted the feature/dudes/umpire-stat-log branch July 2, 2024 20:56
@CusiniM
Copy link
Collaborator

CusiniM commented Jul 2, 2024

this PR has broken my mac builds. Looks like that _SC_AVPHYS_PAGES is not defined on macOS.

@herve-gross
Copy link
Contributor

@CusiniM : try to replace by _SC_PHYS_PAGES

@CusiniM
Copy link
Collaborator

CusiniM commented Jul 2, 2024

@CusiniM : try to replace by _SC_PHYS_PAGES

but are they the same thing? I have patched it like this: #3207

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci: run CUDA builds Allows to triggers (costly) CUDA jobs ci: run integrated tests Allows to run the integrated tests in GEOS CI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update memory statistics table
8 participants