Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SGraph (re-)build optimisations #449

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

reuterbal
Copy link
Collaborator

While profiling the building of the SGraph, I noticed two things:

  1. A non-negligible amount of time was spent on checking if items are part of any of the exclusion lists. The reason that this is costly is the pattern support, which incurs some overhead for repeatedly compiling search patterns. This can be partially reduced by using fnmatch.filter instead of a comprehension with any. On top of that, I tried reducing the amount of pattern compilations overall via some reordering or caching, however, since fnmatch already comes with a built-in pattern cache, this did worsen performance in all cases.
  2. The Module class includes module variables in the list of definitions, which is correct from a Fortran point-of-view but not required for the Scheduler, where we don't capture explicitly the dependency on individual variables but only the module itself. Hence, searching directly only for procedures, interfaces and typedefs brings the search effort down considerably.

Depending on the structure of the graph, this dropped graph building time by 10-20% for me. Much bigger gains could be achieved by caching definitions or dependencies, but this seems a little trickier to do.

Copy link

Documentation for this branch can be viewed at https://sites.ecmwf.int/docs/loki/449/index.html

Copy link
Collaborator

@mlange05 mlange05 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good to me and local ecphys regression also seemed fine. Small, but noticeable improvements during Loki processing - so very much appreciate this.

Will formally approve once test case green, but otherwise GTG from me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants