Add gpu_mig40 to Greatlakes.

[joaander]

Description

Add the gpu_mig40 partition to UMich Great Lakes.

Motivation and Context

Allow users to submit to this new GPU partition.

Checklist:

• I am familiar with the Contributing Guidelines.
• I agree with the terms of the Contributor Agreement.
• My name is on the list of contributors.
• The changes introduced by this pull request are covered by existing or newly introduced tests.
• The package documentation and framework documentation in signac-docs are up to date with these changes.
• I have updated the changelog and added any related issue and pull request numbers for future reference.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (1d0f9fd) 69.43% compared to head (7221806) 69.41%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #811      +/-   ##
==========================================
- Coverage   69.43%   69.41%   -0.03%     
==========================================
  Files          48       48              
  Lines        4397     4397              
  Branches     1065     1065              
==========================================
- Hits         3053     3052       -1     
- Misses       1136     1137       +1     
  Partials      208      208              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cbkerr]

Thanks! Wondering how we would support submitting to both mig40 and gpu with fallback

[joaander]

Thanks! Wondering how we would support submitting to both mig40 and gpu with fallback

The two partitions have differing numbers of GPUs per node which flow uses to calculate the node occupancy. I'm also not certain how SLURM handles multiple GPU jobs with the two partitions - does it assign all to the same partition, or some ranks to one and some to another? Multiple partitions would cause serious problems for MPI jobs and load balancing issues for aggregates.

We could potentially enforce that the "gpu_mig40,gpu" partition is valid only for single GPU jobs. Feel free to implement this and test in a future PR. This might be best done after the ongoing work to refactor directives.