-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to build and run PeleC using GPU? #769
Comments
It's hard to provide detailed instructions for GPU use as the details can vary from system to system. But if you want to run on a system with Nvidia GPUs using cuda and your system is set up properly, all you should need to do is compile as normal, but with When running on GPUs, certain simulation input parameters may benefit from being re-optimized for performance. In particular, you may want larger values for |
It's useful to know that certain things need to be done for certain sites, and AMReX has some supported sites here https://github.com/AMReX-Codes/amrex/tree/development/Tools/GNUMake/sites . The machine query logic is here https://github.com/AMReX-Codes/amrex/blob/development/Tools/GNUMake/Make.machines . |
Dear all, Thank you for your assistance! I try to compile PeleC with nvcc on the WSL but have encountered some challenges, particularly with the Sundial package. Currently, I am able to run PeleC on CPUs without issues, but I am eager to explore the capabilities of GPU acceleration. If it is OK, I would like to keep this issue open to share my future experiences regarding the use of PeleC with GPU computing. Regards, |
Yeah that's fine to leave this issue open and add more detail on any issues you have running on GPUs, which we can then try to address. |
Hello! I have a doubt to clarify. When I first tested the code in CPU parallel mode, by running the basic PMF testcase I had not set MPI=TRUE in the example.inp file. But still the mpirun -np command worked out, to run the PeleC executable. Did I miss out anything? |
Without MPI enabled in PeleC it will run the same application with |
Note that when you compile for MPI, you should have |
Thanks for your clarifications on this! @jrood-nrel @baperry2 . |
I am trying to get Pele to work with GPUs on kestrel and any instructions on the relevant modules to be loaded will be greatly appreciated. So far I've tried using
|
For Kestrel GPUs, you can use the modules specified here, which should also work for PeleC: https://erf.readthedocs.io/en/latest/GettingStarted.html#kestrel-nrel Let us know if there are any issues, it's been a bit since I tested PeleC on Kestrel GPUs and they've been periodically reshuffling the modules as they get the GPUs on line |
thank you @baperry2. this is really helpful. Will let you know how it goes |
I followed the steps outlined ERF's website (with the latest branches of PeleC and Submodules). I seem to run into the following error:
I did try re-making TPL after loading the modules suggested, but still get this error... |
Make sure you've done However, it does seem that there may be another issue here, as when I try it's getting past the step you are seeing but failing to generate the executable after linking. |
The following procedure looks to work but fails to produce an executable towards the end (i.e. goes all the way upto
using MPI+CUDA btw i.e. |
As I mentioned, the setup of the GPU partition of Kestrel has been frustratingly unstable. It appears they have again changed things in a way that makes the prior instructions no longer functional. You should be able to use the following module setup:
And then compile with:
|
I happened to be looking at this as well and I used this:
|
thanks a lot @jrood-nrel @baperry2 I am able to get a linked executable for the PMF case successfully with CUDA. My specific case though still faces the issue. I guess its something to do with the way the PMF functions and data-structures have been defined and will align everything with the way its done in the present case folder |
Dear all,
Hi!
I want to build and run PeleC using GPU, however, I am not able to find any tutorial on installations relevant to GPU or the CUDA enviroment. Can anyone provide some tutorial? Any help will be appreciated!
Thanks!
Regards,
Fan E
The text was updated successfully, but these errors were encountered: