CUDA Misaligned Data

[aescobed]

Hello, I am having an issue with GPU enabled HYPRE. In some of my test cases when I attempt to solve the pressure equations I get a CUDA misaligned data error. Any ideas as to why this is happening? It happens at the following line:
HYPRE_ParCSRGMRESSolve
(
peqn->hypreSlvr,
peqn->hypreParA,
peqn->hypreParRhs,
peqn->hypreParP
);

[victorapm]

Hi @aescobed, could you provide more details about your setup? Specifically:

1. Which version of HYPRE are you using?
2. What GPU architecture are you running on?
3. How was HYPRE configured and built (e.g., any specific CMake options)?
4. What are the matrix and right-hand side sizes in the failing test cases?
5. Does the error occur with all cases you tested or only specific ones?
6. Is your code open source
7. Did you get full stacktrace of the error?

With this information, we can help diagnose the issue more effectively.

Best,
Victor

[aescobed]

Hi @victorapm, here are the answers to your questions, thanks for the response:

1. 2.32.0

2. I am using CUDA, sometimes using the thrust api with -arch=sm_75

3. I used this when configuring petsc
   --prefix=/home/andres/Software/petsc/petsc-3.22.3-cuda-streams2/petsc/install --with-fc=0 --download-f2cblaslapack --download-hdf5 --with-mpi-dir=/home/andres/Software/openmpi/openmpi-5.0.3/openmpi-5.0.3/install --download-hypre --with-64-bit-indices=1 --with-debugging=0 --with-cuda --with-cuda-dir=/home/andres/Software/cuda/cuda_toolkit_12.4/install --with-precision=double --with-device-memory-pool=0 --CUDAFLAGS="-default-stream legacy"

4. matrix (hypreA): 0 170568 0 170568, rhs: 0 170568

5. It only happens with certain test cases, but other test cases which also incorporate a moving IBM seem to work fine so I'm not sure why it doesn't work with these test cases

6. The CPU version is open source: https://github.com/sebastipa/TOSCA
   The GPU version's repo is not public but I can ask my supervisor if I can share it with you.

7. I have been having trouble getting the full stacktrace, I can keep working on that if that would be of benefit to you.

Thank you!

[victorapm]

Thank you for the info! Seeing the full stacktrace would help.