From 0fae2ec868bb2f076bfd8afed91645d9205b50a9 Mon Sep 17 00:00:00 2001 From: Alessio Sclocco Date: Mon, 11 Dec 2023 15:10:31 +0100 Subject: [PATCH 1/4] New version of the GPU programming section. --- _sidebar.md | 2 +- best_practices/language_guides/opencl_cuda.md | 164 ++++++++++++------ 2 files changed, 114 insertions(+), 52 deletions(-) diff --git a/_sidebar.md b/_sidebar.md index 05b46234..80a0636d 100644 --- a/_sidebar.md +++ b/_sidebar.md @@ -11,7 +11,7 @@ * [Bash](/best_practices/language_guides/bash.md) * [JavaScript and TypeScript](/best_practices/language_guides/javascript.md) * [Python](/best_practices/language_guides/python.md) - * [OpenCL and CUDA](/best_practices/language_guides/opencl_cuda.md) + * [GPU](/best_practices/language_guides/opencl_cuda.md) * [R](/best_practices/language_guides/r.md) * [C and C++](/best_practices/language_guides/ccpp.md) * [Fortran](/best_practices/language_guides/fortran.md) diff --git a/best_practices/language_guides/opencl_cuda.md b/best_practices/language_guides/opencl_cuda.md index b02f2451..140f213e 100644 --- a/best_practices/language_guides/opencl_cuda.md +++ b/best_practices/language_guides/opencl_cuda.md @@ -1,69 +1,131 @@ -# OpenCL & CUDA +# GPU Programming Languages *Page maintainer: Alessio Sclocco* [@isazi](https://github.com/isazi) -## Sources for learning - *please add university courses and informative videos* - * Parallel Reduction [[Slides](http://developer.download.nvidia.com/compute/cuda/1.1-Beta/x86_website/projects/reduction/doc/reduction.pdf)] - * GPU Memory bootcamp - Tony Scudiero [[git repo](https://github.com/tscudiero/MemBootcamp)] - * Best Practices [[Slides](https://github.com/tscudiero/MemBootcamp/blob/master/Slides/S5353_Scudiero_Bootcamp1.pdf)] [[Video](http://on-demand.gputechconf.com/gtc/2015/video/S5353.html)] - * Beyond the Best Practices [[Slides](https://github.com/tscudiero/MemBootcamp/blob/master/Slides/S5376-Scudiero_Bootcamp2.pdf)] [[Video](http://on-demand.gputechconf.com/gtc/2015/video/S5376.html)] - * Collaborative Access Patterns [[Slides](https://github.com/tscudiero/MemBootcamp/blob/master/Slides/S6181-Scudiero_Bootcamp3.pdf)] [[Video](http://on-demand.gputechconf.com/gtc/2016/video/s6181-tony-scudiero-bootcamp-3.mp4)] - * CUB: CUDA Collective primitives library [[Git](https://github.com/NVlabs/cub)] [[Slides](http://on-demand.gputechconf.com/gtc/2015/presentation/S5617-Duane-Merrill.pdf)] [[Video](http://on-demand.gputechconf.com/gtc/2015/video/S5617.html)] - * Best Practices Guide by PRACE [[HTML](https://prace-ri.eu/training-support/best-practice-guides/best-practice-guide-gpgpu/)] [[PDF](https://prace-ri.eu/wp-content/uploads/Best-Practice-Guide_GPGPU.pdf)] - +## Learning Resources + +* Carpentries GPU Programming course + * [Lesson material](https://carpentries-incubator.github.io/lesson-gpu-programming/) +* Introduction to CUDA C + * [Slides](http://developer.download.nvidia.com/compute/developertrainingmaterials/presentations/cuda_language/Introduction_to_CUDA_C.pptx) + * [Video](http://on-demand.gputechconf.com/gtc/2012/video/S0624-Monday-Introduction-to-CUDA-C.mp4) +* Introduction to OpenACC + * [Slides](http://developer.download.nvidia.com/compute/developertrainingmaterials/presentations/openacc/Introduction_To_OpenACC.pptx) +* Introduction to HIP Programming + * [Video](https://www.youtube.com/watch?v=3ejUwypP0bI) +* SYCL Introduction and Best Practices + * [Video](https://www.youtube.com/watch?v=TbkrODiVDQY) + ## Documentation -* OpenCL specification [[1.2](https://www.khronos.org/registry/cl/sdk/1.2/docs/man/xhtml/)] [[2.0](https://www.khronos.org/registry/cl/sdk/2.0/docs/man/xhtml/)] -* CUDA Toolkit [[latest](http://docs.nvidia.com/cuda/index.html)] - * [CUDA Programming Guide](http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html) - * [CUDA Runtime API](http://docs.nvidia.com/cuda/cuda-runtime-api/index.html) - - -## Source-to-source translation between CUDA and OpenCL -* vtsynergy (https://github.com/vtsynergy) - * This was shown to work on DAS5 after copying /usr/include/limits.h to $PWD and commenting out the lines around # include_next (122-125) : - "cu2cl-tool host_code.cc device_code.cu -- -DGPU_ON -I$PWD:/usr/include -I/usr/lib/gcc/x86_64-redhat-linux/4.8.2/include". -* cutocl (https://github.com/benvanwerkhoven/cutocl) - -## Overview of libraries -* OpenCL-based libraries - * [CLBlast](https://github.com/CNugteren/CLBlast) - * [clFFT](https://github.com/clMathLibraries/clFFT) -* CUDA-based libraries +* CUDA + * [C programming guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html) + * [Runtime API](https://docs.nvidia.com/cuda/cuda-runtime-api/) + * [Driver API](https://docs.nvidia.com/cuda/cuda-driver-api/index.html) + * [Fortran programming guide](https://docs.nvidia.com/hpc-sdk/compilers/cuda-fortran-prog-guide/index.html) +* HIP + * [Kernel language syntax](https://rocm.docs.amd.com/projects/HIP/en/latest/reference/kernel_language.html) + * [Runtime API](https://rocm.docs.amd.com/projects/HIP/en/latest/.doxygen/docBin/html/modules.html) +* SYCL + * [Specification](https://registry.khronos.org/SYCL/specs/sycl-2020/html/sycl-2020.html) + * [Reference guide](https://www.khronos.org/files/sycl/sycl-2020-reference-guide.pdf) +* OpenCL + * [Guide](https://github.com/KhronosGroup/OpenCL-Guide) + * [API](https://registry.khronos.org/OpenCL/specs/3.0-unified/html/OpenCL_API.html) + * [OpenCL C specification](https://registry.khronos.org/OpenCL/specs/3.0-unified/html/OpenCL_C.html) + * [Reference guide](https://www.khronos.org/files/opencl30-reference-guide.pdf) +* OpenACC + * [Programming guide](https://www.openacc.org/sites/default/files/inline-files/OpenACC_Programming_Guide_0_0.pdf) + * [Reference guide](https://www.openacc.org/sites/default/files/inline-files/API%20Guide%202.7.pdf) +* OpenMP + * [Reference guide](https://www.openmp.org/wp-content/uploads/OpenMPRef-5.0-111802-web.pdf) + +## Source-to-source Translation + +* CUDA to HIP + * [hipify](https://github.com/ROCm/HIPIFY) +* CUDA to SYCL + * [SYCLomatic](https://github.com/oneapi-src/SYCLomatic) +* CUDA to OpenCL + * [cutocl](https://github.com/benvanwerkhoven/cutocl) + +## Overview of Libraries + +* CUDA * [cuBLAS](http://docs.nvidia.com/cuda/cublas/index.html) * [NVBLAS](http://docs.nvidia.com/cuda/nvblas/index.html) * [cuFFT](http://docs.nvidia.com/cuda/cufft/index.html) * [cuGRAPH](https://docs.rapids.ai/api/cugraph/stable/) * [cuRAND](http://docs.nvidia.com/cuda/curand/index.html) * [cuSPARSE](http://docs.nvidia.com/cuda/cusparse/index.html) +* HIP + * [hipBLAS](https://rocm.docs.amd.com/projects/hipBLAS/en/latest/index.html) + * [hipFFT](https://rocm.docs.amd.com/projects/hipFFT/en/latest/index.html) + * [hipRAND](https://rocm.docs.amd.com/projects/hipRAND/en/latest/index.html) + * [hipSPARSE](https://rocm.docs.amd.com/projects/hipSPARSE/en/latest/index.html) +* SYCL + * [OneAPI BLAS](https://www.intel.com/content/www/us/en/docs/onemkl/developer-reference-dpcpp/2024-0/blas-routines.html#GUID-F277361F-49BA-437B-8100-3D2B6BBC3CC1) + * [OneAPI FFT](https://www.intel.com/content/www/us/en/docs/onemkl/developer-reference-dpcpp/2024-0/fourier-transform-functions.html#GUID-BB8891E9-D3D3-40B9-BBB1-2390C4340CDA) + * [OneAPI sparse](https://www.intel.com/content/www/us/en/docs/onemkl/developer-reference-dpcpp/2024-0/sparse-blas-routines.html#GUID-7123D31B-9C7F-4BA7-9792-02A417293E60) + * [OneAPI random number generators](https://www.intel.com/content/www/us/en/docs/onemkl/developer-reference-dpcpp/2024-0/random-number-generators.html#GUID-FFC80D12-C323-4A9F-83E3-D0ACDB686876) +* OpenCL + * [CLBlast](https://github.com/CNugteren/CLBlast) + * [clFFT](https://github.com/clMathLibraries/clFFT) +## Foreign Function Interfaces + +* C++ + * CUDA + * [cudawrappers](https://github.com/nlesc-recruit/cudawrappers) + * OpenCL + * [CLHPP](https://github.com/KhronosGroup/OpenCL-CLHPP) +* Python + * CUDA + * [PyCuda](https://mathema.tician.de/software/pycuda/) + * [CuPy](https://cupy.dev/) + * [cuda-python](https://nvidia.github.io/cuda-python/) + * HIP + * [PyHIP](https://github.com/jatinx/PyHIP) + * SYCL + * [dpctl](https://github.com/IntelPython/dpctl) + * OpenCL + * [PyOpenCL](https://mathema.tician.de/software/pycuda/) +* Julia + * CUDA + * [CUDA.jl](https://github.com/JuliaGPU/CUDA.jl) + * HIP + * [AMDGPU.jl](https://github.com/JuliaGPU/AMDGPU.jl) + * SYCL + * [oneAPI.jl](https://github.com/JuliaGPU/oneAPI.jl) +* Java + * CUDA + * [JCuda](http://www.jcuda.org/) + * OpenCL + * [JOCL](http://www.jocl.org/) -## Foreign Function Interfaces for CUDA and OpenCL - * C++: [[Cuda](https://github.com/eyalroz/cuda-api-wrappers/)], [[OpenCL](https://github.com/KhronosGroup/OpenCL-CLHPP)] - * Python: [[PyCuda](https://mathema.tician.de/software/pycuda/)], [[PyOpenCL](https://mathema.tician.de/software/pycuda/)] - * Java: [[JCuda](http://www.jcuda.org/)], [[JOCL](http://www.jocl.org/)] - - -## Testing - * Unit Testing - * Example of a unit test for CUDA kernel using the [Kernel Tuner](https://github.com/benvanwerkhoven/kernel_tuner/blob/master/examples/cuda/test_vector_add.py) - * [comparing floating-point results](http://docs.nvidia.com/cuda/floating-point/index.html) +## Performance Optimization +* [PRACE best practice guide on modern accelerators](https://zenodo.org/records/5839488) +* [CUDA best practices](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html) +* [OneAPI SYCL best practices](https://www.intel.com/content/www/us/en/docs/oneapi/programming-guide/2023-2/optimize-your-sycl-applications.html) ## Debugging and Profiling Tools - * [Nvidia Visual Profiler](https://developer.nvidia.com/nvidia-visual-profiler) [[User Guide](http://docs.nvidia.com/cuda/profiler-users-guide)] - * [CUDA-GDB](http://docs.nvidia.com/cuda/cuda-gdb/index.html) - * [compute-sanitizer](https://docs.nvidia.com/cuda/compute-sanitizer/index.html) - -## Performance Optimization - * Resources: - * Better Performance at Lower Occupancy [[Slides](http://www.nvidia.com/content/gtc-2010/pdfs/2238_gtc2010.pdf)] [[Video](http://on-demand.gputechconf.com/gtc/2010/video/S12238-Better-Performance-at-Lower-Occupancy.mp4)] - * [Maxwell Tuning Guide](http://docs.nvidia.com/cuda/maxwell-tuning-guide) - * [Pascal Tuning Guide](http://docs.nvidia.com/cuda/pascal-tuning-guide) - - * Generic Auto Tuners: - * [Kernel Tuner](https://github.com/benvanwerkhoven/kernel_tuner) (Python) - * [CLTune](https://github.com/CNugteren/CLTune) (C++) +* CUDA + * [Nsight Systems](https://developer.nvidia.com/nsight-systems) + * [Nsight Compute](https://developer.nvidia.com/nsight-compute) + * [CUDA-GDB](http://docs.nvidia.com/cuda/cuda-gdb/index.html) + * [compute-sanitizer](https://docs.nvidia.com/compute-sanitizer/index.html) +* HIP + * [omniperf](https://github.com/AMDResearch/omniperf) + * [rocprof](https://github.com/ROCm/rocprofiler) +* SYCL + * [onetrace](https://github.com/intel/pti-gpu/tree/master/tools/onetrace) + +## Auto-tuning + +* Kernel Tuner + * [GitHub repository](https://github.com/KernelTuner/kernel_tuner) + * [Documentation](https://kerneltuner.github.io/kernel_tuner/stable/) + * [Tutorial](https://github.com/KernelTuner/kernel_tuner_tutorial) From 9c69489f0162cd0e95c6aec3359e3858ffc31700 Mon Sep 17 00:00:00 2001 From: Alessio Sclocco Date: Tue, 12 Dec 2023 11:00:54 +0100 Subject: [PATCH 2/4] Added abstractions. --- best_practices/language_guides/opencl_cuda.md | 35 ++++++++++++------- 1 file changed, 22 insertions(+), 13 deletions(-) diff --git a/best_practices/language_guides/opencl_cuda.md b/best_practices/language_guides/opencl_cuda.md index 140f213e..38b15199 100644 --- a/best_practices/language_guides/opencl_cuda.md +++ b/best_practices/language_guides/opencl_cuda.md @@ -41,15 +41,6 @@ * OpenMP * [Reference guide](https://www.openmp.org/wp-content/uploads/OpenMPRef-5.0-111802-web.pdf) -## Source-to-source Translation - -* CUDA to HIP - * [hipify](https://github.com/ROCm/HIPIFY) -* CUDA to SYCL - * [SYCLomatic](https://github.com/oneapi-src/SYCLomatic) -* CUDA to OpenCL - * [cutocl](https://github.com/benvanwerkhoven/cutocl) - ## Overview of Libraries * CUDA @@ -73,6 +64,15 @@ * [CLBlast](https://github.com/CNugteren/CLBlast) * [clFFT](https://github.com/clMathLibraries/clFFT) +## Source-to-source Translation + +* CUDA to HIP + * [hipify](https://github.com/ROCm/HIPIFY) +* CUDA to SYCL + * [SYCLomatic](https://github.com/oneapi-src/SYCLomatic) +* CUDA to OpenCL + * [cutocl](https://github.com/benvanwerkhoven/cutocl) + ## Foreign Function Interfaces * C++ @@ -104,11 +104,14 @@ * OpenCL * [JOCL](http://www.jocl.org/) -## Performance Optimization +## High-Level Abstractions -* [PRACE best practice guide on modern accelerators](https://zenodo.org/records/5839488) -* [CUDA best practices](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html) -* [OneAPI SYCL best practices](https://www.intel.com/content/www/us/en/docs/oneapi/programming-guide/2023-2/optimize-your-sycl-applications.html) +* C++ + * [Kokkos](https://github.com/kokkos/kokkos) + * [Raja](https://github.com/LLNL/RAJA) +* Python + * [Numba](https://numba.pydata.org/) + * [pykokkos](https://github.com/kokkos/pykokkos) ## Debugging and Profiling Tools @@ -123,6 +126,12 @@ * SYCL * [onetrace](https://github.com/intel/pti-gpu/tree/master/tools/onetrace) +## Performance Optimization + +* [PRACE best practice guide on modern accelerators](https://zenodo.org/records/5839488) +* [CUDA best practices](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html) +* [OneAPI SYCL best practices](https://www.intel.com/content/www/us/en/docs/oneapi/programming-guide/2023-2/optimize-your-sycl-applications.html) + ## Auto-tuning * Kernel Tuner From d752ce923f0e264ba786e03dcbf83416b4a7af0e Mon Sep 17 00:00:00 2001 From: Alessio Sclocco Date: Tue, 19 Dec 2023 09:47:33 +0100 Subject: [PATCH 3/4] Added OneProf --- best_practices/language_guides/opencl_cuda.md | 1 + 1 file changed, 1 insertion(+) diff --git a/best_practices/language_guides/opencl_cuda.md b/best_practices/language_guides/opencl_cuda.md index 38b15199..be9fcea9 100644 --- a/best_practices/language_guides/opencl_cuda.md +++ b/best_practices/language_guides/opencl_cuda.md @@ -124,6 +124,7 @@ * [omniperf](https://github.com/AMDResearch/omniperf) * [rocprof](https://github.com/ROCm/rocprofiler) * SYCL + * [oneprof](https://github.com/intel/pti-gpu/tree/master/tools/oneprof) * [onetrace](https://github.com/intel/pti-gpu/tree/master/tools/onetrace) ## Performance Optimization From 0281f442b0d2faa527bc45954dbee55e33e640c4 Mon Sep 17 00:00:00 2001 From: Alessio Sclocco Date: Fri, 26 Jan 2024 11:50:44 +0100 Subject: [PATCH 4/4] Added GPU programming wiht Julia. --- best_practices/language_guides/opencl_cuda.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/best_practices/language_guides/opencl_cuda.md b/best_practices/language_guides/opencl_cuda.md index be9fcea9..889759ba 100644 --- a/best_practices/language_guides/opencl_cuda.md +++ b/best_practices/language_guides/opencl_cuda.md @@ -16,6 +16,8 @@ * [Video](https://www.youtube.com/watch?v=3ejUwypP0bI) * SYCL Introduction and Best Practices * [Video](https://www.youtube.com/watch?v=TbkrODiVDQY) +* CSCS GPU Programming with Julia + * [Course recordings](https://github.com/omlins/julia-gpu-course) ## Documentation