diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 00000000000..51b8667db23 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,133 @@ + +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, caste, color, religion, or sexual +identity and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, +diverse, inclusive, and healthy community. + +## Our Standards + +Examples of behavior that contributes to a positive environment for our +community include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +* Focusing on what is best not just for us as individuals, but for the overall + community + +Examples of unacceptable behavior include: + +* The use of sexualized language or imagery, and sexual attention or advances of + any kind +* Trolling, insulting or derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or email address, + without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Enforcement Responsibilities + +Community leaders are responsible for clarifying and enforcing our standards of +acceptable behavior and will take appropriate and fair corrective action in +response to any behavior that they deem inappropriate, threatening, offensive, +or harmful. + +Community leaders have the right and responsibility to remove, edit, or reject +comments, commits, code, wiki edits, issues, and other contributions that are +not aligned to this Code of Conduct, and will communicate reasons for moderation +decisions when appropriate. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when +an individual is officially representing the community in public spaces. +Examples of representing our community include using an official email address, +posting via an official social media account, or acting as an appointed +representative at an online or offline event. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the community leaders responsible for enforcement at +settgast1@llnl.gov. +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the +reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining +the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed +unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing +clarity around the nature of the violation and an explanation of why the +behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series of +actions. + +**Consequence**: A warning with consequences for continued behavior. No +interaction with the people involved, including unsolicited interaction with +those enforcing the Code of Conduct, for a specified period of time. This +includes avoiding interactions in community spaces as well as external channels +like social media. Violating these terms may lead to a temporary or permanent +ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including +sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public +communication with the community for a specified period of time. No public or +private interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, is allowed during this period. +Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behavior, harassment of an +individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within the +community. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 2.1, available at +[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1]. + +Community Impact Guidelines were inspired by +[Mozilla's code of conduct enforcement ladder][Mozilla CoC]. + +For answers to common questions about this code of conduct, see the FAQ at +[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at +[https://www.contributor-covenant.org/translations][translations]. + +[homepage]: https://www.contributor-covenant.org +[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html +[Mozilla CoC]: https://github.com/mozilla/diversity +[FAQ]: https://www.contributor-covenant.org/faq +[translations]: https://www.contributor-covenant.org/translations diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000000..5d9b612c2fd --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,3 @@ +To contribute to GEOS, please refer to the [Contributing](https://geosx-geosx.readthedocs-hosted.com/en/latest/docs/sphinx/developerGuide/Contributing/index_contributing.html) section in the [GEOS developer guide](https://geosx-geosx.readthedocs-hosted.com/en/latest/docs/sphinx/developerGuide/Index.html#). + +To report a bug please open an [issue](https://github.com/GEOS-DEV/GEOS/issues) \ No newline at end of file diff --git a/README.md b/README.md index 000d3b45c7d..6d4cfee25ff 100644 --- a/README.md +++ b/README.md @@ -21,10 +21,13 @@ page for more details on the HPC, numerics, and applied engineering components of this effort. Documentation ---------------------- +------------- -Our documentation is hosted [here](https://geosx-geosx.readthedocs-hosted.com/en/latest/?). +Please visit the [Main documentation for GEOS](https://geosx-geosx.readthedocs-hosted.com/en/latest/?). +If you would like to contribute to GEOS, please see the [developer guide](https://geosx-geosx.readthedocs-hosted.com/en/latest/docs/sphinx/developerGuide/Index.html) + +If you would like to report a bug, please submit an [issue](https://github.com/GEOS-DEV/GEOS/issues/new/choose). Who develops GEOS? ------------------- diff --git a/host-configs/apple/macOS_base.cmake b/host-configs/apple/macOS_base.cmake index c9cf2efc8df..1e5e06c80aa 100644 --- a/host-configs/apple/macOS_base.cmake +++ b/host-configs/apple/macOS_base.cmake @@ -25,15 +25,16 @@ set(ENABLE_CALIPER "OFF" CACHE PATH "" FORCE ) set( BLAS_LIBRARIES ${HOMEBREW_DIR}/opt/lapack/lib/libblas.dylib CACHE PATH "" FORCE ) set( LAPACK_LIBRARIES ${HOMEBREW_DIR}/opt/lapack/lib/liblapack.dylib CACHE PATH "" FORCE ) -set(ENABLE_DOXYGEN OFF CACHE BOOL "" FORCE) +set(ENABLE_DOXYGEN ON CACHE BOOL "" FORCE) +set(ENABLE_SPHINX ON CACHE BOOL "" FORCE) set(ENABLE_MATHPRESSO OFF CACHE BOOL "" FORCE ) set(GEOS_BUILD_SHARED_LIBS ON CACHE BOOL "" FORCE) -#set( DOXYGEN_EXECUTABLE /usr/local/bin/doxygen CACHE PATH "" FORCE ) -#set( SPHINX_EXECUTABLE /usr/local/bin/sphinx-build CACHE PATH "" FORCE ) +set( DOXYGEN_EXECUTABLE ${HOMEBREW_DIR}/bin/doxygen CACHE PATH "" FORCE ) +set( SPHINX_EXECUTABLE ${HOMEBREW_DIR}/opt/sphinx-doc/bin/sphinx-build CACHE PATH "" FORCE ) if(NOT ( EXISTS "${GEOS_TPL_DIR}" AND IS_DIRECTORY "${GEOS_TPL_DIR}" ) ) set(GEOS_TPL_DIR "${CMAKE_SOURCE_DIR}/../../thirdPartyLibs/install-${CONFIG_NAME}-release" CACHE PATH "" FORCE ) diff --git a/host-configs/quick-start-template.cmake b/host-configs/quick-start-template.cmake new file mode 100644 index 00000000000..7170d002b23 --- /dev/null +++ b/host-configs/quick-start-template.cmake @@ -0,0 +1,32 @@ +set( CONFIG_NAME "quick-start" ) + +# Set compilers path +set(CMAKE_C_COMPILER "path-to-gcc/bin/gcc" CACHE PATH "") # This is typically something like /usr/bin/gcc ... or clang +set(CMAKE_CXX_COMPILER "path-to-gcc/bin/g++" CACHE PATH "") # This is typically something like /usr/bin/g++ ... or clang++ +set(ENABLE_FORTRAN OFF CACHE BOOL "" FORCE) + +# Set paths to mpi +set(ENABLE_MPI ON CACHE PATH "") +set(MPI_C_COMPILER "path-to-mpi/bin/mpicc" CACHE PATH "") # This is typically something like /usr/bin/mpicc +set(MPI_CXX_COMPILER "path-to-mpi/bin/mpicxx" CACHE PATH "") # This is typically something like /usr/bin/mpicxx +set(MPIEXEC "path-to-mpi/bin/mpirun" CACHE PATH "") # This is typically something like /usr/bin/mpirun + +# Set paths to blas and lapack +set( BLAS_LIBRARIES "path-to-blas" CACHE PATH "" FORCE ) # This is typically something like /usr/lib64/libblas.so +set( LAPACK_LIBRARIES CACHE PATH "path-to-lapack" FORCE ) # This is typically something like /usr/lib64/liblapack.so + +# Cuda and openMP +set( ENABLE_CUDA OFF CACHE PATH "" FORCE ) +set( ENABLE_OPENMP OFF CACHE PATH "" FORCE ) + +# TPLs +set( ENABLE_TRILINOS OFF CACHE PATH "" FORCE ) +set( ENABLE_CALIPER OFF CACHE PATH "" FORCE ) +set( ENABLE_DOXYGEN OFF CACHE BOOL "" FORCE) +set( ENABLE_MATHPRESSO OFF CACHE BOOL "" FORCE ) + +if(NOT ( EXISTS "${GEOS_TPL_DIR}" AND IS_DIRECTORY "${GEOS_TPL_DIR}" ) ) + set(GEOS_TPL_DIR "${CMAKE_SOURCE_DIR}/../../thirdPartyLibs/install-${CONFIG_NAME}-release" CACHE PATH "" FORCE ) +endif() + +include(${CMAKE_CURRENT_LIST_DIR}/../tpls.cmake) diff --git a/inputFiles/compositionalMultiphaseWell/benchmarks/Egg/deadOilEgg_base_iterative.xml b/inputFiles/compositionalMultiphaseWell/benchmarks/Egg/deadOilEgg_base_iterative.xml index 56c2ade45d2..b0c99c23121 100644 --- a/inputFiles/compositionalMultiphaseWell/benchmarks/Egg/deadOilEgg_base_iterative.xml +++ b/inputFiles/compositionalMultiphaseWell/benchmarks/Egg/deadOilEgg_base_iterative.xml @@ -23,7 +23,8 @@ preconditionerType="mgr" krylovTol="1e-4" krylovAdaptiveTol="1" - krylovWeakestTol="1e-2"/> + krylovWeakestTol="1e-2" + logLevel="1"/> + + + + + + + + + + diff --git a/inputFiles/wellboreECP/ECP_Wellbore_geom02.xml b/inputFiles/wellboreECP/ECP_Wellbore_geom02.xml new file mode 100644 index 00000000000..dcd2f82f4d2 --- /dev/null +++ b/inputFiles/wellboreECP/ECP_Wellbore_geom02.xml @@ -0,0 +1,32 @@ + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/ECP_Wellbore_geom03.xml b/inputFiles/wellboreECP/ECP_Wellbore_geom03.xml new file mode 100644 index 00000000000..d9e0265862c --- /dev/null +++ b/inputFiles/wellboreECP/ECP_Wellbore_geom03.xml @@ -0,0 +1,32 @@ + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/ECP_Wellbore_geom04.xml b/inputFiles/wellboreECP/ECP_Wellbore_geom04.xml new file mode 100644 index 00000000000..f89b5a7df8c --- /dev/null +++ b/inputFiles/wellboreECP/ECP_Wellbore_geom04.xml @@ -0,0 +1,32 @@ + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/ECP_Wellbore_geom05.xml b/inputFiles/wellboreECP/ECP_Wellbore_geom05.xml new file mode 100644 index 00000000000..b6ff8b22bfc --- /dev/null +++ b/inputFiles/wellboreECP/ECP_Wellbore_geom05.xml @@ -0,0 +1,32 @@ + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/ECP_Wellbore_geom06.xml b/inputFiles/wellboreECP/ECP_Wellbore_geom06.xml new file mode 100644 index 00000000000..46653587fab --- /dev/null +++ b/inputFiles/wellboreECP/ECP_Wellbore_geom06.xml @@ -0,0 +1,32 @@ + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/compositionalMultiphaseFlow/ECP_Wellbore_probdef.xml b/inputFiles/wellboreECP/compositionalMultiphaseFlow/ECP_Wellbore_probdef.xml new file mode 100644 index 00000000000..4b426eae484 --- /dev/null +++ b/inputFiles/wellboreECP/compositionalMultiphaseFlow/ECP_Wellbore_probdef.xml @@ -0,0 +1,210 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/compositionalMultiphaseFlow/co2flash.txt b/inputFiles/wellboreECP/compositionalMultiphaseFlow/co2flash.txt new file mode 100644 index 00000000000..ec46548d345 --- /dev/null +++ b/inputFiles/wellboreECP/compositionalMultiphaseFlow/co2flash.txt @@ -0,0 +1 @@ +FlashModel CO2Solubility 1e6 5e7 5e5 367.15 369.15 1 0 diff --git a/inputFiles/wellboreECP/compositionalMultiphaseFlow/level01/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level01/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..d7f365e4ac3 --- /dev/null +++ b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level01/ECP_Wellbore_gpu.xml @@ -0,0 +1,78 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/compositionalMultiphaseFlow/level01/launch_frontier b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level01/launch_frontier new file mode 100644 index 00000000000..550f5402193 --- /dev/null +++ b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level01/launch_frontier @@ -0,0 +1,13 @@ +#!/bin/bash +#SBATCH -N 1 +#SBATCH -n 1 +#SBATCH -t 1:00:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-compflow-geom01.out + +echo `date` +cmd="srun -N 1 -n 1 -c 1 --gpus-per-node=1 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 1 -y 1 -z 1 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/compositionalMultiphaseFlow/level02/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level02/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..968c53afb99 --- /dev/null +++ b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level02/ECP_Wellbore_gpu.xml @@ -0,0 +1,78 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/compositionalMultiphaseFlow/level02/launch_frontier b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level02/launch_frontier new file mode 100644 index 00000000000..15379ad037a --- /dev/null +++ b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level02/launch_frontier @@ -0,0 +1,18 @@ +#!/bin/bash +#SBATCH -N 1 +#SBATCH -n 4 +#SBATCH -t 2:00:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-compflow-geom02.out + +echo `date` +module load cpe/23.12 +module load craype-accel-amd-gfx90a +module load rocm/5.4.3 +export MPICH_GPU_SUPPORT_ENABLED=1 +ldd ${1} +cmd="srun -N 1 -n 4 -c 1 --gpus-per-node=4 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 1 -y 1 -z 4 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/compositionalMultiphaseFlow/level03/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level03/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..299faa6e0d5 --- /dev/null +++ b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level03/ECP_Wellbore_gpu.xml @@ -0,0 +1,78 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/compositionalMultiphaseFlow/level03/launch_frontier b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level03/launch_frontier new file mode 100644 index 00000000000..b64624e817e --- /dev/null +++ b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level03/launch_frontier @@ -0,0 +1,18 @@ +#!/bin/bash +#SBATCH -N 4 +#SBATCH -n 32 +#SBATCH -t 2:00:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-compflow-geom03.out + +echo `date` +module load cpe/23.12 +module load craype-accel-amd-gfx90a +module load rocm/5.4.3 +export MPICH_GPU_SUPPORT_ENABLED=1 +ldd ${1} +cmd="srun -N 4 -n 32 -c 1 --gpus-per-node=8 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 2 -y 2 -z 8 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/compositionalMultiphaseFlow/level04/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level04/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..75790211238 --- /dev/null +++ b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level04/ECP_Wellbore_gpu.xml @@ -0,0 +1,78 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/compositionalMultiphaseFlow/level04/launch_frontier b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level04/launch_frontier new file mode 100644 index 00000000000..8068a2eafb8 --- /dev/null +++ b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level04/launch_frontier @@ -0,0 +1,18 @@ +#!/bin/bash +#SBATCH -N 32 +#SBATCH -n 256 +#SBATCH -t 2:00:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-compflow-geom04.out + +echo `date` +module load cpe/23.12 +module load craype-accel-amd-gfx90a +module load rocm/5.4.3 +export MPICH_GPU_SUPPORT_ENABLED=1 +ldd ${1} +cmd="srun -N 32 -n 256 -c 1 --gpus-per-node=8 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 4 -y 4 -z 16 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/compositionalMultiphaseFlow/level05/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level05/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..bb2a7a50ffe --- /dev/null +++ b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level05/ECP_Wellbore_gpu.xml @@ -0,0 +1,78 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/compositionalMultiphaseFlow/level05/launch_frontier b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level05/launch_frontier new file mode 100644 index 00000000000..7b26b022959 --- /dev/null +++ b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level05/launch_frontier @@ -0,0 +1,18 @@ +#!/bin/bash +#SBATCH -N 256 +#SBATCH -n 2048 +#SBATCH -t 4:00:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-compflow-geom05.out + +echo `date` +module load cpe/23.12 +module load craype-accel-amd-gfx90a +module load rocm/5.4.3 +export MPICH_GPU_SUPPORT_ENABLED=1 +ldd ${1} +cmd="srun -N 256 -n 2048 -c 1 --gpus-per-node=8 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 8 -y 8 -z 32 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/compositionalMultiphaseFlow/level06/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level06/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..bdfaaf7384b --- /dev/null +++ b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level06/ECP_Wellbore_gpu.xml @@ -0,0 +1,78 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/compositionalMultiphaseFlow/level06/launch_frontier b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level06/launch_frontier new file mode 100644 index 00000000000..e8f3c24bf51 --- /dev/null +++ b/inputFiles/wellboreECP/compositionalMultiphaseFlow/level06/launch_frontier @@ -0,0 +1,18 @@ +#!/bin/bash +#SBATCH -N 2048 +#SBATCH -n 16384 +#SBATCH -t 4:00:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-compflow-geom06.out + +echo `date` +module load cpe/23.12 +module load craype-accel-amd-gfx90a +module load rocm/5.4.3 +export MPICH_GPU_SUPPORT_ENABLED=1 +ldd ${1} +cmd="srun -N 2048 -n 16384 -c 1 --gpus-per-node=8 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 16 -y 16 -z 64 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/compositionalMultiphaseFlow/pvtgas.txt b/inputFiles/wellboreECP/compositionalMultiphaseFlow/pvtgas.txt new file mode 100644 index 00000000000..6442bd866a3 --- /dev/null +++ b/inputFiles/wellboreECP/compositionalMultiphaseFlow/pvtgas.txt @@ -0,0 +1,2 @@ +DensityFun SpanWagnerCO2Density 1e6 5e7 5e5 367.15 369.15 1 +ViscosityFun FenghourCO2Viscosity 1e6 5e7 5e5 367.15 369.15 1 diff --git a/inputFiles/wellboreECP/compositionalMultiphaseFlow/pvtliquid.txt b/inputFiles/wellboreECP/compositionalMultiphaseFlow/pvtliquid.txt new file mode 100644 index 00000000000..a4b549a84ca --- /dev/null +++ b/inputFiles/wellboreECP/compositionalMultiphaseFlow/pvtliquid.txt @@ -0,0 +1,2 @@ +DensityFun PhillipsBrineDensity 1e6 5e7 5e5 367.15 369.15 1 0 +ViscosityFun PhillipsBrineViscosity 0 diff --git a/inputFiles/wellboreECP/dispatch.py b/inputFiles/wellboreECP/dispatch.py new file mode 100755 index 00000000000..63249ecf0e2 --- /dev/null +++ b/inputFiles/wellboreECP/dispatch.py @@ -0,0 +1,52 @@ +#!/opt/cray/pe/python/3.11.5/bin/python + +import argparse +import os +import sys + +def main(args): + parser = argparse.ArgumentParser() + parser.add_argument('-a','--account',required=True) + parser.add_argument('-b','--binary',required=True) + parser.add_argument('-c','--caliper',default="") + parser.add_argument('-p','--problem',required=True) + parser.add_argument('-s','--script',default="launch_frontier") + parser.add_argument('-d','--dispatch',default="sbatch") + parser.add_argument('-v','--verbosity',type=int,default=0) + parser.add_argument('-l','--levels',nargs='+') + args = parser.parse_args(args) + + if args.caliper in ("default", "def", "standard", "std"): + profiling="\"-t runtime-report,max_column_width=200,calc.inclusive,output=stdout,mpi-report\"" + else: + profiling=args.caliper + + avail_problems = [ d for d in os.listdir(os.getcwd()) if os.path.isdir(d) ] + + if args.problem not in avail_problems: + print(f"Unkown problem {args.problem}") + raise SystemExit(1) + + os.chdir(os.path.join(os.getcwd(),args.problem)) + prob_dir = os.getcwd() + + levels = list( sorted( ( d for d in os.listdir(prob_dir) if os.path.isdir(d) and d.startswith("level") ), key=lambda v: int(v[5:]) ) ) + if args.levels is not None: + args.levels = [ int(l) for l in args.levels ] + levels = [ l for l in levels if int(l[5:]) in args.levels ] + + if args.verbosity: + print(f"{levels = }") + + launch = ' '.join([args.dispatch, "-A", args.account, args.script, args.binary, profiling]) + if args.verbosity: + print(f"{launch = }") + + for level in levels: + os.chdir(level) + print(os.getcwd()) + os.system(launch) + os.chdir(prob_dir) + +if __name__ == "__main__": + main(sys.argv[1:]) diff --git a/inputFiles/wellboreECP/mechanics/ECP_Wellbore_probdef.xml b/inputFiles/wellboreECP/mechanics/ECP_Wellbore_probdef.xml new file mode 100644 index 00000000000..03565ea7c62 --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/ECP_Wellbore_probdef.xml @@ -0,0 +1,91 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/mechanics/level01/ECP_Wellbore_cpu.xml b/inputFiles/wellboreECP/mechanics/level01/ECP_Wellbore_cpu.xml new file mode 100644 index 00000000000..ff1e6e2786b --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level01/ECP_Wellbore_cpu.xml @@ -0,0 +1,81 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/mechanics/level01/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/mechanics/level01/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..955caa1251f --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level01/ECP_Wellbore_gpu.xml @@ -0,0 +1,87 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/mechanics/level01/launch_frontier b/inputFiles/wellboreECP/mechanics/level01/launch_frontier new file mode 100644 index 00000000000..1e292da4fa3 --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level01/launch_frontier @@ -0,0 +1,13 @@ +#!/bin/bash +#SBATCH -N 1 +#SBATCH -n 1 +#SBATCH -t 0:30:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-mechanics-geom01.out + +echo `date` +cmd="srun -N 1 -n 1 -c 1 --gpus-per-node=1 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 1 -y 1 -z 1 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/mechanics/level02/ECP_Wellbore_cpu.xml b/inputFiles/wellboreECP/mechanics/level02/ECP_Wellbore_cpu.xml new file mode 100644 index 00000000000..0ce1e921b86 --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level02/ECP_Wellbore_cpu.xml @@ -0,0 +1,81 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/mechanics/level02/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/mechanics/level02/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..42b793fd141 --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level02/ECP_Wellbore_gpu.xml @@ -0,0 +1,87 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/mechanics/level02/launch_frontier b/inputFiles/wellboreECP/mechanics/level02/launch_frontier new file mode 100644 index 00000000000..dc892143c51 --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level02/launch_frontier @@ -0,0 +1,18 @@ +#!/bin/bash +#SBATCH -N 1 +#SBATCH -n 8 +#SBATCH -t 0:30:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-mechanics-geom02.out + +echo `date` +module load cpe/23.12 +module load craype-accel-amd-gfx90a +module load rocm/5.4.3 +export MPICH_GPU_SUPPORT_ENABLED=1; +ldd ${1} +cmd="srun -N 1 -n 8 -c 1 --gpus-per-node=8 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 1 -y 1 -z 8 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/mechanics/level03/ECP_Wellbore_cpu.xml b/inputFiles/wellboreECP/mechanics/level03/ECP_Wellbore_cpu.xml new file mode 100644 index 00000000000..f293dbac569 --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level03/ECP_Wellbore_cpu.xml @@ -0,0 +1,81 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/mechanics/level03/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/mechanics/level03/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..2f45ff7d84a --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level03/ECP_Wellbore_gpu.xml @@ -0,0 +1,87 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/mechanics/level03/launch_frontier b/inputFiles/wellboreECP/mechanics/level03/launch_frontier new file mode 100644 index 00000000000..ec1946ef77f --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level03/launch_frontier @@ -0,0 +1,18 @@ +#!/bin/bash +#SBATCH -N 8 +#SBATCH -n 64 +#SBATCH -t 0:30:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-mechanics-geom03.out + +echo `date` +module load cpe/23.12 +module load craype-accel-amd-gfx90a +module load rocm/5.4.3 +export MPICH_GPU_SUPPORT_ENABLED=1 +ldd ${1} +cmd="srun -N 8 -n 64 -c 1 --gpus-per-node=8 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 2 -y 2 -z 16 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/mechanics/level04/ECP_Wellbore_cpu.xml b/inputFiles/wellboreECP/mechanics/level04/ECP_Wellbore_cpu.xml new file mode 100644 index 00000000000..0275158d56e --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level04/ECP_Wellbore_cpu.xml @@ -0,0 +1,81 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/mechanics/level04/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/mechanics/level04/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..0daa1f3b2e1 --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level04/ECP_Wellbore_gpu.xml @@ -0,0 +1,87 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/mechanics/level04/launch_frontier b/inputFiles/wellboreECP/mechanics/level04/launch_frontier new file mode 100644 index 00000000000..30504f5a86b --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level04/launch_frontier @@ -0,0 +1,18 @@ +#!/bin/bash +#SBATCH -N 64 +#SBATCH -n 512 +#SBATCH -t 0:30:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-mechanics-geom04.out + +echo `date` +module load cpe/23.12 +module load craype-accel-amd-gfx90a +module load rocm/5.4.3 +export MPICH_GPU_SUPPORT_ENABLED=1 +ldd ${1} +cmd="srun -N 64 -n 512 -c 1 --gpus-per-node=8 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 4 -y 4 -z 32 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/mechanics/level05/ECP_Wellbore_cpu.xml b/inputFiles/wellboreECP/mechanics/level05/ECP_Wellbore_cpu.xml new file mode 100644 index 00000000000..e175ad452cc --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level05/ECP_Wellbore_cpu.xml @@ -0,0 +1,81 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/mechanics/level05/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/mechanics/level05/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..35793297b48 --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level05/ECP_Wellbore_gpu.xml @@ -0,0 +1,87 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/mechanics/level05/launch_frontier b/inputFiles/wellboreECP/mechanics/level05/launch_frontier new file mode 100644 index 00000000000..d69845ff39a --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level05/launch_frontier @@ -0,0 +1,18 @@ +#!/bin/bash +#SBATCH -N 512 +#SBATCH -n 4096 +#SBATCH -t 0:30:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-mechanics-geom05.out + +echo `date` +module load cpe/23.12 +module load craype-accel-amd-gfx90a +module load rocm/5.4.3 +export MPICH_GPU_SUPPORT_ENABLED=1 +ldd ${1} +cmd="srun -N 512 -n 4096 -c 1 --gpus-per-node=8 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 8 -y 8 -z 64 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/mechanics/level06/ECP_Wellbore_cpu.xml b/inputFiles/wellboreECP/mechanics/level06/ECP_Wellbore_cpu.xml new file mode 100644 index 00000000000..d1901ff9371 --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level06/ECP_Wellbore_cpu.xml @@ -0,0 +1,81 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/mechanics/level06/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/mechanics/level06/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..8ebe7b9238e --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level06/ECP_Wellbore_gpu.xml @@ -0,0 +1,87 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/mechanics/level06/launch_frontier b/inputFiles/wellboreECP/mechanics/level06/launch_frontier new file mode 100644 index 00000000000..2fd2ad14edc --- /dev/null +++ b/inputFiles/wellboreECP/mechanics/level06/launch_frontier @@ -0,0 +1,18 @@ +#!/bin/bash +#SBATCH -N 4096 +#SBATCH -n 32768 +#SBATCH -t 0:30:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-mechanics-geom06.out + +echo `date` +module load cpe/23.12 +module load craype-accel-amd-gfx90a +module load rocm/5.4.3 +export MPICH_GPU_SUPPORT_ENABLED=1 +ldd ${1} +cmd="srun -N 4096 -n 32768 -c 1 --gpus-per-node=8 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 16 -y 16 -z 128 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/postprocess.py b/inputFiles/wellboreECP/postprocess.py new file mode 100755 index 00000000000..914c1abd038 --- /dev/null +++ b/inputFiles/wellboreECP/postprocess.py @@ -0,0 +1,241 @@ +#!/opt/cray/pe/python/3.11.5/bin/python + +import matplotlib.pyplot as plt +import argparse +import glob +import sys +import re +import os + +def parse_log_file(file_path): + """Parse the log file from GEOS and extract required statistics.""" + try: + print(f"Parsing {file_path = }") + with open(file_path, 'r') as file: + data = file.read() + + except FileNotFoundError: + print(f"Error: File not found - {file_path}", file=sys.stderr) + + except Exception as e: + print(f"Error processing file {file_path}: {str(e)}", file=sys.stderr) + + # Regex patterns to extract the required information + patterns = { + 'non_linear_iters': re.compile(r'number of successful nonlinear iterations: (\d+).*number of discarded nonlinear iterations: (\d+)', re.S), + 'linear_iters': re.compile(r'number of successful linear iterations: (\d+).*number of discarded linear iterations: (\d+)', re.S), + 'applysol_time': re.compile(r'apply solution time = (\d+\.\d+) s'), + 'assemble_time': re.compile(r'assemble time = (\d+\.\d+) s'), + 'convcheck_time': re.compile(r'convergence check time = (\d+\.\d+) s'), + 'update_state_time': re.compile(r'update state time = (\d+\.\d+) s'), + 'ls_create_time': re.compile(r'linear solver create time = (\d+.\d+) s'), + 'ls_setup_time': re.compile(r'linear solver setup time = (\d+.\d+) s'), + 'ls_solve_time': re.compile(r'linear solver solve time = (\d+.\d+) s'), + 'run_time': re.compile(r'run time\s+.*\((\d+\.\d+) s\)') + } + + # Dictionary to hold the results + stats = {} + + # Extract and process all patterns + for key, pattern in patterns.items(): + if key in ['non_linear_iters', 'linear_iters']: + xmatch = pattern.search(data) + stats[key] = int(xmatch.group(1)) + int(xmatch.group(2)) if xmatch else 0 + + else: + xmatch = pattern.search(data) + stats[key] = float(xmatch.group(1)) if xmatch else 0 + + # Calculate GEOS time (run time excluding hypre time) + stats['geos_time'] = stats['run_time'] - (stats['ls_create_time'] + stats['ls_setup_time'] + stats['ls_solve_time']) + + # Calculate averages + averages = { + 'avg_geos_time': 'geos_time', + 'avg_ls_create_time': 'ls_create_time', + 'avg_ls_setup_time': 'ls_setup_time', + 'avg_ls_solve_time': 'ls_solve_time', + 'avg_ls_iters': 'linear_iters', + 'avg_run_time': 'run_time' + } + + if stats['non_linear_iters'] <= 0: + print(f"Invalid number of non-linear iterations!", file=sys.stderr) + return None + + for avg_key, total_key in averages.items(): + stats[avg_key] = stats[total_key] / stats['non_linear_iters'] + + # Store level + xmatch = re.search(r'level(\d+)', file_path) + stats['level'] = int(xmatch.group(1)) if xmatch else -1 + + return stats + +def print_results(model, results): + """Print results in markdown table format""" + if not results or not all(results): + print(f"Invalid results!", file=sys.stderr) + return + + # Setup header + headers = ["Level", "GEOS (s)", "Matrix creation (s)", "Hypre setup (s)", "Hypre solve (s)", "Total (s)", "Hypre iters"] + + # Gather all rows data including headers + data_rows = [ + headers, + *[ + [ + "{:02d}".format(result['level']), + "{:.4f}".format(result['avg_geos_time']), + "{:.4f}".format(result['avg_ls_create_time']), + "{:.4f}".format(result['avg_ls_setup_time']), + "{:.4f}".format(result['avg_ls_solve_time']), + "{:.4f}".format(result['avg_run_time']), + "{:.1f}".format(result['avg_ls_iters']), + ] + for result in results + ] + ] + + # Calculate maximum column widths + max_widths = [max(len(row[i]) for row in data_rows) for i in range(len(headers))] + + # Print header row with adjusted widths + header_row = "| " + " | ".join(header.ljust(max_widths[i]) for i, header in enumerate(headers)) + " |" + print(f"\nSummary table for {model}:\n") + print(header_row) + + # Adjust the divider line based on maximum widths + divider_line = "|" + "|".join(["-" * (max_widths[i] + 2) for i in range(len(headers))]) + "|" + print(divider_line) + + # Print each data row with adjusted widths + for row in data_rows[1:]: # Skip the header in data_rows for output + formatted_row = "| " + " | ".join(item.ljust(max_widths[i]) for i, item in enumerate(row)) + " |" + print(formatted_row) + +def plot_results(model, model_data, results, fs=16, ms=10): + if not results: + return + + plt.rcParams.update({'font.size': fs}) + fig, ax = plt.subplots(figsize=(10, 6)) + + # Extracting ranks and global DOFs + levels = [result['level'] for result in results] + ranks = [entry['ranks'] for entry in model_data if entry['level'] in levels] + global_dofs = [entry['global_dofs'] for entry in model_data if entry['level'] in levels] + + # Setting up data for plotting + geos_times = [result['avg_geos_time'] for result in results] + ls_create_times = [result['avg_ls_create_time'] for result in results] + ls_setup_times = [result['avg_ls_setup_time'] for result in results] + ls_solve_times = [result['avg_ls_solve_time'] for result in results] + total_times = [result['avg_run_time'] for result in results] + + # Plotting each metric + ax.plot(ranks, geos_times, 'o-', markersize=ms, label='GEOS') + ax.plot(ranks, ls_create_times, 's-', markersize=ms, label='Matrix Creation') + ax.plot(ranks, ls_setup_times, '^-', markersize=ms, label='Hypre Setup') + ax.plot(ranks, ls_solve_times, 'v-', markersize=ms, label='Hypre Solve') + ax.plot(ranks, total_times, 'd-', markersize=ms, label='Total') + + # Adding labels and legend + ax.set_title(f"Weak scaling - {model}", fontsize=fs+4) + ax.set_xlabel('Number of GPUs (Global DOFs)', labelpad=18, fontsize=fs+1) + ax.set_ylabel('Time (s)', labelpad=18, fontsize=fs+1) + ax.set_xscale('log', base=2) + ax.set_xticks(ranks) + ax.set_xticklabels([f"{r:,} ({d})" for r, d in zip(ranks, global_dofs)]) + ax.legend(fontsize=fs) + + # Show grid lines + ax.grid(True, which='both', linestyle='--', linewidth=0.5, color='gray') # Add gridlines with custom settings + + # Update margins + plt.tight_layout() + + # Save figure to file + figname = f"weakscal_{model}.png" + print(f"Saving figure to file {figname}...") + plt.savefig(figname, format='png', dpi=450) + + # Show the plot + plt.show() + +def find_newest_file(model, level): + """Finds the newest output file under the directory relative to a physics model and for certain problem level (mesh refinement).""" + target_dir = os.path.join(model, f"level{level:02d}") + files = glob.glob(os.path.join(target_dir, '*.out')) + if not files: + return target_dir, None + + # Extract the job ID numbers and sort the files by these IDs + suffixes = {'mechanics': 'mechanics', + 'singlePhaseFlow': 'singlePhaseFlow', + 'compositionalMultiphaseFlow': 'compflow'} + files_with_ids = [(file, int(re.search(rf"(\d+)-{suffixes[model]}", file).group(1))) for file in files] + + # Sort by the job ID number + return target_dir, max(files_with_ids, key=lambda x: x[1])[0] + +def main(): + """Process multiple input files and store results using command line arguments.""" + valid_models = ['mechanics', 'singlePhaseFlow', 'compositionalMultiphaseFlow'] + + parser = argparse.ArgumentParser() + parser.add_argument('-m', '--model', required=True, choices=valid_models, help="Physics model") + parser.add_argument('-l', '--levels', nargs='+', type=int, help="Levels to process") + args = parser.parse_args() + + # Setup problem info + problems = { + "mechanics": [ + {"level": 2, "ranks": 8, "global_dofs": "20.7M"}, + {"level": 3, "ranks": 64, "global_dofs": "162M"}, + {"level": 4, "ranks": 512, "global_dofs": "1.3B"}, + {"level": 5, "ranks": 4096, "global_dofs": "10.2B"}, + {"level": 6, "ranks": 32768, "global_dofs": "81.3B"} + ], + "singlePhaseFlow": [ + {"level": 2, "ranks": 4, "global_dofs": "6.6M"}, + {"level": 3, "ranks": 32, "global_dofs": "52.8M"}, + {"level": 4, "ranks": 256, "global_dofs": "422M"}, + {"level": 5, "ranks": 2048, "global_dofs": "3.4B"}, + {"level": 6, "ranks": 16384, "global_dofs": "27B"} + ], + "compositionalMultiphaseFlow": [ + {"level": 2, "ranks": 4, "global_dofs": "19.8M"}, + {"level": 3, "ranks": 32, "global_dofs": "153M"}, + {"level": 4, "ranks": 256, "global_dofs": "1.2B"}, + {"level": 5, "ranks": 2048, "global_dofs": "9.8B"} + ] + } + + # Set default levels if not passed in + if args.levels is None: + args.levels = [entry['level'] for entry in problems[args.model]] + + # Read log files + results = [] + file_paths = [] + for level in args.levels: + target_dir, file_path = find_newest_file(args.model, level) + if file_path: + file_paths.append(file_path) + else: + print(f"No suitable files found in {target_dir}") + + # Parse statistics + results = [parse_log_file(fp) for fp in file_paths] + + # Print summary table + print_results(args.model, results) + + # Plot results + plot_results(args.model, problems[args.model], results) + +if __name__ == "__main__": + main() diff --git a/inputFiles/wellboreECP/singlePhaseFlow/ECP_Wellbore_probdef.xml b/inputFiles/wellboreECP/singlePhaseFlow/ECP_Wellbore_probdef.xml new file mode 100644 index 00000000000..73411abc18e --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/ECP_Wellbore_probdef.xml @@ -0,0 +1,124 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level01/ECP_Wellbore_cpu.xml b/inputFiles/wellboreECP/singlePhaseFlow/level01/ECP_Wellbore_cpu.xml new file mode 100644 index 00000000000..56163cee621 --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level01/ECP_Wellbore_cpu.xml @@ -0,0 +1,60 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level01/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/singlePhaseFlow/level01/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..81a0b3b84b6 --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level01/ECP_Wellbore_gpu.xml @@ -0,0 +1,64 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level01/launch_frontier b/inputFiles/wellboreECP/singlePhaseFlow/level01/launch_frontier new file mode 100644 index 00000000000..d31e8dd39c3 --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level01/launch_frontier @@ -0,0 +1,13 @@ +#!/bin/bash +#SBATCH -N 1 +#SBATCH -n 1 +#SBATCH -t 0:30:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-singlePhaseFlow-geom01.out + +echo `date` +cmd="srun -N 1 -n 1 -c 1 --gpus-per-node=1 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 1 -y 1 -z 1 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level02/ECP_Wellbore_cpu.xml b/inputFiles/wellboreECP/singlePhaseFlow/level02/ECP_Wellbore_cpu.xml new file mode 100644 index 00000000000..dffc9e5be37 --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level02/ECP_Wellbore_cpu.xml @@ -0,0 +1,60 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level02/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/singlePhaseFlow/level02/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..e784ec8cf0a --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level02/ECP_Wellbore_gpu.xml @@ -0,0 +1,64 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level02/launch_frontier b/inputFiles/wellboreECP/singlePhaseFlow/level02/launch_frontier new file mode 100644 index 00000000000..2741050a32c --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level02/launch_frontier @@ -0,0 +1,18 @@ +#!/bin/bash +#SBATCH -N 1 +#SBATCH -n 4 +#SBATCH -t 0:30:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-singlePhaseFlow-geom02.out + +echo `date` +module load cpe/23.12 +module load craype-accel-amd-gfx90a +module load rocm/5.4.3 +export MPICH_GPU_SUPPORT_ENABLED=1 +ldd ${1} +cmd="srun -N 1 -n 4 -c 1 --gpus-per-node=4 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 1 -y 1 -z 4 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level03/ECP_Wellbore_cpu.xml b/inputFiles/wellboreECP/singlePhaseFlow/level03/ECP_Wellbore_cpu.xml new file mode 100644 index 00000000000..0f58cf96b36 --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level03/ECP_Wellbore_cpu.xml @@ -0,0 +1,60 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level03/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/singlePhaseFlow/level03/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..33bc77be4fc --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level03/ECP_Wellbore_gpu.xml @@ -0,0 +1,65 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level03/launch_frontier b/inputFiles/wellboreECP/singlePhaseFlow/level03/launch_frontier new file mode 100644 index 00000000000..b549f4c4ff5 --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level03/launch_frontier @@ -0,0 +1,18 @@ +#!/bin/bash +#SBATCH -N 4 +#SBATCH -n 32 +#SBATCH -t 0:30:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-singlePhaseFlow-geom03.out + +echo `date` +module load cpe/23.12 +module load craype-accel-amd-gfx90a +module load rocm/5.4.3 +export MPICH_GPU_SUPPORT_ENABLED=1 +ldd ${1} +cmd="srun -N 4 -n 32 -c 1 --gpus-per-node=8 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 2 -y 2 -z 8 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level04/ECP_Wellbore_cpu.xml b/inputFiles/wellboreECP/singlePhaseFlow/level04/ECP_Wellbore_cpu.xml new file mode 100644 index 00000000000..caf966a8b6e --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level04/ECP_Wellbore_cpu.xml @@ -0,0 +1,60 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level04/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/singlePhaseFlow/level04/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..69294a7faca --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level04/ECP_Wellbore_gpu.xml @@ -0,0 +1,66 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level04/launch_frontier b/inputFiles/wellboreECP/singlePhaseFlow/level04/launch_frontier new file mode 100644 index 00000000000..80748a5e6d7 --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level04/launch_frontier @@ -0,0 +1,18 @@ +#!/bin/bash +#SBATCH -N 32 +#SBATCH -n 256 +#SBATCH -t 0:30:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-singlePhaseFlow-geom04.out + +echo `date` +module load cpe/23.12 +module load craype-accel-amd-gfx90a +module load rocm/5.4.3 +export MPICH_GPU_SUPPORT_ENABLED=1 +ldd ${1} +cmd="srun -N 32 -n 256 -c 1 --gpus-per-node=8 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 4 -y 4 -z 16 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level05/ECP_Wellbore_cpu.xml b/inputFiles/wellboreECP/singlePhaseFlow/level05/ECP_Wellbore_cpu.xml new file mode 100644 index 00000000000..2b784e869dd --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level05/ECP_Wellbore_cpu.xml @@ -0,0 +1,60 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level05/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/singlePhaseFlow/level05/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..4bfc1ff0df2 --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level05/ECP_Wellbore_gpu.xml @@ -0,0 +1,65 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level05/launch_frontier b/inputFiles/wellboreECP/singlePhaseFlow/level05/launch_frontier new file mode 100644 index 00000000000..88572e1e96d --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level05/launch_frontier @@ -0,0 +1,18 @@ +#!/bin/bash +#SBATCH -N 256 +#SBATCH -n 2048 +#SBATCH -t 0:30:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-singlePhaseFlow-geom05.out + +echo `date` +module load cpe/23.12 +module load craype-accel-amd-gfx90a +module load rocm/5.4.3 +export MPICH_GPU_SUPPORT_ENABLED=1 +ldd ${1} +cmd="srun -N 256 -n 2048 -c 1 --gpus-per-node=8 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 8 -y 8 -z 32 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level06/ECP_Wellbore_gpu.xml b/inputFiles/wellboreECP/singlePhaseFlow/level06/ECP_Wellbore_gpu.xml new file mode 100644 index 00000000000..2cfaca7d3d5 --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level06/ECP_Wellbore_gpu.xml @@ -0,0 +1,65 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/inputFiles/wellboreECP/singlePhaseFlow/level06/launch_frontier b/inputFiles/wellboreECP/singlePhaseFlow/level06/launch_frontier new file mode 100644 index 00000000000..3791ac44860 --- /dev/null +++ b/inputFiles/wellboreECP/singlePhaseFlow/level06/launch_frontier @@ -0,0 +1,18 @@ +#!/bin/bash +#SBATCH -N 2048 +#SBATCH -n 16384 +#SBATCH -t 0:30:00 +#SBATCH -J geos +#SBATCH -o frontier-%j-singlePhaseFlow-geom06.out + +echo `date` +module load cpe/23.12 +module load craype-accel-amd-gfx90a +module load rocm/5.4.3 +export MPICH_GPU_SUPPORT_ENABLED=1 +ldd ${1} +cmd="srun -N 2048 -n 16384 -c 1 --gpus-per-node=8 --ntasks-per-gpu=1 --gpu-bind=closest ${1} -i ECP_Wellbore_gpu.xml -x 16 -y 16 -z 64 ${2}" +echo "${cmd}" +${cmd} +echo `date` +echo 'Done' diff --git a/src/conf.py b/src/conf.py index bcc5a918a58..fe9c2002d28 100644 --- a/src/conf.py +++ b/src/conf.py @@ -49,17 +49,24 @@ config_src = os.path.join(docs_path, "GeosxConfig.hpp") config_dst = os.path.join(common_path, "GeosxConfig.hpp") - input_dirs = ["coreComponents/common", - "coreComponents/dataRepository", - "coreComponents/fileIO", - "coreComponents/linearAlgebra", - "coreComponents/mesh", - "coreComponents/managers", - "coreComponents/finiteElement/kernelInterface", - "coreComponents/mesh/MeshFields.hpp", - "coreComponents/physicsSolvers/simplePDE/LaplaceFEMKernels.hpp", - "coreComponents/physicsSolvers/solidMechanics", - "coreComponents/finiteVolume"] + input_dirs = [ + "coreComponents/common", + "coreComponents/dataRepository", + "coreComponents/fileIO", + "coreComponents/linearAlgebra", + "coreComponents/mesh", + "coreComponents/finiteElement/elementFormulations", + "coreComponents/finiteElement/kernelInterface", + "coreComponents/mesh/MeshFields.hpp", + "coreComponents/physicsSolvers", + "coreComponents/finiteVolume", + "coreComponents/functions", + "coreComponents/fieldSpecification", + "coreComponents/discretizationMethods", + "coreComponents/events", + "coreComponents/mainInterface" + ] + # Write correct ReadtheDocs path and input directories shutil.copy(doxyfile_src, doxyfile_dst) diff --git a/src/coreComponents/dataRepository/Group.hpp b/src/coreComponents/dataRepository/Group.hpp index b9ece0eba53..1a8cb2358cc 100644 --- a/src/coreComponents/dataRepository/Group.hpp +++ b/src/coreComponents/dataRepository/Group.hpp @@ -143,7 +143,7 @@ class Group using CatalogInterface = dataRepository::CatalogInterface< Group, string const &, Group * const >; /** - * @brief Get the singleton catalog for this class. + * @brief Get the singleton catalog for this Group. * @return reference to the catalog object */ static CatalogInterface::CatalogType & getCatalog(); diff --git a/src/coreComponents/dataRepository/docs/Group.rst b/src/coreComponents/dataRepository/docs/Group.rst index 39547980be3..370ac8937b9 100644 --- a/src/coreComponents/dataRepository/docs/Group.rst +++ b/src/coreComponents/dataRepository/docs/Group.rst @@ -119,3 +119,9 @@ Looping Interface :language: c++ :start-after: //START_SPHINX_INCLUDE_LOOP_INTERFACE :end-before: //END_SPHINX_INCLUDE_LOOP_INTERFACE + + +Doxygen API documentation +------------------------- + +`Group API <../../../doxygen_output/html/classgeos_1_1data_repository_1_1_group.html>`_ diff --git a/src/coreComponents/dataRepository/docs/MappedVector.rst b/src/coreComponents/dataRepository/docs/MappedVector.rst index 260c448d376..22cb211d39b 100644 --- a/src/coreComponents/dataRepository/docs/MappedVector.rst +++ b/src/coreComponents/dataRepository/docs/MappedVector.rst @@ -41,7 +41,7 @@ Element access In addition to these, an STL-conformant iterator interface is available via ``begin()`` and ``end()`` methods. The type iterated over is a key-pointer pair (provided as `value_type` alias). -API documentation ------------------ +Doxygen API documentation +------------------------- -`MappedVector <../../../doxygen_output/html/classgeos_1_1_mapped_vector.html>`_ +`MappedVector API <../../../doxygen_output/html/classgeos_1_1_mapped_vector.html>`_ diff --git a/src/coreComponents/dataRepository/docs/Wrapper.rst b/src/coreComponents/dataRepository/docs/Wrapper.rst index 27f4b8815d9..e7f44040f38 100644 --- a/src/coreComponents/dataRepository/docs/Wrapper.rst +++ b/src/coreComponents/dataRepository/docs/Wrapper.rst @@ -124,7 +124,7 @@ The type ``DefaultValue`` is used to store the default value for the wrapper. ``DefaultValue`` is actually not a type but an alias for another internal struct. As such, it cannot currently be specialized for a user's custom type. -API documentation ------------------ +Doxygen API documentation +------------------------- -`Wrapper <../../../doxygen_output/html/classgeos_1_1data_repository_1_1_wrapper.html>`_ +`Wrapper API <../../../doxygen_output/html/classgeos_1_1data_repository_1_1_wrapper.html>`_ diff --git a/src/coreComponents/dataRepository/docs/dataRepository.rst b/src/coreComponents/dataRepository/docs/dataRepository.rst index 16b82289bba..221cb8823cf 100644 --- a/src/coreComponents/dataRepository/docs/dataRepository.rst +++ b/src/coreComponents/dataRepository/docs/dataRepository.rst @@ -6,14 +6,16 @@ Data Repository The GEOS "Data Repository" is intended to provide the building blocks for the code structure within GEOS. The "Data Repository" provides a general capability to store arbitrary data and objects in a hierarchical -structure, similar to a standard file system. +structure, similar to a standard file system. The "Wrapper" object is a generic container for any object, +and provides a standard interface for accessing an object and performing standard operations. +The "Group" object is a container for "Wrapper" and other "Group" objects. -The components/classes of the data structure that a developer will require some knowledge of are: +The components/classes of the data structure that a developer should have some knowledge of are: .. toctree:: :maxdepth: 2 - /coreComponents/dataRepository/docs/MappedVector /coreComponents/dataRepository/docs/Group /coreComponents/dataRepository/docs/Wrapper /coreComponents/dataRepository/docs/ObjectCatalog + /coreComponents/dataRepository/docs/MappedVector diff --git a/src/coreComponents/mesh/docs/meshDeveloperGuide.rst b/src/coreComponents/mesh/docs/meshDeveloperGuide.rst index c0ba2ca9598..8800dd7d139 100644 --- a/src/coreComponents/mesh/docs/meshDeveloperGuide.rst +++ b/src/coreComponents/mesh/docs/meshDeveloperGuide.rst @@ -135,3 +135,13 @@ tetrahedra, one for all hexahedra, one for all wedges and one for all the pyrami Now that all the classes of the mesh hierarchy has been described, we propose to adapt the diagram presented in :numref:`diagMeshDevFig` to match with the example presented in :numref:`modelMeshDevFig`. + +Direct links to some useful class documentation: + +`ObjectManagerBase API <../../../doxygen_output/html/classgeos_1_1_object_manager_base.html>`_ + +`MeshLevel API <../../../doxygen_output/html/classgeos_1_1_mesh_level.html>`_ + +`NodeManager API <../../../doxygen_output/html/classgeos_1_1_node_manager.html>`_ + +`FaceManager API <../../../doxygen_output/html/classgeos_1_1_face_manager.html>`_ diff --git a/src/coreComponents/physicsSolvers/SolverBase.cpp b/src/coreComponents/physicsSolvers/SolverBase.cpp index a235d1230bf..2ebb1671baf 100644 --- a/src/coreComponents/physicsSolvers/SolverBase.cpp +++ b/src/coreComponents/physicsSolvers/SolverBase.cpp @@ -703,31 +703,7 @@ bool SolverBase::lineSearchWithParabolicInterpolation( real64 const & time_n, return lineSearchSuccess; } -/** - * @brief Eisenstat-Walker adaptive tolerance - * - * This method enables an inexact-Newton method is which the linear solver - * tolerance is chosen based on the nonlinear solver convergence behavior. - * In early Newton iterations, the search direction is usually imprecise, and - * therefore a weak linear convergence tolerance can be chosen to minimize - * computational cost. As the search gets closer to the true solution, however, - * more stringent linear tolerances are necessary to maintain quadratic convergence - * behavior. - * - * The user can set the weakest tolerance allowed, with a default of 1e-3. - * Even weaker values (e.g. 1e-2,1e-1) can be used for further speedup, but may - * occasionally cause convergence problems. Use this parameter with caution. The - * most stringent tolerance is hardcoded to 1e-8, which is sufficient for - * most problems. - * - * See Eisenstat, S.C. and Walker, H.F., 1996. Choosing the forcing terms in an - * inexact Newton method. SIAM Journal on Scientific Computing, 17(1), pp.16-32. - * - * @param newNewtonNorm Residual norm at current iteration - * @param oldNewtonNorm Residual norm at previous iteration - * @param weakestTol Weakest tolerance allowed (default 1e-3). - * @return Adaptive tolerance recommendation - */ + real64 SolverBase::eisenstatWalker( real64 const newNewtonNorm, real64 const oldNewtonNorm, real64 const weakestTol ) diff --git a/src/coreComponents/physicsSolvers/SolverBase.hpp b/src/coreComponents/physicsSolvers/SolverBase.hpp index 5b88227ed2a..702fa445b03 100644 --- a/src/coreComponents/physicsSolvers/SolverBase.hpp +++ b/src/coreComponents/physicsSolvers/SolverBase.hpp @@ -13,6 +13,10 @@ * ------------------------------------------------------------------------------------------------------------ */ +/** + * @file SolverBase.hpp + */ + #ifndef GEOS_PHYSICSSOLVERS_SOLVERBASE_HPP_ #define GEOS_PHYSICSSOLVERS_SOLVERBASE_HPP_ @@ -34,20 +38,54 @@ namespace geos class DomainPartition; +/** + * @class SolverBase + * @brief Base class for all physics solvers + * + * This class provides the base interface for all physics solvers. It provides the basic + * functionality for setting up and solving a linear system, as well as the interface for + * performing a timestep. + */ class SolverBase : public ExecutableGroup { public: + /** + * @brief Constructor for SolverBase + * @param name the name of this instantiation of SolverBase + * @param parent the parent group of this instantiation of SolverBase + */ explicit SolverBase( string const & name, Group * const parent ); + /** + * @brief Move constructor for SolverBase + */ SolverBase( SolverBase && ) = default; + /** + * @brief Destructor for SolverBase + */ virtual ~SolverBase() override; + /** + * @brief Deleted constructor + */ SolverBase() = delete; + + /** + * @brief Deleted copy constructor + */ SolverBase( SolverBase const & ) = delete; + + /** + * @brief Deleted copy assignment operator + */ SolverBase & operator=( SolverBase const & ) = delete; + + /** + * @brief Deleted move assignment operator + */ SolverBase & operator=( SolverBase && ) = delete; /** @@ -56,12 +94,26 @@ class SolverBase : public ExecutableGroup virtual string getCatalogName() const = 0; + /** + * @brief Register wrappers that contain data on the mesh objects + * @param MeshBodies the group of mesh bodies + */ virtual void registerDataOnMesh( Group & MeshBodies ) override; + /** + * @brief Initialization tasks after mesh generation is completed. + */ virtual void initialize_postMeshGeneration() override; + /** + * @brief Generate mesh targets from target regions + * @param meshBodies the group of mesh bodies + */ void generateMeshTargetsFromTargetRegions( Group const & meshBodies ); + /** + * @copydoc ExecutableGroup::cleanup + */ virtual void cleanup( real64 const time_n, integer const cycleNumber, integer const eventCounter, @@ -69,7 +121,7 @@ class SolverBase : public ExecutableGroup DomainPartition & domain ) override; /** - * This method is called when its host event is triggered + * @copydoc ExecutableGroup::execute */ virtual bool execute( real64 const time_n, real64 const dt, @@ -83,6 +135,11 @@ class SolverBase : public ExecutableGroup * @return a reference to linear system matrix of this solver */ ParallelMatrix & getSystemMatrix() { return m_matrix; } + + /** + * @brief Getter for system rhs vector + * @return a reference to linear system right-hand side of this solver + */ ParallelMatrix const & getSystemMatrix() const { return m_matrix; } /** @@ -90,6 +147,11 @@ class SolverBase : public ExecutableGroup * @return a reference to linear system right-hand side of this solver */ ParallelVector & getSystemRhs() { return m_rhs; } + + /** + * @brief Getter for system rhs vector + * @return a reference to linear system right-hand side of this solver + */ ParallelVector const & getSystemRhs() const { return m_rhs; } /** @@ -97,6 +159,11 @@ class SolverBase : public ExecutableGroup * @return a reference to solution vector of this solver */ ParallelVector & getSystemSolution() { return m_solution; } + + /** + * @brief Getter for system solution vector + * @return a reference to solution vector of this solver + */ ParallelVector const & getSystemSolution() const { return m_solution; } /** @@ -104,6 +171,11 @@ class SolverBase : public ExecutableGroup * @return a reference to degree-of-freedom manager of this solver */ DofManager & getDofManager() { return m_dofManager; } + + /** + * @brief Getter for degree-of-freedom manager + * @return a reference to degree-of-freedom manager of this solver + */ DofManager const & getDofManager() const { return m_dofManager; } /** @@ -111,6 +183,11 @@ class SolverBase : public ExecutableGroup * @return a reference to linear system matrix of this solver */ CRSMatrix< real64, globalIndex > & getLocalMatrix() { return m_localMatrix; } + + /** + * @brief Getter for local matrix + * @return a reference to linear system matrix of this solver + */ CRSMatrixView< real64 const, globalIndex const > getLocalMatrix() const { return m_localMatrix.toViewConst(); } /** @@ -213,6 +290,7 @@ class SolverBase : public ExecutableGroup * @param localMatrix the system matrix * @param rhs the system right-hand side vector * @param solution the solution vector + * @param scaleFactor the scaling factor to apply to the solution * @param lastResidual (in) target value below which to reduce residual norm, (out) achieved residual norm * @return return true if line search succeeded, false otherwise * @@ -236,14 +314,16 @@ class SolverBase : public ExecutableGroup /** * @brief Function to perform line search using a parabolic interpolation to find the scaling factor. * @param time_n time at the beginning of the step - * @param dt the perscribed timestep + * @param dt the prescribed timestep * @param cycleNumber the current cycle number * @param domain the domain object * @param dofManager degree-of-freedom manager associated with the linear system * @param localMatrix the system matrix * @param rhs the system right-hand side vector * @param solution the solution vector + * @param scaleFactor the scaling factor to apply to the solution * @param lastResidual (in) target value below which to reduce residual norm, (out) achieved residual norm + * @param residualNormT the residual norm at the end of the line search * @return return true if line search succeeded, false otherwise * */ @@ -298,6 +378,7 @@ class SolverBase : public ExecutableGroup /** * @brief Populate degree-of-freedom manager with fields relevant to this solver + * @param domain the domain containing the mesh and fields * @param dofManager degree-of-freedom manager associated with the linear system */ virtual void @@ -311,6 +392,7 @@ class SolverBase : public ExecutableGroup * @param localMatrix the system matrix * @param rhs the system right-hand side vector * @param solution the solution vector + * @param setSparsity flag to indicate if the sparsity pattern should be set * * @note While the function is virtual, the base class implementation should be * sufficient for most single-physics solvers. @@ -330,8 +412,7 @@ class SolverBase : public ExecutableGroup * @param domain the domain partition * @param dofManager degree-of-freedom manager associated with the linear system * @param localMatrix the system matrix - * @param rhs the system right-hand side vector - * @return the residual for convergence evaluation + * @param localRhs the system right-hand side vector * * This function assembles the residual and the jacobian of the residual wrt the primary * variables. In a stand alone physics solver, this function will fill a single block in the @@ -357,7 +438,7 @@ class SolverBase : public ExecutableGroup * @param domain the domain partition * @param dofManager degree-of-freedom manager associated with the linear system * @param localMatrix the system matrix - * @param rhs the system right-hand side vector + * @param localRhs the system right-hand side vector * * This function applies all boundary conditions to the linear system. This is essentially a * completion of the system assembly, but is separated for use in coupled solvers. @@ -438,12 +519,10 @@ class SolverBase : public ExecutableGroup /** * @brief Function to check system solution for physical consistency and constraint violation - * @param matrix the system matrix - * @param rhs the system right-hand side vector - * @param solution the solution vector + * @param domain the domain partition * @param dofManager degree-of-freedom manager associated with the linear system + * @param localSolution the solution vector * @param scalingFactor factor to scale the solution prior to application - * @param objectManager the object manager that holds the fields we wish to apply the solution to * @return true if solution can be safely applied without violating physical constraints, false otherwise * * @note This function must be overridden in the derived physics solver in order to use an implict @@ -460,7 +539,7 @@ class SolverBase : public ExecutableGroup * @brief Function to determine if the solution vector should be scaled back in order to maintain a known constraint. * @param[in] domain The domain partition. * @param[in] dofManager degree-of-freedom manager associated with the linear system - * @param[in] solution the solution vector + * @param[in] localSolution the solution vector * @return The factor that should be used to scale the solution vector values when they are being applied. */ virtual real64 @@ -470,12 +549,11 @@ class SolverBase : public ExecutableGroup /** * @brief Function to apply the solution vector to the state - * @param matrix the system matrix - * @param rhs the system right-hand side vector - * @param solution the solution vector * @param dofManager degree-of-freedom manager associated with the linear system + * @param localSolution the solution vector * @param scalingFactor factor to scale the solution prior to application - * @param objectManager the object manager that holds the fields we wish to apply the solution to + * @param dt the timestep + * @param domain the domain partition * * This function performs 2 operations: * 1) extract the solution vector for the "blockSystem" parameter, and applies the @@ -517,8 +595,9 @@ class SolverBase : public ExecutableGroup virtual void resetConfigurationToBeginningOfStep( DomainPartition & domain ); /** - * @brief set the simplest configuration state. + * @brief resets the configuration to the default value. * @param domain the domain containing the mesh and fields + * @return a bool that states whether the configuration was reset or not. */ virtual bool resetConfigurationToDefault( DomainPartition & domain ) const; @@ -545,7 +624,7 @@ class SolverBase : public ExecutableGroup /** * @brief perform cleanup for implicit timestep - * @param time_n the time at the beginning of the step + * @param time the time at the beginning of the step * @param dt the desired timestep * @param domain the domain partition * @@ -562,36 +641,79 @@ class SolverBase : public ExecutableGroup DomainPartition & domain ); - /* - * Returns the requirement for the next time-step to the event executing the solver. + /** + * @brief getter for the next timestep size + * @return the next timestep size m_nextDt */ - virtual real64 getTimestepRequest( real64 const GEOS_UNUSED_PARAM( time ) ) override + virtual real64 getTimestepRequest( real64 const ) override {return m_nextDt;}; /**@}*/ + /** + * @brief getter for the next timestep size + * @return the next timestep size m_nextDt + */ real64 getTimestepRequest() {return m_nextDt;}; + /** + * @brief creates a child group of of this SolverBase instantiation + * @param childKey the key of the child type + * @param childName the name of the child + * @return a pointer to the child group + */ virtual Group * createChild( string const & childKey, string const & childName ) override; + /** + * @brief Type alias for catalog interface used by this class. See CatalogInterface. + */ using CatalogInterface = dataRepository::CatalogInterface< SolverBase, string const &, Group * const >; + + /** + * @brief Get the singleton catalog for SolverBase. + * @return reference to the catalog object + */ static CatalogInterface::CatalogType & getCatalog(); + /** + * @brief Structure to hold scoped key names + */ struct viewKeyStruct { + /// @return string for the cflFactor wrapper static constexpr char const * cflFactorString() { return "cflFactor"; } + + /// @return string for the initialDt wrapper static constexpr char const * initialDtString() { return "initialDt"; } + + /// @return string for the maxStableDt wrapper static constexpr char const * maxStableDtString() { return "maxStableDt"; } + + /// @return string for the discretization wrapper static constexpr char const * discretizationString() { return "discretization"; } + + /// @return string for the nextDt targetRegions wrapper static constexpr char const * targetRegionsString() { return "targetRegions"; } + + /// @return string for the meshTargets wrapper static constexpr char const * meshTargetsString() { return "meshTargets"; } + + /// @return string for the writeLinearSystem wrapper static constexpr char const * writeLinearSystemString() { return "writeLinearSystem"; } }; + /** + * @brief Structure to hold scoped key names + */ struct groupKeyStruct { + /// @return string for the linearSolverParameters wrapper static constexpr char const * linearSolverParametersString() { return "LinearSolverParameters"; } + + /// @return string for the nonlinearSolverParameters wrapper static constexpr char const * nonlinearSolverParametersString() { return "NonlinearSolverParameters"; } + + /// @return string for the solverStatistics wrapper static constexpr char const * solverStatisticsString() { return "SolverStatistics"; } }; @@ -624,8 +746,17 @@ class SolverBase : public ExecutableGroup */ R1Tensor const gravityVector() const; + /** + * @brief Check if the solution increments are ok to use + * @param domain the domain partition + * @return true if the solution increments are ok to use, false otherwise + */ virtual bool checkSequentialSolutionIncrements( DomainPartition & domain ) const; + /** + * @brief Save the state of the solver for sequential iteration + * @param domain the domain partition + */ virtual void saveSequentialIterationState( DomainPartition & domain ); /** @@ -664,6 +795,9 @@ class SolverBase : public ExecutableGroup return m_nonlinearSolverParameters; } + /** + * @brief syncronize the nonlinear solver parameters. + */ virtual void synchronizeNonlinearSolverParameters() { /* empty here, overriden in CoupledSolver */ } @@ -729,28 +863,77 @@ class SolverBase : public ExecutableGroup } } - + /** + * @brief return the name of the discretization object + * @return the name of the discretization object + */ string getDiscretizationName() const {return m_discretizationName;} + /** + * @brief function to set the value of m_assemblyCallback + * @param func the function to set m_assemblyCallback to + * @param funcType the type of the function + * @return true if the function was successfully set, false otherwise + * + * This is used to provide a callback function for to be called in the assembly step. + */ virtual bool registerCallback( void * func, const std::type_info & funcType ) final override; + /** + * @brief accessor for the solver statistics. + * @return reference to m_solverStatistics + */ SolverStatistics & getSolverStatistics() { return m_solverStatistics; } + + /** + * @brief const accessor for the solver statistics. + * @return reference to m_solverStatistics + */ SolverStatistics const & getSolverStatistics() const { return m_solverStatistics; } +#if defined(GEOS_USE_PYGEOSX) /** * @brief Return PySolver type. * @return Return PySolver type. */ -#if defined(GEOS_USE_PYGEOSX) virtual PyTypeObject * getPythonType() const override; #endif + /** + * @brief accessor for m_meshTargets + * @return reference to m_meshTargets + */ map< std::pair< string, string >, array1d< string > > const & getMeshTargets() const { return m_meshTargets; } protected: + /** + * @brief Eisenstat-Walker adaptive tolerance + * + * This method enables an inexact-Newton method is which the linear solver + * tolerance is chosen based on the nonlinear solver convergence behavior. + * In early Newton iterations, the search direction is usually imprecise, and + * therefore a weak linear convergence tolerance can be chosen to minimize + * computational cost. As the search gets closer to the true solution, however, + * more stringent linear tolerances are necessary to maintain quadratic convergence + * behavior. + * + * The user can set the weakest tolerance allowed, with a default of 1e-3. + * Even weaker values (e.g. 1e-2,1e-1) can be used for further speedup, but may + * occasionally cause convergence problems. Use this parameter with caution. The + * most stringent tolerance is hardcoded to 1e-8, which is sufficient for + * most problems. + * + * See Eisenstat, S.C. and Walker, H.F., 1996. Choosing the forcing terms in an + * inexact Newton method. SIAM Journal on Scientific Computing, 17(1), pp.16-32. + * + * @param newNewtonNorm Residual norm at current iteration + * @param oldNewtonNorm Residual norm at previous iteration + * @param weakestTol Weakest tolerance allowed (default 1e-3). + * @return Adaptive tolerance recommendation + */ static real64 eisenstatWalker( real64 const newNewtonNorm, real64 const oldNewtonNorm, real64 const weakestTol ); @@ -759,12 +942,19 @@ class SolverBase : public ExecutableGroup * @brief Get the Constitutive Name object * * @tparam CONSTITUTIVE_BASE_TYPE the base type of the constitutive model. - * @param subregion the element subregion on which the constitutive model is registered - * @return the name name of the constitutive model of type @p CONSTITUTIVE_BASE_TYPE registered on the @p subregion. + * @param subRegion the element subregion on which the constitutive model is registered + * @return the name name of the constitutive model of type CONSTITUTIVE_BASE_TYPE registered on the subregion. */ template< typename CONSTITUTIVE_BASE_TYPE > static string getConstitutiveName( ElementSubRegionBase const & subRegion ); + /** + * @brief Get the Constitutive Name object + * + * @tparam CONSTITUTIVE_BASE_TYPE the base type of the constitutive model. + * @param subRegion the particle subregion on which the constitutive model is registered + * @return the name name of the constitutive model of type CONSTITUTIVE_BASE_TYPE registered on the subregion. + */ template< typename CONSTITUTIVE_BASE_TYPE > static string getConstitutiveName( ParticleSubRegionBase const & subRegion ); // particle overload @@ -776,14 +966,46 @@ class SolverBase : public ExecutableGroup */ virtual void setConstitutiveNamesCallSuper( ElementSubRegionBase & subRegion ) const { GEOS_UNUSED_VAR( subRegion ); } + + /** + * @brief Get the Constitutive Model object + * @tparam BASETYPE the base type of the constitutive model. + * @tparam LOOKUP_TYPE the type of the key used to look up the constitutive model. + * @param dataGroup the data group containing the constitutive models. + * @param key the key used to look up the constitutive model. + * @return the constitutive model of type @p BASETYPE registered on the @p dataGroup with the key @p key. + */ template< typename BASETYPE = constitutive::ConstitutiveBase, typename LOOKUP_TYPE > - static BASETYPE const & getConstitutiveModel( dataRepository::Group const & dataGroup, LOOKUP_TYPE const & key ); + static BASETYPE const & getConstitutiveModel( dataRepository::Group const & dataGroup, LOOKUP_TYPE const & key ) + { + dataRepository::Group const & constitutiveModels = dataGroup.getGroup( ElementSubRegionBase::groupKeyStruct::constitutiveModelsString() ); + return constitutiveModels.getGroup< BASETYPE >( key ); + } + /** + * @brief Get the Constitutive Model object + * @tparam BASETYPE the base type of the constitutive model. + * @tparam LOOKUP_TYPE the type of the key used to look up the constitutive model. + * @param dataGroup the data group containing the constitutive models. + * @param key the key used to look up the constitutive model. + * @return the constitutive model of type @p BASETYPE registered on the @p dataGroup with the key @p key. + */ template< typename BASETYPE = constitutive::ConstitutiveBase, typename LOOKUP_TYPE > - static BASETYPE & getConstitutiveModel( dataRepository::Group & dataGroup, LOOKUP_TYPE const & key ); + static BASETYPE & getConstitutiveModel( dataRepository::Group & dataGroup, LOOKUP_TYPE const & key ) + { + dataRepository::Group & constitutiveModels = dataGroup.getGroup( ElementSubRegionBase::groupKeyStruct::constitutiveModelsString() ); + return constitutiveModels.getGroup< BASETYPE >( key ); + } + + + /// Courant–Friedrichs–Lewy factor for the timestep real64 m_cflFactor; + + /// maximum stable time step real64 m_maxStableDt; + + /// timestep of the next cycle real64 m_nextDt; /// name of the FV discretization object in the data repository @@ -792,9 +1014,13 @@ class SolverBase : public ExecutableGroup /// Data structure to handle degrees of freedom DofManager m_dofManager; - /// System matrix, rhs and solution + /// System matrix ParallelMatrix m_matrix; + + /// System right-hand side vector ParallelVector m_rhs; + + /// System solution vector ParallelVector m_solution; /// Local system matrix and rhs @@ -821,8 +1047,10 @@ class SolverBase : public ExecutableGroup /// Timestamp of the last call to setup system Timestamp m_systemSetupTimestamp; + /// Callback function for assembly step std::function< void( CRSMatrix< real64, globalIndex >, array1d< real64 > ) > m_assemblyCallback; + /// Timers for the aggregate profiling of the solver std::map< std::string, std::chrono::system_clock::duration > m_timers; private: @@ -840,11 +1068,25 @@ class SolverBase : public ExecutableGroup */ virtual void setConstitutiveNames( ElementSubRegionBase & subRegion ) const { GEOS_UNUSED_VAR( subRegion ); } + /** + * @brief Solve a nonlinear system using a Newton method + * @param time_n the time at the beginning of the step + * @param dt the desired timestep + * @param cycleNumber the current cycle number + * @param domain the domain partition + * @return true if the nonlinear system was solved, false otherwise + */ bool solveNonlinearSystem( real64 const & time_n, real64 const & dt, integer const cycleNumber, DomainPartition & domain ); + /** + * @brief output information about the cycle to the log + * @param cycleNumber the current cycle number + * @param numOfSubSteps the number of substeps taken + * @param subStepDt the time step size for each substep + */ void logEndOfCycleInformation( integer const cycleNumber, integer const numOfSubSteps, std::vector< real64 > const & subStepDt ) const; @@ -879,21 +1121,6 @@ string SolverBase::getConstitutiveName( ParticleSubRegionBase const & subRegion return validName; } -template< typename BASETYPE, typename LOOKUP_TYPE > -BASETYPE const & SolverBase::getConstitutiveModel( dataRepository::Group const & dataGroup, LOOKUP_TYPE const & key ) -{ - dataRepository::Group const & constitutiveModels = dataGroup.getGroup( ElementSubRegionBase::groupKeyStruct::constitutiveModelsString() ); - - return constitutiveModels.getGroup< BASETYPE >( key ); -} - -template< typename BASETYPE, typename LOOKUP_TYPE > -BASETYPE & SolverBase::getConstitutiveModel( dataRepository::Group & dataGroup, LOOKUP_TYPE const & key ) -{ - Group & constitutiveModels = dataGroup.getGroup( ElementSubRegionBase::groupKeyStruct::constitutiveModelsString() ); - - return constitutiveModels.getGroup< BASETYPE >( key ); -} } // namespace geos diff --git a/src/docs/doxygen/Doxyfile.in b/src/docs/doxygen/Doxyfile.in index f872f18f60c..7038c667890 100644 --- a/src/docs/doxygen/Doxyfile.in +++ b/src/docs/doxygen/Doxyfile.in @@ -32,7 +32,7 @@ DOXYFILE_ENCODING = UTF-8 # title of most generated pages and in a few other places. # The default value is: My Project. -PROJECT_NAME = "GEOSX" +PROJECT_NAME = "GEOS" # The PROJECT_NUMBER tag can be used to enter a project or revision number. This # could be handy for archiving the generated documentation or if some version @@ -767,6 +767,7 @@ INPUT = @PROJECT_SOURCE_DIR@/coreComponents/common \ @PROJECT_SOURCE_DIR@/coreComponents/finiteElement/elementFormulations \ @PROJECT_SOURCE_DIR@/coreComponents/finiteElement/kernelInterface \ @PROJECT_SOURCE_DIR@/coreComponents/mesh/MeshFields.hpp \ + @PROJECT_SOURCE_DIR@/coreComponents/physicsSolvers/SolverBase.hpp \ @PROJECT_SOURCE_DIR@/coreComponents/physicsSolvers/simplePDE/LaplaceFEMKernels.hpp \ @PROJECT_SOURCE_DIR@/coreComponents/physicsSolvers/solidMechanics/kernels/ExplicitFiniteStrain.hpp \ @PROJECT_SOURCE_DIR@/coreComponents/physicsSolvers/solidMechanics/kernels/ExplicitSmallStrain.hpp \ diff --git a/src/docs/sphinx/Doxygen.rst b/src/docs/sphinx/Doxygen.rst index 6e541b988e0..217411e6083 100644 --- a/src/docs/sphinx/Doxygen.rst +++ b/src/docs/sphinx/Doxygen.rst @@ -2,11 +2,17 @@ Doxygen ############################################################################### -The c++ source in GEOS is annotated using doxygen. Our doxygen pages are -linked below. +The GEOS c++ API in is documented using doxygen. +The doxygen `class list pages <../../doxygen_output/html/classes.html>`_ -.. raw:: html +Developers may find it helpful to review the :ref:`KeyComponents` described in the Developer Guide before diving into the doxygen. - GEOS Doxygen Pages +Some key doxygen pages are linked below: -Developers may find it helpful to review the key code components described in the Developer Guide before diving into the doxygen. +`Group API <../../doxygen_output/html/classgeos_1_1data_repository_1_1_group.html>`_ + +`Wrapper API <../../doxygen_output/html/classgeos_1_1data_repository_1_1_wrapper.html>`_ + +`ObjectManagerBase API <../../doxygen_output/html/classgeos_1_1_object_manager_base.html>`_ + +`SolverBase API <../../doxygen_output/html/classgeos_1_1_solver_base.html>`_ \ No newline at end of file diff --git a/src/docs/sphinx/QuickStart.rst b/src/docs/sphinx/QuickStart.rst index 71b94c08e02..0ad5e2acab9 100644 --- a/src/docs/sphinx/QuickStart.rst +++ b/src/docs/sphinx/QuickStart.rst @@ -164,8 +164,6 @@ First, using a terminal, create the ``codes`` directory wherever you like. Inside this directory, we can clone the GEOS repository. We will also use some Git commands to initialize and download the submodules (e.g. ``LvArray``). -Note that most users will not have access to our integrated tests repository, and so we "deinit" (deactivate) this submodule. -Developers who will be working with the integratedTests repository should skip this line. .. code-block:: sh @@ -173,7 +171,6 @@ Developers who will be working with the integratedTests repository should skip t cd GEOS git lfs install git submodule init - git submodule deinit integratedTests git submodule update cd .. @@ -181,10 +178,10 @@ If all goes well, you should have a complete copy of the GEOS source at this poi The most common errors people encounter here have to do with Github not recognizing their authentication settings and/or repository permissions. See the previous section for tips on ensuring your SSH is working properly. -*Note*: The integratedTests submodule is not publicly available, with access limited to the core development team. -This may cause the ``git submodule update`` command to fail -if you forget the ``git submodule deinit integratedTests`` step above. -This submodule is not required for building GEOS. If you see an error message here, however, you may need to initialize and update the submodules manually: +*Note*: Previous versions of GEOS also imported the integratedTests submodule, which is not publicly available (access is limited to the core development team). +This may cause the ``git submodule update`` command to fail. +In that case, run ``git submodule deinit integratedTests`` before ``git submodule update``. +This submodule is not required for building GEOS. .. code-block:: sh @@ -218,11 +215,10 @@ If you are using an older version, you may need to add ``git lfs pull`` after `` The clone ``https://github.com/GEOS-DEV/GEOS.git`` becomes ``git clone git@github.com:GEOS-DEV/GEOS.git``. You may also be willing to insert your credentials in the command line (less secure) ``git clone https://${USER}:${TOKEN}@github.com/GEOS-DEV/GEOS.git``. -Configuration -================ +Configuration +============= -At a minimum, you will need a relatively recent compiler suite installed on your system (e.g. `GCC `_, `Clang `_) as well as `CMake `_. -If you want to run jobs using MPI-based parallelism, you will also need an MPI implementation (e.g. `OpenMPI `_, `MVAPICH `_). +Before proceeding, make sure to have installed all the minimal prerequisites as described in :ref:`Prerequisites` Note that GEOS supports a variety of parallel computing models, depending on the hardware and software environment. Advanced users are referred to the :ref:`BuildGuide` for a discussion of the available configuration options. @@ -237,120 +233,107 @@ If something goes wrong, the first thing the support team will ask you for is th Here, you may need to replace ``cpp`` with the full path to the C++ compiler you would like to use, depending on how your path and any aliases are configured. -GEOS compilations are driven by a cmake ``host-config`` file, which tells the build system about the compilers you are using, where various packages reside, and what options you want to enable. -We have created a number of default hostconfig files for common systems. -You should browse them to see if any are close to your needs: +Defining a Host-Config File +--------------------------- -.. code-block:: sh +GEOS compilations are driven by a CMake ``host-config`` file, which informs the build system about the compilers you are using, where various packages reside, and what options you want to enable. - cd GEOS/host-configs +A template for creating a simple ``host-config`` is provided in ``host-configs/quick-start-template.cmake``. -We maintain host configs (ending in ``.cmake``) for HPC systems at various institutions, as well as ones for common personal systems. -If you cannot find one that matches your needs, we suggest beginning with one of the shorter ones and modifying as needed. -A typical one may look like: +.. literalinclude:: ../../../host-configs/quick-start-template.cmake + :language: sh -.. code-block:: sh +The various ``set()`` commands are used to set variables that control the build. To begin, make a copy of the template file and modify the paths according to the installation locations on your system. - # file: your-platform.cmake - - # detect host and name the configuration file - site_name(HOST_NAME) - set(CONFIG_NAME "your-platform" CACHE PATH "") - message("CONFIG_NAME = ${CONFIG_NAME}") - - # set paths to C, C++, and Fortran compilers. Note that while GEOS does not contain any Fortran code, - # some of the third-party libraries do contain Fortran code. Thus a Fortran compiler must be specified. - set(CMAKE_C_COMPILER "/usr/bin/clang" CACHE PATH "") - set(CMAKE_CXX_COMPILER "/usr/bin/clang++" CACHE PATH "") - set(CMAKE_Fortran_COMPILER "/usr/local/bin/gfortran" CACHE PATH "") - set(ENABLE_FORTRAN OFF CACHE BOOL "" FORCE) - - # enable MPI and set paths to compilers and executable. - # Note that the MPI compilers are wrappers around standard serial compilers. - # Therefore, the MPI compilers must wrap the appropriate serial compilers specified - # in CMAKE_C_COMPILER, CMAKE_CXX_COMPILER, and CMAKE_Fortran_COMPILER. - set(ENABLE_MPI ON CACHE BOOL "") - set(MPI_C_COMPILER "/usr/local/bin/mpicc" CACHE PATH "") - set(MPI_CXX_COMPILER "/usr/local/bin/mpicxx" CACHE PATH "") - set(MPI_Fortran_COMPILER "/usr/local/bin/mpifort" CACHE PATH "") - set(MPIEXEC "/usr/local/bin/mpirun" CACHE PATH "") - - # disable CUDA and OpenMP - set(ENABLE_CUDA OFF CACHE BOOL "" FORCE) - set(ENABLE_OPENMP OFF CACHE BOOL "" FORCE) - - # enable PVTPackage - set(ENABLE_PVTPackage ON CACHE BOOL "" FORCE) - - # enable tests - set(ENABLE_GTEST_DEATH_TESTS ON CACHE BOOL "" FORCE ) - - # define the path to your compiled installation directory - set(GEOS_TPL_DIR "/path/to/your/TPL/installation/dir" CACHE PATH "") - # let GEOS define some third party libraries information for you - include(${CMAKE_CURRENT_LIST_DIR}/tpls.cmake) - -The various ``set()`` commands are used to set environment variables that control the build. -You will see in the above example that we set the C++ compiler to ``/user/bin/clang++`` and so forth. -We also disable CUDA and OpenMP, but enable PVTPackage. -The final line is related to our unit test suite. See the :ref:`BuildGuide` for more details on available options. +We have created a number of default host-config files for common systems. You should browse them to see if any are close to your needs: +We maintain host configuration files (ending in ``.cmake``) for HPC systems at various institutions, as well as for common personal systems. +If you cannot find one that matches your needs, we suggest starting with one of the shorter ones and modifying it as needed. .. note:: If you develop a new ``host-config`` for a particular platform that may be useful for other users, please consider sharing it with the developer team. Compilation -================== +=========== + +The configuration process for both the third-party libraries (TPLs) and GEOS is managed through a Python script called ``config-build.py``. This script simplifies and automates the setup by configuring the build and install directories and by running CMake based on the options set in the host-config file +which is passed as a command-lne argument. The ``config-build.py`` script has several command-line options. Here, we will only use some basic options and rely on default values for many others. During this build process there wil be automatically generated build and install directories for both the TPLs and the main code, +with names consistent with the name specified in the host-config by the variable ``CONFIG_NAME``, i.e. ``build-your-platform-release`` and ``install-your-platform-release``. + +All options can be visualized by running + +.. code-block:: sh + + cd thirdPartyLibs + python scripts/config-build.py -h + +.. note:: + + It is strongly recommended that GEOS and TPLs be configured using the same host configuration file. Below, we assume that you keep this file in, for example, ``GEOS/host-configs/your-platform.cmake``, but the exact location is up to you. -We will begin by compiling the TPLs, followed by the main code. -If you work on an HPC system with other GEOS developers, check with them to see if the TPLs have already been compiled in a shared directory. -If this is the case, you can skip ahead to just compiling the main code. -If you are working on your own machine, you will need to compile both. +Compiling the TPLs +------------------- -We strongly suggest that GEOS and TPLs be built with the same hostconfig file. -Below, we assume that you keep it in, say, ``GEOS/host-configs/your-platform.cmake``, but this is up to you. +.. note:: -We begin with the third-party libraries, and use a python ``config-build.py`` script to configure and build all of the TPLs. -Note that we will request a Release build type, which will enable various optimizations. -The other option is a Debug build, which allows for debugging but will be much slower in production mode. -The TPLS will then be built in a build directory named consistently with your hostconfig file. + If you are working on an HPC system with other GEOS developers, check with them to see if the TPLs have already been compiled in a shared directory. If this is the case, you can skip ahead to just compiling the main code. + If you are working on your own machine, you will need to configure and compile both the TPLs and the main code. + +We begin by configuring the third-party libraries (TPLs) using the ``config-build.py`` script. This script sets up the build directory and runs CMake to generate the necessary build files. .. code-block:: sh cd thirdPartyLibs python scripts/config-build.py -hc ../GEOS/host-configs/your-platform.cmake -bt Release + +The TPLs will be configured in a build directory named consistently with your host configuration file, i.e., ``build-your-platform-release``. + +.. code-block:: sh + cd build-your-platform-release make -Note that building all of the TPLs can take quite a while, so you may want to go get a cup of coffee at this point. -Also note that you should *not* use a parallel ``make -j N`` command to try and speed up the build time. +.. note:: + + Building all of the TPLs can take quite a while, so you may want to go get a cup of coffee at this point. + Also note that you should *not* use a parallel ``make -j N`` command to try and speed up the build time. + +Compiling GEOS +------------------- + +Once the TPLs have been compiler, the next step is to compile the main code. The ``config-build.py`` script is used to configure the build directory. Before running the configuration script, ensure that the path to the TPLs is correctly set in the host configuration file by setting + +.. code-block:: sh + set(GEOS_TPL_DIR "/path/to/your/TPL/installation/dir" CACHE PATH "") -The next step is to compile the main code. -Again, the ``config-build.py`` sets up cmake for you, so the process is very similar. +If you have followed these instructions, the TPLs are installed at the default location, i.e. ``/path/to/your/TPL/thirdPartyLibs/install-your-platform-release``. .. code-block:: sh cd ../../GEOS python scripts/config-build.py -hc host-configs/your-platform.cmake -bt Release - cd build-your-platform-release - make -j4 - make install -The host-config file is the place to set all relevant configuration options. -Note that the path to the previously installed third party libraries is typically specified within this file. An alternative is to set the path ``GEOS_TPL_DIR`` via a cmake command line option, e.g. - .. code-block:: sh python scripts/config-build.py -hc host-configs/your-platform.cmake -bt Release -D GEOS_TPL_DIR=/full/path/to/thirdPartyLibs -We highly recommend using full paths, rather than relative paths, whenever possible. +.. note:: + + We highly recommend using full paths, rather than relative paths, whenever possible. + +Once the configuration process is completed, we proceed with the compilation of the main code and the instalation of geos. + +.. code-block:: sh + + cd build-your-platform-release + make -j4 + make install + The parallel ``make -j 4`` will use four processes for compilation, which can substantially speed up the build if you have a multi-processor machine. You can adjust this value to match the number of processors available on your machine. The ``make install`` command then installs GEOS to a default location unless otherwise specified. - - -If all goes well, a ``geosx`` executable should now be available: +If all goes well, a ``geosx`` executable should now be available .. code-block:: sh diff --git a/src/docs/sphinx/advancedExamples/performanceBenchmarks/Index.rst b/src/docs/sphinx/advancedExamples/performanceBenchmarks/Index.rst index ed9671bbc64..bbca61572c6 100644 --- a/src/docs/sphinx/advancedExamples/performanceBenchmarks/Index.rst +++ b/src/docs/sphinx/advancedExamples/performanceBenchmarks/Index.rst @@ -4,4 +4,155 @@ Performance Benchmarks ######################## +==================================== +Wellbore problem scaling on Frontier +==================================== +.. _wellboreProblemScaling: +The wellbore problem described here was used to test the weak scaling of GEOS on the +`Frontier supercomputer ` at Oak Ridge National +Laboratory. The hardware details of the Frontier system are described in the `Frontier +User Guide `_. + +The wellbore problem consists of a single wellbore with an internal radius of 0.057m, a +casing thickness of 0.006m, with varying length and number elements depending level in the +scaling study. There are 6 levels of scaling summarized in the table below. The number of +ranks used for each level is dependent on the physics solver applied, as the number of +degrees of freedom in the problem changes. + ++-------+------+------+-------+----------------+ +| Level | nr | nt | nz | nelem | ++=======+======+======+=======+================+ +| 1 | 33 | 64 | 391 | 825,792 | ++-------+------+------+-------+----------------+ +| 2 | 33 | 64 | 3125 | 6,600,000 | ++-------+------+------+-------+----------------+ +| 3 | 66 | 128 | 6250 | 52,800,000 | ++-------+------+------+-------+----------------+ +| 4 | 132 | 256 | 12500 | 422,400,000 | ++-------+------+------+-------+----------------+ +| 5 | 264 | 512 | 25000 | 3,379,200,000 | ++-------+------+------+-------+----------------+ +| 6 | 528 | 1024 | 50000 | 27,033,600,000 | ++-------+------+------+-------+----------------+ + +An image of the wellbore geometry is give below: + +.. .. figure:: /coreComponents/physics/docs/Wellbore.png +.. :align: center +.. :width: 500 +.. :figclass: align-center + + +The scaling results for the wellbore problem applied to different physics models on the +Frontier supercomputer are shown below: + +Mechanics +--------- + +.. figure:: weakscal_frontier_mechanics.png + :align: center + :width: 500 + :figclass: align-center + + Weak scaling results for the mechanics model on Frontier. + +Single Phase Flow +----------------- + +.. figure:: weakscal_frontier_singlePhaseFlow.png + :align: center + :width: 500 + :figclass: align-center + + Weak scaling results for the single phase flow model on Frontier. + +Compositional Multiphase Flow +----------------------------- + +.. figure:: weakscal_frontier_compositionalMultiphaseFlow.png + :align: center + :width: 500 + :figclass: align-center + + Weak scaling results for the compositional multiphase flow model on Frontier. + +.. _wellboreRunning: +Running the Performance Study +----------------------------- + +To execute a performance study across different GPU configurations and problem levels for +the wellbore problem on the Frontier supercomputer, follow these steps: + +Prerequisites +~~~~~~~~~~~~~ +Ensure you have access to the Frontier system and that you have access to a valid job +allocation account. For detailed instructions on system access and environment setup, +refer to the `Frontier User Guide +`_. + +Directory Structure +~~~~~~~~~~~~~~~~~~~ +The input files for different problem levels and configurations are organized under +``${GEOS_DIR}/inputFiles/wellboreECP/``. Each physical problem (e.g., +``mechanics``, ``compositionalMultiphaseFlow``, ``singlePhaseFlow``) has its own directory +containing multiple levels of problem refinement as described in the table above. + +Dispatching Jobs +~~~~~~~~~~~~~~~~ +Use the ``dispatch.py`` script to automate the setup and submission of jobs for different +levels of problem refinement and physics models. + +**Usage:** + +.. code-block:: bash + + python3 dispatch.py --acount [account_id] --binary [path_to_binary] --model [model_type] --levels + [specific_levels_to_run] --caliper [def] + +- **--account**: Account name for running jobs on compute nodes. +- **--binary**: Path to the geos executable. +- **--model**: Physics model type, such as ``mechanics``, ``singlePhaseFlow`` or ``compositionalMultiphaseFlow``. +- **--levels**: (optional) Specific levels to run, e.g., ``1 3 5``. If omitted, all levels + are run. +- **--caliper**: (optional) Turn on profiling with `caliper `_. + +**Example:** + +.. code-block:: bash + + python3 dispatch.py --binary /path/to/geos --model mechanics --levels 1 2 3 + +This command launches jobs for levels 1, 2, and 3 under the mechanics problem configuration. + +Analyzing Output +~~~~~~~~~~~~~~~~ +After job completion, utilize the ``postprocess.py`` script to extract and plot performance metrics from the output files. + +**Usage:** + +.. code-block:: bash + + python3 postprocess.py --model [model_type] --levels [specific_levels_to_plot] + +- **--model**: Physics model type, such as ``mechanics``, ``singlePhaseFlow`` or ``compositionalMultiphaseFlow``. +- **--levels**: (optional) Specific levels to plot, e.g., ``2 3 4 5``. If omitted, results + for all available levels are plotted. + +**Example:** + +.. code-block:: bash + + python3 postprocess.py --model mechanics --levels 2 3 4 5 + +This command parses the latest output files in the mechanics directory, selected by the +highest ``[jobID]``. It matches files with the pattern +``[machine_name]-[jobID]-[model_type]-geom[level].out``, extracting average +execution times per non-linear step for the following phases: GEOS, matrix creation, Hypre +setup, and Hypre solve. + +Note +~~~~ +The job scripts designed for Frontier are likely compatible with other Slurm-based +systems, making them reusable across different high-performance computing environments +with minimal adjustments. diff --git a/src/docs/sphinx/advancedExamples/performanceBenchmarks/weakscal_frontier_compositionalMultiphaseFlow.png b/src/docs/sphinx/advancedExamples/performanceBenchmarks/weakscal_frontier_compositionalMultiphaseFlow.png new file mode 100644 index 00000000000..ae2442bcf5c Binary files /dev/null and b/src/docs/sphinx/advancedExamples/performanceBenchmarks/weakscal_frontier_compositionalMultiphaseFlow.png differ diff --git a/src/docs/sphinx/advancedExamples/performanceBenchmarks/weakscal_frontier_mechanics.png b/src/docs/sphinx/advancedExamples/performanceBenchmarks/weakscal_frontier_mechanics.png new file mode 100644 index 00000000000..b797b43a440 Binary files /dev/null and b/src/docs/sphinx/advancedExamples/performanceBenchmarks/weakscal_frontier_mechanics.png differ diff --git a/src/docs/sphinx/advancedExamples/performanceBenchmarks/weakscal_frontier_singlePhaseFlow.png b/src/docs/sphinx/advancedExamples/performanceBenchmarks/weakscal_frontier_singlePhaseFlow.png new file mode 100644 index 00000000000..7944cbba173 Binary files /dev/null and b/src/docs/sphinx/advancedExamples/performanceBenchmarks/weakscal_frontier_singlePhaseFlow.png differ diff --git a/src/docs/sphinx/basicExamples/co2Injection/Example.rst b/src/docs/sphinx/basicExamples/co2Injection/Example.rst index 3076ae46b68..63f0c4f0add 100644 --- a/src/docs/sphinx/basicExamples/co2Injection/Example.rst +++ b/src/docs/sphinx/basicExamples/co2Injection/Example.rst @@ -264,13 +264,13 @@ The simulation can be launched with 4 cores using MPI-parallelism: .. code-block:: console - mpirun -np 4 geosx -i simpleCo2InjTutorial.xml -x 1 -y 1 -z 4 + mpirun -np 4 geosx -i simpleCo2InjTutorial_smoke.xml -x 1 -y 1 -z 4 A restart from a checkpoint file `simpleCo2InjTutorial_restart_000000024.root` is always available thanks to the following command line : .. code-block:: console - mpirun -np 4 geosx -i simpleCo2InjTutorial.xml -r simpleCo2InjTutorial_restart_000000024 -x 1 -y 1 -z 4 + mpirun -np 4 geosx -i simpleCo2InjTutorial_smoke.xml -r simpleCo2InjTutorial_restart_000000024 -x 1 -y 1 -z 4 The output then shows the loading of HDF5 restart files by each core. diff --git a/src/docs/sphinx/buildGuide/AppleMacOS.rst b/src/docs/sphinx/buildGuide/AppleMacOS.rst new file mode 100644 index 00000000000..7b8085035b9 --- /dev/null +++ b/src/docs/sphinx/buildGuide/AppleMacOS.rst @@ -0,0 +1,93 @@ +.. _AppleMacOS: + +Building Apple MacOS +============== + +Install homebrew +---------------- +Taken from the [homebrew website](https://brew.sh) +.. code-block:: + + /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" + (echo; echo 'eval "$(/opt/homebrew/bin/brew shellenv)"') >> ~/.zprofile + + note: this is the command for `zsh`. Other shells will require different commands. Homebrew should provide the correct command after install is complete. + eval "$(/opt/homebrew/bin/brew shellenv)" + +Install packages using homebrew +------------------------------- + +.. code-block:: + + brew install bison cmake gfortran git-lfs open-mpi lapack python3 ninja m4 + echo 'export PATH="/opt/homebrew/opt/bison/bin:$PATH"' >> ~/.zshrc + echo 'export PATH="/opt/homebrew/opt/m4/bin:$PATH"' >> ~/.zshrc + git lfs install + +Clone GEOS +---------- + +.. code-block:: + + git clone git@github.com:GEOS-DEV/GEOS.git + cd GEOS + git submodule init + git submodule update + cd .. + +Clone thirdPartyLibs +-------------------- + +.. code-block:: + + git clone git@github.com:GEOS-DEV/thirdPartyLibs.git + cd thirdPartyLibs + git submodule init + git submodule update + git lfs pull + + +Configure and build thirdPartyLibs +--------------------------------- + +.. code-block:: + + python3 scripts/config-build.py -hc ../GEOS/host-configs/apple/macOS_arm.cmake -bt Release + +You will get a warning you can ignore + +.. code-block:: + + CMake Warning at /Users/settgast1/Codes/geos/GEOS/host-configs/tpls.cmake:10 (message): + 'GEOS_TPL_DIR' does not exist. + + +Continue with the build + +.. code-block:: + + cd build-macOS_arm-release + make + +You will get an error at the end...you can ignore it. + +.. code-block:: + + [100%] Linking CXX executable ../../../tests/blt_mpi_smoke + ld: warning: -commons use_dylibs is no longer supported, using error treatment instead + ld: file not found: @rpath/libquadmath.0.dylib for architecture arm64 + clang: error: linker command failed with exit code 1 (use -v to see invocation) + make[2]: *** [tests/blt_mpi_smoke] Error 1 + make[1]: *** [blt/tests/smoke/CMakeFiles/blt_mpi_smoke.dir/all] Error 2 + make: *** [all] Error 2 + + +Build GEOS +---------- + +.. code-block:: + + cd ../../GEOS + python3 scripts/config-build.py -hc host-configs/apple/macOS_arm.cmake -bt Release --ninja + cd build-macOS_arm-release + ninja geosx diff --git a/src/docs/sphinx/buildGuide/Index.rst b/src/docs/sphinx/buildGuide/Index.rst index afd042d9561..1ed27e84f1f 100644 --- a/src/docs/sphinx/buildGuide/Index.rst +++ b/src/docs/sphinx/buildGuide/Index.rst @@ -18,3 +18,5 @@ Welcome to the GEOS build guide. SpackUberenv ContinuousIntegration + + AppleMacOS diff --git a/src/docs/sphinx/buildGuide/Prerequisites.rst b/src/docs/sphinx/buildGuide/Prerequisites.rst index 78251f997d5..9bbe5ee6174 100644 --- a/src/docs/sphinx/buildGuide/Prerequisites.rst +++ b/src/docs/sphinx/buildGuide/Prerequisites.rst @@ -10,10 +10,10 @@ List of prerequisites Minimal requirements: -- `CMake `_ build system generator (3.17+). +- `CMake `_ build system generator (3.23.1+). - build tools (`GNU make `_ or `ninja `_ on Linux, XCode on MacOS). -- a C++ compiler with full c++17 standard support (`gcc `_ 8.3+ or `clang `_ 10.0+ are recommended). -- `python `_ (2.7+ or 3.6+). +- a C++ compiler with full c++17 standard support (`gcc `_ 12+ or `clang `_ 13.0+ are recommended). +- `python `_ 3.9-3.11 (versions 3.12+ are untested). - :code:`zlib`, :code:`blas` and :code:`lapack` libraries - any compatible MPI runtime and compilers (if building with MPI) diff --git a/src/docs/sphinx/developerGuide/Contributing/Doxygen.rst b/src/docs/sphinx/developerGuide/Contributing/Doxygen.rst index 8a25b96ee0c..1b240c99811 100644 --- a/src/docs/sphinx/developerGuide/Contributing/Doxygen.rst +++ b/src/docs/sphinx/developerGuide/Contributing/Doxygen.rst @@ -239,4 +239,18 @@ Example Current Doxygen ==================================== -`Link to Doxygen <../../../doxygen_output/html/classes.html>`__ +`Link to Doxygen Class directory <../../../../doxygen_output/html/classes.html>`__ + +Direct links to some useful class documentation: + +`Group API <../../../../doxygen_output/html/classgeos_1_1data_repository_1_1_group.html>`_ + +`Wrapper API <../../../../doxygen_output/html/classgeos_1_1data_repository_1_1_wrapper.html>`_ + +`ObjectManagerBase API <../../../../doxygen_output/html/classgeos_1_1_object_manager_base.html>`_ + +`MeshLevel API <../../../../doxygen_output/html/classgeos_1_1_mesh_level.html>`_ + +`NodeManager API <../../../../doxygen_output/html/classgeos_1_1_node_manager.html>`_ + +`FaceManager API <../../../../doxygen_output/html/classgeos_1_1_face_manager.html>`_ \ No newline at end of file diff --git a/src/docs/sphinx/developerGuide/Contributing/index_contributing.rst b/src/docs/sphinx/developerGuide/Contributing/index_contributing.rst index dd03fc486a3..3e34bdc70c8 100644 --- a/src/docs/sphinx/developerGuide/Contributing/index_contributing.rst +++ b/src/docs/sphinx/developerGuide/Contributing/index_contributing.rst @@ -1,3 +1,4 @@ +.. _Contributing: ############################################################################### Contributing ############################################################################### diff --git a/src/docs/sphinx/developerGuide/Index.rst b/src/docs/sphinx/developerGuide/Index.rst index 7e06c68461d..9ecf43b846d 100644 --- a/src/docs/sphinx/developerGuide/Index.rst +++ b/src/docs/sphinx/developerGuide/Index.rst @@ -2,7 +2,10 @@ Developer Guide ############################################################################### -Welcome to the GEOS developer guide. +Welcome to the GEOS developer guide. +This guide serves as a reference for developers contributing to the GEOS project. +The :ref:`Contributing` section provides general details and guidelines for contributing to the GEOS project. +The :ref:`KeyComponents` section provides an overview of the some basic components that of the GEOS project. .. toctree:: :maxdepth: 3 diff --git a/src/docs/sphinx/developerGuide/KeyComponents/index_KeyComponents.rst b/src/docs/sphinx/developerGuide/KeyComponents/index_KeyComponents.rst index 6960fe15c24..68899719971 100644 --- a/src/docs/sphinx/developerGuide/KeyComponents/index_KeyComponents.rst +++ b/src/docs/sphinx/developerGuide/KeyComponents/index_KeyComponents.rst @@ -1,3 +1,5 @@ +.. _KeyComponents: + ############################################################################### Code Components ############################################################################### diff --git a/src/index.rst b/src/index.rst index 9dedc1f4d75..46113205346 100644 --- a/src/index.rst +++ b/src/index.rst @@ -126,6 +126,22 @@ you have suggestions for improving the guides below, please post an issue on our To the Python Tools Documentation + .. grid-item-card:: + + Feature Requests, Reporting Bugs, and Support + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + To make feature requests, report bugs, or get support (after reviewing the user guide) please submit an issue on Github. + + +++ + + .. button-link:: https://github.com/GEOS-DEV/GEOS/issues/new/choose + :expand: + :color: info + :click-parent: + + To the "New issue" page on the GEOS Github repository + ******************** Table of Contents