Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: update trilinos, hypre and fmt. #3214

Merged
merged 25 commits into from
Jul 11, 2024
Merged

chore: update trilinos, hypre and fmt. #3214

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
cfd04a5
udpate raja and chai.
CusiniM Jul 5, 2024
e437040
udpate raja and chai.
CusiniM Jul 5, 2024
f6086d7
udpate trilinos. New path for trilinos CmakeConfig.
CusiniM Jul 8, 2024
1f94377
Merge remote-tracking branch 'origin/develop' into tpls-update/cusini…
CusiniM Jul 8, 2024
bc7cf31
add new images and remove pecan and centos.
CusiniM Jul 8, 2024
a26e01a
Merge branch 'develop' into tpls-update/cusini/update-raja-chai
CusiniM Jul 8, 2024
37e754f
MGR improvements (#3199)
victorapm Jul 9, 2024
78c90c9
Merge branch 'develop' into tpls-update/cusini/update-raja-chai
CusiniM Jul 9, 2024
c493ee7
Update .devcontainer/devcontainer.json
CusiniM Jul 9, 2024
4bf57c7
Update .devcontainer/devcontainer.json
CusiniM Jul 9, 2024
e2fde18
Update devcontainer.json
rrsettgast Jul 9, 2024
6366dfc
Merge branch 'develop' into tpls-update/cusini/update-raja-chai
rrsettgast Jul 9, 2024
4309474
add template parameter.
CusiniM Jul 9, 2024
0f72220
lvarray.
CusiniM Jul 9, 2024
877ce5e
fix lifo storage.
CusiniM Jul 9, 2024
1255c62
fix lifo storage
CusiniM Jul 10, 2024
ec48f36
fmt:fmt to fmt::fmt-header-only
CusiniM Jul 10, 2024
dafdf54
Update .devcontainer/devcontainer.json
CusiniM Jul 10, 2024
84f1137
move fmt find_package.
CusiniM Jul 10, 2024
a7d2f76
add missing const qualifiers.
CusiniM Jul 11, 2024
364fe68
revert raja update related changes
CusiniM Jul 11, 2024
51e2194
tpl tag update
CusiniM Jul 11, 2024
1232013
lvarray update.
CusiniM Jul 11, 2024
442daf8
update lvarray hash and remove rockylinux work.
CusiniM Jul 11, 2024
92c1457
Merge branch 'develop' into tpls-update/cusini/update-raja-chai
CusiniM Jul 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .devcontainer/devcontainer.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"build": {
"dockerfile": "Dockerfile",
"args": {
"GEOS_TPL_TAG": "270-418"
"GEOS_TPL_TAG": "273-446"
CusiniM marked this conversation as resolved.
Show resolved Hide resolved
}
},
"runArgs": [
Expand Down
42 changes: 9 additions & 33 deletions .github/workflows/ci_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,19 +170,6 @@ jobs:
ENABLE_HYPRE: ON
ENABLE_TRILINOS: OFF

- name: Pecan CPU (centos 7.7, gcc 8.2.0, open-mpi 4.0.1, mkl 2019.5)
CMAKE_BUILD_TYPE: Release
DOCKER_REPOSITORY: geosx/pecan-cpu-gcc8.2.0-openmpi4.0.1-mkl2019.5
HOST_CONFIG: host-configs/TOTAL/pecan-CPU.cmake
GCP_BUCKET: geosx/Pecan-CPU

- name: Pangea 2 (centos 7.6, gcc 8.3.0, open-mpi 2.1.5, mkl 2019.3)
CMAKE_BUILD_TYPE: Release
DOCKER_REPOSITORY: geosx/pangea2-gcc8.3.0-openmpi2.1.5-mkl2019.3
ENABLE_HYPRE: ON
ENABLE_TRILINOS: OFF
GCP_BUCKET: geosx/Pangea2

- name: Sherlock CPU (centos 7.9.2009, gcc 10.1.0, open-mpi 4.1.2, openblas 0.3.10)
CMAKE_BUILD_TYPE: Release
DOCKER_REPOSITORY: geosx/sherlock-gcc10.1.0-openmpi4.1.2-openblas0.3.10-zlib1.2.11
Expand Down Expand Up @@ -305,16 +292,6 @@ jobs:
DOCKER_RUN_ARGS: "--cpus=8 --memory=256g --runtime=nvidia --gpus all -v /etc/pki/ca-trust/source/anchors/:/usr/local/share/ca-certificates/llnl:ro"
DOCKER_CERTS_DIR: "/usr/local/share/ca-certificates"
DOCKER_CERTS_UPDATE_COMMAND: "update-ca-certificates"

- name: Centos (7.7, gcc 8.3.1, open-mpi 1.10.7, cuda 11.8.89)
BUILD_AND_TEST_CLI_ARGS: "--no-run-unit-tests --no-install-schema"
CMAKE_BUILD_TYPE: Release
DOCKER_REPOSITORY: geosx/centos7.7-gcc8.3.1-cuda11.8.89
RUNS_ON: streak2
NPROC: 8
DOCKER_RUN_ARGS: "--cpus=8 --memory=128g --runtime=nvidia -v /etc/pki/ca-trust/source/anchors/:/etc/pki/ca-trust/source/anchors/llnl:ro"
DOCKER_CERTS_DIR: "/etc/pki/ca-trust/source/anchors"
DOCKER_CERTS_UPDATE_COMMAND: "update-ca-trust"

# compiler error in ElasticFirstOrderWaveEquationSEMKernel::StressComputation::launch in call to FE_TYPE::computeFirstOrderStiffnessTermX
# - name: Rockylinux (8, clang 17.0.6, cuda 12.5)
Expand All @@ -325,20 +302,19 @@ jobs:
# NPROC: 2
# DOCKER_RUN_ARGS: "--cpus=1 --memory=128g --runtime=nvidia -v /etc/pki/ca-trust/source/anchors/:/etc/pki/ca-trust/source/anchors/llnl:ro"
# DOCKER_CERTS_DIR: "/etc/pki/ca-trust/source/anchors"
# DOCKER_CERTS_UPDATE_COMMAND: "update-ca-trust"

# Below this line, jobs that deploy to Google Cloud.
- name: Pecan GPU (centos 7.7, gcc 8.2.0, open-mpi 4.0.1, mkl 2019.5, cuda 11.5.119)
BUILD_AND_TEST_CLI_ARGS: "--build-exe-only --no-install-schema"
# DOCKER_CERTS_UPDATE_COMMAND: "update-ca-trust"

- name: Rockylinux (8, gcc 8.5, cuda 12.5)
BUILD_AND_TEST_CLI_ARGS: "--no-run-unit-tests --no-install-schema"
CMAKE_BUILD_TYPE: Release
DOCKER_REPOSITORY: geosx/pecan-gpu-gcc8.2.0-openmpi4.0.1-mkl2019.5-cuda11.5.119
HOST_CONFIG: host-configs/TOTAL/pecan-GPU.cmake
DOCKER_REPOSITORY: geosx/rockylinux8-gcc8-cuda12.5
RUNS_ON: streak2
NPROC: 8
DOCKER_RUN_ARGS: "--cpus=8 --memory=128g --runtime=nvidia -v /etc/pki/ca-trust/source/anchors/:/etc/pki/ca-trust/source/anchors/llnl:ro"
NPROC: 2
DOCKER_RUN_ARGS: "--cpus=1 --memory=128g --runtime=nvidia -v /etc/pki/ca-trust/source/anchors/:/etc/pki/ca-trust/source/anchors/llnl:ro"
DOCKER_CERTS_DIR: "/etc/pki/ca-trust/source/anchors"
DOCKER_CERTS_UPDATE_COMMAND: "update-ca-trust"
DOCKER_CERTS_UPDATE_COMMAND: "update-ca-trust"

# Below this line, jobs that deploy to Google Cloud.
- name: Sherlock GPU (centos 7.9.2009, gcc 10.1.0, open-mpi 4.1.2, openblas 0.3.10, cuda 11.7.1,)
BUILD_AND_TEST_CLI_ARGS: "--no-run-unit-tests --no-install-schema"
CMAKE_BUILD_TYPE: Release
Expand Down
2 changes: 1 addition & 1 deletion src/cmake/thirdparty/SetupGeosxThirdParty.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -711,7 +711,7 @@ endif()
if(DEFINED TRILINOS_DIR AND ENABLE_TRILINOS)
message(STATUS "TRILINOS_DIR = ${TRILINOS_DIR}")

include(${TRILINOS_DIR}/lib/cmake/Trilinos/TrilinosConfig.cmake)
include(${TRILINOS_DIR}/lib64/cmake/Trilinos/TrilinosConfig.cmake)

list(REMOVE_ITEM Trilinos_LIBRARIES "gtest")
list(REMOVE_DUPLICATES Trilinos_LIBRARIES)
Expand Down
2 changes: 1 addition & 1 deletion src/coreComponents/LvArray
Open in desktop
Submodule LvArray updated 1 files
+1 −1 host-configs/LLNL/llnl-tpls-base.cmake
2 changes: 2 additions & 0 deletions src/coreComponents/common/Format.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,9 @@
#endif

#ifdef GEOSX_USE_FMT
#ifndef FMT_HEADER_ONLY
CusiniM marked this conversation as resolved.
Show resolved Hide resolved
#define FMT_HEADER_ONLY
#endif
// Differentiate between standalone fmt path and umpire's fmt path
#include "../include/fmt/core.h"
#include "../include/fmt/chrono.h"
Expand Down
28 changes: 17 additions & 11 deletions src/coreComponents/linearAlgebra/interfaces/hypre/HypreMGR.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,23 +73,28 @@ class MGRStrategyBase
{
public:

static constexpr HYPRE_Int numLevels = NLEVEL; ///< Number of levels
static constexpr HYPRE_Int numLevels = NLEVEL; ///< Number of levels

protected:

HYPRE_Int m_numBlocks{ 0 }; ///< Number of different matrix blocks treated separately
HYPRE_Int m_numBlocks{ 0 }; ///< Number of different matrix blocks treated separately

std::vector< HYPRE_Int > m_labels[numLevels]{}; ///< Dof labels kept at each level
HYPRE_Int m_numLabels[numLevels]{ -1 }; ///< Number of dof labels kept
HYPRE_Int * m_ptrLabels[numLevels]{ nullptr }; ///< Pointers to each level's labels, as consumed by MGR
std::vector< HYPRE_Int > m_labels[numLevels]{}; ///< Dof labels kept at each level
HYPRE_Int m_numLabels[numLevels]{ -1 }; ///< Number of dof labels kept
HYPRE_Int * m_ptrLabels[numLevels]{ nullptr }; ///< Pointers to each level's labels, as consumed by MGR

MGRFRelaxationType m_levelFRelaxType[numLevels]; ///< F-relaxation type for each level
HYPRE_Int m_levelFRelaxIters[numLevels]{ -1 }; ///< Number of F-relaxation iterations for each level
MGRInterpolationType m_levelInterpType[numLevels]; ///< Interpolation type for each level
MGRRestrictionType m_levelRestrictType[numLevels]; ///< Restriction type for each level
MGRCoarseGridMethod m_levelCoarseGridMethod[numLevels]; ///< Coarse grid method for each level
MGRFRelaxationType m_levelFRelaxType[numLevels]; ///< F-relaxation type for each level
HYPRE_Int m_levelFRelaxIters[numLevels]{ -1 }; ///< Number of F-relaxation iterations for each level
MGRInterpolationType m_levelInterpType[numLevels]; ///< Interpolation type for each level
MGRRestrictionType m_levelRestrictType[numLevels]; ///< Restriction type for each level
MGRCoarseGridMethod m_levelCoarseGridMethod[numLevels]; ///< Coarse grid method for each level
MGRGlobalSmootherType m_levelGlobalSmootherType[numLevels]; ///< Global smoother type for each level
HYPRE_Int m_levelGlobalSmootherIters[numLevels]{ -1 }; ///< Number of global smoother iterations for each level
HYPRE_Int m_levelGlobalSmootherIters[numLevels]{ -1 }; ///< Number of global smoother iterations for each level
#if GEOS_USE_HYPRE_DEVICE == GEOS_USE_HYPRE_CPU
HYPRE_Real m_coarseGridThreshold{ 1.0e-20 }; ///< Coarse grid truncation threshold
#else
HYPRE_Real m_coarseGridThreshold{ 0.0 }; ///< Coarse grid truncation threshold
#endif
Comment on lines +93 to +97
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't make sense to me? I read it as
"If we GEOS_USE_HYPRE_DEVICE is the same as GEOS_USE_HYPRE_CPU then do something"...but should these ever be the same?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've seen a few test cases with GPUs where threshold > 0 causes a crash in hypre, which does not seem straightforward to resolve. Since threshold > 0 works for CPUs, and for the sake of getting this new truncation functionality used in GEOS, I'm currently setting different threshold according to the architecture. After the GPU path in hypre is improved, I will change back the threshold parameter to the same default for CPUs and GPUs in GEOS


// TODO: the following options are currently commented out in MGR's code.
// Let's consider their use when re-enable in hypre
Expand Down Expand Up @@ -159,6 +164,7 @@ class MGRStrategyBase
GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetCoarseGridMethod( precond.ptr, toUnderlyingPtr( m_levelCoarseGridMethod ) ) );
GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetLevelSmoothType( precond.ptr, toUnderlyingPtr( m_levelGlobalSmootherType ) ) );
GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetLevelSmoothIters( precond.ptr, m_levelGlobalSmootherIters ) );
GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetTruncateCoarseGridThreshold( precond.ptr, m_coarseGridThreshold ) );
GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetNonCpointsToFpoints( precond.ptr, 1 ));
}

Expand Down
3 changes: 1 addition & 2 deletions src/coreComponents/linearAlgebra/interfaces/hypre/HypreUtils.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -515,9 +515,8 @@ enum class MGRCoarseGridMethod : HYPRE_Int
//!< approximated by its diagonal inverse
cprLikeBlockDiag = 3, //!< Non-Galerkin coarse grid computation with dropping strategy: CPR-like approximation with inv(A_FF)
//!< approximated by its block diagonal inverse
approximateInverse = 4, //!< Non-Galerkin coarse grid computation with dropping strategy: inv(A_FF) approximated by sparse approximate
approximateInverse = 4 //!< Non-Galerkin coarse grid computation with dropping strategy: inv(A_FF) approximated by sparse approximate
//!< inverse
galerkinRAI = 5 //!< Galerkin coarse grid computation with arbitrary classical restriction and injective prolongation
};

/**
Expand Down
2 changes: 1 addition & 1 deletion ...oreComponents/linearAlgebra/interfaces/hypre/mgrStrategies/CompositionalMultiphaseFVM.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ class CompositionalMultiphaseFVM : public MGRStrategyBase< 2 >
m_levelFRelaxType[1] = MGRFRelaxationType::none;
m_levelInterpType[1] = MGRInterpolationType::injection;
m_levelRestrictType[1] = MGRRestrictionType::blockColLumped; // True-IMPES
m_levelCoarseGridMethod[1] = MGRCoarseGridMethod::galerkinRAI;
m_levelCoarseGridMethod[1] = MGRCoarseGridMethod::galerkin;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are these settings documented somewhere?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is some documentation via doxygen, see https://geosx-geosx.readthedocs-hosted.com/en/latest/doxygen_output/html/namespacegeos_1_1hypre.html#a05c6983a0ed8e068d0861c91767e83da

We've talked about adding a more detailed description on RTD. Should we re-prioritize that? cc @castelletto1

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant to ask whether there is a discussion on the documentation regarding the strategy? So in this case, for CompositionalMultiphaseFVM there is a strategy that consists of:

     m_levelFRelaxType[0]          = MGRFRelaxationType::jacobi;
     m_levelFRelaxIters[0]         = 1;
     m_levelInterpType[0]          = MGRInterpolationType::jacobi;
     m_levelRestrictType[0]        = MGRRestrictionType::injection;
     m_levelCoarseGridMethod[0]    = MGRCoarseGridMethod::galerkin;
     m_levelGlobalSmootherType[0]  = MGRGlobalSmootherType::none;
     m_levelFRelaxType[1]          = MGRFRelaxationType::none;
     m_levelInterpType[1]          = MGRInterpolationType::injection;
     m_levelRestrictType[1]        = MGRRestrictionType::blockColLumped; // True-IMPES
     m_levelCoarseGridMethod[1]    = MGRCoarseGridMethod::galerkin;
     m_levelGlobalSmootherType[1]  = MGRGlobalSmootherType::ilu0;
     m_levelGlobalSmootherIters[1] = 1;

And is there an easily accessible explanation for all of these options, and why they go together in the geos documentation....or in the hypre documentation? I suspect the hypre documentation is more of an expert level documentation??

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There isn't an easily accessible explanation in neither documentations. I believe hypre's documentation is a better place for a detailed documentation, while in GEOS, we could provide an overview of these preconditioning techniques for each physics model, e.g., the strategy above resembles the commonly used CPR method

m_levelGlobalSmootherType[1] = MGRGlobalSmootherType::ilu0;
m_levelGlobalSmootherIters[1] = 1;
}
Expand Down
2 changes: 1 addition & 1 deletion ...ponents/linearAlgebra/interfaces/hypre/mgrStrategies/CompositionalMultiphaseHybridFVM.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ class CompositionalMultiphaseHybridFVM : public MGRStrategyBase< 3 >
m_levelFRelaxType[1] = MGRFRelaxationType::none;
m_levelInterpType[1] = MGRInterpolationType::jacobi;
m_levelRestrictType[1] = MGRRestrictionType::blockColLumped; // True-IMPES
m_levelCoarseGridMethod[1] = MGRCoarseGridMethod::galerkinRAI;
m_levelCoarseGridMethod[1] = MGRCoarseGridMethod::galerkin;
m_levelGlobalSmootherType[1] = MGRGlobalSmootherType::none;

// Level 2
Expand Down
2 changes: 1 addition & 1 deletion ...ents/linearAlgebra/interfaces/hypre/mgrStrategies/CompositionalMultiphaseReservoirFVM.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ class CompositionalMultiphaseReservoirFVM : public MGRStrategyBase< 3 >
m_levelFRelaxType[2] = MGRFRelaxationType::none;
m_levelInterpType[2] = MGRInterpolationType::injection;
m_levelRestrictType[2] = MGRRestrictionType::blockColLumped; // True-IMPES
m_levelCoarseGridMethod[2] = MGRCoarseGridMethod::galerkinRAI;
m_levelCoarseGridMethod[2] = MGRCoarseGridMethod::galerkin;
m_levelGlobalSmootherType[2] = MGRGlobalSmootherType::ilu0;
m_levelGlobalSmootherIters[2] = 1;
}
Expand Down
2 changes: 1 addition & 1 deletion ...inearAlgebra/interfaces/hypre/mgrStrategies/CompositionalMultiphaseReservoirHybridFVM.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ class CompositionalMultiphaseReservoirHybridFVM : public MGRStrategyBase< 4 >
m_levelFRelaxType[2] = MGRFRelaxationType::none;
m_levelInterpType[2] = MGRInterpolationType::injection;
m_levelRestrictType[2] = MGRRestrictionType::blockColLumped; // True-IMPES
m_levelCoarseGridMethod[2] = MGRCoarseGridMethod::galerkinRAI;
m_levelCoarseGridMethod[2] = MGRCoarseGridMethod::galerkin;
m_levelGlobalSmootherType[2] = MGRGlobalSmootherType::none;

// Level 3
Expand Down
4 changes: 1 addition & 3 deletions src/coreComponents/linearAlgebra/interfaces/hypre/mgrStrategies/MultiphasePoromechanics.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ class MultiphasePoromechanics : public MGRStrategyBase< 3 >
m_levelFRelaxType[2] = MGRFRelaxationType::none;
m_levelInterpType[2] = MGRInterpolationType::injection;
m_levelRestrictType[2] = MGRRestrictionType::blockColLumped; // True-IMPES
m_levelCoarseGridMethod[2] = MGRCoarseGridMethod::galerkinRAI;
m_levelCoarseGridMethod[2] = MGRCoarseGridMethod::galerkin;
m_levelGlobalSmootherType[2] = MGRGlobalSmootherType::ilu0;
m_levelGlobalSmootherIters[2] = 1;
}
Expand All @@ -106,8 +106,6 @@ class MultiphasePoromechanics : public MGRStrategyBase< 3 >
{
setReduction( precond, mgrData );

GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetPMaxElmts( precond.ptr, 0 ));

// Configure the BoomerAMG solver used as F-relaxation for the first level
setMechanicsFSolver( precond, mgrData );

Expand Down
38 changes: 18 additions & 20 deletions ...ents/linearAlgebra/interfaces/hypre/mgrStrategies/MultiphasePoromechanicsReservoirFVM.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,34 +83,34 @@ class MultiphasePoromechanicsReservoirFVM : public MGRStrategyBase< 4 >
setupLabels();

// Level 0
m_levelFRelaxType[0] = MGRFRelaxationType::amgVCycle;
m_levelFRelaxIters[0] = 1;
m_levelInterpType[0] = MGRInterpolationType::jacobi;
m_levelRestrictType[0] = MGRRestrictionType::injection;
m_levelCoarseGridMethod[0] = MGRCoarseGridMethod::nonGalerkin;
m_levelGlobalSmootherType[0] = MGRGlobalSmootherType::none;
m_levelFRelaxType[0] = MGRFRelaxationType::amgVCycle;
m_levelFRelaxIters[0] = 1;
m_levelInterpType[0] = MGRInterpolationType::jacobi;
m_levelRestrictType[0] = MGRRestrictionType::injection;
m_levelCoarseGridMethod[0] = MGRCoarseGridMethod::nonGalerkin;
m_levelGlobalSmootherType[0] = MGRGlobalSmootherType::none;

// Level 1
m_levelFRelaxType[1] = MGRFRelaxationType::gsElimWInverse;
m_levelFRelaxIters[1] = 1;
m_levelInterpType[1] = MGRInterpolationType::blockJacobi;
m_levelRestrictType[1] = MGRRestrictionType::injection;
m_levelCoarseGridMethod[1] = MGRCoarseGridMethod::galerkin;
m_levelGlobalSmootherType[0] = MGRGlobalSmootherType::none;
m_levelFRelaxIters[1] = 1;
m_levelInterpType[1] = MGRInterpolationType::blockJacobi;
m_levelRestrictType[1] = MGRRestrictionType::injection;
m_levelCoarseGridMethod[1] = MGRCoarseGridMethod::galerkin;
m_levelGlobalSmootherType[1] = MGRGlobalSmootherType::none;
rrsettgast marked this conversation as resolved.
Show resolved Hide resolved

// Level 2
m_levelFRelaxType[2] = MGRFRelaxationType::jacobi; //default, i.e. Jacobi
m_levelFRelaxIters[2] = 1;
m_levelInterpType[2] = MGRInterpolationType::jacobi;
m_levelRestrictType[2] = MGRRestrictionType::injection;
m_levelCoarseGridMethod[2] = MGRCoarseGridMethod::galerkin;
m_levelGlobalSmootherType[2] = MGRGlobalSmootherType::none;
m_levelFRelaxType[2] = MGRFRelaxationType::jacobi; //default, i.e. Jacobi
m_levelFRelaxIters[2] = 1;
m_levelInterpType[2] = MGRInterpolationType::jacobi;
m_levelRestrictType[2] = MGRRestrictionType::injection;
m_levelCoarseGridMethod[2] = MGRCoarseGridMethod::galerkin;
m_levelGlobalSmootherType[2] = MGRGlobalSmootherType::none;

// Level 3
m_levelFRelaxType[3] = MGRFRelaxationType::none;
m_levelInterpType[3] = MGRInterpolationType::injection;
m_levelRestrictType[3] = MGRRestrictionType::blockColLumped; // True-IMPES
m_levelCoarseGridMethod[3] = MGRCoarseGridMethod::galerkinRAI;
m_levelCoarseGridMethod[3] = MGRCoarseGridMethod::galerkin;
m_levelGlobalSmootherType[3] = MGRGlobalSmootherType::ilu0;
m_levelGlobalSmootherIters[3] = 1;
}
Expand All @@ -134,8 +134,6 @@ class MultiphasePoromechanicsReservoirFVM : public MGRStrategyBase< 4 >

setReduction( precond, mgrData );

GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetPMaxElmts( precond.ptr, 0 ));

// Configure the BoomerAMG solver used as F-relaxation for the first level
setMechanicsFSolver( precond, mgrData );

Expand Down
2 changes: 0 additions & 2 deletions src/coreComponents/linearAlgebra/interfaces/hypre/mgrStrategies/SinglePhasePoromechanics.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,8 +78,6 @@ class SinglePhasePoromechanics : public MGRStrategyBase< 1 >
{
setReduction( precond, mgrData );

GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetPMaxElmts( precond.ptr, 0 ));

// Configure the BoomerAMG solver used as F-relaxation for the first level
setMechanicsFSolver( precond, mgrData );

Expand Down
5 changes: 1 addition & 4 deletions ...earAlgebra/interfaces/hypre/mgrStrategies/SinglePhasePoromechanicsConformingFractures.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,6 @@

#include "linearAlgebra/interfaces/hypre/HypreMGR.hpp"

#define BRANCH_MGR_FSOLVER

namespace geos
{

Expand Down Expand Up @@ -103,7 +101,6 @@ class SinglePhasePoromechanicsConformingFractures : public MGRStrategyBase< 2 >
HypreMGRData & mgrData )
{
setReduction( precond, mgrData );
GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetPMaxElmts( precond.ptr, 0 ));

// Configure the BoomerAMG solver used as F-relaxation for the second level
GEOS_LAI_CHECK_ERROR( HYPRE_BoomerAMGCreate( &mgrData.mechSolver.ptr ) );
Expand All @@ -121,7 +118,7 @@ class SinglePhasePoromechanicsConformingFractures : public MGRStrategyBase< 2 >
#else
GEOS_LAI_CHECK_ERROR( HYPRE_BoomerAMGSetRelaxOrder( mgrData.mechSolver.ptr, 1 ) );
#endif
GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetFSolverAtLevel( 1, precond.ptr, mgrData.mechSolver.ptr ) );
GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetFSolverAtLevel( precond.ptr, mgrData.mechSolver.ptr, 1 ) );

// Configure the BoomerAMG solver used as mgr coarse solver for the pressure reduced system
setPressureAMG( mgrData.coarseSolver );
Expand Down
5 changes: 1 addition & 4 deletions ...inearAlgebra/interfaces/hypre/mgrStrategies/SinglePhasePoromechanicsEmbeddedFractures.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,6 @@ class SinglePhasePoromechanicsEmbeddedFractures : public MGRStrategyBase< 2 >
m_levelInterpType[1] = MGRInterpolationType::jacobi;
m_levelRestrictType[1] = MGRRestrictionType::injection;
m_levelCoarseGridMethod[1] = MGRCoarseGridMethod::nonGalerkin;

}

/**
Expand All @@ -96,8 +95,6 @@ class SinglePhasePoromechanicsEmbeddedFractures : public MGRStrategyBase< 2 >
{
setReduction( precond, mgrData );

GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetPMaxElmts( precond.ptr, 0 ));

// Configure the BoomerAMG solver used as F-relaxation for the second level
GEOS_LAI_CHECK_ERROR( HYPRE_BoomerAMGCreate( &mgrData.mechSolver.ptr ) );
GEOS_LAI_CHECK_ERROR( HYPRE_BoomerAMGSetTol( mgrData.mechSolver.ptr, 0.0 ) );
Expand All @@ -114,7 +111,7 @@ class SinglePhasePoromechanicsEmbeddedFractures : public MGRStrategyBase< 2 >
#else
GEOS_LAI_CHECK_ERROR( HYPRE_BoomerAMGSetRelaxOrder( mgrData.mechSolver.ptr, 1 ) );
#endif
GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetFSolverAtLevel( 1, precond.ptr, mgrData.mechSolver.ptr ) );
GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetFSolverAtLevel( precond.ptr, mgrData.mechSolver.ptr, 1 ) );

// Configure the BoomerAMG solver used as mgr coarse solver for the pressure reduced system
setPressureAMG( mgrData.coarseSolver );
Expand Down
4 changes: 1 addition & 3 deletions ...nts/linearAlgebra/interfaces/hypre/mgrStrategies/SinglePhasePoromechanicsReservoirFVM.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ class SinglePhasePoromechanicsReservoirFVM : public MGRStrategyBase< 2 >
m_levelGlobalSmootherType[0] = MGRGlobalSmootherType::none;

// Level 1
m_levelFRelaxType[1] = MGRFRelaxationType::gsElimWInverse;
m_levelFRelaxType[1] = MGRFRelaxationType::gsElimWInverse;
m_levelFRelaxIters[1] = 1;
m_levelInterpType[1] = MGRInterpolationType::blockJacobi;
m_levelRestrictType[1] = MGRRestrictionType::injection;
Expand All @@ -93,8 +93,6 @@ class SinglePhasePoromechanicsReservoirFVM : public MGRStrategyBase< 2 >
{
setReduction( precond, mgrData );

GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetPMaxElmts( precond.ptr, 0 ));

// Configure the BoomerAMG solver used as F-relaxation for the first level
setMechanicsFSolver( precond, mgrData );

Expand Down
3 changes: 1 addition & 2 deletions ...mponents/linearAlgebra/interfaces/hypre/mgrStrategies/SolidMechanicsEmbeddedFractures.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ class SolidMechanicsEmbeddedFractures : public MGRStrategyBase< 1 >
setupLabels();

// Level 0
m_levelFRelaxType[0] = MGRFRelaxationType::gsElimWInverse;
m_levelFRelaxType[0] = MGRFRelaxationType::jacobi;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the effect of this?

Copy link
Collaborator Author

@CusiniM CusiniM Jul 9, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My guess is that the gsElimWInverse is not really scalable. Shouldn't we do block jacobi though? It's a block diagonal matrix so wouldn't block jacobi be exact?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we should be doing block jacobi

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, now I recall what's going on here...

Under the hood, hypre does block-Jacobi because of the interpolation choice m_levelInterpType[0] = MGRInterpolationType::blockJacobi. However, this logic is confusing (it shouldn't depend on the interpolation) and hypre should have a specific option for doing block-Jacobi F-relaxation.

I can work on that.

m_levelFRelaxIters[0] = 1;
m_levelInterpType[0] = MGRInterpolationType::blockJacobi;
m_levelRestrictType[0] = MGRRestrictionType::injection;
Expand All @@ -82,7 +82,6 @@ class SolidMechanicsEmbeddedFractures : public MGRStrategyBase< 1 >
HypreMGRData & mgrData )
{
setReduction( precond, mgrData );
GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetPMaxElmts( precond.ptr, 0 ));

// Configure the BoomerAMG solver used as mgr coarse solver for the displacement reduced system
setDisplacementAMG( mgrData.coarseSolver );
Expand Down
2 changes: 0 additions & 2 deletions ...omponents/linearAlgebra/interfaces/hypre/mgrStrategies/ThermalMultiphasePoromechanics.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,8 +109,6 @@ class ThermalMultiphasePoromechanics : public MGRStrategyBase< 3 >
{
setReduction( precond, mgrData );

GEOS_LAI_CHECK_ERROR( HYPRE_MGRSetPMaxElmts( precond.ptr, 0 ));

// CHECK: the mechanics solver setup was missing: was there a reason?
// Configure the BoomerAMG solver used as F-relaxation for the first level
setMechanicsFSolver( precond, mgrData );
Expand Down
Loading
Loading