Feature Request: Lmod integration to use & save user-specified configuration files

[hagertnl]

Proposing an optional flag to runtests.py (or more accurately, to test_harness_driver.py): --lmod-rc=/path/to/rc.lua

This flag provides the path to a Lua file that sets defaults and hides modules, as described by https://lmod.readthedocs.io/en/latest/093_modulerc.html#modulerc-label . This would enable user-specified default environments, which allows tests to depend on the default modules, instead of specifying specific versions, which would require less test maintenance in the future. We can use the LMOD_MODULERCFILE environment variable to achieve this configuration. Note that this does override the OLCF-specified LMOD_MODULERCFILE, but I think that's a necessary risk -- if you are going to set your own defaults, then it seems right that you would ignore the OLCF-specified defaults or copy those defaults into your own rc.lua.

A best practice would be to program known breakages into the build script, to prevent the test from launching if it is known to fail. For example, if ROCm/5.3.0 has a bug in the runtime library that impacts a test, that test should fail in the build stage if ROCm/5.3.0 is detected.