refactor(tests): use devtools fixtures (#1130)

* use fixtures from devtools for loading external models and tracking target exes
* convert adhoc --model, --package and --original-regression opts to pytest fixtures/CLI opts
* use devtools and flopy in build_exes.py and get_exes.py, remove pymake usage
* remove pymake usage from common_regression.py and external model tests
* use GitHub API token for CI step to download executables
* update developer docs

.github/workflows/ci.yml

@@ -149,6 +149,8 @@ jobs:
 
       - name: Get executables
         working-directory: modflow6/autotest
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
         run: pytest -v --durations 0 get_exes.py
 
       - name: Test programs
@@ -221,6 +223,8 @@ jobs:
       
       - name: Get executables
         working-directory: modflow6/autotest
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
         run: pytest -v --durations 0 get_exes.py
 
       - name: Test modflow6
@@ -327,12 +331,16 @@ jobs:
       - name: Get executables
         if: runner.os != 'Windows'
         working-directory: modflow6/autotest
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
         run: pytest -v --durations 0 get_exes.py
 
       - name: Get executables (Windows)
         if: runner.os == 'Windows'
         working-directory: modflow6/autotest
         shell: pwsh
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
         run: pytest -v --durations 0 get_exes.py
 
       - name: Test programs

.github/workflows/docs.yml

@@ -103,6 +103,8 @@ jobs:
 
       - name: Run benchmarks
         working-directory: modflow6/distribution
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
         run: python benchmark.py
 
       - name: Run sphinx

.github/workflows/large.yml

@@ -93,6 +93,8 @@ jobs:
 
       - name: Get executables
         working-directory: modflow6/autotest
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
         run: |
           pytest -v --durations 0 get_exes.py
 

DEVELOPER.md

@@ -152,22 +152,17 @@ We also provide make files which can be used to build MODFLOW 6 with [GNU Make](
 For the build instructions we refer to the [GNU Make Manual](https://www.gnu.org/software/make/manual/).
 
 
-## Running Tests Locally
+## Running Tests
 
-For complete testing as done on the CI, clone your fork of the modflow6-testmodels repository (via either ssh or https, ssh shown here):
+Tests should pass locally before a PR is opened on Github. All the tests are executed by the CI system and a pull request can only be merged with passing tests.
 
-```shell
-git clone [email protected]:<github username>/modflow6-testmodels.git
-```
-* The modflow6-testmodels repository must be cloned in the same directory that contains the modflow6 repository.
-
-To run tests first change directory to the `autotest` folder:
+Tests must be run from the `autotest` folder:
 
 ```shell
 cd modflow6/autotest
 ```
 
-Update your flopy installation by executing
+FloPy plugins must first be updated:
 
 ```shell
 python update_flopy.py
@@ -186,24 +181,79 @@ Unless you built and installed MODFLOW 6 binaries with meson, you will also have
 pytest -v build_exes.py
 ```
 
-Then the tests can be run with commands similar to these:
+Then the tests can be run with `pytest` as usual, for instance:
 
 ```shell
-# Build MODFLOW 6 tests
+# Run all tests with verbose output
 pytest -v
+```
 
-# Build MODFLOW 6 example tests
-pytest -v test_z01_testmodels_mf6.py
+Tests can be run in parallel with the `-n` option, which accepts an integer argument for the number of processes to use. If the value `auto` is provided, `pytest-xdist` will use as many processes as your machine has available CPUs:
 
-# Build MODFLOW 5 to 6 converter example tests
-pytest -v test_z02_testmodels_mf5to6.py
+```shell
+pytest -v -n auto
 ```
 
-The tests can be run in parallel by adding the flag "-n" which accepts an argument for the specific number of processes to use or "auto" to let pytest decide:
+### External model repos
+
+While many tests create models programmatically, the full suite tests MODFLOW 6 against example models stored in the following external repositories:
+
+- [`MODFLOW-USGS/modflow6-testmodels`](https://github.com/MODFLOW-USGS/modflow6-testmodels)
+- [`MODFLOW-USGS/modflow6-largetestmodels`](https://github.com/MODFLOW-USGS/modflow6-largetestmodels)
+- [`MODFLOW-USGS/modflow6-examples`](https://github.com/MODFLOW-USGS/modflow6-examples)
+
+#### Installing external repos
+
+By default, the tests expect these repositories side-by-side with (i.e. in the same parent directory as) the `modflow6` repository. If the repos are somewhere else, you can set the `REPOS_PATH` environment variable to point to their parent directory.
+
+**Note:** a convenient way to persist environment variables needed for tests is to store them in a `.env` file in the `autotest` folder. Each variable should be defined on a separate line, with format `KEY=VALUE`. The `pytest-dotenv` plugin will then automatically load any variables found in this file into the test process' environment.
+
+##### Test models
+
+The test model repos can simply be cloned &mdash; ideally, into the parent directory of the `modflow6` repository, so that repositories live side-by-side:
 
 ```shell
-pytest -v -n auto
+git clone MODFLOW-USGS/modflow6-testmodels
+git clone MODFLOW-USGS/modflow6-largetestmodels
 ```
 
-You should execute the test suites before submitting a PR to Github.
-All the tests are executed on our Continuous Integration infrastructure and a pull request can only be merged once all tests pass.
+##### Example models
+
+First clone the example models repo:
+
+```shell
+git clone MODFLOW-USGS/modflow6-examples
+```
+
+The example models require some setup after cloning. Some extra Python dependencies are required to build the examples: 
+
+```shell
+cd modflow6-examples/etc
+pip install -r requirements.pip.txt
+```
+
+Then, still from the `etc` folder, run:
+
+```shell
+python ci_build_files.py
+```
+
+This will build the examples for subsequent use by the tests.
+
+#### Running external model tests
+
+External model tests are located in their own files:
+
+```shell
+# Run MODFLOW 6 test models
+pytest -v -n auto test_z01_testmodels_mf6.py
+
+# Run MODFLOW 5 to 6 conversion test models
+pytest -v -n auto test_z02_testmodels_mf5to6.py
+
+# Run example models
+pytest -v -n auto test_z03_examples.py
+
+# Run large test models
+pytest -v -n auto test_z03_largetestmodels.py
+```

autotest/build_exes.py

@@ -1,156 +1,31 @@
-# Build targets
+import argparse
+from pathlib import Path
 
-# to use ifort on windows, run this
-# python build_exes.py -fc ifort
+import pytest
+from modflow_devtools.build import meson_build
 
-# can compile only mf6 directly using this command:
-#  python -c "import build_exes; build_exes.test_build_modflow6()"
+from conftest import project_root_path
 
-import os
-import pathlib as pl
-import subprocess as sp
-import sys
-from contextlib import contextmanager
 
-from framework import running_on_CI
+repository = "MODFLOW-USGS/modflow6"
+top_bin_path = project_root_path / "bin"
 
-if running_on_CI():
-    print("running on CI environment")
-    os.environ["PYMAKE_DOUBLE"] = "1"
 
-# set OS dependent extensions
-eext = ""
-soext = ".so"
-if sys.platform.lower() == "win32":
-    eext = ".exe"
-    soext = ".dll"
-elif sys.platform.lower() == "darwin":
-    soext = ".dylib"
+@pytest.fixture
+def bin_path():
+    return top_bin_path
 
-mfexe_pth = "temp/mfexes"
 
-# use the line below to set fortran compiler using environmental variables
-# os.environ["FC"] = "ifort"
-
-# some flags to check for errors in the code
-# add -Werror for compilation to terminate if errors are found
-strict_flags = (
-    "-fall-intrinsics "
-    "-Wtabs -Wline-truncation -Wunused-label "
-    "-Wunused-variable -pedantic -std=f2008 "
-    "-Wcharacter-truncation"
-)
-
-
-@contextmanager
-def set_directory(path: str):
-    """Sets the cwd within the context
-
-    Args:
-        path (Path): The path to the cwd
-
-    Yields:
-        None
-    """
-
-    origin = os.path.abspath(os.getcwd())
-    path = os.path.abspath(path)
-    try:
-        os.chdir(path)
-        print(f"change from {origin} -> {path}")
-        yield
-    finally:
-        os.chdir(origin)
-        print(f"change from {path} -> {origin}")
-
-
-def relpath_fallback(pth):
-    try:
-        # throws ValueError on Windows if pth is on a different drive
-        return os.path.relpath(pth)
-    except ValueError:
-        return os.path.abspath(pth)
-
-
-def create_dir(pth):
-    # create pth directory
-    print(f"creating... {os.path.abspath(pth)}")
-    os.makedirs(pth, exist_ok=True)
-
-    msg = f"could not create... {os.path.abspath(pth)}"
-    assert os.path.exists(pth), msg
-
-
-def set_compiler_environment_variable():
-    fc = None
-
-    # parse command line arguments
-    for idx, arg in enumerate(sys.argv):
-        if arg.lower() == "-fc":
-            fc = sys.argv[idx + 1]
-        elif arg.lower().startswith("-fc="):
-            fc = arg.split("=")[1]
-
-    # determine if fc needs to be set to the FC environmental variable
-    env_var = os.getenv("FC", default="gfortran")
-    if fc is None and fc != env_var:
-        fc = env_var
-
-    # validate Fortran compiler
-    fc_options = (
-        "gfortran",
-        "ifort",
+def test_meson_build(bin_path):
+    meson_build(
+        project_path=project_root_path,
+        build_path=project_root_path / "builddir",
+        bin_path=bin_path
     )
-    if fc not in fc_options:
-        raise ValueError(
-            f"Fortran compiler {fc} not supported. Fortran compile must be "
-            + f"[{', '.join(str(value) for value in fc_options)}]."
-        )
-
-    # set FC environment variable
-    os.environ["FC"] = fc
-
-
-def meson_build(
-    dir_path: str = "..",
-    libdir: str = "bin",
-):
-    set_compiler_environment_variable()
-    is_windows = sys.platform.lower() == "win32"
-    with set_directory(dir_path):
-        cmd = (
-            "meson setup builddir "
-            + f"--bindir={os.path.abspath(libdir)} "
-            + f"--libdir={os.path.abspath(libdir)} "
-            + "--prefix="
-        )
-        if is_windows:
-            cmd += "%CD%"
-        else:
-            cmd += "$(pwd)"
-        if pl.Path("builddir").is_dir():
-            cmd += " --wipe"
-        print(f"setup meson\nrunning...\n  {cmd}")
-        sp.run(cmd, shell=True, check=True)
-
-        cmd = "meson install -C builddir"
-        print(f"build and install with meson\nrunning...\n  {cmd}")
-        sp.run(cmd, shell=True, check=True)
-
-
-def test_create_dirs():
-    pths = [os.path.join("..", "bin"), os.path.join("temp")]
-
-    for pth in pths:
-        create_dir(pth)
-
-    return
-
-
-def test_meson_build():
-    meson_build()
 
 
 if __name__ == "__main__":
-    test_create_dirs()
-    test_meson_build()
+    parser = argparse.ArgumentParser("Rebuild local development version of MODFLOW 6")
+    parser.add_argument("-p", "--path", help="path to bin directory", default=top_bin_path)
+    args = parser.parse_args()
+    test_meson_build(Path(args.path).resolve())