Refactoring, organization & packaging updates (#9)

* refactor/reorganize/rename modules
* remove usgsprograms (only needed in pymake)
* remove meson_build function (provided by pymake)
* refactor download_and_unzip utility function with urllib
* refactor common_regression with pytest_generate_tests hook
* move head/budget comparison & various other utilities to flopy
* update setup.cfg with extra dependency groups and supported python/OS versions
* use pytest --import-mode=importlib in pyproject.toml (as recommended in link above)
* add pytest.ini to project root
* add scripts/pull_request_prepare.py
* expand readme and developer docs
* add shared testing fixtures/markers
* add tests and CI test job
* cache example models in CI test job
* draft CI to create release when changes hit main branch
* use --exit-zero with flake8 and pylint in CI lint job
* add minimal framework for reusable test cases

Co-authored-by: mjreno <[email protected]>

.github/workflows/ci.yml

@@ -0,0 +1,167 @@
+name: CI
+on:
+  push:
+    branches:
+      - main
+      - develop*
+    paths-ignore:
+      - '**.md'
+  pull_request:
+    branches:
+      - main
+      - develop*
+    paths-ignore:
+      - '**.md'
+jobs:
+  lint:
+    name: Lint
+    runs-on: ubuntu-latest
+    steps:
+
+      - name: Checkout repo
+        uses: actions/checkout@v3
+
+      - name: Setup Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: 3.7
+          cache: 'pip'
+          cache-dependency-path: setup.cfg
+
+      - name: Install Python packages
+        run: |
+          pip install .
+          pip install ".[lint]"
+
+      - name: Run isort
+        run: isort --verbose --check --diff modflow_devtools
+
+      - name: Run black
+        run: black --check --diff modflow_devtools
+
+      - name: Run flake8
+        run: flake8 --count --show-source --exit-zero modflow_devtools
+
+      - name: Run pylint
+        run: pylint --jobs=0 --errors-only --exit-zero modflow_devtools
+
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+    steps:
+
+      - name: Checkout repo
+        uses: actions/checkout@v3
+
+      - name: Setup Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: 3.7
+
+      - name: Upgrade pip and install build and twine
+        run: |
+          pip install --upgrade pip
+          pip install build twine
+
+      - name: Base modflow_devtools installation
+        run: |
+          pip --verbose install .
+
+      - name: Print package version
+        run: |
+          python -c "import modflow_devtools; print(modflow_devtools.__version__)"
+
+      - name: Build package
+        run: |
+          python -m build
+      
+      - name: Check distribution
+        run: |
+          twine check --strict dist/*
+
+  test:
+    name: Test
+    needs:
+      - build
+      - lint
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ ubuntu-latest, macos-latest, windows-latest ]
+        python: [ 3.7, 3.8, 3.9, "3.10" ]
+    steps:
+
+      - name: Checkout repo
+        uses: actions/checkout@v3
+        with:
+          path: modflow-devtools
+
+      - name: Checkout modflow6
+        uses: actions/checkout@v3
+        with:
+          repository: MODFLOW-USGS/modflow6
+          path: modflow6
+
+      - name: Checkout modflow6 examples
+        uses: actions/checkout@v3
+        with:
+          repository: MODFLOW-USGS/modflow6-examples
+          path: modflow6-examples
+
+      - name: Checkout modflow6 test models
+        uses: actions/checkout@v3
+        with:
+          repository: MODFLOW-USGS/modflow6-testmodels
+          path: modflow6-testmodels
+
+      - name: Checkout modflow6 large test models
+        uses: actions/checkout@v3
+        with:
+          repository: MODFLOW-USGS/modflow6-largetestmodels
+          path: modflow6-largetestmodels
+
+      - name: Install executables
+        uses: modflowpy/install-modflow-action@v1
+
+      - name: Setup Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python }}
+          cache: 'pip'
+          cache-dependency-path: |
+            modflow-devtools/setup.cfg
+            modflow6-examples/etc/requirements*.txt
+
+      - name: Install Python packages
+        working-directory: modflow-devtools
+        run: |
+          pip install .
+          pip install ".[test]"
+      
+      - name: Cache modflow6 examples
+        id: cache-examples
+        uses: actions/cache@v3
+        with:
+          path: modflow6-examples/examples
+          key: modflow6-examples-${{ hashFiles('modflow6-examples/data/**') }}
+
+      - name: Install extra Python packages
+        if: steps.cache-examples.outputs.cache-hit != 'true'
+        working-directory: modflow6-examples/etc
+        run: |
+          pip install -r requirements.pip.txt
+          pip install -r requirements.usgs.txt
+
+      - name: Build modflow6 example models
+        if: steps.cache-examples.outputs.cache-hit != 'true'
+        working-directory: modflow6-examples/etc
+        run: python ci_build_files.py
+
+      - name: Run tests
+        working-directory: modflow-devtools
+        env:
+          BIN_PATH: ~/.local/bin/modflow
+          REPOS_PATH: ${{ github.workspace }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: pytest -v -n auto --durations 0

.github/workflows/release.yml

@@ -0,0 +1,52 @@
+name: Publish release and package
+on:
+  push:
+    branches:
+      - master
+jobs:
+  release:
+    name: Release
+    runs-on: ubuntu-latest
+    steps:
+
+      - name: Checkout repo
+        uses: actions/checkout@v3
+
+      - name: Setup Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: 3.7
+
+      - name: Upgrade pip and install build and twine
+        run: |
+          pip install --upgrade pip
+          pip install build twine
+
+      - name: Base modflow_devtools installation
+        run: |
+          pip --verbose install .
+
+      - name: Print package version
+        run: |
+          python -c "import modflow_devtools; print(modflow_devtools.__version__)"
+
+      - name: Build package
+        run: |
+          python -m build
+      
+      - name: Check distribution
+        run: |
+          twine check --strict dist/*
+      
+      - name: Create release
+        uses: "marvinpinto/action-automatic-releases@latest"
+        with:
+          repo_token: "${{ secrets.GITHUB_TOKEN }}"
+
+      # todo: set repo secrets and enable
+      # - name: Publish package
+      #   env:
+      #     TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
+      #     TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
+      #   run: |
+      #     twine upload dist/*

DEVELOPER.md

@@ -0,0 +1,64 @@
+# Developing `modflow-devtools`
+
+This document provides guidance to set up a development environment and discusses conventions used in this project.
+
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+
+- [Installation](#installation)
+- [Testing](#testing)
+  - [Environment variables](#environment-variables)
+  - [Running the tests](#running-the-tests)
+  - [Writing new tests](#writing-new-tests)
+    - [Temporary directories](#temporary-directories)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+## Installation
+
+To get started, first fork and clone this repository. Then install the project and core packages as well as linting and testing dependencies:
+
+```shell
+pip install .
+pip install ".[lint, test]"
+```
+
+## Testing
+
+This repository's tests use [`pytest`](https://docs.pytest.org/en/latest/) and several plugins.
+
+### Environment variables
+
+This repository's tests expect a few environment variables:
+
+- `BIN_PATH`: path to MODFLOW 6 and related executables
+- `REPOS_PATH`: the path to MODFLOW 6 example model repositories
+- `GITHUB_TOKEN`: a GitHub authentication token
+
+These may be set manually, but the recommended approach is to configure environment variables in a `.env` file in the project root, for instance:
+
+```
+BIN_PATH=/path/to/modflow/executables
+REPOS_PATH=/path/to/repos
+GITHUB_TOKEN=yourtoken...
+```
+
+The tests use [`pytest-dotenv`](https://github.com/quiqua/pytest-dotenv) to detect and load variables from this file.
+
+**Note:** at minimum, the tests require that the `mf6` executable is present in `BIN_PATH`.
+
+### Running the tests
+
+Tests should be run from the project root. To run the tests in parallel with verbose output:
+
+```shell
+pytest -v -n auto
+```
+
+### Writing new tests
+
+Tests should follow a few conventions for ease of use and maintenance.
+
+#### Temporary directories
+
+Tests which must write to disk should use `pytest`'s built-in `temp_dir` fixture or one of this package's own scoped temporary directory fixtures.

MANIFEST.in

@@ -1,4 +1,3 @@
 global-exclude .DS_Store *.pyc *.pyo *.pyd *.swp *.bak *~ .* *.sh *.yml *.md *.toml
 exclude autotest/*
 include pyproject.toml
-include modflow_devtools/utilities/usgsprograms.txt