Skip to content

Commit

Permalink
docs: expand release docs, update authors (#1660)
Browse files Browse the repository at this point in the history
  • Loading branch information
@wpbonelli
wpbonelli authored Dec 21, 2022
1 parent 89fc16b commit bd51afa
Show file tree
Hide file tree
Showing 3 changed files with 138 additions and 57 deletions.
3 changes: 3 additions & 0 deletions CITATION.cff
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,9 @@ authors:
orcid: https://orcid.org/0000-0001-5909-0010
- family-names: Brakenhoff
given-names: Davíd
- family-names: Bonelli
given-names: Wesley P.
orcid: https://orcid.org/0000-0002-2665-5078
preferred-citation:
type: article
authors:
Expand Down
118 changes: 80 additions & 38 deletions docs/make_release.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,85 +3,123 @@
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->

- [Release automation](#release-automation)
- [Initial steps](#initial-steps)
- [Release procedure](#release-procedure)
- [Release from master branch](#release-from-master-branch)
- [Reinitialize develop branch](#reinitialize-develop-branch)
- [Publish the release](#publish-the-release)
- [PyPI](#pypi)
- [Conda forge](#conda-forge)
- [Automated releases](#automated-releases)
- [Dry runs](#dry-runs)
- [Production mode](#production-mode)
- [Manual releases](#manual-releases)
- [Release from `master` branch](#release-from-master-branch)
- [Reinitialize `develop` branch](#reinitialize-develop-branch)
- [Publish the release](#publish-the-release)
- [PyPI](#pypi)
- [Conda](#conda)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

## Release automation

The FloPy release procedure is mostly automated with GitHub Actions in [`release.yml`](../.github/workflows/release.yml). There are a few manual steps that need to be performed, however:
## Initial steps

The FloPy release procedure is mostly automated with GitHub Actions in [`release.yml`](../.github/workflows/release.yml), but there are a few manual steps to complete first:

1. Update `usgsprograms.txt` in the [GitHub pymake repository](https://github.com/modflowpy/pymake) with the path to the new MODFLOW 6 release. Also update all other targets in `usgsprograms.txt` with the path to new releases.

2. Recompile all of the executables released on the [GitHub executables repository](https://github.com/MODFLOW-USGS/executables) using the `buildall.py` pymake script and Intel compilers for all operating systems.
2. Trigger a new release for the [executables repository](https://github.com/MODFLOW-USGS/executables), either via the GitHub Actions UI or GitHub CLI. See the executables repo's `DEVELOPER.md` for more information. Wait for the release to be published.

3. Update the README.md on the [GitHub executables repository](https://github.com/MODFLOW-USGS/executables) with the information in the `code.md` file created by the `buildall.py` pymake script.
3. Update the authors in `CITATION.cff` for the Software/Code citation for FloPy, if required.


4. Make a new release on the [GitHub executables repository](https://github.com/MODFLOW-USGS/executables) and add all of the operating system specific zip files containing the compiled executables (`linux.zip`, `mac.zip`, `win64.zip`, `win32.zip`). Publish the new release.
## Release procedure

5. Update the authors in `CITATION.cff` for the Software/Code citation for FloPy, if required.


Next, make a release branch from develop (*e.g.* `v3.3.6`). The branch name should be the version number with a `v` prefix. Pushing this branch to GitHub will trigger the release workflow, detailed below. If the branch name ends with `rc` it is considered a release candidate and the workflow is a dry run, stopping after updating version info & plugin classes and running tests and notebooks. If the branch name does not end with `rc` the release is considered approved and the full procedure runs.
Once initial tasks are completed, making a new release involves the following steps:

After updating version information, regenerating plugin classes, and generating a changelog, the approved release workflow creates a draft pull request from the release branch into `master`. Merging this pull request triggers another job to create a draft release. Promoting this draft release to a full release will trigger a final job to publish the release to PyPI and reset the `develop` branch from `master`, incrementing the patch version number on `develop`.
1. Release from `master` branch
2. Reinitialize `develop` branch
3. Publish the package to PyPI
4. Publish the package to Conda

These steps are automated with GitHub Actions in [`release.yml`](../.github/workflows/release.yml), but documentation for running the steps manually is provided in the [Manual releases](#manual-releases) section below.

## Release procedure
### Automated releases

This procedure runs automatically in `release.yml` after a release branch is pushed to GitHub, except for the final step (updating the `conda-forge/flopy-feedstock` repository &mdash; there is a bot which will [automatically detect changes and create a PR](https://github.com/conda-forge/flopy-feedstock/pull/47) to do so).
To begin an automated release, create a release branch from `develop`. The release branch name should be the version number with a `v` prefix (e.g. `v3.3.6`), with an optional `rc` suffix. If the branch name ends with `rc` it is considered a preliminary release candidate. If the branch name does not end with `rc` the release is considered production-ready. The automated workflow for approved, production-ready releases includes more steps than for release candidates.

Pushing the release branch starts the automated workflow. Manual intervention is only necessary to approve autogenerated pull requests and publish the autogenerated release post.

### Release from master branch
When the release branch is pushed to the `modflowpy/flopy` repository, the release workflow begins with the following steps:

- Update MODFLOW 6 dfn files in the repository and MODFLOW 6 package classes by running `python -c 'import flopy; flopy.mf6.utils.generate_classes(branch="master", backup=False)'`

- Run `isort` and `black` on the updated MODFLOW 6 package classes. This can be achieved by running `python scripts/pull_request_prepare.py` from the project root. The commands `isort .` and `black .` can also be run individually instead.
- update version strings to match the version number in the release branch name
- regenerate plugin classes from MODFLOW 6 DFN files
- rerun tests and notebooks
- generate and update changelogs
- build and test the FloPy package

For more detail regarding specific steps, see the [Manual releases](#manual-releases) section below.

#### Dry runs

If the branch name ends with `rc`, the workflow is a dry run, and ends here. No PRs are opened, a release is not created, and the release is not published to PyPI or Conda. The `flopy` package and the changelog are uploaded as artifacts, allowing manual inspection.

#### Production mode

If the branch name does not end with `rc`, the workflow will proceed to open a PR updating `master` from the release branch. Merging this PR to `master` triggers another job to draft a release.

**Note:** the PR should be merged, not squashed. Squashing removes the commit history from the `master` branch and causes `develop` and `master` to diverge, which can cause future PRs updating `master` to replay commits from previous releases.

Publishing the release triggers jobs to publish the `flopy` package to PyPI and open a PR updating `develop` from `master`. This PR also updates version strings, incrementing the patch version number.


### Manual releases

As described above, making a release manually involves the following steps:

1. Release from `master` branch
2. Reinitialize `develop` branch
3. Publish the package to PyPI
4. Publish the package to Conda


#### Release from `master` branch

- Run `python scripts/update_version.py -v <semver>` to update the version number stored in `version.txt` and `flopy/version.py`. For an approved release use the `--approve` flag.

- Use `run_notebooks.py` in the `scripts` directory to rerun all of the notebooks in:
- Update MODFLOW 6 dfn files in the repository and MODFLOW 6 package classes by running `python -c 'import flopy; flopy.mf6.utils.generate_classes(branch="master", backup=False)'`

- Run `isort` and `black` on the `flopy` module. This can be achieved by running `python scripts/pull_request_prepare.py` from the project root. The commands `isort .` and `black .` can also be run individually instead.

- Use `run_notebooks.py` in the `scripts` directory to rerun all notebooks in:

- `examples\Notebooks` directory.
- `examples\Notebooks\groundwater_paper` directory.
- `examples\Notebooks\FAQ` directory.

- Generate a changelog with [git cliff](https://github.com/orhun/git-cliff): `git cliff --unreleased --tag=<version number>`.

- Commit the changes to the release branch and push the commit to the [upstream GitHub repository](https://github.com/modflowpy/flopy).
- Generate a changelog starting from the last release with [git cliff](https://github.com/orhun/git-cliff), for instance: `git cliff --config cliff.toml --unreleased --tag=<release version number>`.

- Build and check the package with:
- Prepend the release changelog to the comprehensive changelog file `docs/version_changes.md`.

```shell
python -m build
twine check --strict dist/*
```
- Commit changes to the release branch and push the commit to the [`modflowpy/flopy` GitHub repository](https://github.com/modflowpy/flopy) (you must have write permissions on the repo).

- Update master branch from the release branch, e.g. by opening and merging a pull request into `master`. The pull request should be merged, *not* squashed, in order to preserve the project's commit history.

- Tag the merge commit to `master` with the version number. Don't forget to commit the tag. Push the commit and tag to GitHub.
- [Make a release](https://github.com/modflowpy/flopy/releases) from the `master` branch. Add the release changelog as the release post contents. Publish the release.

- Make a release on [GitHub website](https://github.com/modflowpy/flopy/releases). Add version changes for [current release](https://github.com/modflowpy/flopy/blob/develop/docs/version_changes.md) from to release text. Publish release.
- GitHub should have tagged the most recent revision of `master` with the release number &mdash; if for some reason this did not occur automatically, tag `master` with the version number (`git tag <version>`) and push the tag to `modflowpy/flopy` (e.g., if you have your fork's remote named `origin` and `modflowpy/flopy` as `upstream`, as is typical, use `git push upstream --tags`).


### Reinitialize develop branch
#### Reinitialize `develop` branch

1. Merge the `master` branch into the `develop` branch.

2. Set the version as appropriate: `python scripts/update_version.py -v <semver>`.

3. Commit and push the updated `develop` branch.
3. Lint Python files: `python scripts/pull_request_prepare.py`

4. Commit and push the updated `develop` branch.

### Publish the release

#### PyPI
#### Publish the release

##### PyPI

1. Make sure the latest `build` and `twine` tools are installed using:

Expand All @@ -103,7 +141,11 @@ twine check --strict dist/*
twine upload dist/*
```
#### Conda forge
##### Conda
For the Conda distribution, there [is a bot](https://github.com/regro-cf-autotick-bot) which will [automatically detect new package versions uploaded to PyPI and create a PR](https://github.com/conda-forge/flopy-feedstock/pull/50) to update the `conda-forge/flopy-feedstock` repository. This PR can be reviewed, updated if needed, and merged to update the package on the `conda-forge` channel.
A Conda update can be performed manually with the following steps, substituting one's own fork of `conda-forge/flopy-feedstock`:
1. Download the `*.tar.gz` file for the just-created release from the [GitHub website](https://github.com/modflowpy/flopy/releases).
Expand Down
74 changes: 55 additions & 19 deletions scripts/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,37 +5,73 @@ This document describes the utility scripts in this directory.
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->

- [`make-release.py`](#make-releasepy)
- [`process_benchmarks.py`](#process_benchmarkspy)
- [`pull_request_prepare.py`](#pull_request_preparepy)
- [`run_notebooks.py`](#run_notebookspy)
- [`update_version.py`](#update_versionpy)
- [Processing benchmarks](#processing-benchmarks)
- [Preparing for PRs](#preparing-for-prs)
- [Running notebooks](#running-notebooks)
- [Updating version](#updating-version)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

## `make-release.py`
## Processing benchmarks

TODO

## `process_benchmarks.py`

The `process_benchmarks.py` script consumes JSON files created by `pytest-benchmark` and creates a CSV file containing benchmark results. The script takes 2 positional arguments:
The `process_benchmarks.py` script converts one or more JSON files created by [`pytest-benchmark`](https://pytest-benchmark.readthedocs.io/en/latest/) into a single CSV file containing benchmark results. The script takes 2 positional arguments:

1. The path to the directory containing the JSON files.
2. The path to the directory to create the results CSV.

## `pull_request_prepare.py`
Input JSON files are expected to be named according to the `pytest-benchmark` format used when the `--benchmark-autosave` is provided:

The `pull_request_prepare.py` script lints Python source code files by running `black` and `isort` on the `flopy` subdirectory. This script should be run before opening a pull request, as CI will fail if the code is not properly formatted.
```shell
<commitid>_<date>_<time>_<isdirty>.json
```

## `run_notebooks.py`
For instance, `e689af57e7439b9005749d806248897ad550eab5_20150811_041632_uncommitted-changes.json`.

TODO
**Note**: the `process_benchmarks.py` script depends on `seaborn`, which is not included as a dependency in either `etc/environment.yml` or in any of the optional groups in `setup.cfg`, since this is the only place it is used in this repository.

## `update_version.py`
## Preparing for PRs

The `update_version.py` script can be used to update the FloPy version number. If the script is run with no argument, the version number is not changed, but an updated timestamp is written to the `flopy/version.py` file. To set the FloPy version number, use the `--version` (short `-v`) option, e.g.:
The `pull_request_prepare.py` script lints Python source code files by running `black` and `isort` on the `flopy` subdirectory. This script should be run before opening a pull request, as CI will fail if the code is not properly formatted. For instance, from the project root:

```shell
python scripts/update_version.py 3.3.6
```
python scripts/pull_request_prepare.py
```

## Running notebooks

The `run_notebooks.py` script runs notebooks located in subdirectories of the `examples` directory, currently including:

- `examples/Notebooks`
- `examples/groundwater_paper/Notebooks`
- `examples/FAQ`

Notebooks are run using `jupytext --from_ipynb --execute <notebook path>`. Note that notebooks are under version control, and running them with this script will produce large changesets in your working tree as execution metadata is updated. You will likely want to discard these changes before committing, e.g. with `git restore examples/Notebooks` (and likewise for other notebook folders). Care should be taken to make sure desired changes to notebook cells are not discarded however.

## Updating version

The `update_version.py` script can be used to update FloPy version numbers. Running the script first updates the version in `version.txt`, then propagates the change to various other places version strings or timestamps are embedded in the repository:

- `flopy/version.py`
- `flopy/DISCLAIMER.md`
- `CITATION.cff`
- `README.md`
- `docs/notebook_examples.md`
- `docs/PyPI_release.md`

The script acquires a file lock before writing to files to make sure only one process edits the files at any given time and prevent desynchronization.

If the script is run with no arguments, the version number is not changed, but updated timestamps are written. To set the version number, use the `--version` (short `-v`) option, e.g.:

```shell
python scripts/update_version.py -v 3.3.6
```

To get the current version number, use the `--get` flag:

```shell
python scripts/update_version.py
```

This simply returns the contents of `version.txt` and does not write any changes to the repository's files.

By default, the script assumes a local development version of FloPy. The `--approve` flag should be used prior to releasing a new FloPy version. This will alter the `DISCLAIMER.md` file, substituting wording to indicate the version is no longer preliminary but approved for official release. See [the release docs](../docs/make_release.md) for more information.

0 comments on commit bd51afa

Please sign in to comment.