automated doc build (#66)

* doc building

* oh yes, it is good

* docstrings, README

.github/workflows/deploy.yml

@@ -2,8 +2,6 @@ name: deploy
 
 on:
   push:
-    branches:
-      - main
     tags:        
       - 'v*'
 
@@ -17,7 +15,6 @@ jobs:
     secrets: inherit
 
   mypy:
-    name: mypy
     uses: ./.github/workflows/mypy.yml
     with:
       job: build
@@ -30,7 +27,6 @@ jobs:
       job: build
     secrets: inherit
 
-
   deploy:
     name: deploy
     runs-on: ubuntu-latest
@@ -40,14 +36,13 @@ jobs:
       - tests
 
     steps:
-
       - name: Checkout
         uses: actions/checkout@v4
 
       - name: Set up Python
         uses: actions/setup-python@v5
         with:
-          python-version: "3.11"
+          python-version: '3.11'
 
       - name: install
         run: |
@@ -66,3 +61,12 @@ jobs:
       - uses: softprops/action-gh-release@v2
         with:
           generate_release_notes: true
+
+  docs:
+    name: docs
+    needs:
+      - deploy
+    uses: ./.github/workflows/docs.yml
+    with:
+      job: build
+    secrets: inherit

.github/workflows/docs.yml

@@ -0,0 +1,43 @@
+name: docs
+
+on:
+  workflow_dispatch:
+  workflow_call:
+    inputs:
+      job:
+        required: true
+        type: string
+    secrets:
+      GH_TOKEN:
+        description: 'A token passed from the caller workflow'
+        required: true
+
+jobs:
+  build:
+    name: publish-docs
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+            python-version: '3.11'
+
+      - name: Install dependencies
+        run: |
+            python -m pip install --upgrade pip
+            pip install pdoc3
+
+      - name: Generate documentation with pdoc3
+        run: |
+            mkdir -p docs
+            pdoc --html meteor --output-dir docs --force
+
+      - name: Deploy to GitHub Pages
+        uses: peaceiris/actions-gh-pages@v4
+        with:
+            github_token: ${{ secrets.GITHUB_TOKEN }}
+            publish_dir: ./docs/meteor
+

.gitignore

@@ -11,6 +11,9 @@ __pycache__/
 *.pdb
 *.cif
 
+# documentation is automatically built
+docs/
+
 # C extensions
 *.so
 
@@ -52,6 +55,7 @@ htmlcov/
 .coverage
 .coverage.*
 .cache
+.ruff_cache
 nosetests.xml
 coverage.xml
 *.cover

README.md

@@ -1,11 +1,14 @@
 # METEOR
-Map Enhancement Tools for Ephemeral Occupancy Refinement
 ----------
 
+**Map Enhancement Tools for Ephemeral Occupancy Refinement**
+
 [![Pytest](https://github.com/alisiafadini/meteor/actions/workflows/tests.yml/badge.svg)](https://github.com/your_username/your_repo/actions/workflows/tests.yml)
 [![Mypy](https://github.com/alisiafadini/meteor/actions/workflows/mypy.yml/badge.svg)](https://github.com/your_username/your_repo/actions/workflows/mypy.yml)
 [![Ruff](https://github.com/alisiafadini/meteor/actions/workflows/lint.yml/badge.svg)](https://github.com/your_username/your_repo/actions/workflows/lint.yml)
 [![codecov](https://codecov.io/gh/alisiafadini/meteor/graph/badge.svg?token=NCYMP06MNS)](https://codecov.io/gh/alisiafadini/meteor)
+![GitHub Tag](https://img.shields.io/github/v/tag/rs-station/meteor)
+
 
 `meteor` is a tool for computing crystallographic difference maps. 
 
@@ -34,22 +37,81 @@ Once installed, you will have two command-line scripts. Ask for more info using
 meteor.diffmap -h
 meteor.phaseboost -h
 ```
-these scripts compute denoised difference maps using the constant-phase approximation _vs._ iterative phase retrieval, respectively.
+these scripts compute denoised difference maps using the constant-phase approximation _vs._ iterative phase retrieval, respectively. See below for additional detail
 
+## the science behind `meteor`
 
-## philosophy: better science through automation
+### philosophy: better science through automation
 
 `meteor` aims to:
 
 1. maximize signal to noise
 2. be objective and reproducible (minimize user choice & bias)
 3. be easy to use
 
-Aim 1 is met using structure factor amplitude weighting (e.g. k-weighting, existing art) and TV denoising (new in the context of crystallography). Aims 2 and 3 are met through automatically setting parameters using negentropy maximization. For all the details, see our paper (coming soon to a preprint server near you).
+Aim 1 is met using structure factor amplitude weighting (e.g. k-weighting, existing art) and [TV denoising](https://en.wikipedia.org/wiki/Total_variation_denoising) (new in the context of crystallography). Aims 2 and 3 are met through automatically setting parameters using negentropy maximization (as in [ICA](https://en.wikipedia.org/wiki/Independent_component_analysis)). For all the details, see our paper (coming soon to a preprint server near you).
+
+
+### isomorphous data, please
+
+METEOR is only for isomorphous difference maps, meaning the lattices/symmetries of the _native_ and _derivative_ datasets are comparable. If you need to compare non-isomorphous lattices, check out [matchmaps](https://github.com/rs-station/matchmaps).
+
+
+## command-line details
+
+`meteor` provides two command-line scripts that most users will want. If you prefer working in Jupyter notebooks or want to develop against `meteor`'s library, refer to the [API documentation](https://rs-station.github.io/meteor/).
+
+Both of `meteor`'s scripts generate difference maps (MTZs). We recommend starting with `meteor.diffmap`. This script applies k-weighting then TV-denoising, and picks parameters for both by maximizing negentropy. It's relatively fast to run (expect about a minute) and has fewer nobs to turn. After you've tried `meteor.diffmap`, you can give `meteor.phaseboost` a try. This script iteratively applys TV denoising, and adjusts the phases of the _derivative_ data to try and produce a denoised map. At the end, it applies a k-weighting and TV denoising pass. It often results in slightly better maps, at the cost of additional compute (many minutes). We also have noticed the algorithm can become unstable in rare circumstances. If you think that might be happening to you, please get in touch.
+
+Note that individual steps in both of these scripts can be turned off or modified using command-line flags, as described below.
+
+One note: in the lingo we adope, we compute _derivative_ minus _native_ maps. Initial phases typically from a model of the _native_ data, computed from a CIF/PDB model. Usually the derivative data are ligand bound, time-resolved-activated, or similar... but in the end, the use case is defined by you!
+
+### meteor.diffmap
+
+Compute difference maps, including k-weighting and TV-denoising options. To see the full help, just run `meteor.diffmap -h`. Here are some examples that highlight common use cases:
+
+Computing a k-weighted, TV-denoised diffmap, with `meteor` making some smart default choices for me:
+```
+meteor.diffmap derivative.mtz native.mtz -s native.pdb
+```
+note that the order of `derivative.mtz` and `native.mtz` matters!
+
+Suppose I have some non-standard column names. Using `gemmi mtz <my.mtz>` might be smart to find out what they should be; then,
+```
+meteor.diffmap derivative.mtz --derivative-amplitude-column F_ON --derivative-uncertainty-column SIGF_ON native.mtz -s native.pdb
+```
+or, equivalently,
+```
+meteor.diffmap derivative.mtz --da F_ON --du SIGF_ON native.mtz -s native.pdb
+```
+
+what if I want to compute a k-weighted map, with k-parameter of `0.05`, and no TV?
+```
+meteor.diffmap derivative.mtz native.mtz -s native.pdb --kweight-mode fixed --kweight-parameter 0.05 --tv-denoise-mode none 
+```
+
+To note and remember:
+
+  - the k-parameter sets how strongly outlier difference structure factor amplitudes are suppressed
+  - k-weighting with k-parameter of `0.0` is NOT the same as with k-weighting totally turned off!
+  - the TV-weight trades off the smoothness (lack of noise) in the final map _vs._ fidelity to the original map. A higher TV weight means more denoising, and a greater departure from the original data
+
+
+### meteor.phaseboost
+
+Compute iterative-TV denoised difference maps. The usage is very similar to `meteor.diffmap`, so let's focus on the three new flags:
+
+  - `--tv-weights-to-scan` sets what TV weights are assessed at every iteration -- you may want to try and play with this. Increasing the number of scanned points will probably result in more stable runs, and perhaps slightly better results, at the cost of computation
+  - `--convergence-tolerance` dictactes the phase change at which the algorithm stops. The default is pretty good, but if you notice the negentropy is still increasing at the end of your run, try lowering this. If instead, you have a lot of maps to denoise and you notice things converge early, you can lower this
+  - `--max-iterations` is mostly just there to force the algorithm to bail out just in case it starts to thrash; so far, we haven't seen cases where it's necessary to adjust this
+
 
+### advanced options
 
-## isomorphous data, please
+If you feel adventerous, check out `meteor/settings.py`. Default values are collected there, and you could ducktype to your heart's content. Not recommended for 99% of users!
 
-METEOR is only for isomorphous difference maps, meaning the lattices/symmetries of the `native` and `derivative` datasets are comparable. If you need to compare non-isomorphous lattices, check out [matchmaps](https://github.com/rs-station/matchmaps).
 
+### what the heck are these `meteor_metadata.json` files?
 
+These are flat text files that contain information about how `meteor` ran. The primary intent is for debugging purposes, but you can also [read them with `meteor`](https://github.com/rs-station/meteor/blob/64f96ca0a293520cbd0163267768ddbfd68c7b0b/meteor/scripts/common.py#L331) or any standard JSON parser, and make some plots to better understand `meteor`'s runtime behavior.

meteor/__init__.py

@@ -1,3 +1,5 @@
+""".. include:: ../README.md"""
+
 __all__ = ["__version__", "version"]
 
 from ._version import __version__, version

meteor/diffmaps.py

@@ -1,3 +1,5 @@
+"""library for computing difference density maps"""
+
 from __future__ import annotations
 
 from collections.abc import Sequence

meteor/io.py

@@ -1,6 +1,4 @@
-"""https://www.ccp4.ac.uk/html/mtzformat.html
-https://www.globalphasing.com/buster/wiki/index.cgi?MTZcolumns
-"""
+"""MTZ input/output helper functions"""
 
 from __future__ import annotations
 

meteor/iterative.py

@@ -1,3 +1,5 @@
+"""iterative TV-based phase retrieval"""
+
 from __future__ import annotations
 
 from collections.abc import Callable

meteor/rsmap.py

@@ -1,3 +1,5 @@
+"""Map class definition and related functions"""
+
 from __future__ import annotations
 
 from collections.abc import Callable

meteor/scale.py

@@ -1,3 +1,5 @@
+"""(anisotropic) scaling of crystallographic datasets"""
+
 from __future__ import annotations
 
 import numpy as np

meteor/scripts/__init__.py

@@ -0,0 +1 @@
+"""`meteor` CLI"""

meteor/scripts/common.py

@@ -1,3 +1,5 @@
+"""shared code for the CLI"""
+
 from __future__ import annotations
 
 import argparse

meteor/scripts/compute_difference_map.py

@@ -1,3 +1,5 @@
+"""source code for meteor.diffmap"""
+
 from __future__ import annotations
 
 from typing import Any

meteor/scripts/compute_iterative_tv_map.py

@@ -1,3 +1,5 @@
+"""source code for meteor.phaseboost"""
+
 from __future__ import annotations
 
 from typing import Any

meteor/settings.py

@@ -1,3 +1,5 @@
+"""runtime settings for `meteor`"""
+
 from __future__ import annotations
 
 import numpy as np
@@ -8,7 +10,10 @@
 MAP_SAMPLING: int = 3
 
 
-# known map labels
+# known map labels, see:
+# https://www.ccp4.ac.uk/html/mtzformat.html
+# https://www.globalphasing.com/buster/wiki/index.cgi?MTZcolumns
+
 OBSERVED_INTENSITY_COLUMNS: list[str] = [
     "I",  # generic
     "IMEAN",  # CCP4

meteor/sfcalc.py

@@ -1,3 +1,5 @@
+"""computing structure factors from models"""
+
 from pathlib import Path
 
 import gemmi

meteor/testing.py

@@ -1,3 +1,5 @@
+"""used during testing"""
+
 from __future__ import annotations
 
 from dataclasses import dataclass

meteor/tv.py

@@ -1,3 +1,5 @@
+"""total variation denoising of maps"""
+
 from __future__ import annotations
 
 import json

meteor/utils.py

@@ -1,3 +1,5 @@
+"""crystallographic helper functions"""
+
 from __future__ import annotations
 
 from collections.abc import Sequence

meteor/validate.py

@@ -1,3 +1,5 @@
+"""map quality assessment: negentropy and negentropy maximization"""
+
 from __future__ import annotations
 
 from collections.abc import Callable, Sequence