Developing the developer guide

.zenodo.json

@@ -23,7 +23,7 @@
 
         {
             "name": "Kukulies, Julia",
-            "affiliation": "University of Gothenburg (Sweden)",
+            "affiliation": "NSF National Center for Atmospheric Research",
             "orcid": "0000-0001-6084-0069"
         },
         {

CONTRIBUTING.md

@@ -2,73 +2,48 @@
 
 __Welcome! We are very happy that you are interested in our project and thanks for taking time to contribute! :)__
 
-We are currently reorganizing our project. So, please check our modifications later.
-
 
 ## Getting Started
 ### Installation & Environment details
-You will find them in the [README.md](https://github.com/climate-processes/tobac/blob/master/README.md).
+You will find them in the [README.md](https://github.com/tobac-project/tobac/blob/master/README.md).
 
 ### Tutorials
-Tutorials have been prepared to provide you further inside to `tobac`s functionality. Please visit the separate [tobac-tutorials](https://github.com/climate-processes/tobac-tutorials) repository here on Github.
-
+Tutorials have been prepared to provide you further inside to `tobac`s functionality. Please have a look in the 
+[examples folder](https://github.com/tobac-project/tobac/tree/main/examples). 
 
 ### Documentation
 You will find our documentation at [https://tobac.readthedocs.io](https://tobac.readthedocs.io).
 
 ### Testing
 The tests are located in the [tests folder](https://github.com/climate-processes/tobac/tree/master/tobac/tests).
 
-
 ## Reporting Bugs
-Please create a new issue on [GitHub](https://github.com/climate-processes/tobac/issues) if it is not listed there, yet.
+Please create a new issue on [GitHub](https://github.com/tobac-project/tobac/issues) if it is not listed there, yet.
 
 ### How to write a good Bug Report?
 * Give it a clear descriptive title.
 * Copy and paste the error message.
 * Describe the steps for reproducing the problem and give an specific example.  
-* Optional: Make a suggestion to fix it.
-
+* Optional: Make a suggestion to fix it. 
 
 ## How to Submit Changes
-* Please read the [README.md](https://github.com/climate-processes/tobac/blob/master/README.md) first, to learn about our project goals and check the [changelog.md]().
-* Before you start a pull request, please make sure that you added [numpydoc docstrings](#docstringExample) to your functions. This way the api documentation will be parsed properly.
-* If it is a larger change or an newly added feature or workflow, please place an example of use in the [tobac-tutorials](https://github.com/climate-processes/tobac-tutorials) repository or adapt the existing examples there.
-* If necessary add a folder or modify a file.
+* Have a look at [our roadmap](https://github.com/tobac-project/tobac-roadmap/blob/master/tobac-roadmap-main.md) first, 
+to learn about our project goals and check the 
+[changelog.md](https://github.com/tobac-project/tobac/blob/main/CHANGELOG.md).
+* More details on the code structure and further help for code contributions can be found in our [developer 
+guide](https://tobac.readthedocs.io/code_structure.html)
+* Before you start a pull request, please make sure that you added [numpydoc 
+docstrings](https://numpydoc.readthedocs.io/en/latest/format.html) to your 
+functions. See [docstring example in the developer guide](https://tobac.readthedocs.io/contributing.html). This way the 
+api documentation will be parsed properly.
+* If it is a larger change or an newly added feature or workflow, please add an example in the [example 
+folder](https://github.com/tobac-project/tobac/tree/main/examples) or adapt the existing examples there.
 * The code should be PEP 8 compliant, as this facilitates our collaboration. Please use the first stable version (22.6.0) of [black](https://black.readthedocs.io/en/stable/) to format your code. When you submit a pull request, all files are checked for formatting.
 * The tobac repository is set up with pre-commit hooks to automatically format your code when commiting changes. Please run the command "pre-commit install" in the root directory of tobac to set up pre-commit formatting.
 
-We aim to respond to all new issues/pull requests as soon as possible, however at times this is not possible due to work commitments.
-
-### Numpydoc Example <a name='docstringExample'>
-```python
-
-   '''
-   calculate centre of gravity and mass forech individual tracked cell in the simulation
-
-
-    Parameters
-    ----------
-    tracks : pandas.DataFram
-        DataFrame containing trajectories of cell centres
-
-    param mass : iris.cube.Cube
-        cube of quantity (need coordinates 'time', 'geopotential_height','projection_x_coordinate' and 
-        'projection_y_coordinate')
-
-    param mask : iris.cube.Cube
-        cube containing mask (int > where belonging to cloud volume, 0 everywhere else )
-
+We aim to respond to all new issues/pull requests as soon as possible, however sometimes this is not possible due to work commitments.
 
-    Returns
-    -------
-    track_out : pandas.DataFrame
-        Dataframe containing t,x,y,z positions of centre of gravity and total cloud mass each tracked cells 
-        at each timestep
-
-    '''
 
-```
 
 ## Slack
 In addition to the workflow here on Github, there's a tobac workspace on Slack [tobac-dev.slack.com](tobac-dev.slack.com) that we use for some additional communication around the project. Please join us there to stay updated about all things tobac that go beyond the detailed work on the code.

doc/code_reviews.rst

@@ -0,0 +1,36 @@
+Code reviews
+------------------
+
+Before anything is merged into the release branch (:code:`RC_*`), we require that two reviewers accept the code changes of a pull request. 
+
+============================
+How to do a code review 
+============================
+
+* Checkout out pull request locally (`how to checkout a pull request locally <https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally>`_)
+
+* Run tests locally
+
+* Go through code and see if it is readable and easy to understand
+
+* Not required, but often useful: test new features with your own data 
+
+
+============================
+Tips and expectations
+============================
+
+
+Doing a code review can be very challenging if you are unfamiliar with the process. Here is a set of documents which might provide a good resource on how to get started:
+
+https://github.com/google/eng-practices
+
+
+=========================
+Conventional comments
+=========================
+
+The comments in a code review should be clear and constructive.
+
+A useful way of highlighting the intention of specific comments is to label them according to `conventional comments <https://conventionalcomments.org/>`_.
+

doc/code_structure.rst

@@ -0,0 +1,72 @@
+Code structure and key design concepts 
+--------------------------------------
+
+==================================
+Modules
+==================================
+
+**tobac** aims to provide a flexible and modular framework which can be seen as a toolbox to create tracking algorithms according to the user's specific research needs. 
+
+The **tobac** package currently consists of three **main modules**:
+
+1. The :py:mod:`tobac.feature_detection` contains methods to identify objects (*features*) in 2D or 3D (3D or 4D when including the time dimensions) gridded data. This is done by identifying contiguous regions above or below one or multiple user-defined thresholds. The module makes use of :py:mod:`scipy.ndimage.label`, a generic image processing method that labels features in an array. The methods in :py:mod:`tobac.feature_detection` are high-level functions that enable a fast and effective feature detection and create easy-to-use output in form of a :py:mod:`pandas.DataFrame` that contains the coordinates and some basic information on each detected feature. The most high-level methods that is commonly used by users is :py:func:`tobac.feature_detection_multithreshold`. 
+
+2. The :py:mod:`tobac.segmentation` module contains methods to define the extent of the identified feature areas or volumes. This step is needed to create a mask of the identified features because the feature detection currently only saves the center points of the features. The segmentation procedure is performed by using the watershedding method, but more methods are to be implemented in the future. Just as the feature detection, this module can handle both 2D and 3D data. 
+
+3. The :py:mod:`tobac.tracking` module is responsible for linking identified features over time. This module makes primarily use of the python package :py:mod:`trackpy`. Note that the linking using :py:mod:`trackpy` is based on particle tracking principles which means that only the feature center positions (not the entire area or volume associated with each feature) are needed to link features over time. Other methods such as tracking based on overlapping areas from the segmented features are to be implemented.
+
+In addition to the main modules, there are three **postprocessing modules**: 
+
+4. The :py:mod:`tobac.merge_split` module provides functionality to identify mergers and splitters in the tracking output and to add labels such that one can reconstruct the parent and child tracks of each cell. 
+
+5. The :py:mod:`tobac.analysis` module contains methods to analyze the tracking output and derive statistics about individual tracks as well as summary statistics of the entire populations of tracks or subsets of the latter. 
+
+6. The :py:mod:`tobac.plotting` module provides methods to visualize the tracking output, for example for creating maps and animations of identified features, segmented areas and tracks.
+
+
+Finally, there are two modules that are primarily **important for developers**:
+
+7. The :py:mod:`tobac.utils` module is a collection of smaller, not necessarily tracking-specific methods that facilitate and support the methods of the main modules. This module has multiple submodules. We separate methods that are rather generic and could also be practical for tobac users who build their own tracking algorithms (:py:mod:`tobac.utils.general`) and methods that mainly facilitate the development of **tobac** (:py:mod:`tobac.utils.internal`). Sometimes, new features come with the need of a whole set of new methods, so it could make sense to save these in their own submodule (see e.g. :py:mod:`tobac.periodic_boundaries`)
+
+8. The :py:mod:`tobac.testing` module provides support for writing of unit tests. This module contains several methods to create simplified test data sets on which the various methods and parameters for feature detection, segmentation, and tracking can be tested. 
+
+For more information on each submodule, refer to the respective source code documentation.
+
+One thing to note is that **tobac** as of now is purely functional. The plan is, however, to move towards a more object-oriented design with base classes for the main operations such as feature detection and tracking. 
+
+
+========
+Examples
+========
+
+To help users get started with **tobac** and to demonstrate the various functionalities, **tobac** hosts several detailed and **illustrated examples** in the form of Jupyter notebooks. They are hosted under the directory `examples/` and be executed by the user. Our readthedocs page also hosts a rendered version of our examples as `gallery <https://tobac.readthedocs.io/en/latest/examples.html>`_
+
+
+============================
+Migrating to xarray and dask
+============================
+
+Currently, **tobac** uses `iris cubes <https://scitools-iris.readthedocs.io/en/latest/userguide/iris_cubes.html>`_ as the 
+primary data container. However, we are currently working on migrating the source code to 
+`xarray <https://docs.xarray.dev/en/stable/>`_ such that all internal functions are based on `xr.DataArray 
+objects <https://docs.xarray.dev/en/stable/generated/xarray.DataArray.html>`_. 
+
+To ensure a robust transition from **iris** to **xarray**, we make use of various decorators that convert input and 
+output data for the main functions without changing their actual code. These decorators are located in the `decorator 
+submodule <https://github.com/tobac-project/tobac/blob/main/tobac/utils/decorators.py>`_. 
+
+In addition, one of our main goals for the future is to fully support `dask <https://www.dask.org/>`_, in order to scale 
+to large datasets and enable parallelization.  
+
+
+
+
+
+
+
+
+
+
+
+
+

doc/conf.py

@@ -19,14 +19,15 @@
     "sphinx_rtd_theme",
     "sphinx.ext.napoleon",
     "nbsphinx",
-   "sphinx_gallery.load_style",
+    "sphinx_gallery.load_style",
 ]
 
 
 html_theme = "sphinx_rtd_theme"
 
 html_static_path = ["_static"]
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+
 
 project = "tobac"
 
@@ -79,3 +80,18 @@ def setup(app):
 napoleon_preprocess_types = False
 napoleon_type_aliases = None
 napoleon_attr_annotations = True
+
+nbsphinx_thumbnails = {
+    "examples/Basics/Idealized-Case-1_Tracking-of-a-Test-Blob-in-2D": "_static/thumbnails/Basics_Idealized-Case-1_Tracking-of-a-Test-Blob-in-2D_Thumbnail.png",
+    "examples/Basics/Idealized-Case-2_Two_crossing_Blobs": "_static/thumbnails/Basics_Idealized-Case-2_Two_crossing_Blobs_Thumbnail.png",
+    "examples/Basics/Methods-and-Parameters-for-Feature-Detection_Part_1": "_static/thumbnails/Basics_Methods-and-Parameters-for-Feature-Detection_Part_1_Thumbnail.png",
+    "examples/Basics/Methods-and-Parameters-for-Feature-Detection_Part_2": "_static/thumbnails/Basics_Methods-and-Parameters-for-Feature-Detection_Part_2_Thumbnail.png",
+    "examples/Basics/Methods-and-Parameters-for-Linking": "_static/thumbnails/Basics_Methods-and-Parameters-for-Linking_Thumbnail.png",
+    "examples/Basics/Methods-and-Parameters-for-Segmentation": "_static/thumbnails/Basics_Methods-and-Parameters-for-Segmentation_Thumbnail.png",
+    "examples/Example_OLR_Tracking_model/Example_OLR_Tracking_model": "_static/thumbnails/Example_OLR_Tracking_model_Thumbnail.png",
+    "examples/Example_OLR_Tracking_satellite/Example_OLR_Tracking_satellite": "_static/thumbnails/Example_OLR_Tracking_satellite_Thumbnail.png",
+    "examples/Example_Precip_Tracking/Example_Precip_Tracking": "_static/thumbnails/Example_Precip_Tracking_Thumbnail.png",
+    "examples/Example_Track_on_Radar_Segment_on_Satellite/Example_Track_on_Radar_Segment_on_Satellite": "_static/thumbnails/Example_Track_on_Radar_Segment_on_Satellite_Thumbnail.png",
+    "examples/Example_Updraft_Tracking/Example_Updraft_Tracking": "_static/thumbnails/Example_Updraft_Tracking_Thumbnail.png",
+    "examples/Example_vorticity_tracking_model/Example_vorticity_tracking_model": "_static/thumbnails/Example_vorticity_tracking_model_Thumbnail.png",
+}