From c0296f58e589266963d67736d53128afba8d34f6 Mon Sep 17 00:00:00 2001 From: Juli Date: Thu, 4 May 2023 09:58:45 -0600 Subject: [PATCH 01/25] suggestions for sections and subsections for developer guide --- doc/code_reviews.rst | 27 ++++++++++++++++++++++++++ doc/code_structure.rst | 19 ++++++++++++++++++ doc/contributing.rst | 44 ++++++++++++++++++++++++++++++++++++++++++ doc/index.rst | 10 ++++++++++ doc/mentoring.rst | 17 ++++++++++++++++ 5 files changed, 117 insertions(+) create mode 100644 doc/code_reviews.rst create mode 100644 doc/code_structure.rst create mode 100644 doc/contributing.rst create mode 100644 doc/mentoring.rst diff --git a/doc/code_reviews.rst b/doc/code_reviews.rst new file mode 100644 index 00000000..480975b7 --- /dev/null +++ b/doc/code_reviews.rst @@ -0,0 +1,27 @@ +Code reviews +------------------ + +========================= +Branch protection rules +========================= + +* main and RC_v* --> 2 reviews required +* dev_* --> 1 review required +* exp_* --> no reviews required + +* dev_* should be created from RC_v* +* exp_* should be created from dev_* + +============================ +How to do a code review +============================ + + +========================= +Expectations +========================= + + +========================= +Conventional comments +========================= diff --git a/doc/code_structure.rst b/doc/code_structure.rst new file mode 100644 index 00000000..6b515451 --- /dev/null +++ b/doc/code_structure.rst @@ -0,0 +1,19 @@ +Code structure and key design koncepts +-------------------------------------- + +================================== +Modules +================================== + +- Explain the modular framework +- Explain the responsibilities of each submodule +- Future plan: create base classes for the main operations like feature detection/ tracking + +============================ +Migrating to xarray and dask +============================ + +- Basics of xarray (xarray.Dataarray class) and dask +- How these are or could be used in tobac +- How to work on this + diff --git a/doc/contributing.rst b/doc/contributing.rst new file mode 100644 index 00000000..05acd587 --- /dev/null +++ b/doc/contributing.rst @@ -0,0 +1,44 @@ +How to contribute +------------------ + +========================= +Code of conduct +========================= + +* Link to code of conduct: https://www.python.org/psf/conduct/ + +========================= +git basics +========================= + +* Create a Github account + +* Branching (link to git tutorials) + +* Create an issue (or select one, look for "good first issue" labels) + +* Link to CONTRIBUTING.md + +* Create a pull request from your fork + +========================= +Branching strategy +========================= + +* RC_* +* dev_* +* exp_* + +========================= +Writing unit tests +========================= + +* Pytest and pytest fixtures +* Test coverage + +========================= +Add examples +========================= + +* jupyter notebooks +* upload example data to zenodo diff --git a/doc/index.rst b/doc/index.rst index 79d57f18..042f6628 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -59,6 +59,16 @@ The project is currently being extended by several contributors to include addit merge_split + +.. toctree:: + :caption: Developer guide + :maxdepth: 2 + + code_structure + contributing + code_reviews + mentoring + .. toctree:: :caption: API Reference :maxdepth: 3 diff --git a/doc/mentoring.rst b/doc/mentoring.rst new file mode 100644 index 00000000..02e14ff3 --- /dev/null +++ b/doc/mentoring.rst @@ -0,0 +1,17 @@ +Mentoring +---------- + + +============================ +Writing code collaboratively +============================ + + +============== +Get a mentor +============== + + +=============== +Pair reviews +=============== From cacabcf1df41aef56e6c4b500e67c01e7d2a3009 Mon Sep 17 00:00:00 2001 From: Juli Date: Wed, 7 Jun 2023 15:07:48 -0600 Subject: [PATCH 02/25] populating the developer guide with some more content --- doc/code_reviews.rst | 11 ------- doc/code_structure.rst | 45 ++++++++++++++++++++++++++-- doc/contributing.rst | 68 +++++++++++++++++++++++++++++++++++------- doc/mentoring.rst | 12 ++++++-- 4 files changed, 108 insertions(+), 28 deletions(-) diff --git a/doc/code_reviews.rst b/doc/code_reviews.rst index 480975b7..42e95fe8 100644 --- a/doc/code_reviews.rst +++ b/doc/code_reviews.rst @@ -1,17 +1,6 @@ Code reviews ------------------ -========================= -Branch protection rules -========================= - -* main and RC_v* --> 2 reviews required -* dev_* --> 1 review required -* exp_* --> no reviews required - -* dev_* should be created from RC_v* -* exp_* should be created from dev_* - ============================ How to do a code review ============================ diff --git a/doc/code_structure.rst b/doc/code_structure.rst index 6b515451..37f6c838 100644 --- a/doc/code_structure.rst +++ b/doc/code_structure.rst @@ -5,9 +5,38 @@ Code structure and key design koncepts Modules ================================== -- Explain the modular framework -- Explain the responsibilities of each submodule -- Future plan: create base classes for the main operations like feature detection/ tracking +**tobac** aims to provide a flexible and modular framework which can be seen as a toolbox to create tracking algorithms according to the user's specific research needs. + + +The **tobac** package currently consists of three main modules: + +1. The :py:mod:`tobac.feature_detection` contains methods to identify features/objects in 2D or 3D (3D or 4D when including the time dimensions) gridded data. This is done by identifying contiguous regions above/below on one or multiple user-defined thresholds. The module makes use of :py:mod:`scipy.ndimage.label`, a generic image processing method that labels features in an array. The methods in this module are high-level functions that enable a fast and effective feature detection and create a user-friendly output with information on each detected feature. The method that is the most important for users is currently `tobac.feature_detection_multithreshold`. + + +2. The :py:mod:`tobac.segmentation` contains methods to define the extent of the identified feature areas or volumes. This is currently done by using watershdding, but more methods are to be implemented. Note that this module can handle both 2D and 3D data. + + +3. The :py:mod:`tobac.tracking` module is responsible for the code that links identified features over time. This module makes use of the python package :py:mod:`trackpy`. Note that currently the linking is based on particle tracking principles which means that only the feature center positions are needed to link features over time. Other methods such as tracking based on overlapping areas from the segmented features are to be implemented. + + +In addition to the main modules, there are three postprocessing modules: + +4. The :py:mod:`tobac.merge_split` module provides functionality to identify mergers and splitters in the tracking output and to add labels such that one can reconstruct the parent and child tracks of each cell. + +5. The :py:mod:`tobac.analysis` module contains methods to analyze the tracking output and derive statistics about individual tracks as well as summary statistics of the entire populations of tracks or subsets of the latter. + +6. The :py:mod:`tobac.plotting` module provides methods to create plots, in particular maps and animations, of the tracking output. + + +Finally, there are two modules that are particularly important for developers: + +7. The :py:mod:`tobac.utils` module is a collection of smaller not necessarily tracking-specific methods that facilitate and support the methods of the main modules. This module has multiple submodules and the most important here is the separation between methods that are more generic and could be used by users as needed (:py:mod:`tobac.utils.general`) and methods that facilitate the development of **tobac** and are therefore primarily for internal use (:py:mod:`tobac.utils.internal`). Sometimes, new features come with the need of a whole set of nee methods, so it could make sense to save these in their own module (e.g. :py:mod:`tobac.periodic_boundaries`) + +8. The :py:mod:`tobac.testing` module provides support for writing of unit tests. In particular, it contains several methods to create simplified test data sets on which the various methods and parameters for feature detection, segmentation, and tracking can be tested. + +For more information on each submodule, refer to the respective source code documentation. + +One thing to note is that **tobac** as of now is purely functional. The plan is, however, to move towards a more object-oriented design with base classes for the main operations such as feature detection and tracking. ============================ Migrating to xarray and dask @@ -17,3 +46,13 @@ Migrating to xarray and dask - How these are or could be used in tobac - How to work on this + + + + + + + + + + diff --git a/doc/contributing.rst b/doc/contributing.rst index 05acd587..71171452 100644 --- a/doc/contributing.rst +++ b/doc/contributing.rst @@ -1,11 +1,11 @@ How to contribute ------------------- +------------------------- ========================= Code of conduct ========================= -* Link to code of conduct: https://www.python.org/psf/conduct/ +* We, the **tobac** community, value diversity and acknowledge that we all come with different experiences and capacities. We strive to foster an inclusive and respectful space for collaboration. To achieve this, we would like to refer to the `Python code of conduct `_ as we follow the same principles for communicating and interacting with each other! ========================= git basics @@ -13,26 +13,65 @@ git basics * Create a Github account -* Branching (link to git tutorials) +The first thing, you need to do is to `create a GitHub account `_ if you do not already have one. -* Create an issue (or select one, look for "good first issue" labels) +* Get familiar with the basics of GitHub and git: -* Link to CONTRIBUTING.md +- `Getting started < https://docs.github.com/en/get-started/quickstart/hello-world>`_ +- `Learn about branches `_ +- `Learn about forks `_ +- `Learn about pull requests < https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests>`_ +- `Learn how to commit and push changes from your local repository< https://docs.github.com/en/migrations/importing-source-code/using-the-command-line-to-import-source-code/adding-locally-hosted-code-to-github>`_ + +* Create an issue + +If you have an idea for a new feature or a suggestion for any kind of code changes, please create an issue for this. We use `our issues `_ to milestones `Link text `_, i.e. the different versions of **tobac** to come. +The issues act, therefore, not only as a place for reporting bugs, but also as a collection of *to do* points. + +You can also work on any issue that was created by somebody else and is already out there. A tip is to look for the **good first issue** label, if you are a new developer. These issues are usually fairly easy to address and can be good to practice our GitHub workflow. + +* Link to CONTRIBUTING.md (?) + +* Tips when working with your local code + +Install tobac package with pip install -e . This allows you to directly test your local code changes as you run tobac. Instead of using the **tobac** version of the latest release, your local version of tobac will be used when you import **tobac** in a python script or IDE. Note that this way of installing a local package will use the code of the checked in branch, so this allows you also to test code while switching between branches. + +- How to locally build the documentation page + + +* Create a pull request from your fork + +We use our personal forks of the tobac repository to create pull requests. This means that you have to first commit and push your local changes to your personal fork and then create a pull request from that fork: + +https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork -* Create a pull request from your fork ========================= Branching strategy ========================= -* RC_* -* dev_* -* exp_* +While you can use any type of branching strategy and naming as you work in your personal fork, we have three branches in the tobac repository: + +* `RC_*` +* `dev_*` +* `hotfix` + +`RC_*` is the release candidate of the next tobac version. The asterisk stands here for the specific tobac version: RC_vx.x.x (e.g. RC_v1.5.0). Pull requests to this branch need two reviews to be accepted before it can be merged into main. + +`dev_*` is the development branch where we experiment with new features. This branch is perfectly suited to collaboratively work on a feature together with other **tobac** developers (see :doc:`mentoring`). In general, this branch is used for long-term, comprehensive code changes that might not be covered by a single pull request and where it might not be conceivable in which future **tobac** version to include the changes. There are no branch protection rules for this branch, which means that collaborators of our GitHub organization can directly push changes to this branch. Note that **dev_** can never directly merged into main, it has be merged into the release candidate branch RC_* first! There can be more than one `dev_*` branch, therefore it we recommand to describe the feature to work on in the respective branch (e.g. `dev_xarray_transition`). + +`hotfix` is the branch we use for hotfixes, i.e. bug fixes that need to be released as fast as possible because it influences people's code. This branch needs only one review before it can directly merged into `main`. + +After a release, we need to make sure that the latest changes of main are merged back into existing `RC_*` and `dev_*` branches and that the branch names are updated accordingly (e.g. replace `RC_v1.5.0` with the next version to come). The hotfix is directly deleted when merged into main. It can be re-created whenever a new hotfix comes up. + +The most important thing to remember from this when you are a new contributor: **The common branch to submit the pull request to is the next release candidate RC_v.x.x.x** + ========================= Writing unit tests ========================= +* CI and other GitHub actions (code formatting etc. ) * Pytest and pytest fixtures * Test coverage @@ -40,5 +79,12 @@ Writing unit tests Add examples ========================= -* jupyter notebooks -* upload example data to zenodo +* Jupyter notebooks +* Upload example data to zenodo + +========================= +Releasing a new version +========================= + + + diff --git a/doc/mentoring.rst b/doc/mentoring.rst index 02e14ff3..16ba0272 100644 --- a/doc/mentoring.rst +++ b/doc/mentoring.rst @@ -1,17 +1,23 @@ -Mentoring ----------- - +Mentoring and Collaboration +---------------------------- ============================ Writing code collaboratively ============================ +We firmly believe that code can only get better if more than two eyes and one brain work on it. Therefore, we aim to write code collaboratively, in particular, when comprehensive refactoring or enhancements of the code are done. In practice, this can be done by creating a *draft pull request* on the `dev*` branch. As explained in our branch strategy, this branch does not undergo any protection rules, so it is meant to experiment with new code and all contributors can directly push to this branch. Creating a draft pull request has the advantage that you have a platform to communicate with others who contribute to the new feature. You can directly see which changes they make, comment these and discuss ways to go forward. ============== Get a mentor ============== +Is it your first time contributing to an open-source project? + +Reach out to the **tobac** developer group and get a mentor, one of the developers, who helps you getting started and explain to you how our workflow works. You are always free to post any questions to GitHub discussions, our Slack channel, or write an email. But sometimes it can be nice to have a specific person to refer to when things seem overwhelming in the beginning. + =============== Pair reviews =============== + +Another great way of collaboration are pair reviews which means that you are reviewing code together with another developer. You can, for example, reach out to us when you have submitted a pull request and would like to talk through the review points with one of the reviewers in order to collaboratively come up with creative solutions to remaining issues. If you are a reviewer, you can offer a pair review to the person who created the pull request and help them addressing certain review points. From 85bcb938a0090d326a3634258b23e97950cd6fd6 Mon Sep 17 00:00:00 2001 From: JuliaKukulies Date: Thu, 8 Jun 2023 15:32:54 +0000 Subject: [PATCH 03/25] fill in some more --- doc/code_structure.rst | 26 ++++++-------- doc/contributing.rst | 82 ++++++++++++++++++++++++------------------ doc/mentoring.rst | 7 ++-- 3 files changed, 61 insertions(+), 54 deletions(-) diff --git a/doc/code_structure.rst b/doc/code_structure.rst index 37f6c838..cedf1ed7 100644 --- a/doc/code_structure.rst +++ b/doc/code_structure.rst @@ -7,32 +7,28 @@ Modules **tobac** aims to provide a flexible and modular framework which can be seen as a toolbox to create tracking algorithms according to the user's specific research needs. +The **tobac** package currently consists of three **main modules**: -The **tobac** package currently consists of three main modules: +1. The :py:mod:`tobac.feature_detection` contains methods to identify objects (*features*) in 2D or 3D (3D or 4D when including the time dimensions) gridded data. This is done by identifying contiguous regions above or below one or multiple user-defined thresholds. The module makes use of :py:mod:`scipy.ndimage.label`, a generic image processing method that labels features in an array. The methods in :py:mod:`tobac.feature_detection` are high-level functions that enable a fast and effective feature detection and create easy-to-use output in form of a :py:mod:`pandas.DataFrame` that contains the coordinates and some basic information on each detected feature. The most high-level methods that is commonly used by users is `tobac.feature_detection_multithreshold`. -1. The :py:mod:`tobac.feature_detection` contains methods to identify features/objects in 2D or 3D (3D or 4D when including the time dimensions) gridded data. This is done by identifying contiguous regions above/below on one or multiple user-defined thresholds. The module makes use of :py:mod:`scipy.ndimage.label`, a generic image processing method that labels features in an array. The methods in this module are high-level functions that enable a fast and effective feature detection and create a user-friendly output with information on each detected feature. The method that is the most important for users is currently `tobac.feature_detection_multithreshold`. +2. The :py:mod:`tobac.segmentation` module contains methods to define the extent of the identified feature areas or volumes. This step is needed to create a mask of the identified features because the feature detection currently only saves the center points of the features. The segmentation procedure is performed by using the watershedding method, but more methods are to be implemented in the future. Just as the feature detection, this module can handle both 2D and 3D data. +3. The :py:mod:`tobac.tracking` module is responsible for linking identified features over time. This module makes primarily use of the python package :py:mod:`trackpy`. Note that the linking using :py:mod:`trackpy` is based on particle tracking principles which means that only the feature center positions (not the entire area or volume associated with each feature) are needed to link features over time. Other methods such as tracking based on overlapping areas from the segmented features are to be implemented. -2. The :py:mod:`tobac.segmentation` contains methods to define the extent of the identified feature areas or volumes. This is currently done by using watershdding, but more methods are to be implemented. Note that this module can handle both 2D and 3D data. - - -3. The :py:mod:`tobac.tracking` module is responsible for the code that links identified features over time. This module makes use of the python package :py:mod:`trackpy`. Note that currently the linking is based on particle tracking principles which means that only the feature center positions are needed to link features over time. Other methods such as tracking based on overlapping areas from the segmented features are to be implemented. - - -In addition to the main modules, there are three postprocessing modules: +In addition to the main modules, there are three **postprocessing modules**: 4. The :py:mod:`tobac.merge_split` module provides functionality to identify mergers and splitters in the tracking output and to add labels such that one can reconstruct the parent and child tracks of each cell. 5. The :py:mod:`tobac.analysis` module contains methods to analyze the tracking output and derive statistics about individual tracks as well as summary statistics of the entire populations of tracks or subsets of the latter. -6. The :py:mod:`tobac.plotting` module provides methods to create plots, in particular maps and animations, of the tracking output. - +6. The :py:mod:`tobac.plotting` module provides methods to visualize the tracking output, for example for creating maps and animations of identified features, segmented areas and tracks. -Finally, there are two modules that are particularly important for developers: + +Finally, there are two modules that are primarily **important for developers**: -7. The :py:mod:`tobac.utils` module is a collection of smaller not necessarily tracking-specific methods that facilitate and support the methods of the main modules. This module has multiple submodules and the most important here is the separation between methods that are more generic and could be used by users as needed (:py:mod:`tobac.utils.general`) and methods that facilitate the development of **tobac** and are therefore primarily for internal use (:py:mod:`tobac.utils.internal`). Sometimes, new features come with the need of a whole set of nee methods, so it could make sense to save these in their own module (e.g. :py:mod:`tobac.periodic_boundaries`) +7. The :py:mod:`tobac.utils` module is a collection of smaller, not necessarily tracking-specific methods that facilitate and support the methods of the main modules. This module has multiple submodules. We separate methods that are rather generic and could also be practical for tobac users who build their own tracking algorithms (:py:mod:`tobac.utils.general`) and methods that mainly facilitate the development of **tobac** (:py:mod:`tobac.utils.internal`). Sometimes, new features come with the need of a whole set of new methods, so it could make sense to save these in their own submodule (see e.g. :py:mod:`tobac.periodic_boundaries`) -8. The :py:mod:`tobac.testing` module provides support for writing of unit tests. In particular, it contains several methods to create simplified test data sets on which the various methods and parameters for feature detection, segmentation, and tracking can be tested. +8. The :py:mod:`tobac.testing` module provides support for writing of unit tests. This module contains several methods to create simplified test data sets on which the various methods and parameters for feature detection, segmentation, and tracking can be tested. For more information on each submodule, refer to the respective source code documentation. @@ -44,7 +40,7 @@ Migrating to xarray and dask - Basics of xarray (xarray.Dataarray class) and dask - How these are or could be used in tobac -- How to work on this +- How to work on the transition diff --git a/doc/contributing.rst b/doc/contributing.rst index 71171452..44a8c621 100644 --- a/doc/contributing.rst +++ b/doc/contributing.rst @@ -1,3 +1,6 @@ +.. + How to contribute to the tobac project + How to contribute ------------------------- @@ -5,73 +8,70 @@ How to contribute Code of conduct ========================= -* We, the **tobac** community, value diversity and acknowledge that we all come with different experiences and capacities. We strive to foster an inclusive and respectful space for collaboration. To achieve this, we would like to refer to the `Python code of conduct `_ as we follow the same principles for communicating and interacting with each other! +We are a multi-institutional and international community that aims to maintain and increase our diversity. We acknowledge that we all come with different experiences and capacities. Therefore, we strive to foster an inclusive and respectful environmentwhere we help and support each other. We welcome any types of contributions and believe that we together can create accessible, reusable, and maintanable code that empowers researchers and enables groundbreaking science. + +We would like to refer to the `Python code of conduct `_ as we follow the same principlesfor communication and working with each other! ========================= git basics ========================= -* Create a Github account - -The first thing, you need to do is to `create a GitHub account `_ if you do not already have one. +* **Create a Github account**: The first thing, you need to do is to `create a GitHub account `_ if you do not already have one. -* Get familiar with the basics of GitHub and git: +* **Get familiar with the basics of GitHub and git**: + * Getting started with the `basics `_ + * Learn about `branches `_ + * Learn about `forks `_ + * Learn about `pull requests `_ + * Learn about `how to commit and push changes from your local repository `_ -- `Getting started < https://docs.github.com/en/get-started/quickstart/hello-world>`_ -- `Learn about branches `_ -- `Learn about forks `_ -- `Learn about pull requests < https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests>`_ -- `Learn how to commit and push changes from your local repository< https://docs.github.com/en/migrations/importing-source-code/using-the-command-line-to-import-source-code/adding-locally-hosted-code-to-github>`_ - -* Create an issue - -If you have an idea for a new feature or a suggestion for any kind of code changes, please create an issue for this. We use `our issues `_ to milestones `Link text `_, i.e. the different versions of **tobac** to come. +* **Create an issue**: If you have an idea for a new feature or a suggestion for any kind of code changes, please create an issue for this. We use `our issues `_ to milestones `Link text `_, i.e. the different versions of **tobac** to come. The issues act, therefore, not only as a place for reporting bugs, but also as a collection of *to do* points. You can also work on any issue that was created by somebody else and is already out there. A tip is to look for the **good first issue** label, if you are a new developer. These issues are usually fairly easy to address and can be good to practice our GitHub workflow. -* Link to CONTRIBUTING.md (?) - -* Tips when working with your local code +* https://github.com/tobac-project/tobac/blob/main/CONTRIBUTING.md -Install tobac package with pip install -e . This allows you to directly test your local code changes as you run tobac. Instead of using the **tobac** version of the latest release, your local version of tobac will be used when you import **tobac** in a python script or IDE. Note that this way of installing a local package will use the code of the checked in branch, so this allows you also to test code while switching between branches. +* **Tips when working with your local code:** -- How to locally build the documentation page + * Install tobac package with :code:`pip install -e` . This allows you to directly test your local code changes as you run tobac. Instead of using the **tobac** version of the latest release, your local version of tobac will be used when you import **tobac** in a python script or IDE. Note that this way of installing a local package will use the code of the checked in branch, so this allows you also to test code while switching between branches. + * You can locally build the documentation page: -* Create a pull request from your fork - -We use our personal forks of the tobac repository to create pull requests. This means that you have to first commit and push your local changes to your personal fork and then create a pull request from that fork: - -https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork +* **Create a pull request from your fork:** We use our personal forks of the tobac repository to create pull requests. This means that you have to first commit and push your local changes to your personal fork and then create a pull request from that fork: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork ========================= -Branching strategy +Our branching strategy ========================= While you can use any type of branching strategy and naming as you work in your personal fork, we have three branches in the tobac repository: -* `RC_*` -* `dev_*` -* `hotfix` +* :code:``RC_*` +* :code:``dev_*` +* :code:``hotfix` + +:code:``RC_*` is the release candidate of the next tobac version. The asterisk stands here for the specific tobac version: RC_vx.x.x (e.g. RC_v1.5.0). Pull requests to this branch need two reviews to be accepted before it can be merged into main. -`RC_*` is the release candidate of the next tobac version. The asterisk stands here for the specific tobac version: RC_vx.x.x (e.g. RC_v1.5.0). Pull requests to this branch need two reviews to be accepted before it can be merged into main. +:code:``dev_*` is the development branch where we experiment with new features. This branch is perfectly suited to collaboratively work on a feature together with other **tobac** developers (see :doc:`mentoring`). In general, this branch is used for long-term, comprehensive code changes that might not be covered by a single pull request and where it might not be conceivable in which future **tobac** version to include the changes. There are no branch protection rules for this branch, which means that collaborators of our GitHub organization can directly push changes to this branch. Note that **dev_** can never directly merged into main, it has be merged into the release candidate branch RC_* first! There can be more than one `dev_*` branch, therefore it we recommand to describe the feature to work on in the respective branch (e.g. :code:`dev_xarray_transition`). -`dev_*` is the development branch where we experiment with new features. This branch is perfectly suited to collaboratively work on a feature together with other **tobac** developers (see :doc:`mentoring`). In general, this branch is used for long-term, comprehensive code changes that might not be covered by a single pull request and where it might not be conceivable in which future **tobac** version to include the changes. There are no branch protection rules for this branch, which means that collaborators of our GitHub organization can directly push changes to this branch. Note that **dev_** can never directly merged into main, it has be merged into the release candidate branch RC_* first! There can be more than one `dev_*` branch, therefore it we recommand to describe the feature to work on in the respective branch (e.g. `dev_xarray_transition`). +:code:``hotfix` is the branch we use for hotfixes, i.e. bug fixes that need to be released as fast as possible because it influences people's code. This branch needs only one review before it can directly merged into :code:``main`. -`hotfix` is the branch we use for hotfixes, i.e. bug fixes that need to be released as fast as possible because it influences people's code. This branch needs only one review before it can directly merged into `main`. +In brief: **Unless you are collaboratively working on a comprehensive feature or on a hotfix, the branch to submit your pull request to is the next release candidate RC_v.x.x.x** -After a release, we need to make sure that the latest changes of main are merged back into existing `RC_*` and `dev_*` branches and that the branch names are updated accordingly (e.g. replace `RC_v1.5.0` with the next version to come). The hotfix is directly deleted when merged into main. It can be re-created whenever a new hotfix comes up. -The most important thing to remember from this when you are a new contributor: **The common branch to submit the pull request to is the next release candidate RC_v.x.x.x** +========================= +GitHub workflow +========================= +* Briefly describe how CI works +* Other github actions such as code format check ========================= Writing unit tests ========================= -* CI and other GitHub actions (code formatting etc. ) +* Make use of :py:mod:`tobac.testing` * Pytest and pytest fixtures * Test coverage @@ -86,5 +86,17 @@ Add examples Releasing a new version ========================= - +This is the checklist of steps for a release of a new **tobac** version: + +* Bump version in `__init__.py `in :code:`hotfix` +* Add changelog in :code:`hotfix` +* Regenerate example notebooks with the new version +* Get the two additional bug fixes into :code:`hotfix` +* Merge :code:`hotfix` into :code:`main` +* Merge :code:`main` into release and dev branches +* Delete :code:`hotfix` branch +* Create release +* Push release to conda-forge: https://github.com/tobac-project/tobac-notes/blob/main/uploading_to_conda-forge.md +* E-mail tobac mailing list +* Create new tag diff --git a/doc/mentoring.rst b/doc/mentoring.rst index 16ba0272..da347431 100644 --- a/doc/mentoring.rst +++ b/doc/mentoring.rst @@ -5,16 +5,15 @@ Mentoring and Collaboration Writing code collaboratively ============================ -We firmly believe that code can only get better if more than two eyes and one brain work on it. Therefore, we aim to write code collaboratively, in particular, when comprehensive refactoring or enhancements of the code are done. In practice, this can be done by creating a *draft pull request* on the `dev*` branch. As explained in our branch strategy, this branch does not undergo any protection rules, so it is meant to experiment with new code and all contributors can directly push to this branch. Creating a draft pull request has the advantage that you have a platform to communicate with others who contribute to the new feature. You can directly see which changes they make, comment these and discuss ways to go forward. +We firmly believe that code can only get better if more than two eyes and one brain work on it. Therefore, we aim to write code collaboratively, in particular, when comprehensive refactoring or enhancements of the code are done. In practice, this can be done by creating a *draft pull request* on the :code:`dev_*` branch. As explained in our :doc:`our branching strategy `, this branch does not undergo any protection rules. It is meant to experiment with new code and all collaborators of the `tobac-project organization `_ can directly push to this branch. Creating a draft pull request has the advantage of facilitating the communication with other developers who contribute to the same new feature. You can directly see which changes they make, comment these and discuss ways to go forward. ============== Get a mentor ============== -Is it your first time contributing to an open-source project? - -Reach out to the **tobac** developer group and get a mentor, one of the developers, who helps you getting started and explain to you how our workflow works. You are always free to post any questions to GitHub discussions, our Slack channel, or write an email. But sometimes it can be nice to have a specific person to refer to when things seem overwhelming in the beginning. +**Is this your first time contributing to an open-source project?** +Reach out to the **tobac** developer group and get a mentor! One of our developers will help you getting started and explain how our workflow works. You are, of course, always free to post any questions to GitHub discussions, our Slack channel, or write an email. But sometimes it can also be nice to have a specific person to refer to when things seem overwhelming in the beginning. =============== Pair reviews From e0bed08cffe3d2950ac08129515ffad104e04593 Mon Sep 17 00:00:00 2001 From: JuliaKukulies Date: Thu, 8 Jun 2023 15:58:41 +0000 Subject: [PATCH 04/25] fixing bullet points --- doc/contributing.rst | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/doc/contributing.rst b/doc/contributing.rst index 44a8c621..aed3aca8 100644 --- a/doc/contributing.rst +++ b/doc/contributing.rst @@ -19,11 +19,11 @@ git basics * **Create a Github account**: The first thing, you need to do is to `create a GitHub account `_ if you do not already have one. * **Get familiar with the basics of GitHub and git**: - * Getting started with the `basics `_ - * Learn about `branches `_ - * Learn about `forks `_ - * Learn about `pull requests `_ - * Learn about `how to commit and push changes from your local repository `_ + * Getting started with the `basics `_ + * Learn about `branches `_ + * Learn about `forks `_ + * Learn about `pull requests `_ + * Learn about `how to commit and push changes from your local repository `_ * **Create an issue**: If you have an idea for a new feature or a suggestion for any kind of code changes, please create an issue for this. We use `our issues `_ to milestones `Link text `_, i.e. the different versions of **tobac** to come. The issues act, therefore, not only as a place for reporting bugs, but also as a collection of *to do* points. @@ -31,16 +31,17 @@ The issues act, therefore, not only as a place for reporting bugs, but also as a You can also work on any issue that was created by somebody else and is already out there. A tip is to look for the **good first issue** label, if you are a new developer. These issues are usually fairly easy to address and can be good to practice our GitHub workflow. * https://github.com/tobac-project/tobac/blob/main/CONTRIBUTING.md + +* **Create a pull request from your fork:** We use our personal forks of the tobac repository to create pull requests. This means that you have to first commit and push your local changes to your personal fork and then create a pull request from that fork: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork -* **Tips when working with your local code:** - - * Install tobac package with :code:`pip install -e` . This allows you to directly test your local code changes as you run tobac. Instead of using the **tobac** version of the latest release, your local version of tobac will be used when you import **tobac** in a python script or IDE. Note that this way of installing a local package will use the code of the checked in branch, so this allows you also to test code while switching between branches. - * You can locally build the documentation page: +* **Tips when working with your local code:** + * Install tobac package with :code:`pip install -e` . This allows you to directly test your local code changes as you run tobac. Instead of using the **tobac** version of the latest release, your local version of tobac will be used when you import **tobac** in a python script or IDE. Note that this way of installing a local package will use the code of the checked in branch, so this allows you also to test code while switching between branches. -* **Create a pull request from your fork:** We use our personal forks of the tobac repository to create pull requests. This means that you have to first commit and push your local changes to your personal fork and then create a pull request from that fork: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork + * You can locally build the documentation page: + ========================= Our branching strategy ========================= From 48756ce0a87708eb6894051ddb207fae57b9664c Mon Sep 17 00:00:00 2001 From: JuliaKukulies Date: Thu, 8 Jun 2023 16:13:03 +0000 Subject: [PATCH 05/25] add some things for code review --- doc/code_reviews.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/doc/code_reviews.rst b/doc/code_reviews.rst index 42e95fe8..7cbd2f50 100644 --- a/doc/code_reviews.rst +++ b/doc/code_reviews.rst @@ -1,16 +1,31 @@ Code reviews ------------------ +Before anything is merged into the release branch (:code:`RC_*`), we require that two reviewers accept the code changes of a pull request. + ============================ How to do a code review ============================ +* Checkout out pull request locally + +* Run tests locally + +* Go through code and see if it is readable and easy to understand + +* Not required, but often useful: test new features with your own data ========================= Expectations ========================= + ========================= Conventional comments ========================= + +The comments in a code review should be clear and constructive. + +A useful way of highlighting the intention of a specific comments is to label them according to `conventional comments `_. + From 1dfd5d0bf6aeef957df365b12fc5740c11b24e65 Mon Sep 17 00:00:00 2001 From: Juli Date: Tue, 13 Jun 2023 11:18:58 -0600 Subject: [PATCH 06/25] added suggestions from Kolya --- doc/code_reviews.rst | 11 ++++++++--- doc/contributing.rst | 12 ++++++++---- doc/mentoring.rst | 5 ++++- 3 files changed, 20 insertions(+), 8 deletions(-) diff --git a/doc/code_reviews.rst b/doc/code_reviews.rst index 7cbd2f50..e35bb8bc 100644 --- a/doc/code_reviews.rst +++ b/doc/code_reviews.rst @@ -15,10 +15,15 @@ How to do a code review * Not required, but often useful: test new features with your own data -========================= -Expectations -========================= +============================ +Tips and expectations +============================ + + +Doing a code review can be very challenging if you are unfamiliar with the process. Here is a set of documents might provide a good resource on how to get started: + +https://github.com/google/eng-practices ========================= diff --git a/doc/contributing.rst b/doc/contributing.rst index aed3aca8..3ab327df 100644 --- a/doc/contributing.rst +++ b/doc/contributing.rst @@ -35,13 +35,17 @@ You can also work on any issue that was created by somebody else and is already * **Create a pull request from your fork:** We use our personal forks of the tobac repository to create pull requests. This means that you have to first commit and push your local changes to your personal fork and then create a pull request from that fork: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork -* **Tips when working with your local code:** - * Install tobac package with :code:`pip install -e` . This allows you to directly test your local code changes as you run tobac. Instead of using the **tobac** version of the latest release, your local version of tobac will be used when you import **tobac** in a python script or IDE. Note that this way of installing a local package will use the code of the checked in branch, so this allows you also to test code while switching between branches. +=================================== +Tips on working on your local code +=================================== - * You can locally build the documentation page: +* Install tobac package with :code:`pip install -e` . This allows you to directly test your local code changes as you run tobac. Instead of using the **tobac** version of the latest release, your local version of tobac will be used when you import **tobac** in a python script or IDE. Note that this way of installing a local package will use the code of the checked in branch, so this allows you also to test code while switching between branches. + +* You can locally build the documentation page: + +* Writing `meaningful commit messages `_ can be very helpful for you and people who review your code to better understand the code changes. - ========================= Our branching strategy ========================= diff --git a/doc/mentoring.rst b/doc/mentoring.rst index da347431..98f1f984 100644 --- a/doc/mentoring.rst +++ b/doc/mentoring.rst @@ -5,7 +5,10 @@ Mentoring and Collaboration Writing code collaboratively ============================ -We firmly believe that code can only get better if more than two eyes and one brain work on it. Therefore, we aim to write code collaboratively, in particular, when comprehensive refactoring or enhancements of the code are done. In practice, this can be done by creating a *draft pull request* on the :code:`dev_*` branch. As explained in our :doc:`our branching strategy `, this branch does not undergo any protection rules. It is meant to experiment with new code and all collaborators of the `tobac-project organization `_ can directly push to this branch. Creating a draft pull request has the advantage of facilitating the communication with other developers who contribute to the same new feature. You can directly see which changes they make, comment these and discuss ways to go forward. +We firmly believe that code can only get better if more than two eyes and one brain work on it. Therefore, we aim to write code collaboratively, in particular, when comprehensive refactoring or enhancements of the code are done. In practice, this can be done by creating a **draft pull request**. This makes it really easy to iteratively improve a pull request with the feedback from others until the pull request is ready for review. + + +When you work on a comprehensive feature with multiple developers, it is recommended to create a draft pull request on the :code:`dev_*` branch. As explained in our :doc:`our branching strategy `, this branch does not undergo any protection rules. It is meant to experiment with new code and all collaborators of the `tobac-project organization `_ can directly push to this branch. Creating a draft pull request has the advantage of facilitating the communication with other developers who contribute to the same new feature. You can directly see which changes they make, comment these and discuss ways to go forward. ============== Get a mentor From b6f87e7c771e93e130fe767a72c186baeb95c67f Mon Sep 17 00:00:00 2001 From: Kolya Lettl Date: Thu, 15 Jun 2023 15:02:21 +0200 Subject: [PATCH 07/25] fix: small grammar mistake --- doc/code_reviews.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/code_reviews.rst b/doc/code_reviews.rst index e35bb8bc..e74a5b9f 100644 --- a/doc/code_reviews.rst +++ b/doc/code_reviews.rst @@ -21,7 +21,7 @@ Tips and expectations ============================ -Doing a code review can be very challenging if you are unfamiliar with the process. Here is a set of documents might provide a good resource on how to get started: +Doing a code review can be very challenging if you are unfamiliar with the process. Here is a set of documents which might provide a good resource on how to get started: https://github.com/google/eng-practices From 60853b3b2351881261508f3b36bac7e7be52de2c Mon Sep 17 00:00:00 2001 From: Kolya Lettl Date: Thu, 15 Jun 2023 15:06:56 +0200 Subject: [PATCH 08/25] fix: small grammar mistake --- doc/code_reviews.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/code_reviews.rst b/doc/code_reviews.rst index e74a5b9f..922311d4 100644 --- a/doc/code_reviews.rst +++ b/doc/code_reviews.rst @@ -32,5 +32,5 @@ Conventional comments The comments in a code review should be clear and constructive. -A useful way of highlighting the intention of a specific comments is to label them according to `conventional comments `_. +A useful way of highlighting the intention of specific comments is to label them according to `conventional comments `_. From 79157486b0521de27d46e775846caff1f60240c0 Mon Sep 17 00:00:00 2001 From: Kolya Lettl Date: Thu, 15 Jun 2023 15:34:04 +0200 Subject: [PATCH 09/25] chore: added link to github's documentation on how to checkout a pr locally --- doc/code_reviews.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/code_reviews.rst b/doc/code_reviews.rst index 922311d4..da8c2c90 100644 --- a/doc/code_reviews.rst +++ b/doc/code_reviews.rst @@ -7,7 +7,7 @@ Before anything is merged into the release branch (:code:`RC_*`), we require tha How to do a code review ============================ -* Checkout out pull request locally +* Checkout out pull request locally (`how to checkout a pull request locally `_) * Run tests locally From 1146f1cb2a47e70fee4f4206dcfd821076c98969 Mon Sep 17 00:00:00 2001 From: Fabian Senf Date: Fri, 7 Jul 2023 15:51:12 +0200 Subject: [PATCH 10/25] added description for sphinx-based rendering now formatted as RST file --- doc/testing_sphinx-based_rendering.rst | 140 +++++++++++++++++++++++++ 1 file changed, 140 insertions(+) create mode 100644 doc/testing_sphinx-based_rendering.rst diff --git a/doc/testing_sphinx-based_rendering.rst b/doc/testing_sphinx-based_rendering.rst new file mode 100644 index 00000000..4ea6c0b4 --- /dev/null +++ b/doc/testing_sphinx-based_rendering.rst @@ -0,0 +1,140 @@ +How to Check if Sphinx-based Rendering of ``tobac`` is Working? +=============================================================== + +The workflow has been tested in a linux system. We aim to build a static +website out of the documentation material present in ``tobac``. + +1. Preparing the Local Environment +---------------------------------- + +- **choose a separate place for your testing** + + I will use the temporary directory ``/tmp/website-testing`` which I + need to create. You can use a dedicated place of your choice … + + .. code:: bash + + > mkdir /tmp/website-testing + > cd /tmp/website-testing + + I will indicate my position now with the ``/tmp/website-testing>`` + prompt. + +- **get the official repository** + + .. code:: bash + + /tmp/website-testing> git clone https://github.com/tobac-project/tobac + + You might like to test a certain remote branch ```` then do: + + .. code:: bash + + git fetch --all + git checkout -t origin/ + +- **Python environment** + + - create a python virtual env + ``bash /tmp/website-testing> python -m venv .python3-venv`` + + - and install requirements \```bash # deactivation conda is only + necessary if your loaded conda before … /tmp/website-testing> + conda deactivate + + # activate the new env and upgrade ``pip`` /tmp/website-testing> + source .python3-venv/bin/activate /tmp/website-testing> pip + install –upgrade pip + + # now everything is installed into the local python env! + /tmp/website-testing> pip install -r tobac/doc/requirements.txt + + # and also install RTD scheme /tmp/website-testing> pip install + sphinx_rtd_theme + + \`\`\ ````\ pip\ ``-based installation takes a bit of time, but is much faster than``\ conda`. + If the installation runs without problems, you are ready to build + the website. + +2. Building the Website +----------------------- + +Actually, only few steps are needed to build the website, i.e. + +- **running sphinx for rendering** + + .. code:: bash + + /tmp/website-testing> cd tobac + + /tmp/website-testing/tobac> sphinx-build -b html doc doc/_build/html + + If no severe error appeared + +- **view the HTML content** + + .. code:: bash + + /tmp/website-testing/tobac> firefox doc/_build/html/index.html + +3. Parsing Your Local Changes +----------------------------- + +Now, we connect to your locally hosted ``tobac`` repository and your +development branch. + +- **connect to your local repo**: Assume your repo is located at + ``/tmp/tobac-testing/tobac``, then add a new remote alias and fetch + all content with + + .. code:: bash + + /tmp/website-testing/tobac> git remote add local-repo /tmp/tobac-testing/tobac + /tmp/website-testing/tobac> git fetch --all + +- **check your development branch out**: Now, assume the your + development branch is called ``my-devel``, then do + + .. code:: bash + + # to get a first overview on available branches + /tmp/website-testing/tobac> git branch --all + + # and then actually get your development branch + /tmp/website-testing/tobac> git checkout -b my-devel local-repo/my-devel + + You should see your developments, now … + +- **build and view website again** + + .. code:: bash + + /tmp/website-testing/tobac> sphinx-build -M clean doc doc/_build + /tmp/website-testing/tobac> sphinx-build -b html doc doc/_build/html + /tmp/website-testing/tobac> firefox _build/html/index.html + + +Option: Check Rendering of a Pull requests +------------------------------------------ + +- **check the pull request out**: Now, assume the PR has the ID ```` and you define the branch name ``BRANCH_NAME`` as you like + + .. code:: bash + + # to get PR shown as dedicated branch + /tmp/website-testing/tobac> git fetch upstream pull/ID/head:BRANCH_NAME + + # and then actually get this PR as branch + /tmp/website-testing/tobac> git checkout BRANCH_NAME + + You should see the PR now ... + +- **build and view website again** + + .. code:: bash + + /tmp/website-testing/tobac> sphinx-build -M clean doc doc/_build + /tmp/website-testing/tobac> sphinx-build -b html doc doc/_build/html + /tmp/website-testing/tobac> firefox _build/html/index.html + + From 52b20fd54680516fe310b079be2e8c72f2e768bb Mon Sep 17 00:00:00 2001 From: Juli Date: Fri, 7 Jul 2023 18:20:40 -0600 Subject: [PATCH 11/25] added page about sphinx rendering to index.rst --- doc/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/index.rst b/doc/index.rst index 042f6628..f04f4e4c 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -66,6 +66,7 @@ The project is currently being extended by several contributors to include addit code_structure contributing + testing_sphinx-based_rendering code_reviews mentoring From 15e81294adda17ab160a3d7ebbecd71b1cb44684 Mon Sep 17 00:00:00 2001 From: Juli Date: Fri, 7 Jul 2023 18:49:34 -0600 Subject: [PATCH 12/25] check --- doc/testing_sphinx-based_rendering.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/testing_sphinx-based_rendering.rst b/doc/testing_sphinx-based_rendering.rst index 4ea6c0b4..a6e0659c 100644 --- a/doc/testing_sphinx-based_rendering.rst +++ b/doc/testing_sphinx-based_rendering.rst @@ -1,5 +1,6 @@ -How to Check if Sphinx-based Rendering of ``tobac`` is Working? -=============================================================== +How to check the Sphinx-based rendering +--------------------------------------- + The workflow has been tested in a linux system. We aim to build a static website out of the documentation material present in ``tobac``. From 1a000bef46d76cbb173c28811d1bb613490c5c96 Mon Sep 17 00:00:00 2001 From: Fabian Senf Date: Fri, 1 Sep 2023 14:24:37 +0200 Subject: [PATCH 13/25] docs: Revised dev guide - mainly minor spelling and formatting corrections - added exclude pattern to conf.py to limit rendering to the actual content - changed toctree: excl. sphinx-based doc -> it is now linked in contributing.rst --- doc/code_structure.rst | 10 ++++- doc/conf.py | 4 ++ doc/contributing.rst | 35 +++++++++------- doc/index.rst | 1 - doc/mentoring.rst | 2 +- doc/testing_sphinx-based_rendering.rst | 55 ++++++++++++++++---------- 6 files changed, 69 insertions(+), 38 deletions(-) diff --git a/doc/code_structure.rst b/doc/code_structure.rst index cedf1ed7..cea93fdd 100644 --- a/doc/code_structure.rst +++ b/doc/code_structure.rst @@ -9,7 +9,7 @@ Modules The **tobac** package currently consists of three **main modules**: -1. The :py:mod:`tobac.feature_detection` contains methods to identify objects (*features*) in 2D or 3D (3D or 4D when including the time dimensions) gridded data. This is done by identifying contiguous regions above or below one or multiple user-defined thresholds. The module makes use of :py:mod:`scipy.ndimage.label`, a generic image processing method that labels features in an array. The methods in :py:mod:`tobac.feature_detection` are high-level functions that enable a fast and effective feature detection and create easy-to-use output in form of a :py:mod:`pandas.DataFrame` that contains the coordinates and some basic information on each detected feature. The most high-level methods that is commonly used by users is `tobac.feature_detection_multithreshold`. +1. The :py:mod:`tobac.feature_detection` contains methods to identify objects (*features*) in 2D or 3D (3D or 4D when including the time dimensions) gridded data. This is done by identifying contiguous regions above or below one or multiple user-defined thresholds. The module makes use of :py:mod:`scipy.ndimage.label`, a generic image processing method that labels features in an array. The methods in :py:mod:`tobac.feature_detection` are high-level functions that enable a fast and effective feature detection and create easy-to-use output in form of a :py:mod:`pandas.DataFrame` that contains the coordinates and some basic information on each detected feature. The most high-level methods that is commonly used by users is :py:func:`tobac.feature_detection_multithreshold`. 2. The :py:mod:`tobac.segmentation` module contains methods to define the extent of the identified feature areas or volumes. This step is needed to create a mask of the identified features because the feature detection currently only saves the center points of the features. The segmentation procedure is performed by using the watershedding method, but more methods are to be implemented in the future. Just as the feature detection, this module can handle both 2D and 3D data. @@ -34,6 +34,14 @@ For more information on each submodule, refer to the respective source code docu One thing to note is that **tobac** as of now is purely functional. The plan is, however, to move towards a more object-oriented design with base classes for the main operations such as feature detection and tracking. + +======== +Examples +======== + +To help users get started with **tobac** and to demonstrate the various functionalities, **tobac** hosts several detailed and **illustrated examples** in the form of Jupyter notebooks. They are hosted under the directory `examples/` and be executed by the user. + + ============================ Migrating to xarray and dask ============================ diff --git a/doc/conf.py b/doc/conf.py index 2b8c717a..16b21aa5 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -26,6 +26,9 @@ html_static_path = ["_static"] +exclude_patterns = [ + "_build", +] project = "tobac" @@ -34,6 +37,7 @@ # allow dropdowns collapse_navigation = False + # Include our custom CSS (currently for special table config) def setup(app): app.add_css_file("theme_overrides.css") diff --git a/doc/contributing.rst b/doc/contributing.rst index 3ab327df..7cd5eddb 100644 --- a/doc/contributing.rst +++ b/doc/contributing.rst @@ -4,13 +4,16 @@ How to contribute ------------------------- +Step-by-step overview: https://github.com/tobac-project/tobac/blob/main/CONTRIBUTING.md + + ========================= Code of conduct ========================= -We are a multi-institutional and international community that aims to maintain and increase our diversity. We acknowledge that we all come with different experiences and capacities. Therefore, we strive to foster an inclusive and respectful environmentwhere we help and support each other. We welcome any types of contributions and believe that we together can create accessible, reusable, and maintanable code that empowers researchers and enables groundbreaking science. +We are a multi-institutional and international community that aims to maintain and increase our diversity. We acknowledge that we all come with different experiences and capacities. Therefore, we strive to foster an inclusive and respectful environment where we help and support each other. We welcome any types of contributions and believe that we together can create accessible, reusable, and maintanable code that empowers researchers and enables groundbreaking science. -We would like to refer to the `Python code of conduct `_ as we follow the same principlesfor communication and working with each other! +We would like to refer to the `Python code of conduct `_ as we follow the same principles for communication and working with each other! ========================= git basics @@ -25,12 +28,12 @@ git basics * Learn about `pull requests `_ * Learn about `how to commit and push changes from your local repository `_ -* **Create an issue**: If you have an idea for a new feature or a suggestion for any kind of code changes, please create an issue for this. We use `our issues `_ to milestones `Link text `_, i.e. the different versions of **tobac** to come. -The issues act, therefore, not only as a place for reporting bugs, but also as a collection of *to do* points. +* **Create an issue**: If you have an idea for a new feature or a suggestion for any kind of code changes, please create an issue for this. We sort `our issues `_ into `milestones `_ to priorized work and manage our workflow, i.e. the different versions of **tobac** to come. + + The issues act, therefore, not only as a place for reporting bugs, but also as a collection of *to do* points. -You can also work on any issue that was created by somebody else and is already out there. A tip is to look for the **good first issue** label, if you are a new developer. These issues are usually fairly easy to address and can be good to practice our GitHub workflow. +* **Work on an issue**: You can also work on any issue that was created by somebody else and is already out there. A tip is to look for the **good first issue** label, if you are a new developer. These issues are usually fairly easy to address and can be good to practice our GitHub workflow. -* https://github.com/tobac-project/tobac/blob/main/CONTRIBUTING.md * **Create a pull request from your fork:** We use our personal forks of the tobac repository to create pull requests. This means that you have to first commit and push your local changes to your personal fork and then create a pull request from that fork: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork @@ -40,27 +43,31 @@ You can also work on any issue that was created by somebody else and is already Tips on working on your local code =================================== -* Install tobac package with :code:`pip install -e` . This allows you to directly test your local code changes as you run tobac. Instead of using the **tobac** version of the latest release, your local version of tobac will be used when you import **tobac** in a python script or IDE. Note that this way of installing a local package will use the code of the checked in branch, so this allows you also to test code while switching between branches. +* Install tobac package with :code:`pip install -e` + * This allows you to directly test your local code changes as you run tobac. Instead of using the **tobac** version of the latest release, your local version of tobac will be used when you import **tobac** in a python script or IDE. + * *Note that* this way of installing a local package will use the code of the checked in branch, so this allows you also to test code while switching between branches. -* You can locally build the documentation page: +* You can locally **build the documentation page**: + * see :doc:`testing_sphinx-based_rendering` * Writing `meaningful commit messages `_ can be very helpful for you and people who review your code to better understand the code changes. + ========================= Our branching strategy ========================= While you can use any type of branching strategy and naming as you work in your personal fork, we have three branches in the tobac repository: -* :code:``RC_*` -* :code:``dev_*` -* :code:``hotfix` +* :code:`RC_*` +* :code:`dev_*` +* :code:`hotfix` -:code:``RC_*` is the release candidate of the next tobac version. The asterisk stands here for the specific tobac version: RC_vx.x.x (e.g. RC_v1.5.0). Pull requests to this branch need two reviews to be accepted before it can be merged into main. +:code:`RC_*` is the release candidate of the next tobac version. The asterisk stands here for the specific tobac version: RC_vx.x.x (e.g. RC_v1.5.0). Pull requests to this branch need two reviews to be accepted before it can be merged into main. -:code:``dev_*` is the development branch where we experiment with new features. This branch is perfectly suited to collaboratively work on a feature together with other **tobac** developers (see :doc:`mentoring`). In general, this branch is used for long-term, comprehensive code changes that might not be covered by a single pull request and where it might not be conceivable in which future **tobac** version to include the changes. There are no branch protection rules for this branch, which means that collaborators of our GitHub organization can directly push changes to this branch. Note that **dev_** can never directly merged into main, it has be merged into the release candidate branch RC_* first! There can be more than one `dev_*` branch, therefore it we recommand to describe the feature to work on in the respective branch (e.g. :code:`dev_xarray_transition`). +:code:`dev_*` is the development branch where we experiment with new features. This branch is perfectly suited to collaboratively work on a feature together with other **tobac** developers (see :doc:`mentoring`). In general, this branch is used for long-term, comprehensive code changes that might not be covered by a single pull request and where it might not be conceivable in which future **tobac** version to include the changes. There are no branch protection rules for this branch, which means that collaborators of our GitHub organization can directly push changes to this branch. Note that **dev_** can never directly merged into main, it has be merged into the release candidate branch :code:`RC_*` first! There can be more than one `dev_*` branch, therefore it we recommand to describe the feature to work on in the respective branch (e.g. :code:`dev_xarray_transition`). -:code:``hotfix` is the branch we use for hotfixes, i.e. bug fixes that need to be released as fast as possible because it influences people's code. This branch needs only one review before it can directly merged into :code:``main`. +:code:`hotfix` is the branch we use for hotfixes, i.e. bug fixes that need to be released as fast as possible because it influences people's code. This branch needs only one review before it can directly merged into :code:`main`. In brief: **Unless you are collaboratively working on a comprehensive feature or on a hotfix, the branch to submit your pull request to is the next release candidate RC_v.x.x.x** diff --git a/doc/index.rst b/doc/index.rst index f04f4e4c..042f6628 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -66,7 +66,6 @@ The project is currently being extended by several contributors to include addit code_structure contributing - testing_sphinx-based_rendering code_reviews mentoring diff --git a/doc/mentoring.rst b/doc/mentoring.rst index 98f1f984..c956af5e 100644 --- a/doc/mentoring.rst +++ b/doc/mentoring.rst @@ -8,7 +8,7 @@ Writing code collaboratively We firmly believe that code can only get better if more than two eyes and one brain work on it. Therefore, we aim to write code collaboratively, in particular, when comprehensive refactoring or enhancements of the code are done. In practice, this can be done by creating a **draft pull request**. This makes it really easy to iteratively improve a pull request with the feedback from others until the pull request is ready for review. -When you work on a comprehensive feature with multiple developers, it is recommended to create a draft pull request on the :code:`dev_*` branch. As explained in our :doc:`our branching strategy `, this branch does not undergo any protection rules. It is meant to experiment with new code and all collaborators of the `tobac-project organization `_ can directly push to this branch. Creating a draft pull request has the advantage of facilitating the communication with other developers who contribute to the same new feature. You can directly see which changes they make, comment these and discuss ways to go forward. +When you work on a comprehensive feature with multiple developers, it is recommended to create a draft pull request on the :code:`dev_*` branch. As explained in :doc:`our branching strategy `, this branch does not undergo any protection rules. It is meant to experiment with new code and all collaborators of the `tobac-project organization `_ can directly push to this branch. Creating a draft pull request has the advantage of facilitating the communication with other developers who contribute to the same new feature. You can directly see which changes they make, comment these and discuss ways to go forward. ============== Get a mentor diff --git a/doc/testing_sphinx-based_rendering.rst b/doc/testing_sphinx-based_rendering.rst index a6e0659c..f0a7443e 100644 --- a/doc/testing_sphinx-based_rendering.rst +++ b/doc/testing_sphinx-based_rendering.rst @@ -5,8 +5,9 @@ How to check the Sphinx-based rendering The workflow has been tested in a linux system. We aim to build a static website out of the documentation material present in ``tobac``. +================================== 1. Preparing the Local Environment ----------------------------------- +================================== - **choose a separate place for your testing** @@ -31,34 +32,44 @@ website out of the documentation material present in ``tobac``. .. code:: bash - git fetch --all - git checkout -t origin/ + /tmp/website-testing/tobac> git fetch --all + /tmp/website-testing/tobac> git checkout -t origin/ - **Python environment** - create a python virtual env - ``bash /tmp/website-testing> python -m venv .python3-venv`` - - and install requirements \```bash # deactivation conda is only - necessary if your loaded conda before … /tmp/website-testing> - conda deactivate + .. code:: bash + + /tmp/website-testing> python -m venv .python3-venv + + + - and install requirements + + .. code:: bash + + # deactivation conda is only necessary if your loaded conda before … + /tmp/website-testing> conda deactivate + + # activate the new env and upgrade ``pip`` + /tmp/website-testing> source .python3-venv/bin/activate + /tmp/website-testing> pip install –upgrade pip + + # now everything is installed into the local python env! + /tmp/website-testing> pip install -r tobac/doc/requirements.txt - # activate the new env and upgrade ``pip`` /tmp/website-testing> - source .python3-venv/bin/activate /tmp/website-testing> pip - install –upgrade pip + # and also install RTD scheme + /tmp/website-testing> pip install sphinx_rtd_theme - # now everything is installed into the local python env! - /tmp/website-testing> pip install -r tobac/doc/requirements.txt + `pip`-based installation takes a bit of time, but is much faster than `conda`. + - # and also install RTD scheme /tmp/website-testing> pip install - sphinx_rtd_theme +If the installation runs without problems, you are ready to build the website. - \`\`\ ````\ pip\ ``-based installation takes a bit of time, but is much faster than``\ conda`. - If the installation runs without problems, you are ready to build - the website. -2. Building the Website ------------------------ +================================== +1. Building the Website +================================== Actually, only few steps are needed to build the website, i.e. @@ -78,8 +89,9 @@ Actually, only few steps are needed to build the website, i.e. /tmp/website-testing/tobac> firefox doc/_build/html/index.html +================================== 3. Parsing Your Local Changes ------------------------------ +================================== Now, we connect to your locally hosted ``tobac`` repository and your development branch. @@ -115,8 +127,9 @@ development branch. /tmp/website-testing/tobac> firefox _build/html/index.html +========================================== Option: Check Rendering of a Pull requests ------------------------------------------- +========================================== - **check the pull request out**: Now, assume the PR has the ID ```` and you define the branch name ``BRANCH_NAME`` as you like From 1475c1d9c2dc2aacf6185b184328768f0a82fe22 Mon Sep 17 00:00:00 2001 From: JuliaKukulies Date: Wed, 10 Apr 2024 08:40:15 -0600 Subject: [PATCH 14/25] modified contributing.md and populated github workflow section in dev guide --- CONTRIBUTING.md | 27 ++++++++++++++------------- doc/code_structure.rst | 2 +- doc/contributing.rst | 13 +++++++++++-- 3 files changed, 26 insertions(+), 16 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6664863a..1be43b22 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,16 +2,14 @@ __Welcome! We are very happy that you are interested in our project and thanks for taking time to contribute! :)__ -We are currently reorganizing our project. So, please check our modifications later. - ## Getting Started ### Installation & Environment details -You will find them in the [README.md](https://github.com/climate-processes/tobac/blob/master/README.md). +You will find them in the [README.md](https://github.com/tobac-project/tobac/blob/master/README.md). ### Tutorials -Tutorials have been prepared to provide you further inside to `tobac`s functionality. Please visit the separate [tobac-tutorials](https://github.com/climate-processes/tobac-tutorials) repository here on Github. - +Tutorials have been prepared to provide you further inside to `tobac`s functionality. Please have a look in the +[examples folder](https://github.com/tobac-project/tobac/tree/main/examples). ### Documentation You will find our documentation at [https://tobac.readthedocs.io](https://tobac.readthedocs.io). @@ -19,22 +17,25 @@ You will find our documentation at [https://tobac.readthedocs.io](https://tobac. ### Testing The tests are located in the [tests folder](https://github.com/climate-processes/tobac/tree/master/tobac/tests). - ## Reporting Bugs -Please create a new issue on [GitHub](https://github.com/climate-processes/tobac/issues) if it is not listed there, yet. +Please create a new issue on [GitHub](https://github.com/tobac-project/tobac/issues) if it is not listed there, yet. ### How to write a good Bug Report? * Give it a clear descriptive title. * Copy and paste the error message. * Describe the steps for reproducing the problem and give an specific example. -* Optional: Make a suggestion to fix it. - +* Optional: Make a suggestion to fix it. ## How to Submit Changes -* Please read the [README.md](https://github.com/climate-processes/tobac/blob/master/README.md) first, to learn about our project goals and check the [changelog.md](). -* Before you start a pull request, please make sure that you added [numpydoc docstrings](#docstringExample) to your functions. This way the api documentation will be parsed properly. -* If it is a larger change or an newly added feature or workflow, please place an example of use in the [tobac-tutorials](https://github.com/climate-processes/tobac-tutorials) repository or adapt the existing examples there. -* If necessary add a folder or modify a file. +* Have a look at [our roadmap](https://github.com/tobac-project/tobac-roadmap/blob/master/tobac-roadmap-main.md) first, +to learn about our project goals and check the +[changelog.md](https://github.com/tobac-project/tobac/blob/main/CHANGELOG.md). +* More details on the code structure and further help for code contributions can be found in our [developer +guide](https://tobac.readthedocs.io/code_structure.html) +* Before you start a pull request, please make sure that you added [numpydoc docstrings](#docstringExample) to your +functions. This way the api documentation will be parsed properly. +* If it is a larger change or an newly added feature or workflow, please add an example in the [example +folder](https://github.com/tobac-project/tobac/tree/main/examples) or adapt the existing examples there. * The code should be PEP 8 compliant, as this facilitates our collaboration. Please use the first stable version (22.6.0) of [black](https://black.readthedocs.io/en/stable/) to format your code. When you submit a pull request, all files are checked for formatting. * The tobac repository is set up with pre-commit hooks to automatically format your code when commiting changes. Please run the command "pre-commit install" in the root directory of tobac to set up pre-commit formatting. diff --git a/doc/code_structure.rst b/doc/code_structure.rst index cea93fdd..ea94a94b 100644 --- a/doc/code_structure.rst +++ b/doc/code_structure.rst @@ -1,4 +1,4 @@ -Code structure and key design koncepts +Code structure and key design concepts -------------------------------------- ================================== diff --git a/doc/contributing.rst b/doc/contributing.rst index 7cd5eddb..92197db8 100644 --- a/doc/contributing.rst +++ b/doc/contributing.rst @@ -76,8 +76,17 @@ In brief: **Unless you are collaboratively working on a comprehensive feature or GitHub workflow ========================= -* Briefly describe how CI works -* Other github actions such as code format check +We use several [GitHub actions](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions) to +assure continuous integration and to enable an efficient code development and release process. Our workflow +configuration can be found in +[.github/workflows](https://github.com/tobac-project/tobac/tree/main/.github/workflows) and encompass + +* check that code is formatted using the latest stable version of black +* linting of the latest code changes that checks the code quality and results in a score compared to the most recent +released version +* check of the zenodo JSON file that ensures that the citation is correct +* check that all unit tests pass (including testing on multiple operating testings) +* check that the example jupyter notebooks run without problems ========================= Writing unit tests From 3e624318089848a868c42cb693408a36537cb0e6 Mon Sep 17 00:00:00 2001 From: JuliaKukulies Date: Wed, 10 Apr 2024 09:03:42 -0600 Subject: [PATCH 15/25] added text for unit testing and example notebooks --- doc/contributing.rst | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/doc/contributing.rst b/doc/contributing.rst index 92197db8..4a5464ae 100644 --- a/doc/contributing.rst +++ b/doc/contributing.rst @@ -85,23 +85,37 @@ configuration can be found in * linting of the latest code changes that checks the code quality and results in a score compared to the most recent released version * check of the zenodo JSON file that ensures that the citation is correct -* check that all unit tests pass (including testing on multiple operating testings) +* check that all unit tests pass (including testing on multiple operating testings) and report test coverage * check that the example jupyter notebooks run without problems +* check that the readthedocs page is properly built ========================= Writing unit tests ========================= -* Make use of :py:mod:`tobac.testing` -* Pytest and pytest fixtures -* Test coverage +We use unit tests that ensure that the functions of each module and submodule work properly. If you add a new +functionality, you should also add a unit test. All tests are located in the [test +folder](https://github.com/tobac-project/tobac/tree/main/tobac/tests). The module :py:mod:`tobac.testing` may help to +create simple, idealized cases where objects can be tracked to test if the new features result in the expected outcome. + +If you are unsure on how to contruct tests and run tests locally, you can find additional documentation on +[pytest](https://docs.pytest.org/en/7.1.x/getting-started.html) and [pytest +fixtures](https://docs.pytest.org/en/6.2.x/fixture.html). + +You will also notice that we report the test coverage, i.e. how much of our current code is triggered and thus tested by +the unit tests. When you submit a pull request, you will see if your code changes have increased or decreased the test +coverage. Ideally, test coverage should not decrease, so please make sure to add appropriate unit tests that cover +all newly added functions. ========================= Add examples ========================= -* Jupyter notebooks -* Upload example data to zenodo +In addition to the unit tests, we aim to provide examples on how to use all functionalities and how to choose different +tracking parameters. These [examples](https://github.com/tobac-project/tobac/tree/main/examples) are in form of jupyter +notebooks and can be based on simple, idealized test cases or real data. We strongly encourage the use of real data that +is publlicly accessible, but another option for new examples with real data is to upload the data tpo our [zenodo +repository](https://zenodo.org/records/10863405). ========================= Releasing a new version From 57c76ac03de785d235e68e8f30b9d0add2144dd8 Mon Sep 17 00:00:00 2001 From: JuliaKukulies Date: Wed, 10 Apr 2024 09:24:22 -0600 Subject: [PATCH 16/25] added basic description of xarray migration --- doc/code_structure.rst | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/doc/code_structure.rst b/doc/code_structure.rst index ea94a94b..3c65937d 100644 --- a/doc/code_structure.rst +++ b/doc/code_structure.rst @@ -46,9 +46,19 @@ To help users get started with **tobac** and to demonstrate the various function Migrating to xarray and dask ============================ -- Basics of xarray (xarray.Dataarray class) and dask -- How these are or could be used in tobac -- How to work on the transition +Currently, **tobac** uses [iris cubes](https://scitools-iris.readthedocs.io/en/latest/userguide/iris_cubes.html) as the +primary data container. However, we are currently working on migrating the source code to +[xarray](https://docs.xarray.dev/en/stable/) such that all internal functions are based on [xr.DataArray +objects](https://docs.xarray.dev/en/stable/generated/xarray.DataArray.html). + +To ensure a robust transition from **iris** to **xarray**, we make use of various decorators that convert input and +output data for the main functions without changing their actual code. These decorators are located in the [decorator +submodule](https://github.com/tobac-project/tobac/blob/main/tobac/utils/decorators.py). + +In addition, one of our main goals for the future is to fully support [dask](https://www.dask.org/), in order to scale +to large datasets and enable parallelization. + + From 9ed042011047936513c6ae138f520c60faf72d7c Mon Sep 17 00:00:00 2001 From: JuliaKukulies Date: Wed, 10 Apr 2024 09:26:36 -0600 Subject: [PATCH 17/25] updated my affiliation in json file --- .zenodo.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.zenodo.json b/.zenodo.json index 8d6ff6c8..2e74bee0 100644 --- a/.zenodo.json +++ b/.zenodo.json @@ -23,7 +23,7 @@ { "name": "Kukulies, Julia", - "affiliation": "University of Gothenburg (Sweden)", + "affiliation": "National Center for Atmospheric Research", "orcid": "0000-0001-6084-0069" }, { From 6aeb9e1ae2849ca2774f9cf1f5187b9e252111dd Mon Sep 17 00:00:00 2001 From: JuliaKukulies Date: Wed, 10 Apr 2024 09:47:43 -0600 Subject: [PATCH 18/25] moved docstrings example to developer guide to keep contributing.md more concise and have everything at one place --- CONTRIBUTING.md | 6 ++++-- doc/contributing.rst | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1be43b22..c2e602ac 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -32,8 +32,10 @@ to learn about our project goals and check the [changelog.md](https://github.com/tobac-project/tobac/blob/main/CHANGELOG.md). * More details on the code structure and further help for code contributions can be found in our [developer guide](https://tobac.readthedocs.io/code_structure.html) -* Before you start a pull request, please make sure that you added [numpydoc docstrings](#docstringExample) to your -functions. This way the api documentation will be parsed properly. +* Before you start a pull request, please make sure that you added [numpydoc +docstrings](https://numpydoc.readthedocs.io/en/latest/format.html) to your +functions. See [docstring example in the developer guide](https://tobac.readthedocs.io/contributing.html).This way the +api documentation will be parsed properly. * If it is a larger change or an newly added feature or workflow, please add an example in the [example folder](https://github.com/tobac-project/tobac/tree/main/examples) or adapt the existing examples there. * The code should be PEP 8 compliant, as this facilitates our collaboration. Please use the first stable version (22.6.0) of [black](https://black.readthedocs.io/en/stable/) to format your code. When you submit a pull request, all files are checked for formatting. diff --git a/doc/contributing.rst b/doc/contributing.rst index 4a5464ae..d9747dfb 100644 --- a/doc/contributing.rst +++ b/doc/contributing.rst @@ -37,6 +37,40 @@ git basics * **Create a pull request from your fork:** We use our personal forks of the tobac repository to create pull requests. This means that you have to first commit and push your local changes to your personal fork and then create a pull request from that fork: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork +=================================== +Writing proper documentation +=================================== + +Please provide **Numpy Docstrings** for all new functions. + +**Example**: + +`` + ''' + calculate centre of gravity and mass forech individual tracked cell in the simulation + + + Parameters + ---------- + tracks : pandas.DataFram + DataFrame containing trajectories of cell centres + + param mass : iris.cube.Cube + cube of quantity (need coordinates 'time', 'geopotential_height','projection_x_coordinate' and + 'projection_y_coordinate') + + param mask : iris.cube.Cube + cube containing mask (int > where belonging to cloud volume, 0 everywhere else ) + + + Returns + ------- + track_out : pandas.DataFrame + Dataframe containing t,x,y,z positions of centre of gravity and total cloud mass each tracked cells + at each timestep + + ''' +`` =================================== From 008d73659438890289a54658d7e1edcebd9e3ecc Mon Sep 17 00:00:00 2001 From: JuliaKukulies Date: Wed, 10 Apr 2024 09:56:44 -0600 Subject: [PATCH 19/25] format code example --- doc/contributing.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/contributing.rst b/doc/contributing.rst index d9747dfb..3800d09b 100644 --- a/doc/contributing.rst +++ b/doc/contributing.rst @@ -45,7 +45,8 @@ Please provide **Numpy Docstrings** for all new functions. **Example**: -`` +.. code:: + ''' calculate centre of gravity and mass forech individual tracked cell in the simulation @@ -70,7 +71,7 @@ Please provide **Numpy Docstrings** for all new functions. at each timestep ''' -`` + =================================== From 65dbff38ca668aac122701f13c7d790003a46728 Mon Sep 17 00:00:00 2001 From: JuliaKukulies Date: Wed, 10 Apr 2024 10:02:45 -0600 Subject: [PATCH 20/25] updates --- doc/contributing.rst | 3 +-- doc/index.rst | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/doc/contributing.rst b/doc/contributing.rst index 3800d09b..5b10a6f5 100644 --- a/doc/contributing.rst +++ b/doc/contributing.rst @@ -4,8 +4,7 @@ How to contribute ------------------------- -Step-by-step overview: https://github.com/tobac-project/tobac/blob/main/CONTRIBUTING.md - +Step-by-step overview of most important points: https://github.com/tobac-project/tobac/blob/main/CONTRIBUTING.md ========================= Code of conduct diff --git a/doc/index.rst b/doc/index.rst index e61562a5..1b215f20 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -68,7 +68,7 @@ The project is currently being extended by several contributors to include addit .. toctree:: :caption: Developer guide - :maxdepth: 2 + :maxdepth: 3 code_structure contributing From 02284ece2b90cd1e6027e632f0b21e6ca2295ce0 Mon Sep 17 00:00:00 2001 From: fsenf <43379813+fsenf@users.noreply.github.com> Date: Fri, 19 Apr 2024 07:29:47 +0000 Subject: [PATCH 21/25] updates during review: - reformatted links - spell corrections - removed docstring from CONTRIBUTING file --- CONTRIBUTING.md | 30 +----------------------------- doc/code_structure.rst | 14 +++++++------- doc/contributing.rst | 35 +++++++++++++++++------------------ 3 files changed, 25 insertions(+), 54 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c2e602ac..c56ebbe8 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -41,38 +41,10 @@ folder](https://github.com/tobac-project/tobac/tree/main/examples) or adapt the * The code should be PEP 8 compliant, as this facilitates our collaboration. Please use the first stable version (22.6.0) of [black](https://black.readthedocs.io/en/stable/) to format your code. When you submit a pull request, all files are checked for formatting. * The tobac repository is set up with pre-commit hooks to automatically format your code when commiting changes. Please run the command "pre-commit install" in the root directory of tobac to set up pre-commit formatting. -We aim to respond to all new issues/pull requests as soon as possible, however at times this is not possible due to work commitments. +We aim to respond to all new issues/pull requests as soon as possible, however sometimes this is not possible due to work commitments. -### Numpydoc Example -```python - - ''' - calculate centre of gravity and mass forech individual tracked cell in the simulation - Parameters - ---------- - tracks : pandas.DataFram - DataFrame containing trajectories of cell centres - - param mass : iris.cube.Cube - cube of quantity (need coordinates 'time', 'geopotential_height','projection_x_coordinate' and - 'projection_y_coordinate') - - param mask : iris.cube.Cube - cube containing mask (int > where belonging to cloud volume, 0 everywhere else ) - - - Returns - ------- - track_out : pandas.DataFrame - Dataframe containing t,x,y,z positions of centre of gravity and total cloud mass each tracked cells - at each timestep - - ''' - -``` - ## Slack In addition to the workflow here on Github, there's a tobac workspace on Slack [tobac-dev.slack.com](tobac-dev.slack.com) that we use for some additional communication around the project. Please join us there to stay updated about all things tobac that go beyond the detailed work on the code. diff --git a/doc/code_structure.rst b/doc/code_structure.rst index 3c65937d..cbc1b23e 100644 --- a/doc/code_structure.rst +++ b/doc/code_structure.rst @@ -39,23 +39,23 @@ One thing to note is that **tobac** as of now is purely functional. The plan is, Examples ======== -To help users get started with **tobac** and to demonstrate the various functionalities, **tobac** hosts several detailed and **illustrated examples** in the form of Jupyter notebooks. They are hosted under the directory `examples/` and be executed by the user. +To help users get started with **tobac** and to demonstrate the various functionalities, **tobac** hosts several detailed and **illustrated examples** in the form of Jupyter notebooks. They are hosted under the directory `examples/` and be executed by the user. Our readthedocs page also hosts a rendered version of our examples as `gallery `_ ============================ Migrating to xarray and dask ============================ -Currently, **tobac** uses [iris cubes](https://scitools-iris.readthedocs.io/en/latest/userguide/iris_cubes.html) as the +Currently, **tobac** uses `iris cubes `_ as the primary data container. However, we are currently working on migrating the source code to -[xarray](https://docs.xarray.dev/en/stable/) such that all internal functions are based on [xr.DataArray -objects](https://docs.xarray.dev/en/stable/generated/xarray.DataArray.html). +`xarray `_ such that all internal functions are based on `xr.DataArray +objects `_. To ensure a robust transition from **iris** to **xarray**, we make use of various decorators that convert input and -output data for the main functions without changing their actual code. These decorators are located in the [decorator -submodule](https://github.com/tobac-project/tobac/blob/main/tobac/utils/decorators.py). +output data for the main functions without changing their actual code. These decorators are located in the `decorator +submodule `_. -In addition, one of our main goals for the future is to fully support [dask](https://www.dask.org/), in order to scale +In addition, one of our main goals for the future is to fully support `dask `_, in order to scale to large datasets and enable parallelization. diff --git a/doc/contributing.rst b/doc/contributing.rst index 5b10a6f5..90326497 100644 --- a/doc/contributing.rst +++ b/doc/contributing.rst @@ -27,7 +27,7 @@ git basics * Learn about `pull requests `_ * Learn about `how to commit and push changes from your local repository `_ -* **Create an issue**: If you have an idea for a new feature or a suggestion for any kind of code changes, please create an issue for this. We sort `our issues `_ into `milestones `_ to priorized work and manage our workflow, i.e. the different versions of **tobac** to come. +* **Create an issue**: If you have an idea for a new feature or a suggestion for any kind of code changes, please create an issue for this. We sort `our issues `_ into `milestones `_ to priorize work and manage our workflow, i.e. the different versions of **tobac** to come. The issues act, therefore, not only as a place for reporting bugs, but also as a collection of *to do* points. @@ -47,7 +47,7 @@ Please provide **Numpy Docstrings** for all new functions. .. code:: ''' - calculate centre of gravity and mass forech individual tracked cell in the simulation + Calculates centre of gravity and mass for each individually tracked cell in the simulation. Parameters @@ -99,7 +99,7 @@ While you can use any type of branching strategy and naming as you work in your :code:`RC_*` is the release candidate of the next tobac version. The asterisk stands here for the specific tobac version: RC_vx.x.x (e.g. RC_v1.5.0). Pull requests to this branch need two reviews to be accepted before it can be merged into main. -:code:`dev_*` is the development branch where we experiment with new features. This branch is perfectly suited to collaboratively work on a feature together with other **tobac** developers (see :doc:`mentoring`). In general, this branch is used for long-term, comprehensive code changes that might not be covered by a single pull request and where it might not be conceivable in which future **tobac** version to include the changes. There are no branch protection rules for this branch, which means that collaborators of our GitHub organization can directly push changes to this branch. Note that **dev_** can never directly merged into main, it has be merged into the release candidate branch :code:`RC_*` first! There can be more than one `dev_*` branch, therefore it we recommand to describe the feature to work on in the respective branch (e.g. :code:`dev_xarray_transition`). +:code:`dev_*` is the development branch where we experiment with new features. This branch is perfectly suited to collaboratively work on a feature together with other **tobac** developers (see :doc:`mentoring`). In general, this branch is used for long-term, comprehensive code changes that might not be covered by a single pull request and where it might not be conceivable in which future **tobac** version to include the changes. There are no branch protection rules for this branch, which means that collaborators of our GitHub organization can directly push changes to this branch. Note that **dev_** can never directly merged into main, it has be merged into the release candidate branch :code:`RC_*` first! There can be more than one `dev_*` branch, therefore it we recommend to describe the feature to work on in the respective branch (e.g. :code:`dev_xarray_transition`). :code:`hotfix` is the branch we use for hotfixes, i.e. bug fixes that need to be released as fast as possible because it influences people's code. This branch needs only one review before it can directly merged into :code:`main`. @@ -110,16 +110,15 @@ In brief: **Unless you are collaboratively working on a comprehensive feature or GitHub workflow ========================= -We use several [GitHub actions](https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions) to +We use several `GitHub actions `_ to assure continuous integration and to enable an efficient code development and release process. Our workflow configuration can be found in -[.github/workflows](https://github.com/tobac-project/tobac/tree/main/.github/workflows) and encompass +`.github/workflows `_ and encompass * check that code is formatted using the latest stable version of black -* linting of the latest code changes that checks the code quality and results in a score compared to the most recent -released version +* linting of the latest code changes that checks the code quality and results in a score compared to the most recent released version * check of the zenodo JSON file that ensures that the citation is correct -* check that all unit tests pass (including testing on multiple operating testings) and report test coverage +* check that all unit tests pass (including testing on multiple operating systems) and report test coverage * check that the example jupyter notebooks run without problems * check that the readthedocs page is properly built @@ -128,13 +127,13 @@ Writing unit tests ========================= We use unit tests that ensure that the functions of each module and submodule work properly. If you add a new -functionality, you should also add a unit test. All tests are located in the [test -folder](https://github.com/tobac-project/tobac/tree/main/tobac/tests). The module :py:mod:`tobac.testing` may help to +functionality, you should also add a unit test. All tests are located in the `test +folder `_ The module :py:mod:`tobac.testing` may help to create simple, idealized cases where objects can be tracked to test if the new features result in the expected outcome. -If you are unsure on how to contruct tests and run tests locally, you can find additional documentation on -[pytest](https://docs.pytest.org/en/7.1.x/getting-started.html) and [pytest -fixtures](https://docs.pytest.org/en/6.2.x/fixture.html). +If you are unsure on how to construct tests and run tests locally, you can find additional documentation on +`pytest `_ and `pytest +fixtures `_. You will also notice that we report the test coverage, i.e. how much of our current code is triggered and thus tested by the unit tests. When you submit a pull request, you will see if your code changes have increased or decreased the test @@ -146,10 +145,10 @@ Add examples ========================= In addition to the unit tests, we aim to provide examples on how to use all functionalities and how to choose different -tracking parameters. These [examples](https://github.com/tobac-project/tobac/tree/main/examples) are in form of jupyter +tracking parameters. These `examples `_ are in form of jupyter notebooks and can be based on simple, idealized test cases or real data. We strongly encourage the use of real data that -is publlicly accessible, but another option for new examples with real data is to upload the data tpo our [zenodo -repository](https://zenodo.org/records/10863405). +is publicly accessible, but another option for new examples with real data is to either upload the data to our `zenodo +repository `_ or create your own data upload on zenodo. Please include the name "tobac" in the data title for the latter. ========================= Releasing a new version @@ -157,10 +156,10 @@ Releasing a new version This is the checklist of steps for a release of a new **tobac** version: -* Bump version in `__init__.py `in :code:`hotfix` +* Bump version in :code:`__init__.py` in :code:`hotfix` * Add changelog in :code:`hotfix` * Regenerate example notebooks with the new version -* Get the two additional bug fixes into :code:`hotfix` +* Get the additional bug fixes into :code:`hotfix` * Merge :code:`hotfix` into :code:`main` * Merge :code:`main` into release and dev branches * Delete :code:`hotfix` branch From 37049953a451b694fc1a76684c01465cff09953b Mon Sep 17 00:00:00 2001 From: Julia Kukulies <44163060+JuliaKukulies@users.noreply.github.com> Date: Fri, 19 Apr 2024 07:32:27 -0600 Subject: [PATCH 22/25] Update CONTRIBUTING.md Co-authored-by: fsenf <43379813+fsenf@users.noreply.github.com> --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c56ebbe8..98aee8b8 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -34,7 +34,7 @@ to learn about our project goals and check the guide](https://tobac.readthedocs.io/code_structure.html) * Before you start a pull request, please make sure that you added [numpydoc docstrings](https://numpydoc.readthedocs.io/en/latest/format.html) to your -functions. See [docstring example in the developer guide](https://tobac.readthedocs.io/contributing.html).This way the +functions. See [docstring example in the developer guide](https://tobac.readthedocs.io/contributing.html). This way the api documentation will be parsed properly. * If it is a larger change or an newly added feature or workflow, please add an example in the [example folder](https://github.com/tobac-project/tobac/tree/main/examples) or adapt the existing examples there. From f75c85020ff6f96846ac9314c1c95755f2d07579 Mon Sep 17 00:00:00 2001 From: Julia Kukulies <44163060+JuliaKukulies@users.noreply.github.com> Date: Fri, 19 Apr 2024 07:39:06 -0600 Subject: [PATCH 23/25] addressed Fabians comments --- doc/contributing.rst | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/doc/contributing.rst b/doc/contributing.rst index 90326497..c458a69d 100644 --- a/doc/contributing.rst +++ b/doc/contributing.rst @@ -120,7 +120,6 @@ configuration can be found in * check of the zenodo JSON file that ensures that the citation is correct * check that all unit tests pass (including testing on multiple operating systems) and report test coverage * check that the example jupyter notebooks run without problems -* check that the readthedocs page is properly built ========================= Writing unit tests @@ -156,15 +155,14 @@ Releasing a new version This is the checklist of steps for a release of a new **tobac** version: -* Bump version in :code:`__init__.py` in :code:`hotfix` -* Add changelog in :code:`hotfix` +* Bump version in :code:`__init__.py` in :code:`RC_vXXX` +* Add changelog in :code:`RC_vXXX` * Regenerate example notebooks with the new version -* Get the additional bug fixes into :code:`hotfix` -* Merge :code:`hotfix` into :code:`main` -* Merge :code:`main` into release and dev branches -* Delete :code:`hotfix` branch +* Merge :code:`RC_vXXX` into :code:`main` +* Merge updated :code:`main` branch back into release and dev branches +* Delete :code:`RC_vXXX` branch * Create release * Push release to conda-forge: https://github.com/tobac-project/tobac-notes/blob/main/uploading_to_conda-forge.md -* E-mail tobac mailing list * Create new tag +* E-mail tobac mailing list From cf3c6bd9e922836450f6afa4896cf468a8e7668b Mon Sep 17 00:00:00 2001 From: Julia Kukulies Date: Fri, 19 Jul 2024 14:08:08 -0600 Subject: [PATCH 24/25] zenodo json formatting --- .zenodo.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.zenodo.json b/.zenodo.json index 2e74bee0..65805358 100644 --- a/.zenodo.json +++ b/.zenodo.json @@ -23,7 +23,7 @@ { "name": "Kukulies, Julia", - "affiliation": "National Center for Atmospheric Research", + "affiliation": "NSF National Center for Atmospheric Research", "orcid": "0000-0001-6084-0069" }, { From 29f0a33184f636cb4c96919b9e121584451bad60 Mon Sep 17 00:00:00 2001 From: Sean Freeman Date: Tue, 23 Jul 2024 15:35:26 -0500 Subject: [PATCH 25/25] update codecov action to v4 --- .github/workflows/codecov-CI.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/codecov-CI.yml b/.github/workflows/codecov-CI.yml index ef2ef259..c20c96c5 100644 --- a/.github/workflows/codecov-CI.yml +++ b/.github/workflows/codecov-CI.yml @@ -27,7 +27,7 @@ jobs: mamba install --quiet --yes --file requirements.txt coverage pytest-cov && python -m coverage run -m pytest --cov=./ --cov-report=xml - name: Upload Coverage to Codecov - uses: codecov/codecov-action@v3 + uses: codecov/codecov-action@v4 with: token: ${{ secrets.CODECOV_TOKEN }} fail_ci_if_error: true