From c76f83deaae5bd575549d055fd130da18009a6fe Mon Sep 17 00:00:00 2001 From: Timothy Willard <9395586+TimothyWillard@users.noreply.github.com> Date: Tue, 29 Oct 2024 11:22:27 -0400 Subject: [PATCH 1/7] Add index to development, init git/github docs --- documentation/gitbook/development/README.md | 6 ++++++ documentation/gitbook/development/git-and-github-usage.md | 3 +++ 2 files changed, 9 insertions(+) create mode 100644 documentation/gitbook/development/README.md create mode 100644 documentation/gitbook/development/git-and-github-usage.md diff --git a/documentation/gitbook/development/README.md b/documentation/gitbook/development/README.md new file mode 100644 index 000000000..ba720ad4b --- /dev/null +++ b/documentation/gitbook/development/README.md @@ -0,0 +1,6 @@ +# Development + +This section covers development/contribution guidelines, including tutorials on how to setup your environment and guides on how we use git/GitHub. + +* [Git and GitHub Usage](./git-and-github-usage.md) +* [Python Guidelines for Developers](./python-guidelines-for-developers.md) diff --git a/documentation/gitbook/development/git-and-github-usage.md b/documentation/gitbook/development/git-and-github-usage.md new file mode 100644 index 000000000..65142c185 --- /dev/null +++ b/documentation/gitbook/development/git-and-github-usage.md @@ -0,0 +1,3 @@ +# Git and GitHub Usage + +We now use a modified gitflow style workflow for working with git and GitHub. For a detailed overview of this topic please refer to [Atlassian's article on Gitflow workflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow). From 45612b245f350165aca279dc36b7c93bbdbb732c Mon Sep 17 00:00:00 2001 From: Timothy Willard <9395586+TimothyWillard@users.noreply.github.com> Date: Tue, 29 Oct 2024 11:44:49 -0400 Subject: [PATCH 2/7] Add more details to git/GitHub dev docs --- .../development/git-and-github-usage.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/documentation/gitbook/development/git-and-github-usage.md b/documentation/gitbook/development/git-and-github-usage.md index 65142c185..6d5972929 100644 --- a/documentation/gitbook/development/git-and-github-usage.md +++ b/documentation/gitbook/development/git-and-github-usage.md @@ -1,3 +1,22 @@ # Git and GitHub Usage We now use a modified gitflow style workflow for working with git and GitHub. For a detailed overview of this topic please refer to [Atlassian's article on Gitflow workflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow). + + +## New Features + +New features should be developed in a new branch checked out from the `dev` branch and then merged back into the `dev` branch via a PR on GitHub when ready for review. These feature branches can be deleted after merging into `dev`, unless someone from operations requests that it be kept around. For example, operations may want to merge the feature into their operational branch to get new functionality in advance of a release. + + +## Hot Fixes + +Hot fixes should be developed in a new branch checked out from the `main` branch and merged back into the `main` branch via a PR on GitHub when ready for review. After successfully merging into `main` the hotfix branch should then be merged into `dev`, making appropriate adjustments to stabilize the feature. The priority for hot fixes is to correct a major issue quickly, so it is okay to delay detailed testing/documentation until merging into `dev`. + + +## Creating Releases + +Periodically releases will be created by merging the `dev` branch into `main` via a PR on GitHub and creating a new release the `main` branch after merging. These PRs should avoid discussion of individual feature changes, those discussions should be reserved for and handled in the feature PRs. If there is a feature that poses a significant problem in the process of creating a new release those changes should be treated like a new feature. The main purpose of this PR is to: + +1. Resolve merge conflicts generated by hot fixes, +2. Making minor edits to documentation to make it clearer or more cohesive, and +3. Updating the `NEWS.md` file with a summary of the changes included in the release. From a0959428269973d032a77c88455d258917fdf710 Mon Sep 17 00:00:00 2001 From: Timothy Willard <9395586+TimothyWillard@users.noreply.github.com> Date: Tue, 29 Oct 2024 11:49:29 -0400 Subject: [PATCH 3/7] Add development links to `SUMMARY.md` --- documentation/gitbook/SUMMARY.md | 3 ++- documentation/gitbook/development/README.md | 3 --- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/documentation/gitbook/SUMMARY.md b/documentation/gitbook/SUMMARY.md index ec75af31a..d49888786 100644 --- a/documentation/gitbook/SUMMARY.md +++ b/documentation/gitbook/SUMMARY.md @@ -58,8 +58,9 @@ * [Useful commands](how-to-run/useful-commands.md) * [Tips, tricks, FAQ](how-to-run/tips-tricks-faq.md) -## 🗜️ Development +## [Development](./development/README.md) +* [Git and GitHub Usage](./development/git-and-github-usage.md) * [Guidelines for contributors](development/python-guidelines-for-developers.md) ## Deprecated pages diff --git a/documentation/gitbook/development/README.md b/documentation/gitbook/development/README.md index ba720ad4b..b7daa2b95 100644 --- a/documentation/gitbook/development/README.md +++ b/documentation/gitbook/development/README.md @@ -1,6 +1,3 @@ # Development This section covers development/contribution guidelines, including tutorials on how to setup your environment and guides on how we use git/GitHub. - -* [Git and GitHub Usage](./git-and-github-usage.md) -* [Python Guidelines for Developers](./python-guidelines-for-developers.md) From 051a6368ae78c17baa5c83076970038d7b203135 Mon Sep 17 00:00:00 2001 From: Timothy Willard <9395586+TimothyWillard@users.noreply.github.com> Date: Tue, 29 Oct 2024 12:17:42 -0400 Subject: [PATCH 4/7] Add description of operations to git/GitHub --- documentation/gitbook/development/git-and-github-usage.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/documentation/gitbook/development/git-and-github-usage.md b/documentation/gitbook/development/git-and-github-usage.md index 6d5972929..971f914ce 100644 --- a/documentation/gitbook/development/git-and-github-usage.md +++ b/documentation/gitbook/development/git-and-github-usage.md @@ -20,3 +20,8 @@ Periodically releases will be created by merging the `dev` branch into `main` vi 1. Resolve merge conflicts generated by hot fixes, 2. Making minor edits to documentation to make it clearer or more cohesive, and 3. Updating the `NEWS.md` file with a summary of the changes included in the release. + + +## Operations + +Operational work should be developed in a new branch checked out from the `main` branch. Pre-released features can be merged directly into this operational branch from the corresponding feature branch as needed via a git merge or rebase not a GitHub PR. After the operational cycle is over, the operations branch **should not** be deleted, instead should be kept around for archival reasons. Operational work needs to move quickly and usually does not involve documenting or testing code and is therefore unsuitable for merging into `dev` or `main` directly. Instead potential features should be extracted from an operations branch into a feature branch using [git cherry-pick](https://git-scm.com/docs/git-cherry-pick) and then modified into an appropriates state for merging into `dev` like a feature branch. From 0d1ef153b515e5ad2e5e67850a45f27ab1bc0f18 Mon Sep 17 00:00:00 2001 From: Timothy Willard <9395586+TimothyWillard@users.noreply.github.com> Date: Tue, 29 Oct 2024 14:52:49 -0400 Subject: [PATCH 5/7] Add branch prefixing to docs --- documentation/gitbook/development/git-and-github-usage.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/documentation/gitbook/development/git-and-github-usage.md b/documentation/gitbook/development/git-and-github-usage.md index 971f914ce..e9e9b3153 100644 --- a/documentation/gitbook/development/git-and-github-usage.md +++ b/documentation/gitbook/development/git-and-github-usage.md @@ -5,12 +5,12 @@ We now use a modified gitflow style workflow for working with git and GitHub. Fo ## New Features -New features should be developed in a new branch checked out from the `dev` branch and then merged back into the `dev` branch via a PR on GitHub when ready for review. These feature branches can be deleted after merging into `dev`, unless someone from operations requests that it be kept around. For example, operations may want to merge the feature into their operational branch to get new functionality in advance of a release. +New features should be developed in a new branch checked out from the `dev` branch and then merged back into the `dev` branch via a PR on GitHub when ready for review. These feature branches can be deleted after merging into `dev`, unless someone from operations requests that it be kept around. For example, operations may want to merge the feature into their operational branch to get new functionality in advance of a release. By convention feature branches should be prefixed with `feature//`, I.e. `feature/99/cool-new-thing`. ## Hot Fixes -Hot fixes should be developed in a new branch checked out from the `main` branch and merged back into the `main` branch via a PR on GitHub when ready for review. After successfully merging into `main` the hotfix branch should then be merged into `dev`, making appropriate adjustments to stabilize the feature. The priority for hot fixes is to correct a major issue quickly, so it is okay to delay detailed testing/documentation until merging into `dev`. +Hot fixes should be developed in a new branch checked out from the `main` branch and merged back into the `main` branch via a PR on GitHub when ready for review. After successfully merging into `main` the hot fix branch should then be merged into `dev`, making appropriate adjustments to stabilize the feature. The priority for hot fixes is to correct a major issue quickly, so it is okay to delay detailed testing/documentation until merging into `dev`. By convention hot fix branches should be prefixed with `hotfix/`, I.e. `hotfix/important-fix-to-something`, and then converted into a feature branch after merging into main. ## Creating Releases @@ -24,4 +24,4 @@ Periodically releases will be created by merging the `dev` branch into `main` vi ## Operations -Operational work should be developed in a new branch checked out from the `main` branch. Pre-released features can be merged directly into this operational branch from the corresponding feature branch as needed via a git merge or rebase not a GitHub PR. After the operational cycle is over, the operations branch **should not** be deleted, instead should be kept around for archival reasons. Operational work needs to move quickly and usually does not involve documenting or testing code and is therefore unsuitable for merging into `dev` or `main` directly. Instead potential features should be extracted from an operations branch into a feature branch using [git cherry-pick](https://git-scm.com/docs/git-cherry-pick) and then modified into an appropriates state for merging into `dev` like a feature branch. +Operational work should be developed in a new branch checked out from the `main` branch if there are modifications needed to the `flepiMoP` codebase. Pre-released features can be merged directly into this operational branch from the corresponding feature branch as needed via a git merge or rebase not a GitHub PR. After the operational cycle is over, the operations branch **should not** be deleted, instead should be kept around for archival reasons. Operational work needs to move quickly and usually does not involve documenting or testing code and is therefore unsuitable for merging into `dev` or `main` directly. Instead potential features should be extracted from an operations branch into a feature branch using [git cherry-pick](https://git-scm.com/docs/git-cherry-pick) and then modified into an appropriates state for merging into `dev` like a feature branch. By convention operations branch names should be prefixed with `operations/`, I.e. `operations/flu-SMH-2023-24`. From 605e0680af4876d5982d0b6179d2ec2e5ba02757 Mon Sep 17 00:00:00 2001 From: Timothy Willard <9395586+TimothyWillard@users.noreply.github.com> Date: Mon, 18 Nov 2024 16:26:46 -0500 Subject: [PATCH 6/7] Remove outdated sections from python guidelines --- .../python-guidelines-for-developers.md | 55 ------------------- 1 file changed, 55 deletions(-) diff --git a/documentation/gitbook/development/python-guidelines-for-developers.md b/documentation/gitbook/development/python-guidelines-for-developers.md index 75a7e60ab..a937ca8ae 100644 --- a/documentation/gitbook/development/python-guidelines-for-developers.md +++ b/documentation/gitbook/development/python-guidelines-for-developers.md @@ -78,58 +78,3 @@ For those using a Mac or Linux system for development this command is also avail ```bash cp -f bin/pre-commit .git/hooks/ ``` - -#### Structure of the main classes - -The code is structured so that each of the main classes **owns** a config segment, and only this class should parse and build the related object. To access this information, other classes first need to build the object. - -{% hint style="warning" %} -Below, this page is still underconstruction -{% endhint %} - -The main classes are: - -* `Coordinates:` this is a light class that stores all the coordinates needed by every other class (e.g the time serie -* `Parameter` -* `Compartments` -* `Modifers` -* `Seeding`, -* `InitialConditions` -* a `writeDF` -* function to plot -* (TODO: detail pipeline internal API) - -### Batch folder - -Here are some notes useful to improve the batch submission: - -Setup site wide Rprofile. - -``` -export R_PROFILE=$COVID_PATH/slurm_batch/Rprofile -``` - -> SLURM copies your environment variables by default. You don't need to tell it to set a variable on the command line for sbatch. Just set the variable in your environment before calling sbatch. - -> There are two useful environment variables that SLURM sets up when you use job arrays: - -> SLURM\_ARRAY\_JOB\_ID, specifies the array's master job ID number. SLURM\_ARRAY\_TASK\_ID, specifies the job array index number. https://help.rc.ufl.edu/doc/Using\_Variables\_in\_SLURM\_Jobs - -SLURM does not support using variables in the #SBATCH lines within a job script (for example, #SBATCH -N=$REPS will NOT work). A very limited number of variables are available in the #SBATCH just as %j for JOB ID. However, values passed from the command line have precedence over values defined in the job script. and you could use variables in the command line. For example, you could set the job name and output/error files can be passed on the sbatch command line: - -``` -RUNTYPE='test' -RUNNUMBER=5 -sbatch --job-name=$RUNTYPE.$RUNNUMBER.run --output=$RUNTYPE.$RUNUMBER.txt --export=A=$A,b=$b jobscript.sbatch -``` - -However note in this example, the output file doesn't have the job ID which is not available from the command line, only inside the sbatch shell script. - -#### File descriptions - -launch\_job.py and runner.py for non inference job - -inference\_job.py launch a slurm or aws job, where it uses - -* \`inference\_runner.sh\` and inference\_copy.sh for aws -* ;batch/inference\_job.run for slurm From 5aea02dad8b861f508c0ac19dbc4862c44664a6a Mon Sep 17 00:00:00 2001 From: Timothy Willard <9395586+TimothyWillard@users.noreply.github.com> Date: Mon, 18 Nov 2024 16:31:36 -0500 Subject: [PATCH 7/7] Add minor notes about docs in workflow --- documentation/gitbook/development/git-and-github-usage.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/gitbook/development/git-and-github-usage.md b/documentation/gitbook/development/git-and-github-usage.md index e9e9b3153..1cde17fcc 100644 --- a/documentation/gitbook/development/git-and-github-usage.md +++ b/documentation/gitbook/development/git-and-github-usage.md @@ -5,12 +5,12 @@ We now use a modified gitflow style workflow for working with git and GitHub. Fo ## New Features -New features should be developed in a new branch checked out from the `dev` branch and then merged back into the `dev` branch via a PR on GitHub when ready for review. These feature branches can be deleted after merging into `dev`, unless someone from operations requests that it be kept around. For example, operations may want to merge the feature into their operational branch to get new functionality in advance of a release. By convention feature branches should be prefixed with `feature//`, I.e. `feature/99/cool-new-thing`. +New features should be developed in a new branch checked out from the `dev` branch and then merged back into the `dev` branch via a PR on GitHub when ready for review. These feature branches can be deleted after merging into `dev`, unless someone from operations requests that it be kept around. For example, operations may want to merge the feature into their operational branch to get new functionality in advance of a release. By convention feature branches should be prefixed with `feature//`, I.e. `feature/99/cool-new-thing`. Feature branch should also include edits to the GitBook documentation that describe their changes. ## Hot Fixes -Hot fixes should be developed in a new branch checked out from the `main` branch and merged back into the `main` branch via a PR on GitHub when ready for review. After successfully merging into `main` the hot fix branch should then be merged into `dev`, making appropriate adjustments to stabilize the feature. The priority for hot fixes is to correct a major issue quickly, so it is okay to delay detailed testing/documentation until merging into `dev`. By convention hot fix branches should be prefixed with `hotfix/`, I.e. `hotfix/important-fix-to-something`, and then converted into a feature branch after merging into main. +Hot fixes should be developed in a new branch checked out from the `main` branch and merged back into the `main` branch via a PR on GitHub when ready for review. After successfully merging into `main` the hot fix branch should then be merged into `dev`, making appropriate adjustments to stabilize the feature. The priority for hot fixes is to correct a major issue quickly, so it is okay to delay detailed testing/documentation until merging into `dev`. By convention hot fix branches should be prefixed with `hotfix/`, I.e. `hotfix/important-fix-to-something`, and then converted into a feature branch after merging into main. These do not have to include edits to the GitBook documentation, but if the hotfix conflicts with what is described in the GitBook documentation it's **strongly recommended**. ## Creating Releases