Auto Generate environment.yml
#339
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Heavily inspired by the original `batch/slurm_init.sh` script. The init script is a run once script that takes care of installation of dependencies and setup whereas prerun sets env vars needed per a run.
Initial version of the HPC install script, some what inspired by the slurm init script.
* Changed how the R arrow version is formatted for readability. * Changed the final output command to print diagnostic info correctly.
Added slurm's --partition flag to the `batch/inference_job_launcher.py` script for usage on UNC's Longleaf cluster.
The longleaf specific init/pre-run scripts are now surpassed by the generic `build/hpc_install.sh` script.
Remove the --partition flag for the slurm partition to use from the inference job launcher script. This will be handled in a new flepiscripts script.
* Changed `flepiMoP` git clone to use ssh instead of http to allow for edits from HPC. * Add `set -e` to error clearly on a command failure. * Install `gempyor` from cloned `flepiMoP` repo directly, yet to do the same for R packages.
Swapped out manual install from GitHub with the `build/setup.R` script which handles a few dependencies and installs CLIs as executables.
In light of disucssion about how directories are structured on longleaf split out the project path into a work directory. Still need to do the same for rockfish, for now assuming work and user dirs are the same there.
* Keep the work directory as where project is supposed to go. * Move flepiMoP source and conda env to $HOME.
Out sourced the per run setup into `build/flepi_init.sh` so users are not forced to update/reinstall just to run.
Use `devtools::install` for `install.packages` for better handling of source package installs.
The install script throws unexpected warnings about being unable to install arrow, even though arrow is already installed by conda.
The `inference::install_cli` function now installs to the bin folder provided by the conda environment. Co-authored-by: Carl A. B. Pearson <[email protected]>
Users now must call the init script themselves.
Moved R dependencies install from individually for each of the custom R packages into the conda environment. This should alleviate warnings relating to arrow install and streamline the install of R dependencies.
Add an explicit call to `inference::install_cli()` after flepiMoP custom R packages.
The covidcast package is not available through conda-forge, so has to be installed through CRAN.
* Downgrade R to 4.3 to resolve r-MMWRweek. * Correct directory change and missing repo errors.
Custom R script to generate a consistent conda `environment.yml` file.
* Added dnachun to the channels for osx-arm64 builds of r-truncnorm and r-ggraph. * Added an explicit r-sf dependency for covidcast that has to be installed manually outside of the environment file. * Bug fix to check if file exists before comparing.
Custom GitHub action to run the `build/create_environment_yml.R` script and then add those changes to a pull request if there are any.
Change the ref from a particular commit to the branch of the PR so the HEAD is attached.
Make it clear that the commit is made by a GitHub action by putting that in the commit message title.
|
Any reason for PR close while keeping the branch ? Hard to do ? or something for later |
|
No, just forgot to do so while working on this yesterday. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Describe your changes.
This pull request adds a script called
build/create_environment_yml.Rto generate a consistent condaenvironment.ymlfile that can be used to install the rest offlepiMoPinto. It also runs this script in a GitHub action that will automatically regenerate the file when one of the dependencies has changed.What does your pull request address? Tag relevant issues.
This pull request takes another step towards GH-191 by extracting the conda environment file portion of GH-329's
build/hpc_install.shinto it's own thing.Tag relevant team members.
n/a at the moment, draft PR