Bug/missing maap cfg #104
Merged
Bug/missing maap cfg #104
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
marjo-luc
approved these changes
Mar 26, 2024
bsatoriu
added a commit
that referenced
this pull request
Apr 4, 2024
* Feature/update maap help tour (#61) updated maap help jupyter extension to version 1.0.0 * Bugs/presigned url (#62) Main fix for the presigned url bug was changing 'maapsec/environment' to 'jupyter-server-extension/getConfig' There was another bug caused by a race condition that was causing users to not be able to get the presigned url about every 10th time. What fixed this bug is calling updateKeycloakToken upon the launch of the extension and also trying to call updateKeycloakToken again if getting the keycloak user information fails to get I confirmed this bug was fixed by trying to get the presigned url 25 times and it worked Successful build in vanilla here: https://repo.dit.maap-project.org/root/maap-workspaces/-/pipelines/3389 * Added memory extension (#59) added changes from cluttered commit branch * hysds base-image [skip ci] (#65) * Feature/libmamba solver (#64) * upgraded maap help jupyter extension version to 1.0.0 * reverted back changes * changed r docker to use libmamba solver * changed mamba install to conda * converted all images use libmamba solver like R image * added lang tz and debian frontend commands to vanilla docker image * copied pangeo dockerfile into vanilla * first line of packages back to vanilla versions * updated package versions for vanilla to be same as pangeo * updated isce2 package versions to match vanilla * updating continuumio/miniconda3 for rgedi and making other 3 top lines consistent * removed changes from rgedi * Update stac_ipyleaflet to 0.3.5 for Pangeo (#66) * Updated pinned version of stac_ipyleaflet to 0.3.5 for pangeo jupyter workspace. * Added environment variables to Pangeo environment yml. * Updated entrypoint bash script so the the base environment is activated before launching jupyter so that environment variables are accessible within jupyter. --------- Co-authored-by: Alex <[email protected]> * Remove hysds base_image [skip-ci] (#68) * Bugs/quick fix libmamba (#69) * removed libmamba sovler from conda env update * added solver to environment.yml * added conda libmamba solver version * added back in solvers and pinned version for conda-libmamba-solver * added back in space * Bugs/presigned url (#70) * changed dit url to test * changed dit back and changed ops to be maap-ops-workspace instead of dataset * Bugs/git tour link (#71) * manual build for change to maap help * updated npm version of maap help * updated custom ext versions (#72) * Feature/update maap py (#73) upgraded all maap py versions to 3.1.2 * bumped jupyter server extension (#74) * update server ext * bumped server ext * Bump maap-py to v3.1.3 (#75) * Update Dockerfile * Bugs/libmamba default (#77) * added conda config setting libmamba solver to entrypoint.sh * updated maap help to version 1.0.1 * converted back to 1.0.0 * trying to create .condarc file * removed touch because might work without it * upgraded conda libmamba solver * undid change because forgot I had PR open for this branch * added --system tag and removed conda config from entrypoint * added the system tag to all base image dockerfiles * Update entrypoint.sh (#78) * Bugs/double submit null username (#79) * updated maap help to version 1.0.1 * converted back to 1.0.0 * upgraded dps version to add feature disabling submit button/ saying last submitted info and fixing the null username bug * Features/update workspace dependencies (#80) * Updated how conda packages installs are handled at the base image layer for Pangeo to use an environment.yml file like we do for its respective jupyterlab image. * Updated to miniconda 23.5.2-0 for the pangeo base image. * Updated list of packages for the base and jupyter images for the Pangeo Workspace. * updated maap help to version 1.0.1 * converted back to 1.0.0 * Clean up of pangeo base environment.yml * Reverted the miniconda version of base pangeo image to 22.11.1. * Updated jupyterlab3 pangeo docker conda solver parameter * Modified base pangeo image conda env update solver parameter spec * explicility installing libarchive fron conda-forge to test resolution of issue with libmamba. * Testing base pangeo conda install parameter updates. * Test builds using new version of miniconda. * Test another build configuration on gitlab using new pinned version of conda-libmamba-solver * Bumped miniconda version to 23.10.0-1 and removed conda-libmamba-solver since libmamba is the default solver in this new version of miniconda. * Updated Pangeo base image and environment yml so that packages are installed in a new environment named with the workspace type to resolve issues with libarchive/libmamba and the installation of packages. * Updated Pangeo jupyterlab dockerfile and environment yml to use new custom environment instead of base environment. * Moved a couple jupyter python packages from dockerfile to the environment yml. * Updated entrypoint to test new environment variable that stores the type of workspace being run to ensure jupyter can be launched appropriately. * Patch issue with ARG values not carrying over to downstream build stages in docker file. Also replaced use of BUILD_IMAGE_TYPE with ENV var, "WORKSPACE_TYPE". * Pinned jupyterlab package to 3.6.1 to fix build issue with jupyterlab_plotly. * Pinned additional packages to assist with resolving jupyterlab build issues. * Commented line "conda init" in entrypoint.sh. * Reverted commenting out "conda init" in entrypoint that was previously done for testing. * Testing changes "~/.bash_profile" to properly set the default conda environment for terminals. * Updated path variable to point to our custom conda environment bin instead of the conda base env bin path. * Updated how the default environment is activated for terminals. * removed jupyter-server from envirionment yml as the jupyterlab installation takes care of installing the jupyter_server package already. * Added Makefile which contains commonly run recipes when modifying workspaces locally and needing to test building them. * Updated notebook lib path and jupyter server lib path to point to the appropriate custom conda environment. * current attempt at fixing r workspace issues, building for base image but not for jupyterlab3 * r image building successfully locally * added conda lists * Added perl package to r base environment.yml to resolve issue with r-tinytex not installing. * Added string replacement commands to remove conda env related commands from .bash_profile that are no longer needed. * Updated jupyterlab3's dockerfile so that it modifies the "jupyterlab3/shared/environment.yml" by doing a string replacement on the workspace environment name for workspaces that don't have a custom environment.yml. * r image building locally * pinned versions * updated pinned versions * changed version for r-dplyr * new botocore version * added aiobotocore dependency * removed aiobotocore from install * removed botocore install * added half of the packages to see if I can get rid of the botocore warning * removed some more environment.yml packages because still getting botocore error * removed even more packages because still getting botocore warnining * removed more packages because still getting botocore warning * Added conda list to dockerfiles for testing on ADE. * added maap py into pip install from source * updated git install from source * added v to version tag for maappy * added a bunch of conda lists trying to debug * installing maap py to root * corrected installing into root bug * added run pwd line to try to debug * tried setting the working directory to root before git clone * tried adding another pwd command * removed . from git clone * set working directory to maap py * added back in . for git clone * Updated packages for base and jupyterlab reported by data team that are needed in Python based workspaces. * Updated the folder parameter for where maap-py is installed to hopefully resolve maap-py installation issues. * Moved maap py conf environment variable to base environment yml. * commented out some of the code trying to test maap py again * commented back in code before adding in Anils code that worked * cleaning up jupyterlab3 dockerfile so is consistent * removed extra ; * removed pip install of maap py * updated botocore version in base environment.yml * added binary library files to r install * Removed commented ENV command in pangeo dockerfile. * Updated miniconda version for base vanilla image. * Updated base vanilla image to use an environment yml to store it's configuration environment. Vanilla base image now relies on a custom vanilla conda environment. * Updated jupyterlab shared environment yml so that 1) missing packages needed by data team are included, 2) each workspace uses a custom conda environment, and 3) updated entrypoint so that .bash_profile is cleaned up. * added r-car * pinned r-car version and removed botocore3 * added maap py to environment.yml to avoid botocore dependency errors * Added help documentation to Makefile along with other minor improvements. * commented out git clone for r image * Pinned pangeo packages and removed duplicates. * Pinned packages for vanilla and shared workspace environment ymls. Added missing jupyter package to shared environment yml. * Bumped stac_ipyleaflet to version 0.3.6 for pangeo workspace. * created isce3 environment.yml and pinned r binary library versions * fixed name of fsspec * added isce3 to the jupyterlab3 dockerfile * removed dependencies from isce3 environment.yml * added back in dependencies and removed environment.yml for isce3 in jupyter image * removed packages specific to isce3 * added in 3 packages but isce3 to see if that is the problem * added back in isce3 * removed all vanilla packages and just trying to install base isce3 packages * pinned python version * attempting to install r packages via script * pinned versions for fsspec, hvplot, isce3, xarray * added and modified r scripts and removed awscli trying to fix maappy dependency error * trying to add awscli after maappy install * capitalized Fgmutils * added packages to environment.yml * changed maappy branch to bugs/dependency issues * upgraded awscli version * unpinned versions for vanilla environment.yml * removed all r packages except python and r * unpinned python version * changed back to v3.1.3 tag for maap-py * downgraded awscli to 2.14.1 * upgraded awscali to 2.14.2 * removed pinnection vertsions for cryptography and prompt-toolkit * r image should be building now, doesnt include Fgmutils * unpinned versions to fix maap-py bug in vanilla workspace * pinned vanilla versions and changed to dependecies branch for pangeo and isce3 * changed branch for maappy to version with ipython pinned * pinned botocore version and added conda list * moved conda list command and just pushed maappy pinning boto3=1.33.13 * removed botocore from r/environment.yml * changed to correctly use repos and removed print statements * fixed but for Fgmutils install * pinned package versions that were dependency of Fgmutils * added r channel to r environment.yml * downgraded version of r-devemf * downgraded r and r-base to 4.2.3 to be compatible with r-devemf * changed to r version 4.2 * downgraded r essentials to 4.2 * Added geocube to vanilla base image which was added to pangeo but accidentally omitted from vanilla. * Added additional packages requested by user group for python based workspaces. * Removed pins for vanilla and pangeo base image to test new image builds on GitLab with additional packages. * Updated pinned versions for vanilla base image as well as shared environment yml for jupyterlab. Also bumped jupyterlab to 3.6.3 and added missing package, rio-tiler. * Added pinned version information for pangeo base and jupyterlab images. Also modified version information for jupyterlab to 3.6.3 to resolve version descrepancy with pangeo-notebook package. * Removed pin to test installation of the morecantile package. * Fixed typo of morecantile package in pangeo base image. * Repinned morecantile package for pangeo and vanilla base images. * removed print statements and conda lists * added conda config lines just for testing * added descriptions necessary to use install2.r with GPL license * removed lines making conda forge default because that should be a different PR * updated all maap-py branches to be new 3.1.4 release --------- Co-authored-by: Anil Natha <[email protected]> * Feature/conda forge default (#81) * updated maap help to version 1.0.1 * converted back to 1.0.0 * added in lines that set conda forge as the default and create .condarc * fixed typo * Feature/missing r package (#83) * updated maap help to version 1.0.1 * converted back to 1.0.0 * added in lines that set conda forge as the default and create .condarc * added r-BiocManager=1.30.22 package * removed sfarrow just for testing * added back in sfarrow package * added arrow package to environment.yml * added r-arrow dependency * removed arrow package and pinned r-arrow * fixed pinned r-arrow version to 14.0.1 * updated version of libpq-dev to avoid build error * Add `stac_ipyleaflet` (#84) * Added stac-ipyleaflet to shared environment yaml. * Minor commenting and visual improvements to Makefile * Feature/triaged job folder (#88) * updated maap help to version 1.0.1 * converted back to 1.0.0 * added in lines that set conda forge as the default and create .condarc * updated all devfiles to have triaged_job folder * changed instances of triaged_job to triaged-jobs * Fix s3fs sidecar image urls (#89) fix s3fs sidecar image Co-authored-by: bsatoriu <[email protected]> * Bug/update dps version (#90) * updated maap help to version 1.0.1 * converted back to 1.0.0 * added in lines that set conda forge as the default and create .condarc * increased dps version to 0.5.9 * Bug/jlab launch error (#94) * manual install for jupyter server extension * changed to npm package of jupyter server extension I published * test pypi install of maap jupyter server extension with jupyter_server version pinned * upgraded version of test pypi package maap jupyter server extension * increased version maap jupyter server extension test pypi after rebuilding the entire extension before publishing * upgraded to 1.2.3 of maap jupyter server extension pypi version * removed comment * Add additional packages to Pangeo (#93) Added additional, pinned packages to pangeo base and jupyter lab images. * Awscli bug (#95) * added the groff package to vanilla * added groff to all images * pinned version for groff * Feature/move rio tiler (#86) * updated maap help to version 1.0.1 * converted back to 1.0.0 * added in lines that set conda forge as the default and create .condarc * added rio tiler to the base image instead of jupyterlab3 image * removed rio tiler from all jupyterlab3 environment.yml and added to base images * Removed ability to build and launch ISCE2 and RGEDI images/workspaces. (#96) Removed ability to build ISCE2 and RGEDI images. * Feature/update maap help (#97) added maap help jupyter extension install to pip and v1.2.2=1 * Feature/update s3fs (#98) updated s3fs to 2023.12.2 * Bug/resolve merge conflicts (#101) * fixed merge conflicts * finished fixing merge conflicts * merge conflicts makefile * upgrade maap-py to v3.1.5 (#102) * Update devfile metadata (#103) * Bug/missing maap cfg (#104) * changed maap-py to be using branch commenting out check for maap.cfg * switched back to v3.1.5 and removed MAAP_CONF * removed conda lists * Bug/maap py dependency errors (#106) * changed maap-py to be using branch commenting out check for maap.cfg * switched back to v3.1.5 and removed MAAP_CONF * added dependencies from installing maap-py first * merged in Brians errors and testing vanilla by pinning boto3 in maap-py and making awscli=2.15.0 * only pinning minor version for awscli vanilla * testing fix that solved vanilla aiobotocore error * try upgrading awscli to 2.14.6 * upgraded awscli for isce3 to 2.14.6 * isce3 awscli to 2.14.5 * awscli 2.14.6 and unpinned most other package versions * pinned back all vanilla versions but awscli 2.14.2 * changed all maap-py branches to bug/unpinned-versions and boto3 versions to 1.34.41 * added maap-py var back into isce3 and cleaned up some conda lists * added boto3 to r image * added in jupyter server extension so correctly testing pinning xmltodict * installing jupyter server extension 1.3.4 * removed extra conda lists and changed to maap-py version v3.1.5 --------- Co-authored-by: bsatoriu <[email protected]> Co-authored-by: Anil Natha <[email protected]> Co-authored-by: Alex <[email protected]> Co-authored-by: Marjorie Lucas <[email protected]> Co-authored-by: Sujen Shah <[email protected]> Co-authored-by: Anil Natha <[email protected]> Co-authored-by: bsatoriu <[email protected]> Co-authored-by: bsatoriu <[email protected]>
grallewellyn
added a commit
that referenced
this pull request
Apr 4, 2024
* Feature/update maap help tour (#61) updated maap help jupyter extension to version 1.0.0 * Bugs/presigned url (#62) Main fix for the presigned url bug was changing 'maapsec/environment' to 'jupyter-server-extension/getConfig' There was another bug caused by a race condition that was causing users to not be able to get the presigned url about every 10th time. What fixed this bug is calling updateKeycloakToken upon the launch of the extension and also trying to call updateKeycloakToken again if getting the keycloak user information fails to get I confirmed this bug was fixed by trying to get the presigned url 25 times and it worked Successful build in vanilla here: https://repo.dit.maap-project.org/root/maap-workspaces/-/pipelines/3389 * Added memory extension (#59) added changes from cluttered commit branch * hysds base-image [skip ci] (#65) * Feature/libmamba solver (#64) * upgraded maap help jupyter extension version to 1.0.0 * reverted back changes * changed r docker to use libmamba solver * changed mamba install to conda * converted all images use libmamba solver like R image * added lang tz and debian frontend commands to vanilla docker image * copied pangeo dockerfile into vanilla * first line of packages back to vanilla versions * updated package versions for vanilla to be same as pangeo * updated isce2 package versions to match vanilla * updating continuumio/miniconda3 for rgedi and making other 3 top lines consistent * removed changes from rgedi * Update stac_ipyleaflet to 0.3.5 for Pangeo (#66) * Updated pinned version of stac_ipyleaflet to 0.3.5 for pangeo jupyter workspace. * Added environment variables to Pangeo environment yml. * Updated entrypoint bash script so the the base environment is activated before launching jupyter so that environment variables are accessible within jupyter. --------- Co-authored-by: Alex <[email protected]> * Remove hysds base_image [skip-ci] (#68) * Bugs/quick fix libmamba (#69) * removed libmamba sovler from conda env update * added solver to environment.yml * added conda libmamba solver version * added back in solvers and pinned version for conda-libmamba-solver * added back in space * Bugs/presigned url (#70) * changed dit url to test * changed dit back and changed ops to be maap-ops-workspace instead of dataset * Bugs/git tour link (#71) * manual build for change to maap help * updated npm version of maap help * updated custom ext versions (#72) * Feature/update maap py (#73) upgraded all maap py versions to 3.1.2 * bumped jupyter server extension (#74) * update server ext * bumped server ext * Bump maap-py to v3.1.3 (#75) * Update Dockerfile * Bugs/libmamba default (#77) * added conda config setting libmamba solver to entrypoint.sh * updated maap help to version 1.0.1 * converted back to 1.0.0 * trying to create .condarc file * removed touch because might work without it * upgraded conda libmamba solver * undid change because forgot I had PR open for this branch * added --system tag and removed conda config from entrypoint * added the system tag to all base image dockerfiles * Update entrypoint.sh (#78) * Bugs/double submit null username (#79) * updated maap help to version 1.0.1 * converted back to 1.0.0 * upgraded dps version to add feature disabling submit button/ saying last submitted info and fixing the null username bug * Features/update workspace dependencies (#80) * Updated how conda packages installs are handled at the base image layer for Pangeo to use an environment.yml file like we do for its respective jupyterlab image. * Updated to miniconda 23.5.2-0 for the pangeo base image. * Updated list of packages for the base and jupyter images for the Pangeo Workspace. * updated maap help to version 1.0.1 * converted back to 1.0.0 * Clean up of pangeo base environment.yml * Reverted the miniconda version of base pangeo image to 22.11.1. * Updated jupyterlab3 pangeo docker conda solver parameter * Modified base pangeo image conda env update solver parameter spec * explicility installing libarchive fron conda-forge to test resolution of issue with libmamba. * Testing base pangeo conda install parameter updates. * Test builds using new version of miniconda. * Test another build configuration on gitlab using new pinned version of conda-libmamba-solver * Bumped miniconda version to 23.10.0-1 and removed conda-libmamba-solver since libmamba is the default solver in this new version of miniconda. * Updated Pangeo base image and environment yml so that packages are installed in a new environment named with the workspace type to resolve issues with libarchive/libmamba and the installation of packages. * Updated Pangeo jupyterlab dockerfile and environment yml to use new custom environment instead of base environment. * Moved a couple jupyter python packages from dockerfile to the environment yml. * Updated entrypoint to test new environment variable that stores the type of workspace being run to ensure jupyter can be launched appropriately. * Patch issue with ARG values not carrying over to downstream build stages in docker file. Also replaced use of BUILD_IMAGE_TYPE with ENV var, "WORKSPACE_TYPE". * Pinned jupyterlab package to 3.6.1 to fix build issue with jupyterlab_plotly. * Pinned additional packages to assist with resolving jupyterlab build issues. * Commented line "conda init" in entrypoint.sh. * Reverted commenting out "conda init" in entrypoint that was previously done for testing. * Testing changes "~/.bash_profile" to properly set the default conda environment for terminals. * Updated path variable to point to our custom conda environment bin instead of the conda base env bin path. * Updated how the default environment is activated for terminals. * removed jupyter-server from envirionment yml as the jupyterlab installation takes care of installing the jupyter_server package already. * Added Makefile which contains commonly run recipes when modifying workspaces locally and needing to test building them. * Updated notebook lib path and jupyter server lib path to point to the appropriate custom conda environment. * current attempt at fixing r workspace issues, building for base image but not for jupyterlab3 * r image building successfully locally * added conda lists * Added perl package to r base environment.yml to resolve issue with r-tinytex not installing. * Added string replacement commands to remove conda env related commands from .bash_profile that are no longer needed. * Updated jupyterlab3's dockerfile so that it modifies the "jupyterlab3/shared/environment.yml" by doing a string replacement on the workspace environment name for workspaces that don't have a custom environment.yml. * r image building locally * pinned versions * updated pinned versions * changed version for r-dplyr * new botocore version * added aiobotocore dependency * removed aiobotocore from install * removed botocore install * added half of the packages to see if I can get rid of the botocore warning * removed some more environment.yml packages because still getting botocore error * removed even more packages because still getting botocore warnining * removed more packages because still getting botocore warning * Added conda list to dockerfiles for testing on ADE. * added maap py into pip install from source * updated git install from source * added v to version tag for maappy * added a bunch of conda lists trying to debug * installing maap py to root * corrected installing into root bug * added run pwd line to try to debug * tried setting the working directory to root before git clone * tried adding another pwd command * removed . from git clone * set working directory to maap py * added back in . for git clone * Updated packages for base and jupyterlab reported by data team that are needed in Python based workspaces. * Updated the folder parameter for where maap-py is installed to hopefully resolve maap-py installation issues. * Moved maap py conf environment variable to base environment yml. * commented out some of the code trying to test maap py again * commented back in code before adding in Anils code that worked * cleaning up jupyterlab3 dockerfile so is consistent * removed extra ; * removed pip install of maap py * updated botocore version in base environment.yml * added binary library files to r install * Removed commented ENV command in pangeo dockerfile. * Updated miniconda version for base vanilla image. * Updated base vanilla image to use an environment yml to store it's configuration environment. Vanilla base image now relies on a custom vanilla conda environment. * Updated jupyterlab shared environment yml so that 1) missing packages needed by data team are included, 2) each workspace uses a custom conda environment, and 3) updated entrypoint so that .bash_profile is cleaned up. * added r-car * pinned r-car version and removed botocore3 * added maap py to environment.yml to avoid botocore dependency errors * Added help documentation to Makefile along with other minor improvements. * commented out git clone for r image * Pinned pangeo packages and removed duplicates. * Pinned packages for vanilla and shared workspace environment ymls. Added missing jupyter package to shared environment yml. * Bumped stac_ipyleaflet to version 0.3.6 for pangeo workspace. * created isce3 environment.yml and pinned r binary library versions * fixed name of fsspec * added isce3 to the jupyterlab3 dockerfile * removed dependencies from isce3 environment.yml * added back in dependencies and removed environment.yml for isce3 in jupyter image * removed packages specific to isce3 * added in 3 packages but isce3 to see if that is the problem * added back in isce3 * removed all vanilla packages and just trying to install base isce3 packages * pinned python version * attempting to install r packages via script * pinned versions for fsspec, hvplot, isce3, xarray * added and modified r scripts and removed awscli trying to fix maappy dependency error * trying to add awscli after maappy install * capitalized Fgmutils * added packages to environment.yml * changed maappy branch to bugs/dependency issues * upgraded awscli version * unpinned versions for vanilla environment.yml * removed all r packages except python and r * unpinned python version * changed back to v3.1.3 tag for maap-py * downgraded awscli to 2.14.1 * upgraded awscali to 2.14.2 * removed pinnection vertsions for cryptography and prompt-toolkit * r image should be building now, doesnt include Fgmutils * unpinned versions to fix maap-py bug in vanilla workspace * pinned vanilla versions and changed to dependecies branch for pangeo and isce3 * changed branch for maappy to version with ipython pinned * pinned botocore version and added conda list * moved conda list command and just pushed maappy pinning boto3=1.33.13 * removed botocore from r/environment.yml * changed to correctly use repos and removed print statements * fixed but for Fgmutils install * pinned package versions that were dependency of Fgmutils * added r channel to r environment.yml * downgraded version of r-devemf * downgraded r and r-base to 4.2.3 to be compatible with r-devemf * changed to r version 4.2 * downgraded r essentials to 4.2 * Added geocube to vanilla base image which was added to pangeo but accidentally omitted from vanilla. * Added additional packages requested by user group for python based workspaces. * Removed pins for vanilla and pangeo base image to test new image builds on GitLab with additional packages. * Updated pinned versions for vanilla base image as well as shared environment yml for jupyterlab. Also bumped jupyterlab to 3.6.3 and added missing package, rio-tiler. * Added pinned version information for pangeo base and jupyterlab images. Also modified version information for jupyterlab to 3.6.3 to resolve version descrepancy with pangeo-notebook package. * Removed pin to test installation of the morecantile package. * Fixed typo of morecantile package in pangeo base image. * Repinned morecantile package for pangeo and vanilla base images. * removed print statements and conda lists * added conda config lines just for testing * added descriptions necessary to use install2.r with GPL license * removed lines making conda forge default because that should be a different PR * updated all maap-py branches to be new 3.1.4 release --------- Co-authored-by: Anil Natha <[email protected]> * Feature/conda forge default (#81) * updated maap help to version 1.0.1 * converted back to 1.0.0 * added in lines that set conda forge as the default and create .condarc * fixed typo * Feature/missing r package (#83) * updated maap help to version 1.0.1 * converted back to 1.0.0 * added in lines that set conda forge as the default and create .condarc * added r-BiocManager=1.30.22 package * removed sfarrow just for testing * added back in sfarrow package * added arrow package to environment.yml * added r-arrow dependency * removed arrow package and pinned r-arrow * fixed pinned r-arrow version to 14.0.1 * updated version of libpq-dev to avoid build error * Add `stac_ipyleaflet` (#84) * Added stac-ipyleaflet to shared environment yaml. * Minor commenting and visual improvements to Makefile * Feature/triaged job folder (#88) * updated maap help to version 1.0.1 * converted back to 1.0.0 * added in lines that set conda forge as the default and create .condarc * updated all devfiles to have triaged_job folder * changed instances of triaged_job to triaged-jobs * Fix s3fs sidecar image urls (#89) fix s3fs sidecar image Co-authored-by: bsatoriu <[email protected]> * Bug/update dps version (#90) * updated maap help to version 1.0.1 * converted back to 1.0.0 * added in lines that set conda forge as the default and create .condarc * increased dps version to 0.5.9 * Bug/jlab launch error (#94) * manual install for jupyter server extension * changed to npm package of jupyter server extension I published * test pypi install of maap jupyter server extension with jupyter_server version pinned * upgraded version of test pypi package maap jupyter server extension * increased version maap jupyter server extension test pypi after rebuilding the entire extension before publishing * upgraded to 1.2.3 of maap jupyter server extension pypi version * removed comment * Add additional packages to Pangeo (#93) Added additional, pinned packages to pangeo base and jupyter lab images. * Awscli bug (#95) * added the groff package to vanilla * added groff to all images * pinned version for groff * Feature/move rio tiler (#86) * updated maap help to version 1.0.1 * converted back to 1.0.0 * added in lines that set conda forge as the default and create .condarc * added rio tiler to the base image instead of jupyterlab3 image * removed rio tiler from all jupyterlab3 environment.yml and added to base images * Removed ability to build and launch ISCE2 and RGEDI images/workspaces. (#96) Removed ability to build ISCE2 and RGEDI images. * Feature/update maap help (#97) added maap help jupyter extension install to pip and v1.2.2=1 * Feature/update s3fs (#98) updated s3fs to 2023.12.2 * Bug/resolve merge conflicts (#101) * fixed merge conflicts * finished fixing merge conflicts * merge conflicts makefile * upgrade maap-py to v3.1.5 (#102) * Update devfile metadata (#103) * Bug/missing maap cfg (#104) * changed maap-py to be using branch commenting out check for maap.cfg * switched back to v3.1.5 and removed MAAP_CONF * removed conda lists * Bug/maap py dependency errors (#106) * changed maap-py to be using branch commenting out check for maap.cfg * switched back to v3.1.5 and removed MAAP_CONF * added dependencies from installing maap-py first * merged in Brians errors and testing vanilla by pinning boto3 in maap-py and making awscli=2.15.0 * only pinning minor version for awscli vanilla * testing fix that solved vanilla aiobotocore error * try upgrading awscli to 2.14.6 * upgraded awscli for isce3 to 2.14.6 * isce3 awscli to 2.14.5 * awscli 2.14.6 and unpinned most other package versions * pinned back all vanilla versions but awscli 2.14.2 * changed all maap-py branches to bug/unpinned-versions and boto3 versions to 1.34.41 * added maap-py var back into isce3 and cleaned up some conda lists * added boto3 to r image * added in jupyter server extension so correctly testing pinning xmltodict * installing jupyter server extension 1.3.4 * removed extra conda lists and changed to maap-py version v3.1.5 * Bug/add maap py var (#107) added back in maap-py variable * rio-tiler==6.2.8 --------- Co-authored-by: Grace Llewellyn <[email protected]> Co-authored-by: Anil Natha <[email protected]> Co-authored-by: Alex <[email protected]> Co-authored-by: Marjorie Lucas <[email protected]> Co-authored-by: Sujen Shah <[email protected]> Co-authored-by: Anil Natha <[email protected]> Co-authored-by: bsatoriu <[email protected]> Co-authored-by: grallewellyn <[email protected]>
bsatoriu
added a commit
that referenced
this pull request
Jun 26, 2024
* Feature/update maap help tour (#61) updated maap help jupyter extension to version 1.0.0 * Bugs/presigned url (#62) Main fix for the presigned url bug was changing 'maapsec/environment' to 'jupyter-server-extension/getConfig' There was another bug caused by a race condition that was causing users to not be able to get the presigned url about every 10th time. What fixed this bug is calling updateKeycloakToken upon the launch of the extension and also trying to call updateKeycloakToken again if getting the keycloak user information fails to get I confirmed this bug was fixed by trying to get the presigned url 25 times and it worked Successful build in vanilla here: https://repo.dit.maap-project.org/root/maap-workspaces/-/pipelines/3389 * Added memory extension (#59) added changes from cluttered commit branch * hysds base-image [skip ci] (#65) * Feature/libmamba solver (#64) * upgraded maap help jupyter extension version to 1.0.0 * reverted back changes * changed r docker to use libmamba solver * changed mamba install to conda * converted all images use libmamba solver like R image * added lang tz and debian frontend commands to vanilla docker image * copied pangeo dockerfile into vanilla * first line of packages back to vanilla versions * updated package versions for vanilla to be same as pangeo * updated isce2 package versions to match vanilla * updating continuumio/miniconda3 for rgedi and making other 3 top lines consistent * removed changes from rgedi * Update stac_ipyleaflet to 0.3.5 for Pangeo (#66) * Updated pinned version of stac_ipyleaflet to 0.3.5 for pangeo jupyter workspace. * Added environment variables to Pangeo environment yml. * Updated entrypoint bash script so the the base environment is activated before launching jupyter so that environment variables are accessible within jupyter. --------- Co-authored-by: Alex <[email protected]> * Remove hysds base_image [skip-ci] (#68) * Bugs/quick fix libmamba (#69) * removed libmamba sovler from conda env update * added solver to environment.yml * added conda libmamba solver version * added back in solvers and pinned version for conda-libmamba-solver * added back in space * Bugs/presigned url (#70) * changed dit url to test * changed dit back and changed ops to be maap-ops-workspace instead of dataset * Bugs/git tour link (#71) * manual build for change to maap help * updated npm version of maap help * updated custom ext versions (#72) * Feature/update maap py (#73) upgraded all maap py versions to 3.1.2 * bumped jupyter server extension (#74) * update server ext * bumped server ext * Bump maap-py to v3.1.3 (#75) * Update Dockerfile * Bugs/libmamba default (#77) * added conda config setting libmamba solver to entrypoint.sh * updated maap help to version 1.0.1 * converted back to 1.0.0 * trying to create .condarc file * removed touch because might work without it * upgraded conda libmamba solver * undid change because forgot I had PR open for this branch * added --system tag and removed conda config from entrypoint * added the system tag to all base image dockerfiles * Update entrypoint.sh (#78) * Bugs/double submit null username (#79) * updated maap help to version 1.0.1 * converted back to 1.0.0 * upgraded dps version to add feature disabling submit button/ saying last submitted info and fixing the null username bug * Features/update workspace dependencies (#80) * Updated how conda packages installs are handled at the base image layer for Pangeo to use an environment.yml file like we do for its respective jupyterlab image. * Updated to miniconda 23.5.2-0 for the pangeo base image. * Updated list of packages for the base and jupyter images for the Pangeo Workspace. * updated maap help to version 1.0.1 * converted back to 1.0.0 * Clean up of pangeo base environment.yml * Reverted the miniconda version of base pangeo image to 22.11.1. * Updated jupyterlab3 pangeo docker conda solver parameter * Modified base pangeo image conda env update solver parameter spec * explicility installing libarchive fron conda-forge to test resolution of issue with libmamba. * Testing base pangeo conda install parameter updates. * Test builds using new version of miniconda. * Test another build configuration on gitlab using new pinned version of conda-libmamba-solver * Bumped miniconda version to 23.10.0-1 and removed conda-libmamba-solver since libmamba is the default solver in this new version of miniconda. * Updated Pangeo base image and environment yml so that packages are installed in a new environment named with the workspace type to resolve issues with libarchive/libmamba and the installation of packages. * Updated Pangeo jupyterlab dockerfile and environment yml to use new custom environment instead of base environment. * Moved a couple jupyter python packages from dockerfile to the environment yml. * Updated entrypoint to test new environment variable that stores the type of workspace being run to ensure jupyter can be launched appropriately. * Patch issue with ARG values not carrying over to downstream build stages in docker file. Also replaced use of BUILD_IMAGE_TYPE with ENV var, "WORKSPACE_TYPE". * Pinned jupyterlab package to 3.6.1 to fix build issue with jupyterlab_plotly. * Pinned additional packages to assist with resolving jupyterlab build issues. * Commented line "conda init" in entrypoint.sh. * Reverted commenting out "conda init" in entrypoint that was previously done for testing. * Testing changes "~/.bash_profile" to properly set the default conda environment for terminals. * Updated path variable to point to our custom conda environment bin instead of the conda base env bin path. * Updated how the default environment is activated for terminals. * removed jupyter-server from envirionment yml as the jupyterlab installation takes care of installing the jupyter_server package already. * Added Makefile which contains commonly run recipes when modifying workspaces locally and needing to test building them. * Updated notebook lib path and jupyter server lib path to point to the appropriate custom conda environment. * current attempt at fixing r workspace issues, building for base image but not for jupyterlab3 * r image building successfully locally * added conda lists * Added perl package to r base environment.yml to resolve issue with r-tinytex not installing. * Added string replacement commands to remove conda env related commands from .bash_profile that are no longer needed. * Updated jupyterlab3's dockerfile so that it modifies the "jupyterlab3/shared/environment.yml" by doing a string replacement on the workspace environment name for workspaces that don't have a custom environment.yml. * r image building locally * pinned versions * updated pinned versions * changed version for r-dplyr * new botocore version * added aiobotocore dependency * removed aiobotocore from install * removed botocore install * added half of the packages to see if I can get rid of the botocore warning * removed some more environment.yml packages because still getting botocore error * removed even more packages because still getting botocore warnining * removed more packages because still getting botocore warning * Added conda list to dockerfiles for testing on ADE. * added maap py into pip install from source * updated git install from source * added v to version tag for maappy * added a bunch of conda lists trying to debug * installing maap py to root * corrected installing into root bug * added run pwd line to try to debug * tried setting the working directory to root before git clone * tried adding another pwd command * removed . from git clone * set working directory to maap py * added back in . for git clone * Updated packages for base and jupyterlab reported by data team that are needed in Python based workspaces. * Updated the folder parameter for where maap-py is installed to hopefully resolve maap-py installation issues. * Moved maap py conf environment variable to base environment yml. * commented out some of the code trying to test maap py again * commented back in code before adding in Anils code that worked * cleaning up jupyterlab3 dockerfile so is consistent * removed extra ; * removed pip install of maap py * updated botocore version in base environment.yml * added binary library files to r install * Removed commented ENV command in pangeo dockerfile. * Updated miniconda version for base vanilla image. * Updated base vanilla image to use an environment yml to store it's configuration environment. Vanilla base image now relies on a custom vanilla conda environment. * Updated jupyterlab shared environment yml so that 1) missing packages needed by data team are included, 2) each workspace uses a custom conda environment, and 3) updated entrypoint so that .bash_profile is cleaned up. * added r-car * pinned r-car version and removed botocore3 * added maap py to environment.yml to avoid botocore dependency errors * Added help documentation to Makefile along with other minor improvements. * commented out git clone for r image * Pinned pangeo packages and removed duplicates. * Pinned packages for vanilla and shared workspace environment ymls. Added missing jupyter package to shared environment yml. * Bumped stac_ipyleaflet to version 0.3.6 for pangeo workspace. * created isce3 environment.yml and pinned r binary library versions * fixed name of fsspec * added isce3 to the jupyterlab3 dockerfile * removed dependencies from isce3 environment.yml * added back in dependencies and removed environment.yml for isce3 in jupyter image * removed packages specific to isce3 * added in 3 packages but isce3 to see if that is the problem * added back in isce3 * removed all vanilla packages and just trying to install base isce3 packages * pinned python version * attempting to install r packages via script * pinned versions for fsspec, hvplot, isce3, xarray * added and modified r scripts and removed awscli trying to fix maappy dependency error * trying to add awscli after maappy install * capitalized Fgmutils * added packages to environment.yml * changed maappy branch to bugs/dependency issues * upgraded awscli version * unpinned versions for vanilla environment.yml * removed all r packages except python and r * unpinned python version * changed back to v3.1.3 tag for maap-py * downgraded awscli to 2.14.1 * upgraded awscali to 2.14.2 * removed pinnection vertsions for cryptography and prompt-toolkit * r image should be building now, doesnt include Fgmutils * unpinned versions to fix maap-py bug in vanilla workspace * pinned vanilla versions and changed to dependecies branch for pangeo and isce3 * changed branch for maappy to version with ipython pinned * pinned botocore version and added conda list * moved conda list command and just pushed maappy pinning boto3=1.33.13 * removed botocore from r/environment.yml * changed to correctly use repos and removed print statements * fixed but for Fgmutils install * pinned package versions that were dependency of Fgmutils * added r channel to r environment.yml * downgraded version of r-devemf * downgraded r and r-base to 4.2.3 to be compatible with r-devemf * changed to r version 4.2 * downgraded r essentials to 4.2 * Added geocube to vanilla base image which was added to pangeo but accidentally omitted from vanilla. * Added additional packages requested by user group for python based workspaces. * Removed pins for vanilla and pangeo base image to test new image builds on GitLab with additional packages. * Updated pinned versions for vanilla base image as well as shared environment yml for jupyterlab. Also bumped jupyterlab to 3.6.3 and added missing package, rio-tiler. * Added pinned version information for pangeo base and jupyterlab images. Also modified version information for jupyterlab to 3.6.3 to resolve version descrepancy with pangeo-notebook package. * Removed pin to test installation of the morecantile package. * Fixed typo of morecantile package in pangeo base image. * Repinned morecantile package for pangeo and vanilla base images. * removed print statements and conda lists * added conda config lines just for testing * added descriptions necessary to use install2.r with GPL license * removed lines making conda forge default because that should be a different PR * updated all maap-py branches to be new 3.1.4 release --------- Co-authored-by: Anil Natha <[email protected]> * Feature/conda forge default (#81) * updated maap help to version 1.0.1 * converted back to 1.0.0 * added in lines that set conda forge as the default and create .condarc * fixed typo * Feature/missing r package (#83) * updated maap help to version 1.0.1 * converted back to 1.0.0 * added in lines that set conda forge as the default and create .condarc * added r-BiocManager=1.30.22 package * removed sfarrow just for testing * added back in sfarrow package * added arrow package to environment.yml * added r-arrow dependency * removed arrow package and pinned r-arrow * fixed pinned r-arrow version to 14.0.1 * updated version of libpq-dev to avoid build error * Add `stac_ipyleaflet` (#84) * Added stac-ipyleaflet to shared environment yaml. * Minor commenting and visual improvements to Makefile * Feature/triaged job folder (#88) * updated maap help to version 1.0.1 * converted back to 1.0.0 * added in lines that set conda forge as the default and create .condarc * updated all devfiles to have triaged_job folder * changed instances of triaged_job to triaged-jobs * Fix s3fs sidecar image urls (#89) fix s3fs sidecar image Co-authored-by: bsatoriu <[email protected]> * Bug/update dps version (#90) * updated maap help to version 1.0.1 * converted back to 1.0.0 * added in lines that set conda forge as the default and create .condarc * increased dps version to 0.5.9 * Bug/jlab launch error (#94) * manual install for jupyter server extension * changed to npm package of jupyter server extension I published * test pypi install of maap jupyter server extension with jupyter_server version pinned * upgraded version of test pypi package maap jupyter server extension * increased version maap jupyter server extension test pypi after rebuilding the entire extension before publishing * upgraded to 1.2.3 of maap jupyter server extension pypi version * removed comment * Add additional packages to Pangeo (#93) Added additional, pinned packages to pangeo base and jupyter lab images. * Awscli bug (#95) * added the groff package to vanilla * added groff to all images * pinned version for groff * Feature/move rio tiler (#86) * updated maap help to version 1.0.1 * converted back to 1.0.0 * added in lines that set conda forge as the default and create .condarc * added rio tiler to the base image instead of jupyterlab3 image * removed rio tiler from all jupyterlab3 environment.yml and added to base images * Removed ability to build and launch ISCE2 and RGEDI images/workspaces. (#96) Removed ability to build ISCE2 and RGEDI images. * Feature/update maap help (#97) added maap help jupyter extension install to pip and v1.2.2=1 * Feature/update s3fs (#98) updated s3fs to 2023.12.2 * Bug/resolve merge conflicts (#101) * fixed merge conflicts * finished fixing merge conflicts * merge conflicts makefile * upgrade maap-py to v3.1.5 (#102) * Update devfile metadata (#103) * Bug/missing maap cfg (#104) * changed maap-py to be using branch commenting out check for maap.cfg * switched back to v3.1.5 and removed MAAP_CONF * removed conda lists * Bug/maap py dependency errors (#106) * changed maap-py to be using branch commenting out check for maap.cfg * switched back to v3.1.5 and removed MAAP_CONF * added dependencies from installing maap-py first * merged in Brians errors and testing vanilla by pinning boto3 in maap-py and making awscli=2.15.0 * only pinning minor version for awscli vanilla * testing fix that solved vanilla aiobotocore error * try upgrading awscli to 2.14.6 * upgraded awscli for isce3 to 2.14.6 * isce3 awscli to 2.14.5 * awscli 2.14.6 and unpinned most other package versions * pinned back all vanilla versions but awscli 2.14.2 * changed all maap-py branches to bug/unpinned-versions and boto3 versions to 1.34.41 * added maap-py var back into isce3 and cleaned up some conda lists * added boto3 to r image * added in jupyter server extension so correctly testing pinning xmltodict * installing jupyter server extension 1.3.4 * removed extra conda lists and changed to maap-py version v3.1.5 * Bug/add maap py var (#107) added back in maap-py variable * rio-tiler==6.2.8 * Feature/extract jupyter extension installs (#110) * extracted all jupyter extensions except for jupyter server extension * extracted jupyter server extension install * commented out enabling jupyter server extension * commented back in enable for maap_jupyter_server_extension and added installs to pangeo environment.yml * changed maap_jupyter_server_extension to jupyter_server_extension * Bug/remove conda run (#112) * switched to conda init and installing conda/pip directly into vanilla * adding comments about where to remove from in jupyterlab3/dockerfile * changed maap-py install and added back in package vanilla environment.yml * extracted all jupyter extensions except for jupyter server extension * extracted jupyter server extension install * commented out enabling jupyter server extension * commented back in enable for maap_jupyter_server_extension and added installs to pangeo environment.yml * changed enabling maap_jupyter_server_extension back to jupyter_server_extension * added conda install of nodejs so can use npm * added installing jupyterlab to the dockerfile so we can run jupyter labextension install commands * changed to location of jupyter for jupyter install commands * removed install of nodejs and jupyterlab and added path to npm for install * added which npm because getting error installing with that node version * moved npm install command * attempting to set the path for npm to install * setting the path variable and converting the path to jupyter back to jupyter * changed jupyterlab-plotly to npm install * removed exporting the path in entrypoint.sh to see if it still works * added npm and jupyter commands into source activate command * trying to just hardcode in vanilla * changed to a single source activate line in both dockerfiles * added shell bin/bash command * added /bin/bash -c command in front of source activate * adding bin/bash to more commands * SHELL for source activate lines and conda init in base image dockerfile * deleted comments * updated all images to conda init and install maap-py correctly * source activating for maap-py installs * removed SHELL bin/bash from jupyterlab3/dockerfile and added r script into r source activate * added comment for commenting out SHELL * updated conda list commands to activate the environment first * added conda list to above command in jupyterlab3 but keeping base images separate because extracting maap-py soon * Feature/jlab4 upgrade (#113) * jlab upgrade to v4.1.6 * updated other jupyter packages * test * test * test * test * test * test * test * added jupyter-server * added jupyter-server * test * test * test * test * test * comment about not upgrading notebook * added plotly to the environment.yml * ext updates * removed plotly from the pip install for environment.yml * updated maap jupyter server ext * added xmltodict * version updates * updated shared env yaml * version updates * removed version * version update * test * test * test * test * lock packages * update paths * extracted extensions and disabling announcements again * added xmltodict to R and bumped jupyterlab to 4 for pangeo and pinned versions shared environment * added back in jupyter lab build/clean commands * added back in the npm install jupyterlab plotly line * unpinned versions to resolve dependency conflicts * removed rise package * pinned pangeo versions --------- Co-authored-by: grallewellyn <[email protected]> * Feature/extract plotly (#115) removed npm install jupyterlab-plotly and jlpm cache clean * Feature/update packages (#114) * successful build for updating vanilla packages * upgraded all base image packages from conda update --all * unpinned problematic versions * updated jupyterlab3 environment.ymls * changed plotly install from jupyterlab extension to npm * nodejs upgrade to 20.1.0 * commented out rise package for pangeo * attempting to debug problems with pangeo and r images * added unpinned vanilla packages to the R base image * added in upgrade changes to jupyterlab4 and unpinned R versions * added back in rise and nb_conda * pinned r and pangeo packages * downgraded numba one more version * downgraded plotly and numba * downgraded plotly * downgraded nodejs and pinned version of jupyter server extension * upgraded nodejs to 20.1.0 because error building * converting back recent changes because vanilla started failing * removed jupyterlab-plotly install line and added xmltodict install * added other changes back and added npm install jupyterlab-plotly line back in * changed jupyter server extension to the version with bug fix and removed extensions from environment.ymls * put pip installs for extensions into source activate line * added lonboard package * added jupyter_server package to pangeo * moved the conda install anywidget command * added channel for anywidget install * removed adding anywidget to base * testing which line is failing when removing jupyterlab-plotly install * added lonboard to all images * removed installing pip and upgraded version plotly and commented xmltodict install * added back in pip to jupyterlab/environment.yml * removed comments * removed jupyter-packaging and comments * Add devfile docker path env vars (#117) Co-authored-by: bsatoriu <[email protected]> * Feature/rename workspaces (#116) * Updated all references of "vanilla" to be "python". * Replaced jupyter logo with python logo for python workspace. * Updated dev file references that are using the term "vanilla" to instead use "python". * Updated workspace devfiles. * Removed extra space in python devfile image reference. --------- Co-authored-by: grallewellyn <[email protected]> * Feature/add unzip (#118) * added unzip package to all base images * added version for unzip * Create devfile_compile.py (#121) * Create devfile_compile.py * use abs directory for compile file * Add missing links property to all devfiles --------- Co-authored-by: bsatoriu <[email protected]> * Feature/no maap cfg (#120) * changed to community-909 branch of maap-py and removed MAAP_CONF and added exporting MAAP_API_HOST * Install maap-py from git using pip * Re-add SHELL command * added MAAP_API_HOST env variable to the devfiles * added sujens suggested changes to MAAP_API_HOST * merged in develop and added v4.0.0 tag to maap-py install --------- Co-authored-by: Sujen Shah <[email protected]> Co-authored-by: Sujen Shah <[email protected]> --------- Co-authored-by: Grace Llewellyn <[email protected]> Co-authored-by: Anil Natha <[email protected]> Co-authored-by: Alex <[email protected]> Co-authored-by: Marjorie Lucas <[email protected]> Co-authored-by: Sujen Shah <[email protected]> Co-authored-by: Anil Natha <[email protected]> Co-authored-by: bsatoriu <[email protected]> Co-authored-by: grallewellyn <[email protected]> Co-authored-by: Sujen Shah <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Needed to remove MAAP_CONF to get maap-py to successfully work in the ADE
Fixes: MAAP-Project/Community#909