From 4495d7d06506ff66815b3788352156f431845881 Mon Sep 17 00:00:00 2001 From: YuviPanda Date: Mon, 25 Mar 2024 18:42:19 -0700 Subject: [PATCH 1/3] Document how to test if buckets are fully set up --- docs/howto/features/buckets.md | 69 ++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/docs/howto/features/buckets.md b/docs/howto/features/buckets.md index c7150fb67d..8d782bd56e 100644 --- a/docs/howto/features/buckets.md +++ b/docs/howto/features/buckets.md @@ -83,6 +83,75 @@ on why users want this! 4. Get this change deployed, and users should now be able to use the buckets! Currently running users might have to restart their pods for the change to take effect. +### Testing access to buckets + +Once bucket access has been set up, we should test to make sure users can write +to and read from it. + +#### AWS + +1. Login to the hub, and open a Terminal in JupyterLab + +2. Check if the AWS CLI is installed by running the `aws` command - many base images + already include this package. If not, you can do a local installation with: + + ```bash + curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" + unzip awscliv2.zip + export PATH=$(pwd)aws/dist/:$PATH + ``` + + ```{note} + This could have been as simple as a `pip install`, but [AWS does not support it](https://github.com/aws/aws-cli/issues/4947) + ``` + +3. Create a temporary file, which we will then copy over to our scratch bucket. + + ```bash + echo 'hi' > temp-test-file + ``` + +4. Copy the file over to S3, under `$SCRATCH_BUCKET` or `$PERSISTENT_BUCKET` (based on + which one we are going to be testing). + + ```bash + aws s3 cp temp-test-file $SCRATCH_BUCKET/temp-test-file + ``` + + This should succeed with a message like `upload: ./temp-test-file to s3://openscapeshub-scratch/yuvipanda/temp-test-file` + +5. Let's list our bucket to make sure the file is there. + + ```bash + $ aws s3 ls $SCRATCH_BUCKET/ + 2024-03-26 01:38:53 3 temp-test-file + ``` + + ```{note} + The trailing `/` is important. + ``` + + ```{note} + If testing `$PERSISTENT_BUCKET`, use that environment variable instead + ``` + +6. Copy the file back from s3, to make sure we can read. + + ```bash + $ aws s3 cp $SCRATCH_BUCKET/temp-test-file back-here + download: s3://openscapeshub-scratch/yuvipanda/temp-test-file to ./back-here + $ cat back-here + hi + ``` + + We have verified this all works! + +7. Clean up our files so we don't cost the community money in the long run. + + ```bash + aws s3 rm $SCRATCH_BUCKET/temp-test-file + rm temp-test-file back-here + ``` ## Allowing public, readonly to buckets from outside the JupyterHub From a36100909945ad9c8112adbf5a33ecb78cb86d6a Mon Sep 17 00:00:00 2001 From: YuviPanda Date: Tue, 26 Mar 2024 14:28:35 -0700 Subject: [PATCH 2/3] Check for the bucket environment variables too --- docs/howto/features/buckets.md | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/docs/howto/features/buckets.md b/docs/howto/features/buckets.md index 8d782bd56e..bf872bf994 100644 --- a/docs/howto/features/buckets.md +++ b/docs/howto/features/buckets.md @@ -92,7 +92,22 @@ to and read from it. 1. Login to the hub, and open a Terminal in JupyterLab -2. Check if the AWS CLI is installed by running the `aws` command - many base images +2. Look for the envirionment variables we just set (`SCRATCH_BUCKET` and/or `PERSISTENT_BUCKET`), make + sure they are showing up correctly: + + ```bash + env | grep _BUCKET + ``` + + They should end with the name of your JupyterHub user. For example, here is the output + on the openscapes hub, when my JupyterHub username is `yuvipanda`: + + ``` + PERSISTENT_BUCKET=s3://openscapeshub-persistent/yuvipanda + SCRATCH_BUCKET=s3://openscapeshub-scratch/yuvipanda + ``` + +3. Check if the AWS CLI is installed by running the `aws` command - many base images already include this package. If not, you can do a local installation with: ```bash @@ -105,13 +120,13 @@ to and read from it. This could have been as simple as a `pip install`, but [AWS does not support it](https://github.com/aws/aws-cli/issues/4947) ``` -3. Create a temporary file, which we will then copy over to our scratch bucket. +4. Create a temporary file, which we will then copy over to our scratch bucket. ```bash echo 'hi' > temp-test-file ``` -4. Copy the file over to S3, under `$SCRATCH_BUCKET` or `$PERSISTENT_BUCKET` (based on +5. Copy the file over to S3, under `$SCRATCH_BUCKET` or `$PERSISTENT_BUCKET` (based on which one we are going to be testing). ```bash @@ -120,7 +135,7 @@ to and read from it. This should succeed with a message like `upload: ./temp-test-file to s3://openscapeshub-scratch/yuvipanda/temp-test-file` -5. Let's list our bucket to make sure the file is there. +6. Let's list our bucket to make sure the file is there. ```bash $ aws s3 ls $SCRATCH_BUCKET/ @@ -135,7 +150,7 @@ to and read from it. If testing `$PERSISTENT_BUCKET`, use that environment variable instead ``` -6. Copy the file back from s3, to make sure we can read. +7. Copy the file back from s3, to make sure we can read. ```bash $ aws s3 cp $SCRATCH_BUCKET/temp-test-file back-here @@ -146,7 +161,7 @@ to and read from it. We have verified this all works! -7. Clean up our files so we don't cost the community money in the long run. +8. Clean up our files so we don't cost the community money in the long run. ```bash aws s3 rm $SCRATCH_BUCKET/temp-test-file From 6d67e460903aed5e5fcc12369e1a4b935f616412 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Tue, 26 Mar 2024 21:29:41 +0000 Subject: [PATCH 3/3] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- docs/howto/features/buckets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto/features/buckets.md b/docs/howto/features/buckets.md index bf872bf994..24e7e5afa4 100644 --- a/docs/howto/features/buckets.md +++ b/docs/howto/features/buckets.md @@ -92,7 +92,7 @@ to and read from it. 1. Login to the hub, and open a Terminal in JupyterLab -2. Look for the envirionment variables we just set (`SCRATCH_BUCKET` and/or `PERSISTENT_BUCKET`), make +2. Look for the environment variables we just set (`SCRATCH_BUCKET` and/or `PERSISTENT_BUCKET`), make sure they are showing up correctly: ```bash