From dd068e5cbded66789902b18ccbdddce80e8d7ff6 Mon Sep 17 00:00:00 2001 From: Tarashish Mishra Date: Fri, 20 Sep 2024 15:16:14 +0530 Subject: [PATCH 01/10] Document how to configure per-user storage quota --- docs/howto/features/storage-quota.md | 130 +++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 docs/howto/features/storage-quota.md diff --git a/docs/howto/features/storage-quota.md b/docs/howto/features/storage-quota.md new file mode 100644 index 0000000000..97196cc455 --- /dev/null +++ b/docs/howto/features/storage-quota.md @@ -0,0 +1,130 @@ +# Configure per-user storage quotas + +This guide explains how to enable and configure per-user storage quotas. + +## Enabling jupyter-home-nfs + +To be able to configure per-user storage quotas, we need to run an in-cluster NFS server using [`jupyter-home-nfs`](https://github.com/sunu/jupyter-home-nfs). This can be enabled by setting `jupyter-home-nfs.enabled` to `true` in the hub's values file. + +jupyter-home-nfs expects a reference to an pre-provisioned disk. Here's an example of how to configure that on AWS and GCP. + +### AWS + +```yaml +basehub: + jupyter-home-nfs: + enabled: true + eks: + enabled: true + volumeId: vol-0a1246ee2e07372d0 +``` + + +### GCP + +```yaml +basehub: + jupyter-home-nfs: + enabled: true + gke: + enabled: true + volumeId: projects/jupyter-nfs/zones/us-central1-f/disks/jupyter-nfs-home-directories +``` + +Once these changes are deployed, we should have a new NFS server running in our cluster through the `jupyter-home-nfs` Helm chart. + +## Migrating existing home directories + +If there are existing home directories, we need to migrate them to the new NFS server. For this, we will create a throwaway pod with both the existing home directories and the new NFS server mounted, and we will copy the contents from the existing home directories to the new NFS server. + +Here's an example of how to do this: + +```bash +# Create a throwaway pod with both the existing home directories and the new NFS server mounted +deployer exec root-homes --extra_nfs_server= --extra_nfs_base_path=/ --extra_nfs_mount_path=/new-nfs-volume + +# Copy the existing home directories to the new NFS server while keeping the original permissions +rsync -av --info=progress2 /root-homes/ /new-nfs-volume/ +``` + +Make sure the path structure of the existing home directories matches the path structure of the home directories in the new NFS storage volume. For example, if an existing home directory is located at `/root-homes/staging/user`, we should expect to find it at `/new-nfs-volume/staging/user`. + +## Switching to the new NFS server + +Now that we have a new NFS server running in our cluster, and we have migrated the existing home directories to the new NFS server, we can switch the hub to use the new NFS server. This can be done by updating the `basehub.nfs.pv.serverIP` field in the hub's values file. + +```yaml +basehub: + nfs: + pv: + serverIP: +``` + +Note that Kubernetes doesn't allow changing an existing PersistentVolume. So we need to delete the existing PersistentVolume first. + +```bash + kubectl delete pv ${HUB_NAME}-home-nfs --wait=false + kubectl --namespace $HUB_NAME delete pvc home-nfs --wait=false + kubectl --namespace $HUB_NAME delete pod -l component=shared-dirsize-metrics + kubectl --namespace $HUB_NAME delete pod -l component=shared-volume-metrics +``` + +After this, we should be able to deploy the hub and have it use the new NFS server: + +```bash +deployer deploy +``` + +## Enforcing storage quotas + +Now we can set quotas for each user and configure the path to monitor for storage quota enforcement. + +This can be done by updating `basehub.jupyter-home-nfs.quotaEnforcer` in the hub's values file. For example, to set a quota of 10GB for the user `staging`, we would add the following to the user's values file: + +```yaml +basehub: + jupyter-home-nfs: + quotaEnforcer: + hardQuota: "10" # in GB + path: "/export/staging" +``` + +The `path` field is the path to the parent directory of the user's home directory in the NFS server. The `hardQuota` field is the maximum allowed size of the user's home directory in GB. + +To deploy the changes, we need to run the following command: + +```bash +deployer deploy +``` + +Once this is deployed, the hub will automatically enforce the storage quota for each user. If a user's home directory exceeds the quota, the user's pod may not be able to start successfully. + +## Troubleshooting + +### Checking the NFS server +To check whether the NFS server is running properly, we can run the following command in the NFS server pod in the nfs-server container: + +```bash +showmount -e 0.0.0.0 +``` + +If the NFS server is running properly, we should see the path to the NFS server's export directory. For example: + +```bash +Export list for 0.0.0.0: +/export * +``` + +If we don't see the path to the export directory, the NFS server is not running properly and we need to check the logs for the NFS server pod. + +### Debugging quota enforcement + +We can check the current usage and quotas for each user by running the following command in the NFS server pod in the `enforce-xfs-quota` container: + +```bash +xfs_quota -x -c 'report -N -p' +``` + +This should show us the list of directories being monitored for quota enforcement, and the current usage and quotas for each of the home directories. + +If a user exceeds their quota, the user's pod will not be able to start successfully. A hub admin will need to free up space in the user's home directory to allow the pod to start. \ No newline at end of file From c2eb7ac791ce4a1726c1c0b553f123b384df7ca9 Mon Sep 17 00:00:00 2001 From: Tarashish Mishra Date: Fri, 20 Sep 2024 15:29:30 +0530 Subject: [PATCH 02/10] update documentation --- docs/howto/features/storage-quota.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/docs/howto/features/storage-quota.md b/docs/howto/features/storage-quota.md index 97196cc455..8b49f1ca14 100644 --- a/docs/howto/features/storage-quota.md +++ b/docs/howto/features/storage-quota.md @@ -31,7 +31,19 @@ basehub: volumeId: projects/jupyter-nfs/zones/us-central1-f/disks/jupyter-nfs-home-directories ``` -Once these changes are deployed, we should have a new NFS server running in our cluster through the `jupyter-home-nfs` Helm chart. +This changes can be deployed by running the following command: + +```bash +deployer deploy +``` + +Once these changes are deployed, we should have a new NFS server running in our cluster through the `jupyter-home-nfs` Helm chart. We can get the IP address of the NFS server by running the following command: + +```bash +kubectl -n get svc -nfs-service +``` + +To check whether the NFS server is running properly, see the [Troubleshooting](#troubleshooting) section. ## Migrating existing home directories @@ -101,7 +113,7 @@ Once this is deployed, the hub will automatically enforce the storage quota for ## Troubleshooting -### Checking the NFS server +### Checking the NFS server is running properly To check whether the NFS server is running properly, we can run the following command in the NFS server pod in the nfs-server container: ```bash From 882d752abb18c32079b5c072a507396ebab8dbca Mon Sep 17 00:00:00 2001 From: Tarashish Mishra Date: Fri, 20 Sep 2024 15:31:37 +0530 Subject: [PATCH 03/10] add storage-quota to table of contents --- docs/howto/features/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/howto/features/index.md b/docs/howto/features/index.md index 1a2371eb94..99fc731918 100644 --- a/docs/howto/features/index.md +++ b/docs/howto/features/index.md @@ -28,4 +28,5 @@ private-nbgitpuller.md rocker.md shared-db.md static-sites.md +storage-quota.md ``` \ No newline at end of file From e0e07da7f8ba4ca2d47ad15e952c5da78b3dd8cb Mon Sep 17 00:00:00 2001 From: Sarah Gibson Date: Fri, 20 Sep 2024 13:07:52 +0100 Subject: [PATCH 04/10] Use a tab-set for vendor-specific examples --- docs/howto/features/storage-quota.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/howto/features/storage-quota.md b/docs/howto/features/storage-quota.md index 8b49f1ca14..1cef06d002 100644 --- a/docs/howto/features/storage-quota.md +++ b/docs/howto/features/storage-quota.md @@ -8,8 +8,9 @@ To be able to configure per-user storage quotas, we need to run an in-cluster NF jupyter-home-nfs expects a reference to an pre-provisioned disk. Here's an example of how to configure that on AWS and GCP. -### AWS - +`````{tab-set} +````{tab-item} AWS +:sync: aws-key ```yaml basehub: jupyter-home-nfs: @@ -18,10 +19,10 @@ basehub: enabled: true volumeId: vol-0a1246ee2e07372d0 ``` +```` - -### GCP - +````{tab-item} GCP +:sync: gcp-key ```yaml basehub: jupyter-home-nfs: @@ -30,6 +31,8 @@ basehub: enabled: true volumeId: projects/jupyter-nfs/zones/us-central1-f/disks/jupyter-nfs-home-directories ``` +```` +````` This changes can be deployed by running the following command: From 5a8ea2add1c3373d3193d2db7242b2b98098da97 Mon Sep 17 00:00:00 2001 From: Sarah Gibson Date: Fri, 20 Sep 2024 13:10:48 +0100 Subject: [PATCH 05/10] Remove basehub key and add note to include it if deploying a daskhub --- docs/howto/features/storage-quota.md | 38 ++++++++++++++-------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/howto/features/storage-quota.md b/docs/howto/features/storage-quota.md index 1cef06d002..5a4c07c662 100644 --- a/docs/howto/features/storage-quota.md +++ b/docs/howto/features/storage-quota.md @@ -2,6 +2,10 @@ This guide explains how to enable and configure per-user storage quotas. +```{note} +Nest all config examples under a `basehub` key if deployingb this for a daskhub. +``` + ## Enabling jupyter-home-nfs To be able to configure per-user storage quotas, we need to run an in-cluster NFS server using [`jupyter-home-nfs`](https://github.com/sunu/jupyter-home-nfs). This can be enabled by setting `jupyter-home-nfs.enabled` to `true` in the hub's values file. @@ -12,24 +16,22 @@ jupyter-home-nfs expects a reference to an pre-provisioned disk. Here's an examp ````{tab-item} AWS :sync: aws-key ```yaml -basehub: - jupyter-home-nfs: +jupyter-home-nfs: + enabled: true + eks: enabled: true - eks: - enabled: true - volumeId: vol-0a1246ee2e07372d0 + volumeId: vol-0a1246ee2e07372d0 ``` ```` ````{tab-item} GCP :sync: gcp-key ```yaml -basehub: - jupyter-home-nfs: +jupyter-home-nfs: + enabled: true + gke: enabled: true - gke: - enabled: true - volumeId: projects/jupyter-nfs/zones/us-central1-f/disks/jupyter-nfs-home-directories + volumeId: projects/jupyter-nfs/zones/us-central1-f/disks/jupyter-nfs-home-directories ``` ```` ````` @@ -69,10 +71,9 @@ Make sure the path structure of the existing home directories matches the path s Now that we have a new NFS server running in our cluster, and we have migrated the existing home directories to the new NFS server, we can switch the hub to use the new NFS server. This can be done by updating the `basehub.nfs.pv.serverIP` field in the hub's values file. ```yaml -basehub: - nfs: - pv: - serverIP: +nfs: + pv: + serverIP: ``` Note that Kubernetes doesn't allow changing an existing PersistentVolume. So we need to delete the existing PersistentVolume first. @@ -97,11 +98,10 @@ Now we can set quotas for each user and configure the path to monitor for storag This can be done by updating `basehub.jupyter-home-nfs.quotaEnforcer` in the hub's values file. For example, to set a quota of 10GB for the user `staging`, we would add the following to the user's values file: ```yaml -basehub: - jupyter-home-nfs: - quotaEnforcer: - hardQuota: "10" # in GB - path: "/export/staging" +jupyter-home-nfs: + quotaEnforcer: + hardQuota: "10" # in GB + path: "/export/staging" ``` The `path` field is the path to the parent directory of the user's home directory in the NFS server. The `hardQuota` field is the maximum allowed size of the user's home directory in GB. From a5fc93be8382988743b6e68c86aeaa0c141897de Mon Sep 17 00:00:00 2001 From: Sarah Gibson Date: Fri, 20 Sep 2024 13:18:29 +0100 Subject: [PATCH 06/10] Improve some formatting --- docs/howto/features/storage-quota.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/docs/howto/features/storage-quota.md b/docs/howto/features/storage-quota.md index 5a4c07c662..c63ea1c278 100644 --- a/docs/howto/features/storage-quota.md +++ b/docs/howto/features/storage-quota.md @@ -36,15 +36,19 @@ jupyter-home-nfs: ```` ````` -This changes can be deployed by running the following command: +These changes can be deployed by running the following command: ```bash deployer deploy ``` -Once these changes are deployed, we should have a new NFS server running in our cluster through the `jupyter-home-nfs` Helm chart. We can get the IP address of the NFS server by running the following command: +Once these changes are deployed, we should have a new NFS server running in our cluster through the `jupyter-home-nfs` Helm chart. We can get the IP address of the NFS server by running the following commands: ```bash +# Authenticate with the cluster +deployer use-cluster-credentials + +# Retrieve the service IP kubectl -n get svc -nfs-service ``` @@ -79,10 +83,10 @@ nfs: Note that Kubernetes doesn't allow changing an existing PersistentVolume. So we need to delete the existing PersistentVolume first. ```bash - kubectl delete pv ${HUB_NAME}-home-nfs --wait=false - kubectl --namespace $HUB_NAME delete pvc home-nfs --wait=false - kubectl --namespace $HUB_NAME delete pod -l component=shared-dirsize-metrics - kubectl --namespace $HUB_NAME delete pod -l component=shared-volume-metrics +kubectl delete pv ${HUB_NAME}-home-nfs --wait=false +kubectl --namespace $HUB_NAME delete pvc home-nfs --wait=false +kubectl --namespace $HUB_NAME delete pod -l component=shared-dirsize-metrics +kubectl --namespace $HUB_NAME delete pod -l component=shared-volume-metrics ``` After this, we should be able to deploy the hub and have it use the new NFS server: @@ -117,6 +121,7 @@ Once this is deployed, the hub will automatically enforce the storage quota for ## Troubleshooting ### Checking the NFS server is running properly + To check whether the NFS server is running properly, we can run the following command in the NFS server pod in the nfs-server container: ```bash From 9e020cb67788371e17bf907ebf20f6979989c4b6 Mon Sep 17 00:00:00 2001 From: Sarah Gibson Date: Fri, 20 Sep 2024 13:20:01 +0100 Subject: [PATCH 07/10] Add some x-refs --- docs/howto/features/storage-quota.md | 3 ++- docs/topic/infrastructure/storage-layer.md | 2 ++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/howto/features/storage-quota.md b/docs/howto/features/storage-quota.md index c63ea1c278..fd3f383eb4 100644 --- a/docs/howto/features/storage-quota.md +++ b/docs/howto/features/storage-quota.md @@ -1,3 +1,4 @@ +(howto:configure-storage-quota)= # Configure per-user storage quotas This guide explains how to enable and configure per-user storage quotas. @@ -147,4 +148,4 @@ xfs_quota -x -c 'report -N -p' This should show us the list of directories being monitored for quota enforcement, and the current usage and quotas for each of the home directories. -If a user exceeds their quota, the user's pod will not be able to start successfully. A hub admin will need to free up space in the user's home directory to allow the pod to start. \ No newline at end of file +If a user exceeds their quota, the user's pod will not be able to start successfully. A hub admin will need to free up space in the user's home directory to allow the pod to start, using [the `allusers` feature](topic:storage:allusers). diff --git a/docs/topic/infrastructure/storage-layer.md b/docs/topic/infrastructure/storage-layer.md index f976bd202a..55a74490fb 100644 --- a/docs/topic/infrastructure/storage-layer.md +++ b/docs/topic/infrastructure/storage-layer.md @@ -1,3 +1,4 @@ +(topic:storage)= # User home directory storage All users on all the hubs get a home directory with persistent storage. @@ -71,6 +72,7 @@ This volumeMount is **NOT `readOnly`**, so admins can write to it. This feature comes from the [custom KubeSpawner](https://github.com/2i2c-org/infrastructure/tree/HEAD/helm-charts/basehub/values.yaml#L182) that the our community hubs use, that allows providing extra configuration for admin users only. ``` +(topic:storage:allusers)= #### (Optional) The `allusers` directory Can be mounted **just for admins**, showing the contents of `//`. From a9baa095d50e59aa65ec4429a8d848efa1aa5565 Mon Sep 17 00:00:00 2001 From: Tarashish Mishra Date: Sat, 21 Sep 2024 10:47:56 +0530 Subject: [PATCH 08/10] Document how to create a disk for the nfs server --- docs/howto/features/storage-quota.md | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/docs/howto/features/storage-quota.md b/docs/howto/features/storage-quota.md index fd3f383eb4..a20c3e0be0 100644 --- a/docs/howto/features/storage-quota.md +++ b/docs/howto/features/storage-quota.md @@ -7,6 +7,26 @@ This guide explains how to enable and configure per-user storage quotas. Nest all config examples under a `basehub` key if deployingb this for a daskhub. ``` +## Creating a pre-provisioned disk + +The in-cluster NFS server uses a pre-provisioned disk to store the users' home directories. We don't use a dynamically provisioned volume because we want to be able to reuse the same disk even when the Kubernetes cluster is deleted and recreated. So the first step is to create a disk that will be used to store the users' home directories. + +For hubs running on AWS, we can create a disk through Terraform by adding a block like the following to the [tfvars file of the hub](https://github.com/2i2c-org/infrastructure/tree/main/terraform/aws/projects): + +```hcl +ebs_volumes = { + "staging" = { + size = 100 + type = "gp3" + name_suffix = "staging" + tags = {} + } +} +``` + +This will create a disk with a size of 100GB for the `staging` hub that we can reference when configuring the NFS server. + + ## Enabling jupyter-home-nfs To be able to configure per-user storage quotas, we need to run an in-cluster NFS server using [`jupyter-home-nfs`](https://github.com/sunu/jupyter-home-nfs). This can be enabled by setting `jupyter-home-nfs.enabled` to `true` in the hub's values file. @@ -100,7 +120,7 @@ deployer deploy Now we can set quotas for each user and configure the path to monitor for storage quota enforcement. -This can be done by updating `basehub.jupyter-home-nfs.quotaEnforcer` in the hub's values file. For example, to set a quota of 10GB for the user `staging`, we would add the following to the user's values file: +This can be done by updating `basehub.jupyter-home-nfs.quotaEnforcer` in the hub's values file. For example, to set a quota of 10GB for all users on the `staging` hub, we would add the following to the hub's values file: ```yaml jupyter-home-nfs: From 245efa67b98473095a8331f8d6d692ae4d64722f Mon Sep 17 00:00:00 2001 From: Sarah Gibson <44771837+sgibson91@users.noreply.github.com> Date: Mon, 23 Sep 2024 09:22:28 +0100 Subject: [PATCH 09/10] fix typo --- docs/howto/features/storage-quota.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto/features/storage-quota.md b/docs/howto/features/storage-quota.md index a20c3e0be0..4778186355 100644 --- a/docs/howto/features/storage-quota.md +++ b/docs/howto/features/storage-quota.md @@ -4,7 +4,7 @@ This guide explains how to enable and configure per-user storage quotas. ```{note} -Nest all config examples under a `basehub` key if deployingb this for a daskhub. +Nest all config examples under a `basehub` key if deploying this for a daskhub. ``` ## Creating a pre-provisioned disk From db8c5060fff7ca360e9f066acea6160174ff8be1 Mon Sep 17 00:00:00 2001 From: Sarah Gibson <44771837+sgibson91@users.noreply.github.com> Date: Mon, 23 Sep 2024 09:30:46 +0100 Subject: [PATCH 10/10] Update docs/howto/features/storage-quota.md --- docs/howto/features/storage-quota.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto/features/storage-quota.md b/docs/howto/features/storage-quota.md index 4778186355..0d4e3726b6 100644 --- a/docs/howto/features/storage-quota.md +++ b/docs/howto/features/storage-quota.md @@ -11,7 +11,7 @@ Nest all config examples under a `basehub` key if deploying this for a daskhub. The in-cluster NFS server uses a pre-provisioned disk to store the users' home directories. We don't use a dynamically provisioned volume because we want to be able to reuse the same disk even when the Kubernetes cluster is deleted and recreated. So the first step is to create a disk that will be used to store the users' home directories. -For hubs running on AWS, we can create a disk through Terraform by adding a block like the following to the [tfvars file of the hub](https://github.com/2i2c-org/infrastructure/tree/main/terraform/aws/projects): +For infrastructure running on AWS, we can create a disk through Terraform by adding a block like the following to the [tfvars file of the hub](https://github.com/2i2c-org/infrastructure/tree/main/terraform/aws/projects): ```hcl ebs_volumes = {