[Platform Initiative] Implement per-user home directory quotas

[yuvipanda]

This issue tracks rolling out 'per user home directory quotas' as an initiative within the product & services team.

Team

• Person responsible for this currently - @sgibson91 (TBD, based on a clearer definition of this role)
• Tech Lead - @yuvipanda
• Infrastructure Engineer - @sgibson91 and @sunu (from Development Seed!)
• Application Engineer - @sunu

External Links

• Objective JH-1: Enforce home directory quotas per-user NASA-IMPACT/veda-jupyterhub#41
• Objective JH-1: Automated backups and alerting for user limited home directories NASA-IMPACT/veda-jupyterhub#56

Definition of Done

• We have migrated at least two production hubs to using the new system
• "Home directory quotas enforced?" becomes an option when setting up a new hub. This should default to yes
• It’s been tested with one or more users (if applicable)
• It’s been deployed to a production cluster
• It has been shown to be replicably deployable
• It is well documented
• It has been added to our product menu: https://docs.google.com/forms/d/1Gx3v-KaruL6K9VanuwatKEr3qOLJi8ezrcsvGDtZDEQ/edit
• Lessons learned have been documented

[sgibson91]

We have migrated at least two production hubs to using the new system

@Gman0909 @yuvipanda Do you have suggestions/preferences for which community to be the other one migrated with VEDA? And does a conversation with them need to happen?

[yuvipanda]

I'm actually thinking it could be AWI-CIROH, because I saw a message from them about wanting quotas. Let's confirm with @jmunroe.

[yuvipanda]

Since our initial rollout was in AWS, I spoke instead to NMFS-Openscapes and Cryocloud. They're both very happy to be the next. Since NMFS-Openscapes is much newer, let's start there.

[sgibson91]

@sunu is currently working on the following two epics:

• [EPIC] [devseed] Set up automated backups using EBS snapshots for our home directory Volumes in AWS #5002
• [EPIC] [devseed] Set up alerting for an EBS volume getting close to full on AWS, and a runbook on how to resize it when needed #5004

The next tasks he'll be tackling are:

• Enable automated backups of EBS volumes on VEDA staging hub #5061
• Enable disk size alerting for AWS EBS volumes on VEDA staging hub #5062

I am tackling the migration effort, following Yuvi's suggestion above, starting with #5066, then #5067, before returning to #4835

[sgibson91]

@sunu has made great progress on #5061 and #5062 and I feel good that we'll close out #5002 and #5004 before the end of the year.

@yuvipanda and I agreed that only ONE of either #4835 or #5067 is required to close this epic, and CryoCloud will be ready to go during the next sprint.

That just leaves #5187 which Yuvi and I agreed we'll tackle in the new year

[sgibson91]

I extracted the blog post (#4869) out of #4835 and added it as a subtask here so that we finish this initiative with publication of the feature.

[Gman0909]

Love that, thanks @sgibson91