Limit how much each user can consume on /tmp

[yuvipanda]

Context

This came up while I was documenting how /tmp works in 2i2c-org/docs#218. Since it's possible for a single user to use up all of /tmp, this can cause issues based on which two users are on a node.

Approach 1: ephemeral storage limit

Kubernetes allows us to limit the amount of 'ephemeral storage' a user pod can use - https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#setting-requests-and-limits-for-local-ephemeral-storage. This counts not just towards /tmp but also things like temporary conda / pip package installs (since these are also in the container).

The following diff sets this up in our infrastructure, setting a guarantee of 256MB of space and a limit of 5GB of space:

diff --git a/helm-charts/basehub/values.yaml b/helm-charts/basehub/values.yaml
index 200c1676..5dc210b3 100644
--- a/helm-charts/basehub/values.yaml
+++ b/helm-charts/basehub/values.yaml
@@ -601,6 +601,10 @@ jupyterhub:
         # Increase timeout for Jupyter server to become 'ready', until
         # https://github.com/2i2c-org/infrastructure/issues/2047 is fixed
         http_timeout: 120
+        extra_resource_guarantees:
+          ephemeral-storage: 256Mi
+        extra_resource_limits:
+          ephemeral-storage: 5Gi
       Authenticator:
         # Don't allow test username to login into the hub
         # The test service will still be able to create this hub username

However, this will just kill the user's pod if they go over the limit:

5m17s       Normal    Killing               pod/jupyter-yuvipanda                           Stopping container notebook
5m17s       Warning   Evicted               pod/jupyter-yuvipanda                           Pod ephemeral local storage usage exceeds the total limit of containers 5Gi.

This isn't the most graceful - ideally we'd instead just error out in user code when /tmp is full. This is a poor user experience, and we should not do this.

Approach 2: Setup an emptyDir under /tmp for each user

Kubernetes emptyDir also allows us to set a size limit, and I believe this will be enforced a bit better. We can mount an emptyDir volume under /tmp for all users, and set a limit there.

I haven't tested this approach out.

[yuvipanda]

I don't think this is a high priority item until a community actually runs into this problem and asks for a solution