GH action to generate report #199

asmacdo · 2024-09-25T20:07:00Z

Fixes #177

Step 1: Create Skeleton

I've verified that when a user-node is available (created by running a tiny jupyterhub), the job pod schedules on that node. I then shut down my jupyterhub and all user-nodes scaled down. I reran this job, and Karpenter successfully scaled up a new spot node, the pod was scheduled on it, ran successfully, was deleted, and the node cleaned up. Step 1 complete!

Step 2 Generate Report

Connect Pod to EFS
List users
du each user
du shared
collate data into report
Double Check that nodes come up and down successfully
Run job several times in 1 day, check next day for EFS usage spike (IIUC we should be fine because EFS is Bursting mode)

Step 3 Push Report

Create private GitHub repository to store reports
Configure bot permission to push to repo
push report to repo on complete

Questions to answer:

If a SPOT node is preempted, can we redeploy again later?

kabilar · 2024-09-26T14:53:56Z

.github/workflows/report.yaml

+  pull_request:
+    branches:
+      - main


Should we also run this on a weekly basis?

It's on PR push just so I can test easily, but yes, 1/week sounds good to me. @kabilar Do you have a preference for what day/time?

Great, thank you. How about Mondays at 6am EST? We can then review the report on Monday mornings.

kabilar · 2024-11-07T00:05:29Z

Hi @asmacdo, just checking in to see how this report generation is going? Thanks.

asmacdo · 2024-11-11T18:51:24Z

Ran into some problems

The du script ran for about 50 minutes and then the pod disappeared without logs.

Worse it kicked my jupyterhub pod as well as another user.

[I 2024-11-11 17:57:57.999 JupyterHub log:192] 200 GET /hub/error/503?url=%2Fuser%2Fasmacdo%2Fterminals%2Fwebsocket%2F1 (@100.64.247.104) 7.52ms
[W 2024-11-11 17:57:59.266 JupyterHub base:1254] User asmacdo server stopped, with exit code: 1
[I 2024-11-11 17:57:59.266 JupyterHub proxy:357] Removing user asmacdo from proxy (/user/asmacdo/)

I think this means we need to take a different approach. By setting resource limits, we should have isolated our job from the other pods, but since I have no other logs about what happened here I think we need to take a more conservative approach that is completely isolated from user pods.

I did it this way because I thought it would be simpler, but if theres any chance that we affect a running user pod, we would be better off directly deploying a separate EC2 instance and bind the EFS directly, avoiding Kubernetes altogether.

kabilar · 2024-11-14T20:58:32Z

we would be better off directly deploying a separate EC2 instance and bind the EFS directly, avoiding Kubernetes altogether.

Thanks @asmacdo. That makes sense.

yarikoptic · 2024-12-12T17:55:37Z

.github/scripts/cleanup-ec2.sh

@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+
+set -e


Suggested change

set -e

set -eu

yarikoptic · 2024-12-12T17:55:53Z

.github/scripts/launch-ec2.sh

@@ -0,0 +1,133 @@
+#!/usr/bin/env bash
+
+set -e


Suggested change

set -e

set -eu

yarikoptic · 2024-12-12T18:06:52Z

.github/scripts/launch-ec2.sh

+LOCAL_SCRIPTS_DIR=".github/scripts"
+REMOTE_SCRIPTS_DIR="/home/ec2-user/scripts"
+MOUNT_POINT="/mnt/efs"
+ENV_FILE=".ec2-session.env"


I dislike the fact that it would be just dumping into some hidden file in my current directory.
Could we make it dumped into some tmpdir and may be establish there env variable with its path so cleanup script could take it from that environment

Suggested change

ENV_FILE=".ec2-session.env"

ENV_FILE="/run/user/$(id -u)/ec2-session.env"

but then we might want to add logic to react if file already exist -- since that would mean likely that cleanup did not remove it and instance might still be running etc.

yarikoptic · 2024-12-12T18:07:32Z

.github/scripts/cleanup-ec2.sh

+set -e
+
+# Load environment variables from the file if they are not already set
+ENV_FILE=".ec2-session.env"


Suggested change

ENV_FILE=".ec2-session.env"

ENV_FILE="/run/user/$(id -u)/ec2-session.env"

yarikoptic

left comments on the report and walking.

The plan:

see if more recent AMI with newer than 3.7 python available (e.g. 3.10 to be future proof somewhat)
run the create-file-index.py across all the users
- could be done in parallel and under duct following http://www.oneukrainian.com/tmp/run_manifests.sh example
scp those reports somewhere
while running develop produce-report.py further:
- change logic so we compute sizes for all directories
- go through the list of folders and match to the desired to be monitored/informed about https://github.com/dandi/dandi-hub/pull/188/files#diff-b7bf13dc481c474b9080a061c3154f0e645619658cb9bd469383938de07a7439R16
  - ideally compose a dict/json record with those, do not render immediately. e.g.

{
   "total":
    {
      "file_size": 123123123123, 
      "file_count":123
    },
   "caches": {
        "__pycache__": {  # count directories which last path component is `__pycache__`
             "file_size": 12123123,
             "file_count": 12,
             "directories": [
                  "a/e/__pycache__"
                  "a/b/d/e/__pycache__"
                  "a/e/__pycache__/c/__pycache__"  # should not happen in the wild
             ],
        "pip_cache": {
               "file_size": 13,
               "file_count": 1,
               "directories": [
                    "blah/.cache/pip
               ]
   }

.github/scripts/create-file-index.py

yarikoptic · 2024-12-12T18:24:37Z

.github/scripts/create-file-index.py

+import gzip
+from datetime import datetime
+
+def list_files_with_metadata(directory, output_file):


write a simple test, could even probably be in this file, where you populate directory with nested folders and symlinks and you know the ground truth to aim for and compare against.

yarikoptic · 2024-12-12T18:28:59Z

.github/scripts/create-file-index.py

+
+    files_metadata = []
+
+    for root, dirs, files in os.walk(directory):


FTR. walk seems to be already doing desired (right) thing and does not follow symlinked folders. We get

In [14]: list(os.walk('/tmp/1234')) Out[14]: [('/tmp/1234', ['infinitum', 'subdir', 'linkgood'], []), ('/tmp/1234/subdir', ['subdir2'], ['file']), ('/tmp/1234/subdir/subdir2', [], ['file2'])]

for

❯ tree /tmp/1234 /tmp/1234 ├── infinitum -> /tmp/1234 ├── linkgood -> subdir └── subdir ├── file └── subdir2 └── file2

note: we do not monitor empty folders below

.github/scripts/produce-report.py

asmacdo · 2024-12-12T23:01:26Z

see if more recent AMI with newer than 3.7 python available (e.g. 3.10 to be future proof somewhat)

Amazon Linux 2023 comes with 3.9, which is sufficient. I briefly tried ubuntu 22.04, (with 3.12) but that would add complexity, requiring amazon-efs-utils to be installed from source.

asmacdo · 2024-12-16T18:55:06Z

Update:

The automation is incomplete
- the launch script sets up an ec2 instance with the scripts
- scripts must be executed by hand
- produced data must be retrieved by hand
Running in parallel (8 simultaneous) the file indexing takes about 16 minutes
The file indexes are 499MB

Heres an example of the json output produced by the report generator:

{
  "total_size": 238294213,
  "file_count": 3639,
  "caches": {
    "pycache": {
      "total_size": 22782820,
      "file_count": 1431,
      "directories": [
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/extern/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/jaraco/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/jaraco/text/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/importlib_resources/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/packaging/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/more_itertools/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_distutils/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_distutils/command/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/extern/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/config/_validate_pyproject/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/config/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/jaraco/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/jaraco/text/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/importlib_resources/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/importlib_metadata/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/tomli/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/packaging/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/more_itertools/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/command/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/_distutils_hack/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/index/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/operations/install/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/resolution/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/webencodings/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pkg_resources/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/certifi/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/tenacity/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/distro/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/tomli/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/util/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/contrib/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/contrib/_securetransport/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/packages/backports/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/packages/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pygments/styles/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pygments/filters/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pygments/lexers/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/extern/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/jaraco/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/jaraco/text/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/importlib_resources/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/packaging/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/more_itertools/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_distutils/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_distutils/command/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/extern/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/config/_validate_pyproject/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/config/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/jaraco/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/jaraco/text/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/importlib_resources/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/importlib_metadata/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/tomli/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/packaging/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/more_itertools/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/command/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/_distutils_hack/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/index/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/req/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/operations/install/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/resolution/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/models/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/network/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/webencodings/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pkg_resources/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/certifi/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/tenacity/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/distro/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/tomli/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/util/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/contrib/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/contrib/_securetransport/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/packages/backports/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/packages/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pygments/styles/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pygments/filters/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pygments/lexers/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/con_duct/__pycache__",
        "asmacdo/dandi-notebooks/000971/lernerlab/seiler_2024/__pycache__"
      ]
    },
    "user_cache": {
      "total_size": 10852503,
      "file_count": 81,
      "directories": [
        "asmacdo/.cache"
      ]
    },
    "yarn_cache": {
      "total_size": 0,
      "file_count": 0,
      "directories": []
    },
    "pip_cache": {
      "total_size": 10852503,
      "file_count": 80,
      "directories": [
        "asmacdo/.cache/pip"
      ]
    },
    "nwb_cache": {
      "total_size": 0,
      "file_count": 0,
      "directories": []
    }
  }
}

Notably, the total size 238294213 is less than reported by du.
du --apparent: 247232941
du: 252841984

(Also increase timeout)

Python 3.9 is new enough. Ubuntu was not ideal because the EFS mounting require amazonl-efs-utils which is available via package manage for amazon linux, but requires building from source on Ubuntu.

gzip will only be needed if we want to upload to s3

=== Do not change lines below === { "chain": [], "cmd": "black .github/scripts/calculate-directory-stats.py", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [ ".github/scripts/calculate-directory-stats.py" ], "pwd": "." } ^^^ Do not change lines above ^^^

Holding the entire file index in memory would be risky, especially with zarr files

Whether there is an error or not, write all fields for each file. This allows us to read the tsv file and unpack the same number of values for each line.

kabilar reviewed Sep 26, 2024

View reviewed changes

asmacdo force-pushed the cron-data-usage-report branch 3 times, most recently from a4b44f9 to 965a81e Compare November 11, 2024 16:55

asmacdo force-pushed the cron-data-usage-report branch from 81b43e0 to 747f0a4 Compare December 2, 2024 17:47

yarikoptic reviewed Dec 12, 2024

View reviewed changes

.github/scripts/cleanup-ec2.sh

@@ -0,0 +1,63 @@

#!/usr/bin/env bash

set -e

Copy link

Member

yarikoptic Dec 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

set -e

set -eu

yarikoptic reviewed Dec 12, 2024

View reviewed changes

.github/scripts/launch-ec2.sh

@@ -0,0 +1,133 @@

#!/usr/bin/env bash

set -e

Copy link

Member

yarikoptic Dec 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

set -e

set -eu

yarikoptic reviewed Dec 12, 2024

View reviewed changes

yarikoptic requested changes Dec 12, 2024

View reviewed changes

yarikoptic mentioned this pull request Dec 19, 2024

Delete files from backup that don't match any inventory items dandi/s3invsync#33

Closed

asmacdo added 14 commits January 17, 2025 12:18

Inital commit to add GH action to generate report

30e9cbf

Assume Jupyterhub Provisioning Role

3bcba91

Fixup: indent

b5cdcf3

Rename job

6e118cc

Add assumed role to update-kubeconfig

5062f08

No need to add ProvisioningRole to masters

d21a3a9

Deploy a pod to the cluster, and schedule with Karpenter

403028f

Fixup: correct path to pod manifest

92b9925

Fixup again ugh, rename file

478a31f

Delete Pod even if previous step times out

9db914e

(Also increase timeout)

Hack out initial du

8458d01

tmp comment out job deployment, test dockerhub build

7999455

Fixup hyphens for image name

d2e65de

Write file to output location

5e9e7df

asmacdo added 28 commits January 17, 2025 12:18

Roll back to AL 2023

cc29df5

Python 3.9 is new enough. Ubuntu was not ideal because the EFS mounting require amazonl-efs-utils which is available via package manage for amazon linux, but requires building from source on Ubuntu.

drop gzip, just write json

295361c

gzip will only be needed if we want to upload to s3

include target dir in relative paths

a667c04

Second script will not produce user report, but directory stats json

a91beb0

inital algorithm hackout

9371982

Clean up and refactor for simplicity

8cead5a

Add basic tests

86a7c72

test multiple directories in root

fc1cab1

comment about [:-1]

2308aed

support abspaths

84754fe

test propagation with files in all dirs

16e4890

Write files to disk as they are inspected

528833d

Holding the entire file index in memory would be risky, especially with zarr files

Comment out column headers in output

3c0e7f7

Write all fields for every file

260c69d

Whether there is an error or not, write all fields for each file. This allows us to read the tsv file and unpack the same number of values for each line.

Convert to reading tsv

87dd8ca

Fixup: update test to match tsv-read data

e0e0a32

update for renamed script

41aaa2a

install pip

25e27eb

install parallel

204b70e

install dependencies in launch script

64d653e

Output to tmp, accept only 1 arg, target dir

6475f11

add up sizes

b67c063

print useful info as index is created

3241473

dont fail if output dir exists

f4eb101

Create a report dict with only relevant stats

13e0e75

output data reports

a7e6991

Remove unused

845df00

asmacdo force-pushed the cron-data-usage-report branch from fcd9531 to 845df00 Compare January 17, 2025 18:19

asmacdo mentioned this pull request Jan 21, 2025

(aspirational hack project) Auto-generate EFS usage report & Log storage #173

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH action to generate report #199

GH action to generate report #199

asmacdo commented Sep 25, 2024 •

edited

Loading

kabilar Sep 26, 2024

asmacdo Sep 26, 2024

kabilar Sep 26, 2024 •

edited

Loading

kabilar commented Nov 7, 2024

asmacdo commented Nov 11, 2024 •

edited

Loading

kabilar commented Nov 14, 2024

yarikoptic Dec 12, 2024

yarikoptic Dec 12, 2024

yarikoptic Dec 12, 2024

yarikoptic Dec 12, 2024

yarikoptic left a comment

yarikoptic Dec 12, 2024

yarikoptic Dec 12, 2024

asmacdo commented Dec 12, 2024

asmacdo commented Dec 16, 2024

	ENV_FILE=".ec2-session.env"
	ENV_FILE="/run/user/$(id -u)/ec2-session.env"


		files_metadata = []

		for root, dirs, files in os.walk(directory):

GH action to generate report #199

Are you sure you want to change the base?

GH action to generate report #199

Conversation

asmacdo commented Sep 25, 2024 • edited Loading

Step 1: Create Skeleton

Step 2 Generate Report

Step 3 Push Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kabilar Sep 26, 2024 • edited Loading

Choose a reason for hiding this comment

kabilar commented Nov 7, 2024

asmacdo commented Nov 11, 2024 • edited Loading

kabilar commented Nov 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yarikoptic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asmacdo commented Dec 12, 2024

asmacdo commented Dec 16, 2024

asmacdo commented Sep 25, 2024 •

edited

Loading

kabilar Sep 26, 2024 •

edited

Loading

asmacdo commented Nov 11, 2024 •

edited

Loading