GH action to generate report #199

asmacdo · 2024-09-25T20:07:00Z

Fixes #177

Step 1: Create Skeleton

I've verified that when a user-node is available (created by running a tiny jupyterhub), the job pod schedules on that node. I then shut down my jupyterhub and all user-nodes scaled down. I reran this job, and Karpenter successfully scaled up a new spot node, the pod was scheduled on it, ran successfully, was deleted, and the node cleaned up. Step 1 complete!

Step 2 Generate Report

Connect Pod to EFS
List users
du each user
du shared
collate data into report
Double Check that nodes come up and down successfully
Run job several times in 1 day, check next day for EFS usage spike (IIUC we should be fine because EFS is Bursting mode)

Step 3 Push Report

Create private GitHub repository to store reports
Configure bot permission to push to repo
push report to repo on complete

Questions to answer:

If a SPOT node is preempted, can we redeploy again later?

(Also increase timeout)

kabilar · 2024-09-26T14:53:56Z

.github/workflows/report.yaml

+  pull_request:
+    branches:
+      - main


Should we also run this on a weekly basis?

It's on PR push just so I can test easily, but yes, 1/week sounds good to me. @kabilar Do you have a preference for what day/time?

Great, thank you. How about Mondays at 6am EST? We can then review the report on Monday mornings.

kabilar · 2024-11-07T00:05:29Z

Hi @asmacdo, just checking in to see how this report generation is going? Thanks.

job shouldnt take that long, but this is wall time, includes docker pull, etc

took almost 60 sec to start up

Using ubuntu for more modern python (3.12 vs 3.7 with previous or 3.9 with another amazon linux)

Python 3.9 is new enough. Ubuntu was not ideal because the EFS mounting require amazonl-efs-utils which is available via package manage for amazon linux, but requires building from source on Ubuntu.

asmacdo · 2024-12-12T23:01:26Z

see if more recent AMI with newer than 3.7 python available (e.g. 3.10 to be future proof somewhat)

Amazon Linux 2023 comes with 3.9, which is sufficient. I briefly tried ubuntu 22.04, (with 3.12) but that would add complexity, requiring amazon-efs-utils to be installed from source.

gzip will only be needed if we want to upload to s3

=== Do not change lines below === { "chain": [], "cmd": "black .github/scripts/calculate-directory-stats.py", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [ ".github/scripts/calculate-directory-stats.py" ], "pwd": "." } ^^^ Do not change lines above ^^^

Holding the entire file index in memory would be risky, especially with zarr files

Whether there is an error or not, write all fields for each file. This allows us to read the tsv file and unpack the same number of values for each line.

asmacdo · 2024-12-16T18:55:06Z

Update:

The automation is incomplete
- the launch script sets up an ec2 instance with the scripts
- scripts must be executed by hand
- produced data must be retrieved by hand
Running in parallel (8 simultaneous) the file indexing takes about 16 minutes
The file indexes are 499MB

Heres an example of the json output produced by the report generator:

{
  "total_size": 238294213,
  "file_count": 3639,
  "caches": {
    "pycache": {
      "total_size": 22782820,
      "file_count": 1431,
      "directories": [
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/extern/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/jaraco/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/jaraco/text/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/importlib_resources/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/packaging/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pkg_resources/_vendor/more_itertools/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_distutils/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_distutils/command/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/extern/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/config/_validate_pyproject/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/config/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/jaraco/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/jaraco/text/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/importlib_resources/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/importlib_metadata/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/tomli/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/packaging/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/_vendor/more_itertools/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/setuptools/command/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/_distutils_hack/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/index/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/req/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/operations/install/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/resolution/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/models/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/network/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/webencodings/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pkg_resources/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/certifi/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/tenacity/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/distro/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/tomli/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/util/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/contrib/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/contrib/_securetransport/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/packages/backports/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/urllib3/packages/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pygments/styles/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pygments/filters/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pygments/lexers/__pycache__",
        "asmacdo/venvs/my_venv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/extern/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/jaraco/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/jaraco/text/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/importlib_resources/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/packaging/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pkg_resources/_vendor/more_itertools/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_distutils/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_distutils/command/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/extern/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/config/_validate_pyproject/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/config/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/jaraco/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/jaraco/text/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/importlib_resources/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/importlib_metadata/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/tomli/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/packaging/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/_vendor/more_itertools/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/setuptools/command/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/_distutils_hack/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/index/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/cli/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/req/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/operations/build/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/operations/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/operations/install/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/resolution/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/resolution/legacy/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/models/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/vcs/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/locations/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/commands/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/network/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/utils/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/metadata/importlib/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/metadata/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_internal/distributions/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/webencodings/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pkg_resources/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/idna/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/requests/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/msgpack/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/rich/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/certifi/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/colorama/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/colorama/tests/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/tenacity/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/distro/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/cachecontrol/caches/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/tomli/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/packaging/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pyparsing/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pyparsing/diagram/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/util/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/contrib/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/contrib/_securetransport/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/packages/backports/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/urllib3/packages/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/distlib/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/chardet/cli/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/chardet/metadata/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/chardet/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/resolvelib/compat/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/resolvelib/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/platformdirs/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pygments/styles/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pygments/filters/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pygments/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pygments/lexers/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/pip/_vendor/pygments/formatters/__pycache__",
        "asmacdo/venvs/myenv/lib/python3.11/site-packages/con_duct/__pycache__",
        "asmacdo/dandi-notebooks/000971/lernerlab/seiler_2024/__pycache__"
      ]
    },
    "user_cache": {
      "total_size": 10852503,
      "file_count": 81,
      "directories": [
        "asmacdo/.cache"
      ]
    },
    "yarn_cache": {
      "total_size": 0,
      "file_count": 0,
      "directories": []
    },
    "pip_cache": {
      "total_size": 10852503,
      "file_count": 80,
      "directories": [
        "asmacdo/.cache/pip"
      ]
    },
    "nwb_cache": {
      "total_size": 0,
      "file_count": 0,
      "directories": []
    }
  }
}

Notably, the total size 238294213 is less than reported by du.
du --apparent: 247232941
du: 252841984

asmacdo added 10 commits September 25, 2024 15:06

Inital commit to add GH action to generate report

d16abb2

Assume Jupyterhub Provisioning Role

713d64c

Fixup: indent

519360c

Rename job

e6f4814

Add assumed role to update-kubeconfig

72496f4

No need to add ProvisioningRole to masters

8428d3a

Deploy a pod to the cluster, and schedule with Karpenter

e170b59

Fixup: correct path to pod manifest

bfce046

Fixup again ugh, rename file

0993129

Delete Pod even if previous step times out

87027d2

(Also increase timeout)

kabilar reviewed Sep 26, 2024

View reviewed changes

asmacdo added 2 commits November 8, 2024 12:36

Hack out initial du

686f686

tmp comment out job deployment, test dockerhub build

ff52971

asmacdo force-pushed the cron-data-usage-report branch from f53e820 to ff52971 Compare November 8, 2024 18:41

asmacdo added 15 commits November 8, 2024 12:42

Fixup hyphens for image name

ca6db89

Write file to output location

d228f9d

use kubectl cp to retrieve report

68f707f

Combine run blocks to use vars

ad6b589

Mount efs and pass arg to du script

f18e8b7

Comment out repo pushing, lets see if the report runs

387cfc1

Restrict job to asmacdo for testing

04b4193

Sanity check. Just list the directories

a443081

Job was deployed, but never assigned to node, back to sanity check

99ac264

change from job to pod

6ee89b2

deploy pod to same namespace as pvc

a8f6ed3

Use ns in action

664853b

increase timeout to 60s

e35c974

job shouldnt take that long, but this is wall time, includes docker pull, etc

fixup: image name in manifest

a8af5f2

increase timeout to 150

024cf6e

took almost 60 sec to start up

asmacdo added 2 commits December 12, 2024 16:45

Update AMI to ubuntu

5a207bc

Using ubuntu for more modern python (3.12 vs 3.7 with previous or 3.9 with another amazon linux)

Roll back to AL 2023

8ce97ee

Python 3.9 is new enough. Ubuntu was not ideal because the EFS mounting require amazonl-efs-utils which is available via package manage for amazon linux, but requires building from source on Ubuntu.

asmacdo added 25 commits December 13, 2024 11:29

drop gzip, just write json

f7fe412

gzip will only be needed if we want to upload to s3

include target dir in relative paths

e9904c8

Second script will not produce user report, but directory stats json

7da2aae

inital algorithm hackout

41a65ed

Clean up and refactor for simplicity

8eb0f06

Add basic tests

40947ef

test multiple directories in root

0e9c065

comment about [:-1]

e4794de

support abspaths

ee2c3b1

test propagation with files in all dirs

541f1f3

Write files to disk as they are inspected

ac364fb

Holding the entire file index in memory would be risky, especially with zarr files

Comment out column headers in output

05609a1

Write all fields for every file

7560db2

Whether there is an error or not, write all fields for each file. This allows us to read the tsv file and unpack the same number of values for each line.

Convert to reading tsv

502ff76

Fixup: update test to match tsv-read data

639f279

update for renamed script

96490e5

install pip

64d69a7

install parallel

6f3dae5

install dependencies in launch script

b38be7d

Output to tmp, accept only 1 arg, target dir

f0b0709

add up sizes

326bb55

print useful info as index is created

c881287

dont fail if output dir exists

a3505f9

Create a report dict with only relevant stats

fcd9531

yarikoptic mentioned this pull request Dec 19, 2024

Delete files from backup that don't match any inventory items dandi/s3invsync#33

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH action to generate report #199

GH action to generate report #199

asmacdo commented Sep 25, 2024 •

edited

Loading

kabilar Sep 26, 2024

asmacdo Sep 26, 2024

kabilar Sep 26, 2024 •

edited

Loading

kabilar commented Nov 7, 2024

asmacdo commented Dec 12, 2024

asmacdo commented Dec 16, 2024

GH action to generate report #199

Are you sure you want to change the base?

GH action to generate report #199

Conversation

asmacdo commented Sep 25, 2024 • edited Loading

Step 1: Create Skeleton

Step 2 Generate Report

Step 3 Push Report

kabilar Sep 26, 2024

Choose a reason for hiding this comment

asmacdo Sep 26, 2024

Choose a reason for hiding this comment

kabilar Sep 26, 2024 • edited Loading

Choose a reason for hiding this comment

kabilar commented Nov 7, 2024

asmacdo commented Dec 12, 2024

asmacdo commented Dec 16, 2024

asmacdo commented Sep 25, 2024 •

edited

Loading

kabilar Sep 26, 2024 •

edited

Loading