Code

GH action to generate report#199

Code

Draft

asmacdo wants to merge 94 commits intomaindandi/dandi-hub:mainfrom cron-data-usage-reportdandi/dandi-hub:cron-data-usage-reportCopy head branch name to clipboard

+763-63

Commits

Commits on Sep 25, 2024

• Inital commit to add GH action to generate report

  asmacdo

  committed
  d16abb2

• Assume Jupyterhub Provisioning Role

  asmacdo

  committed
  713d64c

• Fixup: indent

  asmacdo

  committed
  519360c

• Rename job

  asmacdo

  committed
  e6f4814

• Add assumed role to update-kubeconfig

  asmacdo

  committed
  72496f4

• No need to add ProvisioningRole to masters

  asmacdo

  committed
  8428d3a

• Deploy a pod to the cluster, and schedule with Karpenter

  asmacdo

  committed
  e170b59

• Fixup: correct path to pod manifest

  asmacdo

  committed
  bfce046

• Fixup again ugh, rename file

  asmacdo

  committed
  0993129

• Delete Pod even if previous step times out

  Show description for 87027d2
  asmacdo

  committed
  87027d2

Commits on Nov 8, 2024

• Hack out initial du

  asmacdo

  committed
  686f686

• tmp comment out job deployment, test dockerhub build

  asmacdo

  committed
  ff52971

• Fixup hyphens for image name

  asmacdo

  committed
  ca6db89

• Write file to output location

  asmacdo

  committed
  d228f9d

• use kubectl cp to retrieve report

  asmacdo

  committed
  68f707f

• Combine run blocks to use vars

  asmacdo

  committed
  ad6b589

• Mount efs and pass arg to du script

  asmacdo

  committed
  f18e8b7

• Comment out repo pushing, lets see if the report runs

  asmacdo

  committed
  387cfc1

• Restrict job to asmacdo for testing

  asmacdo

  committed
  04b4193

• Sanity check. Just list the directories

  asmacdo

  committed
  a443081

• Job was deployed, but never assigned to node, back to sanity check

  asmacdo

  committed
  99ac264

• change from job to pod

  asmacdo

  committed
  6ee89b2

• deploy pod to same namespace as pvc

  asmacdo

  committed
  a8f6ed3

• Use ns in action

  asmacdo

  committed
  664853b

• increase timeout to 60s

  Show description for e35c974
  asmacdo

  committed
  e35c974

• fixup: image name in manifest

  asmacdo

  committed
  a8af5f2

• increase timeout to 150

  Show description for 024cf6e
  asmacdo

  committed
  024cf6e

• override entrypoint so i can debug with exec

  asmacdo

  committed
  49c346e

• bound /home actually meant path was /home/home/asmacdo

  asmacdo

  committed
  0191c85

• Create output dir prior to writing report

  asmacdo

  committed
  3eb9157

Commits on Nov 11, 2024

• pod back to job

  asmacdo

  committed
  676a00e

• Fixup use the correct job api

  asmacdo

  committed
  c085751

• Add namespace to pod retrieval

  asmacdo

  committed
  3e18a37

• write directly to pv to test job

  asmacdo

  committed
  0fa5ece

• fixup script fstring

  asmacdo

  committed
  e1ecbc3

• no retry on failure, we were spinning up 5 pods, lets just fail 1 time

  asmacdo

  committed
  082d3cc

• Fixup backup limit job not template

  asmacdo

  committed
  d46ea44

• Initial report

  asmacdo

  committed
  965a81e

• disable report

  Show description for 7366d2d
  asmacdo

  committed
  7366d2d

Commits on Dec 2, 2024

• deploy ec2 instance directly

  asmacdo

  committed
  747f0a4

• Update AMI image

  asmacdo

  committed
  6156e21

• update sg and subnet

  asmacdo

  committed
  588892c

• terminate even if job fails

  asmacdo

  committed
  958630b

• debug: print public ip

  asmacdo

  committed
  e24a666

• explicitly allocate public ip for ec2 instance

  asmacdo

  committed
  0e58f10

Commits on Dec 6, 2024

• Add WIP scripts

  asmacdo

  committed
  5c28c0e

• rm old unused

  asmacdo

  committed
  21811dd

• initial commit of scripts

  asmacdo

  committed
  97de713

• clean up launch script

  asmacdo

  committed
  644f8c3

• make scripe executable

  asmacdo

  committed
  e176592

• fixup cleanup script

  asmacdo

  committed
  a101f18

• add a name to elastic ip (for easier manual cleanup)

  asmacdo

  committed
  615baf2

• Exit on fail

  asmacdo

  committed
  bb8f25a

• Add permission for aws ec2 wait instance-status-ok

  asmacdo

  committed
  a8a615a

• Upload scripts to instance

  asmacdo

  committed
  8157a12

• explicitly return

  asmacdo

  committed
  d3f6f52

Commits on Dec 11, 2024

• output session variables to file

  Show description for f1f687f
  asmacdo

  committed
  f1f687f

• modify cleanup script to retrieve instance from temporary file

  asmacdo

  committed
  a10bc2a

• All ec2 persmissions granted

  Show description for 7fd340d
  asmacdo

  committed
  7fd340d

• Add EFS mount (hardcoded)

  asmacdo

  committed
  1649b35

• No pager for termination

  asmacdo

  committed
  30aa60c

• force pseudo-terminal, otherwise hangs after yum install

  asmacdo

  committed
  cc845d4

• Add doublequotes to variable usage for proper expansion

  asmacdo

  committed
  7854124

• Fixup -t goes on ssh, not scp

  asmacdo

  committed
  9fbad37

• Mount as a single command, since we dont have access to pty

  asmacdo

  committed
  4fc9dde

• add todos for manual steps

  asmacdo

  committed
  86e645e

• Disable job for now

  Show description for c614004
  asmacdo

  committed
  c614004

Commits on Dec 12, 2024

• Update AMI to ubuntu

  Show description for 5a207bc
  asmacdo

  committed
  5a207bc

• Roll back to AL 2023

  Show description for 8ce97ee
  asmacdo

  committed
  8ce97ee

Commits on Dec 13, 2024

• drop gzip, just write json

  Show description for f7fe412
  asmacdo

  committed
  f7fe412

• include target dir in relative paths

  asmacdo

  committed
  e9904c8

• Second script will not produce user report, but directory stats json

  asmacdo

  committed
  7da2aae

• inital algorithm hackout

  asmacdo

  committed
  41a65ed

• Clean up and refactor for simplicity

  asmacdo

  committed
  8eb0f06

• Add basic tests

  asmacdo

  committed
  40947ef

• test multiple directories in root

  asmacdo

  committed
  0e9c065

• comment about [:-1]

  asmacdo

  committed
  e4794de

Commits on Dec 14, 2024

• support abspaths

  asmacdo

  committed
  ee2c3b1

• [DATALAD RUNCMD] blacken

  Show description for e1dcd63
  asmacdo

  committed
  e1dcd63

• test propagation with files in all dirs

  asmacdo

  committed
  541f1f3

Commits on Dec 15, 2024

• Write files to disk as they are inspected

  Show description for ac364fb
  asmacdo

  committed
  ac364fb

• Comment out column headers in output

  asmacdo

  committed
  05609a1

• Write all fields for every file

  Show description for 7560db2
  asmacdo

  committed
  7560db2

• Convert to reading tsv

  asmacdo

  committed
  502ff76

• Fixup: update test to match tsv-read data

  asmacdo

  committed
  639f279

• update for renamed script

  asmacdo

  committed
  96490e5

• install pip

  asmacdo

  committed
  64d69a7

• install parallel

  asmacdo

  committed
  6f3dae5

• install dependencies in launch script

  asmacdo

  committed
  b38be7d

• Output to tmp, accept only 1 arg, target dir

  asmacdo

  committed
  f0b0709

Commits on Dec 16, 2024

• add up sizes

  asmacdo

  committed
  326bb55

• print useful info as index is created

  asmacdo

  committed
  c881287

• dont fail if output dir exists

  asmacdo

  committed
  a3505f9

• Create a report dict with only relevant stats

  asmacdo

  committed
  fcd9531