Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

running on the cluster takes forever #77

Open
nikihowe opened this issue Jun 16, 2021 · 1 comment
Open

running on the cluster takes forever #77

nikihowe opened this issue Jun 16, 2021 · 1 comment

Comments

@nikihowe
Copy link 

/Users/niki/Documents/constrained_vs_regularized/buddy-venv/bin/python /Users/niki/Documents/constrained_vs_regularized/mnist_classifier.py
INFO:root:Name your run in the pop-up window!
experiment_id: [CLUSTER] no_id
INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_8.2p1)
INFO:paramiko.transport:Authentication (publickey) successful!
/tmp/experiment_buddy-CsM9XfKYFk
INFO:paramiko.transport.sftp:[chan 2] Opened sftp connection (server version 3)
Slurmctld(primary) at slurm is UP
Slurmctld(backup) at slurmctl is DOWN
HEAD is now at b561923 Add torch and torchvision to requirements
To github.com:manuel-delverme/constrained_vs_regularized.git
 * [new tag]         snapshot/master/54ec8a6c75a692514e62cdacc5347da52002b479 -> snapshot/master/54ec8a6c75a692514e62cdacc5347da52002b479
Switched to branch 'master'
monitor your run on https://wandb.ai/
bash -l /tmp/experiment_buddy-CsM9XfKYFk//run_experiment.sh [email protected]:manuel-delverme/constrained_vs_regularized.git mnist_classifier.py 54ec8a6c75a692514e62cdacc5347da52002b479
  0%|          | 0/1 [00:00<?, ?it/s]"[DEPLOY LOG] script realpath: /tmp/experiment_buddy-CsM9XfKYFk/run_experiment.sh"
"[DEPLOY LOG] scripts home: /tmp/experiment_buddy-CsM9XfKYFk"
"[DEPLOY LOG] cd /home/mila/h/howeniko/experiments/"
"[DEPLOY LOG] downloading source code from [email protected]:manuel-delverme/constrained_vs_regularized.git to ./tmp.n3cFFKIn3X"
Warning: Permanently added the RSA host key for IP address '140.82.112.4' to the list of known hosts.
Note: checking out '54ec8a6c75a692514e62cdacc5347da52002b479'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 54ec8a6 [CLUSTER] no_id
"[DEPLOY LOG] pwd is now /home/mila/h/howeniko/experiments/tmp.n3cFFKIn3X"
"[DEPLOY LOG] Refreshing modules..."
The following modules were not unloaded:
  (Use "module --force purge" to unload all):

  1) gcc/7.4.0   2) Mila
[=== Module python/3.7 loaded ===]
/tmp/experiment_buddy-CsM9XfKYFk/common.sh: line 41: /home/mila/h/howeniko/venv/bin/activate: No such file or directory
"[DEPLOY LOG] Setting up venv @ /home/mila/h/howeniko/venv..."
created virtual environment CPython3.7.6.final.0-64 in 29533ms
  creator CPython3Posix(dest=/home/mila/h/howeniko/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/mila/h/howeniko/.local/share/virtualenv)
    added seed packages: pip==20.2.2, setuptools==56.2.0, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
"[DEPLOY LOG] Using shared venv @ /home/mila/h/howeniko/venv"
Collecting pip
  Using cached pip-21.1.2-py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.2
    Uninstalling pip-20.2.2:
      Successfully uninstalled pip-20.2.2
Successfully installed pip-21.1.2
  Running command git clone -q https://github.com/ministry-of-silly-code/experiment_buddy.git /home/mila/h/howeniko/venv/src/experiment-buddy
  Running command git rev-parse -q --verify 'sha^875823cf75cf176680e61d1a45bbe56e3e303747'
  Running command git fetch -q https://github.com/ministry-of-silly-code/experiment_buddy.git 875823cf75cf176680e61d1a45bbe56e3e303747
  Running command git checkout -q 875823cf75cf176680e61d1a45bbe56e3e303747
Obtaining experiment_buddy from git+https://github.com/ministry-of-silly-code/experiment_buddy.git@875823cf75cf176680e61d1a45bbe56e3e303747#egg=experiment_buddy (from -r requirements.txt (line 14))
  Cloning https://github.com/ministry-of-silly-code/experiment_buddy.git (to revision 875823cf75cf176680e61d1a45bbe56e3e303747) to /home/mila/h/howeniko/venv/src/experiment-buddy
Collecting aiohttp==3.7.4.post0
  Downloading aiohttp-3.7.4.post0-cp37-cp37m-manylinux2014_x86_64.whl (1.3 MB)
Collecting async-timeout==3.0.1
  Using cached async_timeout-3.0.1-py3-none-any.whl (8.2 kB)
Collecting attrs==21.2.0
  Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting bcrypt==3.2.0
  Using cached bcrypt-3.2.0-cp36-abi3-manylinux2010_x86_64.whl (63 kB)
Collecting certifi==2021.5.30
  Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting cffi==1.14.5
  Downloading cffi-1.14.5-cp37-cp37m-manylinux1_x86_64.whl (402 kB)
Collecting chardet==4.0.0
  Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting click==8.0.1
  Using cached click-8.0.1-py3-none-any.whl (97 kB)
Collecting cloudpickle==1.6.0
  Using cached cloudpickle-1.6.0-py3-none-any.whl (23 kB)
Collecting configparser==5.0.2
  Using cached configparser-5.0.2-py3-none-any.whl (19 kB)
Collecting cryptography==3.4.7
  Using cached cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
Collecting cycler==0.10.0
  Using cached cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Collecting docker-pycreds==0.4.0
  Using cached docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)
Collecting fabric==2.6.0
  Using cached fabric-2.6.0-py2.py3-none-any.whl (53 kB)
Collecting funcy==1.16
  Using cached funcy-1.16-py2.py3-none-any.whl (32 kB)
Collecting gitdb==4.0.7
  Using cached gitdb-4.0.7-py3-none-any.whl (63 kB)
Collecting GitPython==3.1.17
  Using cached GitPython-3.1.17-py3-none-any.whl (166 kB)
Collecting importlib-metadata==4.5.0
  Using cached importlib_metadata-4.5.0-py3-none-any.whl (17 kB)
Collecting invoke==1.5.0
  Using cached invoke-1.5.0-py3-none-any.whl (211 kB)
Collecting kiwisolver==1.3.1
  Using cached kiwisolver-1.3.1-cp37-cp37m-manylinux1_x86_64.whl (1.1 MB)
Collecting matplotlib==3.4.2
  Using cached matplotlib-3.4.2-cp37-cp37m-manylinux1_x86_64.whl (10.3 MB)
Collecting multidict==5.1.0
  Downloading multidict-5.1.0-cp37-cp37m-manylinux2014_x86_64.whl (142 kB)
Collecting numpy==1.20.3
  Using cached numpy-1.20.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.3 MB)
Collecting paramiko==2.7.2
  Downloading paramiko-2.7.2-py2.py3-none-any.whl (206 kB)
Collecting pathlib2==2.3.5
  Downloading pathlib2-2.3.5-py2.py3-none-any.whl (18 kB)
Collecting pathtools==0.1.2
  Using cached pathtools-0.1.2.tar.gz (11 kB)
Collecting Pillow==8.2.0
  Using cached Pillow-8.2.0-cp37-cp37m-manylinux1_x86_64.whl (3.0 MB)
Collecting promise==2.3
  Using cached promise-2.3.tar.gz (19 kB)
Collecting protobuf==3.17.2
  Downloading protobuf-3.17.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
Collecting psutil==5.8.0
  Downloading psutil-5.8.0-cp37-cp37m-manylinux2010_x86_64.whl (296 kB)
Collecting PyNaCl==1.4.0
  Downloading PyNaCl-1.4.0-cp35-abi3-manylinux1_x86_64.whl (961 kB)
Collecting PyYAML==5.4.1
  Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)
Collecting requests==2.25.1
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting sentry-sdk==1.1.0
  Using cached sentry_sdk-1.1.0-py2.py3-none-any.whl (131 kB)
Collecting shortuuid==1.0.1
  Downloading shortuuid-1.0.1-py3-none-any.whl (7.5 kB)
Collecting six==1.16.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting smmap==4.0.0
  Downloading smmap-4.0.0-py2.py3-none-any.whl (24 kB)
Collecting subprocess32==3.5.4
  Downloading subprocess32-3.5.4.tar.gz (97 kB)
Collecting tensorboardX==2.2
  Using cached tensorboardX-2.2-py2.py3-none-any.whl (120 kB)
Collecting tqdm==4.61.0
  Downloading tqdm-4.61.0-py2.py3-none-any.whl (75 kB)
Collecting typing-extensions==3.10.0.0
  Using cached typing_extensions-3.10.0.0-py3-none-any.whl (26 kB)
WARNING: The candidate selected for download or install is a yanked version: 'gitpython' candidate (version 3.1.17 at https://files.pythonhosted.org/packages/27/da/6f6224fdfc47dab57881fe20c0d1bc3122be290198ba0bf26a953a045d92/GitPython-3.1.17-py3-none-any.whl#sha256=29fe82050709760081f588dd50ce83504feddbebdc4da6956d02351552b1c135 (from https://pypi.org/simple/gitpython/) (requires-python:>=3.5))
Reason for being yanked: Breaks older versions of python 3.5, 3.5.2 notably, due to typing
Collecting urllib3==1.26.5
  Downloading urllib3-1.26.5-py2.py3-none-any.whl (138 kB)
Collecting wandb==0.10.31
  Using cached wandb-0.10.31-py2.py3-none-any.whl (1.8 MB)
Collecting yarl==1.6.3
  Downloading yarl-1.6.3-cp37-cp37m-manylinux2014_x86_64.whl (294 kB)
Collecting zipp==3.4.1
  Downloading zipp-3.4.1-py3-none-any.whl (5.2 kB)
Collecting torchvision~=0.9.1
  Downloading torchvision-0.9.1-cp37-cp37m-manylinux1_x86_64.whl (17.4 MB)
Collecting torch~=1.8.1
  Using cached torch-1.8.1-cp37-cp37m-manylinux1_x86_64.whl (804.1 MB)
Building wheels for collected packages: pathtools, promise, subprocess32
  Building wheel for pathtools (setup.py): started
  Building wheel for pathtools (setup.py): finished with status 'done'
  Created wheel for pathtools: filename=pathtools-0.1.2-py3-none-any.whl size=8784 sha256=e16596d3fa45ae1d175fd68e2a2e8ccb01ac55fad88039efe4f750e4888956be
  Stored in directory: /home/mila/h/howeniko/.cache/pip/wheels/3e/31/09/fa59cef12cdcfecc627b3d24273699f390e71828921b2cbba2
  Building wheel for promise (setup.py): started
  Building wheel for promise (setup.py): finished with status 'done'
  Created wheel for promise: filename=promise-2.3-py3-none-any.whl size=21494 sha256=0888e9776a12125b16e0afce01608bb98a78707e64e0dd8ba64b1085d8d2a718
  Stored in directory: /home/mila/h/howeniko/.cache/pip/wheels/29/93/c6/762e359f8cb6a5b69c72235d798804cae523bbe41c2aa8333d
  Building wheel for subprocess32 (setup.py): started
  Building wheel for subprocess32 (setup.py): finished with status 'done'
  Created wheel for subprocess32: filename=subprocess32-3.5.4-py3-none-any.whl size=6488 sha256=647668f1d888bd3204aa9e1da64e1fb86f02fd0f028618411691afebe0c27367
  Stored in directory: /home/mila/h/howeniko/.cache/pip/wheels/50/ca/fa/8fca8d246e64f19488d07567547ddec8eb084e8c0d7a59226a
Successfully built pathtools promise subprocess32
Installing collected packages: zipp, typing-extensions, smmap, six, cffi, urllib3, PyNaCl, multidict, importlib-metadata, gitdb, cryptography, chardet, certifi, bcrypt, yarl, subprocess32, shortuuid, sentry-sdk, requests, PyYAML, psutil, protobuf, promise, Pillow, pathtools, pathlib2, paramiko, numpy, kiwisolver, invoke, GitPython, docker-pycreds, cycler, configparser, click, attrs, async-timeout, wandb, tqdm, torch, tensorboardX, matplotlib, funcy, fabric, cloudpickle, aiohttp, torchvision, experiment-buddy
  Attempting uninstall: zipp
    Found existing installation: zipp 3.0.0
    Not uninstalling zipp at /cvmfs/ai.mila.quebec/apps/x86_64/debian/python/3.7/lib/python3.7/site-packages, outside environment /home/mila/h/howeniko/venv
    Can't uninstall 'zipp'. No files were found to uninstall.
  Attempting uninstall: six
    Found existing installation: six 1.14.0
    Not uninstalling six at /cvmfs/ai.mila.quebec/apps/x86_64/debian/python/3.7/lib/python3.7/site-packages, outside environment /home/mila/h/howeniko/venv
    Can't uninstall 'six'. No files were found to uninstall.
  Attempting uninstall: cffi
    Found existing installation: cffi 1.14.3
    Not uninstalling cffi at /cvmfs/ai.mila.quebec/apps/x86_64/debian/python/3.7/lib/python3.7/site-packages, outside environment /home/mila/h/howeniko/venv
    Can't uninstall 'cffi'. No files were found to uninstall.
  Attempting uninstall: urllib3
    Found existing installation: urllib3 1.25.10
    Not uninstalling urllib3 at /cvmfs/ai.mila.quebec/apps/x86_64/debian/python/3.7/lib/python3.7/site-packages, outside environment /home/mila/h/howeniko/venv
    Can't uninstall 'urllib3'. No files were found to uninstall.
  Attempting uninstall: importlib-metadata
    Found existing installation: importlib-metadata 1.5.0
    Not uninstalling importlib-metadata at /cvmfs/ai.mila.quebec/apps/x86_64/debian/python/3.7/lib/python3.7/site-packages, outside environment /home/mila/h/howeniko/venv
    Can't uninstall 'importlib-metadata'. No files were found to uninstall.
  Attempting uninstall: cryptography
    Found existing installation: cryptography 3.1.1
    Not uninstalling cryptography at /cvmfs/ai.mila.quebec/apps/x86_64/debian/python/3.7/lib/python3.

^ I've been waiting for it to not uninstall cryptography for at least 20 minutes...
@manuel-delverme
Copy link
Member

manuel-delverme commented Jun 16, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@manuel-delverme @nikihowe