Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

YOWO not using GPUs? #83

Open
ljorquera opened this issue Jun 4, 2021 · 1 comment
Open

YOWO not using GPUs? #83

ljorquera opened this issue Jun 4, 2021 · 1 comment

Comments

@ljorquera
Copy link

Hello

First of all, thanks a lot for making this available.

I am using YOWO to train with a custom dataset. I prepared my data in the same format than AVA and used the ava.cfg configuration to run.

It is running OK but very slowly (almost 3 hours for each epoch, with total number of trainable parameters equal to 121381770). The machine where I am running has 4 GPUs but I have several reasons to think that YOWO is not using them:

  • It makes no difference if I set the variable CUDA_VISIBLE_DEVICES to '0,1,2,3' or '0' or '', the speed is the same
  • I am monitoring GPU usage with nvidia-smi and most of the time I see that GPU usage is 0
  • If I check CPU usage, python is using 100% all the time

I added code to main.py to see if the GPUs are accesible and I can confirm that they are. The following

print(f" cuda available {torch.cuda.is_available()}") print(f" current device {torch.cuda.current_device()}") print(f" device 0 {torch.cuda.device(0)}") print(f" device count {torch.cuda.device_count()}")

Produces

cuda available True current device 0 device 0 <torch.cuda.device object at 0x7fdcf4a2ae80> device count 4
My question is, do I need to change something else to make YOWO use the GPUs?

@Griffintaur
Copy link

I am also facing the same any update on this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants