New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OOM Issue (A6000 GPU, Batch Size 8 per GPU) #144

Open

2minkyulee opened this issue May 27, 2024 · 0 comments

2minkyulee commented May 27, 2024

Thanks for your great work!

I'm having an Out of Memory issue with the following configuration:
(Probably the default training setting and identical VRAM size)

batch size per GPU = 8
A6000 GPUs (48G VRAM)
gt_size = 256

Disabling the CUDA prefetcher didn't help.
A batch size of 7 per GPU works fine, so it seems to be a slight OOM problem.

Gradient checkpointing also worked, but led to slower training. It would be better if we have alternatives.

Are there any configurations that I might have missed to reduce VRAM usage?
Or do you have any other suggestions?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment