Are you testing with 7.1 GB weights?

[Michoko92]

Hi,

I can't wait to try your code when the weights are finally released! However, I'm curious to know which weights you're using, as it seems that the ones that were sent to testers are bigger (7.1 gb) than the ones that will be released within a couple of days. SD staff announced the final optimized weights would be around 2 gb in size. Does that mean that your code could run even faster on such a small weight? Or is it already the one you make your tests with?

Cheers!

[basujindal]

Hi, SD released two types of weights, 4.3 Gb and 7.7Gb (the larger weights have ema checkpoints included). I am using the 4.3 Gb weights. I am guessing that the 2Gb weights recently announced are just the 4.3 Gb weights converted to half precision which effectively reduces their size from 4.3Gb --> 2.1Gb. But I am not sure since they may be doing other optimizations too.

I am already converting the weights to half-precision in my code, so if that is the case, I don't think my code will run faster due to the reduction in weight size. To reduce the required VRAM further, this repo uses other modifications like splitting the model into multiple parts and moving them into the GPU only when required. This increases the inference time a little bit, but one can generate larger images.

[GucciFlipFlops1917]

Thanks for your hard work, man. It's much appreciated between the cornering of the market by server-oriented companies like OpenAI and the chip shortage, making it difficult to get your hands on higher-end hardware. Your fork is a refinement of the already-consumer friendly Stable Diffusion project and a breath of fresh air that I haven't truly had since VQGAN.

[Michoko92]

Hi Basu,

Thank you for your kind reply, this is very interesting information and much appreciated. I am also a fellow RTX 2060 owner, and am extremely interested in using your SD optimization (I've been a happy beta tester and fell in love with the technology).

I have a few extra questions, if I may:

• From what you just wrote, it may seem that a smaller 2 GB model would at least require less splitting, theoretically speeding things up a bit? Or maybe allow bigger resolutions?
• Speaking of resolutions, I was generally using the portrait format (512x704) during the beta test. If I read your Readme correctly, I can hope to do this with your implementation too? By checking my remaining VRAM with GPU-Z, I have about 580 MB of VRAM in use, so I have more or less 5.5 GB of VRAM left, I guess.
• If you have two GPUs in your PC (a discrete Nvidia GPU and an onboard GPU), is there a way to clearly specify the GPU you want to use, or this isn't necessary?

Thanks again, and keep up the great work! Exciting times, for sure!

[basujindal]

I agree this technology is amazing, thanks to the people at stabilityAI.

To answer your questions,

• In the case of the original SD repo, the 2GB model will definitely speed up inference and allow bigger resolutions. But this repo is already reducing the model size to around 2.1GB by converting the weights to half precision. So unless the smaller model is using optimizations other than half-precision, I don't think that this repo will gain an advantage from using the smaller weights. Regardless of the weight size, splitting the model always allows for generating a larger resolution at the expense of inference speed.

• Yes, you can generate 512x704 images on a 6GB GPU. But I think you may be able to use a larger batch size if you close the processes using 580 MB of your GPU VRAM.

• Currently, there is no way to specify the GPU in this repo; I will make the necessary changes and push the update in a short while.

Thanks!

[Michoko92]

Awesome! Thank you so much!

I tracked down some apps, and managed to only use 338 MB of VRAM. I suppose I'm good to go. Can't wait to see what will be possible. Cheers! :)