Training GPU requirement

[Sm0kyWu]

Hi! Thanks for the amazing code!

I would like to ask about the requirement of training. Currently, I am using a single A100 with 40G RAM. My training code follows Moore. The problem is that no matter what video size I use (I even tried 64*64), it will be out of memory.

Could you please kindly share some information about training?

Thanks!

[TZYSJTU]

Hi! Thanks for the amazing code!

I would like to ask about the requirement of training. Currently, I am using a single A100 with 40G RAM. My training code follows Moore. The problem is that no matter what video size I use (I even tried 64*64), it will be out of memory.

Could you please kindly share some information about training?

Thanks!

8x80GB deepspeed zero2

[FangSen9000]

@Sm0kyWu If you use the training code for the mole thread, does it need some modification and approximately how much extra time will it take?

[Sm0kyWu]

@FangSen9000 I haven't tried to train the Moore code. For Musepose you can directly use the training script from Moore. Single 40G A100 takes around 40 hours for 10000 steps of stage 1 (bs 6, 768x768, deepspeed zero2).