How to finetune the infinity based on your provided weights?

[xuanyuzhang21]

Thanks for your great work and code release! Could you provide any instructions for finetuning infinity based on your provided checkpoints?

[JeyesHan]

@xuanyuzhang21 Thanks for your appreciation to Infinity. Finetuning is very simple. You can run train.sh with --rush_resume [infinity_2B.pth].

[xuanyuzhang21]

Thanks for your help~

[hakanmuluk]

Firstly, thanks for the great work! However, there happens lots of shape mismatches when I directly use --rush_resume with the model you have provided, is there a solution for it?

[JeyesHan]

@hakanmuluk Have you ever solved the problems? If not, could you pull latest commit and try again?

[Kaiwen-Zhu]

@JeyesHan A minor typo in README: --rush_resume=[infinity_vae_d32reg.pth] -> --rush_resume=[infinity_2b_reg.pth]

[hakanmuluk]

@JeyesHan No, I could not solve them. I tried with the latest commit and the shape mismatches looks disappeared but I still get meaningless images. However, the loss and accuracy starts in a better region suggesting that the model is loaded without any problem. I think that model is finetuned but there is something incompatible with the training and infer.sh so I cannot get good images in the inference but get images similar to the one shared in the other issue about finetuning. Is the infer.sh compatible with the trained version of the 2b model you have provided? Also, there are some problems in the code: visualizer is not defined in train.py, there should be something like enable_model_cache in the interactive_ipynb.

[metildachee]

Hello @xuanyuzhang21 if its possible could you please share how much GPU was necessary for your training set? I tried using rush_resume on 2b and vae_ckpt on 32b but 40GPU wasn't sufficient. Did you use more than that? Looking forward! Thank you very much! 🙏🏻

[xuanyuzhang21]

@metildachee I finetuned the model via a single 80GB a100.

[JeyesHan]

@hakanmuluk Can you post your b1_stdout.txt and interactive_ipynb file here? Such that I can find the problem.