Unseen characters

[luisaocl]

Hello! Do I need to train the model on all possible combinations of subjects that i may want to represent? Also in case of already independenlty trained LoRAs? Thank you!

[petergerten]

The biggest disadvantage of MuDi / Dreambooth compares to other methods that do not require finetuning seems that a lot of examples are required. Methods like OneActor (https://johnneywang.github.io/OneActor-webpage/) get away with one single image.

If someone figure out a way to get high quality results incl. adaptation to unseen combinations with one image using MuDI, I would be very interested.

[agwmon]

Thank you for your interest in our work!

@luisaocl As you mentioned, test-time training methods like DreamBooth have the major drawback of requiring training for each subject individually, and our method also requires training for combinations of subjects. However, as shown in Figure 38 of our paper, training multiple subjects into a single LoRA (though time-consuming) eliminates the need to consider every combination separately.

Additionally, in Section 5.4 of the paper, we propose a method to utilize independently trained LoRAs, so please refer to that as well.

@petergerten Regarding the issue of MuDI(DreamBooth) requiring multiple data points, we have observed that FLUX achieves very high generalization performance even with a single image. We also believe that this issue can be addressed indirectly through various other studies by using synthetic references. (https://arxiv.org/abs/2403.14987v2, https://arxiv.org/abs/2306.00983)

[petergerten]

@agwmon thanks!

I tried several training runs with the example parameters and see strong over fitting when using one image and FLUX. While there is still prompt adherence, the characters become "locked" in the posture of the one reference image. I will look into creating synthetic references.