Using captions & CLIP with Lora?

[ClaytonFarr]

It sounds like using captions can improve training in Lora significantly (example). @cloneofsimo does that sound right to you? @brian6091 I see you have an option for captions in your Colab as well.

Do folks have any tips on using captions in general, or with Lora specifically?

• Caption content? — Are folks using whatever text CLIP spits out, or tailoring this in some ways?
• Include token word(s) in caption? — Is this necessary or helpful to add to each caption?
• Using captions outside of filenames with Lora? — E.g. can captions be in matching .txt files or a single desc.csv file?
• Integrating CLIP Interrogator into Lora pipeline? — I see references to loading CLIP output into the dataloader class, but I haven't been able to find any full-fledged examples yet.

Any help or guidance would be much appreciated.

And, if there's a resource on using captions or CLIP effectively in training that I haven't been able to find yet, feel free to pawn me over to that as well. Thanks!

[brian6091]

I actually haven't gotten around to testing captions. I'd be happy to help if you want to try it out with my repo. For now, it's set up to use filenames (although separate text files would not be hard to add), and the class images all get the same class prompt.

You might find discussions over at the Stable Diffusion Dreambooth discord helpful: https://discord.gg/JFsWqZFb3e

[ClaytonFarr]

Thanks @brian6091. I’ll try some options out to see what results I can get.

If it’s possible to easily add an option to read the captions from external file(s), that’d be a big help I suspect. Seems like most CLIP output is longer than what can be contained in a file name.

Thanks!

[ExponentialML]

@ClaytonFarr LoRA support is a bit behind on this repos development, but you can currently do what @cloneofsimo suggested here using AUTOMATIC's ui and the Dreambooth extension with the LoRA option checked under the "Advanced".

Simply use [filewords] for both the image & class prompts.

[brian6091]

@ClaytonFarr
I just cleaned up the caption option on my repo: https://github.com/brian6091/Dreambooth
You can use filenames or separate text files to provide the captions. Happy to help if you are interested in trying it out.

[ClaytonFarr]

@brian6091 Thanks for the quick add on that feature - I'll start some tests today to see what I can find.

@ExponentialML Thanks for the tip about A1111 DB extension; I didn't realize it supported Lora yet.

[Boldor83]

unfortunatly [filewords] doenst work for me in automatic1111, it works for hypernet but when i try it in lora dreambooth it doens't look like it reads the prompt in the .txt files at all. if i instead copy one of the prompts and use it instead of [filewords] it works. but i can't use different prompts for each image that way.

Edit: the results after the model is finished still look good, maybe its just not working for the sample prompt?
I also get interessting results when i change the class prompt during training :), something thats easier to test when the training is this fast.

[cloneofsimo]

Maybe! But can't say for sure. Maybe having multiple prompt per image might help as well. Thanks for the helpful suggestion!

[cloneofsimo]

Another source suggesting that caption helps, but not CLIP integrator captions. https://www.reddit.com/r/StableDiffusion/comments/zcr644/make_better_dreambooth_style_models_by_using/

[ClaytonFarr]

@cloneofsimo This is a great thread, thanks for the heads up. I'm curious to see if / how a similar approach can be taken for training subjects instead of styles.

Also interesting to see him avoiding using CLIP for the captions.

I'm still a bit confused on the applicability of the keywords CLIP kicks out. My understanding was CLIP's output being what the model already understands based on what it's seeing in the training images.

When and whether to use, ignore, or append to this when training those images is still a bit blurry.

[ClaytonFarr]

Another question on the CLIP train - what are folks favorite methods for batch processing images with CLIP? Using a tool like Automatic1111 web ui, a script, or something else?

[hdon96]

I used this notebook I found here for batch renaming with the clip model used in stable diffusion 2/2.1.

[ClaytonFarr]

For others digging into captioning, it looks like the EveryDream repo and Discord are a great resource for this info as well (a Dreambooth-esque tool that uses captioning extensively). There's also a set of accompanying tools to help with batch captioning and the like.

Still digging into to it's info & tooling, but it looks interesting in its own right and as way to help prep training content for Lora.

[ballerburg9005]

I have heard a lot of talk about captioning, however I have a bit of a hard time finding any concrete examples whatsoever.