What's the difference betwee sample_gs_* and sample_scaled_gs_*

[Y-ichen]

When I was reproducing the results for the paper, I found that files named sample_gs_* and sample_scaled_gs_* are contemporary produced.
The results for sample_scaled_gs_* are good:

However I don't know what does a samples_gs_* file means:

Therefore I want to know the difference between them.
Besides, what makes sample_scaled_gs_* that better than sample_gs_* ? Does the input image set acts as a role for producing sample_scaled_gs_* ? (for example, is the start_code for diffusion model initialized from one image from the given image set?)

[rinongal]

Hi,

You can read more info about the outputs here: #19, #34

tl;dr: The difference is that samples_scaled_gs uses a classifier guidance scale of 5.0, while samples_gs uses a guidance scale of 1.0. You only care about samples_scaled_gs since you're going to use high guidance scales at inference anyhow.

However, If your unscaled samples look too much like your concept, that's a good sign of overfitting.

Also - if you are trying to reproduce - keep in mind the paper predates Stable Diffusion and uses the original LDM. You're not going to get similar results with SD.

[Y-ichen]

Thanks! That's a good explanation! Actually I am using stable diffusion 1.5 and trying to do some experiments with it, I think SD works well.
If I want to add my own loss, where in the code should I add it? I am not familiar with pytorch_lightning and have difficult to find where the loss is fixed. The loss I want to add is about the generated image with an gt_image given by myself.

[rinongal]

You can add losses inside this function:

textual_inversion/ldm/models/diffusion/ddpm.py

Line 1053 in 26ed44f

def p_losses(self, x_start, cond, t, noise=None):

You'll probably have to understand how to pipe your data into that function, however.