Problems encountered in training

[shiminxuan003]

Thanks for your great work!  We tried to retrain with 512 resolution, but when inference, the face did not change. Is there a mistake in training? Tensorboard is as follow, Is it normal?

[felixrosberg]

Hi,

Well I notice that the faces ia not properly cropped as an input into ArcFace and the perceptual ArcFace. Did you use my alignment code to process the data? And did you add cropping into the training code before inputing tnto ArcFace?

Also what batch size do you use?

[shiminxuan003]

Using your alignment code, there are many situations where there are only partial faces，as follow.So we used the celeba dataset

The batch size we use is 1, is it the problem? We will try a larger batchsize

[felixrosberg]

It is really important that the face is correctly aligned. It could be that the detector struggles if you use too high resolution images. Quick fix would be to resize the image before inputing into the detector. That should be all as the alignment code scales the detection based of the original width and height.

im = Image.open(im_folder + fld + '/' + img)
im = np.asarray(im)

im_h, im_w, _ = im.shape

if im_h > opt.min_size and im_w > opt.min_size:

    ann = face_detector(np.expand_dims(im <- RESIZE THIS, axis=0)).numpy()[0]
    lm_align = np.array([[ann[4] * im_w, ann[5] * im_h],
                         [ann[6] * im_w, ann[7] * im_h],
                         [ann[8] * im_w, ann[9] * im_h],
                         [ann[10] * im_w, ann[11] * im_h],
                         [ann[12] * im_w, ann[13] * im_h]],
                        dtype=np.float32)

It does look like it did not find the face properly, as it would make sure that the eyes are all horizontal. The cropping could be adjusted by the shrink_factor (should be 1.0 generally, less than that require a central crop of 0.75 (keeping 75% of the central image, I mention this in the README as well) of the image before each call to ArcFace and inside the IFSR loss (which also uses ArcFace).

Now batch_size 1 is very problematic, due to how target and source is sampled. The code makes sure that there is at least ONE pair that is the same, meaning you are only passing target source pairs that is the same image. This would just teach the model to reconstruct the input. I would try at least batch size 4. The reason for this, is to provide a consistant reconstruction loss signal and avoid dividing by 0. The default chance that source and target is the same is 20%.

[ruanjiyang]

can i use dlib to align the face for both training dataset and the face for inference? thanks.

[felixrosberg]

can i use dlib to align the face for both training dataset and the face for inference? thanks.

I assume so, but you need to make sure that the target eye, nose and mouth positions match that of ArcFace is expecting. Which probably means that you have write some code yourself as I am not sure if dlib aligns in the same way. You can see prepared locations in same file as the norm_align function.

But I recommend you use the alignment code here as you only need to download the face detector provided on my huggingface.

[ruanjiyang]

can i use dlib to align the face for both training dataset and the face for inference? thanks.

I assume so, but you need to make sure that the target eye, nose and mouth positions match that of ArcFace is expecting. Which probably means that you have write some code yourself as I am not sure if dlib aligns in the same way. You can see prepared locations in same file as the norm_align function.

But I recommend you use the alignment code here as you only need to download the face detector provided on my huggingface.

thanks for your comments. let me try dlib and report to you ASAP.

[zengjingming]

I also run train.py to train a model. and the face did not change either.My input dataset are cropped and aligned. not sure what's wrong.