Need help on training LoRA model

[CaledoniaProject]

I'm trying to train LoRA with human faces then create photo with existing txt2img models.

Environment

• AWS g4dn.xlarge instance (T4 GPU, 16GB vRAM)
• kohya_ss master branch
• downloaded 24 images online, cropped and keep only faces

Training data and results models

• Download crop-test-done.tar.gz from https://easyupload.io/13vbh8

Training steps

I first use BLIP to caption the images, and use img, as prefix / trigger word.

And confirm the caption is succeed:

root@ip-xxxx:/data/crop-test/img/50_img woman# strings *txt
img, a woman with long hair and a black dress
img, a woman with long hair and earrings posing for a picture
img, a woman with long hair and earrings posing for a picture
img, a woman with a big smile on her face
img, a close up of a woman with red lipstick
img, a woman with long hair and a black dress
img, a woman with long hair and a black dress
img, a woman with long hair and a red lipstick
img, a woman with long blonde hair and red lipstick
img, a woman with long hair and a necklace
...

Then I started the training process

1. Training -> Models -> Pretrained model name or path，I picked stabilityai/stable-diffusion-xl-base-1.0
2. Folder -> Output directory for trained model，I picked /data/crop-test/model
3. Folder -> Image folder (containing training images subfolders)，I picked /data/crop-test/img
4. Parameters -> Basic, check No half VAE

The training process is successful:

(venv) root@ip-172-31-30-185:/home/kohya_ss/package# ./gui.sh --share
Warning: LD_LIBRARY_PATH environment variable is not set.
Certain functionalities may not work correctly.
Please ensure that the required libraries are properly configured.

If you use WSL2 you may want to: export LD_LIBRARY_PATH=/usr/lib/wsl/lib/

12:05:26-185279 INFO     Kohya_ss GUI version: v24.1.7
12:05:26-258257 INFO     Submodule initialized and updated.
12:05:26-259637 INFO     nVidia toolkit detected
12:05:28-843697 INFO     Torch 2.1.2+cu118
12:05:28-988088 INFO     Torch backend: nVidia CUDA 11.8 cuDNN 8700
12:05:29-018737 INFO     Torch detected GPU: Tesla T4 VRAM 14918 Arch (7, 5) Cores 40
12:05:29-025589 INFO     Python version is 3.10.12 (main, Jan 17 2025, 14:35:34) [GCC 11.4.0]
12:05:29-027176 INFO     Verifying modules installation status from /home/kohya_ss/package/requirements_linux.txt...
12:05:29-031445 INFO     Verifying modules installation status from requirements.txt...
2025-01-25 12:05:33.860689: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2025-01-25 12:05:33.911873: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-01-25 12:05:33.911921: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-01-25 12:05:33.913381: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-01-25 12:05:33.927310: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2025-01-25 12:05:33.929022: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-01-25 12:05:35.732607: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
12:05:40-607170 INFO     headless: False
12:05:40-691975 INFO     Using shell=True when running external commands...
/home/kohya_ss/package/venv/lib/python3.10/site-packages/gradio/analytics.py:106: UserWarning: IMPORTANT: You are using gradio version 4.43.0, however version 4.44.1 is available, please upgrade.
--------
  warnings.warn(
Running on local URL:  http://127.0.0.1:7860
Running on public URL: https://xxxx.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)
12:05:57-434344 INFO     SDXL model selected. Setting sdxl parameters
12:06:02-306668 INFO     SDXL model selected. Setting sdxl parameters
12:06:15-520353 INFO     Start training LoRA Standard ...
12:06:15-521888 INFO     Validating lr scheduler arguments...
12:06:15-523131 INFO     Validating optimizer arguments...
12:06:15-524087 INFO     Validating /data/crop-test/log existence and writability... SUCCESS
12:06:15-525141 INFO     Validating /data/crop-test/model existence and writability... SUCCESS
12:06:15-526304 INFO     Validating stabilityai/stable-diffusion-xl-base-1.0 existence... SUCCESS
12:06:15-527309 INFO     Validating /data/crop-test/img existence... SUCCESS
12:06:15-528331 INFO     Folder 50_img woman: 50 repeats found
12:06:15-529470 INFO     Folder 50_img woman: 24 images found
12:06:15-530403 INFO     Folder 50_img woman: 24 * 50 = 1200 steps
12:06:15-531361 INFO     Regulatization factor: 1
12:06:15-532252 INFO     Total steps: 1200
12:06:15-533133 INFO     Train batch size: 1
12:06:15-534001 INFO     Gradient accumulation steps: 1
12:06:15-534896 INFO     Epoch: 1
12:06:15-535786 INFO     Max train steps: 1600
12:06:15-536643 INFO     stop_text_encoder_training = 0
12:06:15-537544 INFO     lr_warmup_steps = 160
12:06:15-539077 INFO     Saving training config to /data/crop-test/model/last_20250125-120615.json...
12:06:15-540462 INFO     Executing command: /home/kohya_ss/package/venv/bin/accelerate launch --dynamo_backend no --dynamo_mode default --mixed_precision fp16 --num_processes 1
                         --num_machines 1 --num_cpu_threads_per_process 2 /home/kohya_ss/package/sd-scripts/sdxl_train_network.py --config_file
                         /data/crop-test/model/config_lora-20250125-120615.toml
12:06:15-544535 INFO     Command executed.
2025-01-25 12:06:24.777838: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2025-01-25 12:06:24.823094: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-01-25 12:06:24.823142: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-01-25 12:06:24.824557: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-01-25 12:06:24.831922: I external/local_tsl/tsl/cuda/cudart_stub.cc:31] Could not find cuda drivers on your machine, GPU will not be used.
2025-01-25 12:06:24.832176: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-01-25 12:06:25.870362: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2025-01-25 12:06:27 INFO     Loading settings from /data/crop-test/model/config_lora-20250125-120615.toml...                                                        train_util.py:4174
                    INFO     /data/crop-test/model/config_lora-20250125-120615                                                                                      train_util.py:4193
2025-01-25 12:06:27 INFO     prepare tokenizers                                                                                                                 sdxl_train_util.py:138
                    INFO     update token length: 75                                                                                                            sdxl_train_util.py:163
                    INFO     Using DreamBooth method.                                                                                                             train_network.py:172
2025-01-25 12:06:28 INFO     prepare images.                                                                                                                        train_util.py:1815
                    INFO     found directory /data/crop-test/img/50_img woman contains 24 image files                                                               train_util.py:1762
                    INFO     1200 train images with repeating.                                                                                                      train_util.py:1856
                    INFO     0 reg images.                                                                                                                          train_util.py:1859
                    WARNING  no regularization images / 正則化画像が見つかりませんでした                                                                            train_util.py:1864
                    INFO     [Dataset 0]                                                                                                                            config_util.py:572
                               batch_size: 1
                               resolution: (512, 512)
                               enable_bucket: True
                               network_multiplier: 1.0
                               min_bucket_reso: 256
                               max_bucket_reso: 2048
                               bucket_reso_steps: 64
                               bucket_no_upscale: True

                               [Subset 0 of Dataset 0]
                                 image_dir: "/data/crop-test/img/50_img woman"
                                 image_count: 24
                                 num_repeats: 50
                                 shuffle_caption: False
                                 keep_tokens: 0
                                 keep_tokens_separator:
                                 caption_separator: ,
                                 secondary_separator: None
                                 enable_wildcard: False
                                 caption_dropout_rate: 0.0
                                 caption_dropout_every_n_epoches: 0
                                 caption_tag_dropout_rate: 0.0
                                 caption_prefix: None
                                 caption_suffix: None
                                 color_aug: False
                                 flip_aug: False
                                 face_crop_aug_range: None
                                 random_crop: False
                                 token_warmup_min: 1,
                                 token_warmup_step: 0,
                                 alpha_mask: False,
                                 is_reg: False
                                 class_tokens: img woman
                                 caption_extension: .txt


                    INFO     [Dataset 0]                                                                                                                            config_util.py:578
                    INFO     loading image sizes.                                                                                                                    train_util.py:911
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 4340.24it/s]
                    INFO     make buckets                                                                                                                            train_util.py:917
                    WARNING  min_bucket_reso and max_bucket_reso are ignored if bucket_no_upscale is set, because bucket reso is defined by image size automatically train_util.py:934
                             / bucket_no_upscaleが指定された場合は、bucketの解像度は画像サイズから自動計算されるため、min_bucket_resoとmax_bucket_resoは無視されます
                    INFO     number of images (including repeats) / 各bucketの画像枚数（繰り返し回数を含む）                                                         train_util.py:963
                    INFO     bucket 0: resolution (256, 256), count: 150                                                                                             train_util.py:968
                    INFO     bucket 1: resolution (320, 320), count: 50                                                                                              train_util.py:968
                    INFO     bucket 2: resolution (320, 384), count: 50                                                                                              train_util.py:968
                    INFO     bucket 3: resolution (384, 384), count: 100                                                                                             train_util.py:968
                    INFO     bucket 4: resolution (384, 448), count: 50                                                                                              train_util.py:968
                    INFO     bucket 5: resolution (448, 384), count: 50                                                                                              train_util.py:968
                    INFO     bucket 6: resolution (448, 448), count: 300                                                                                             train_util.py:968
                    INFO     bucket 7: resolution (448, 512), count: 200                                                                                             train_util.py:968
                    INFO     bucket 8: resolution (448, 576), count: 50                                                                                              train_util.py:968
                    INFO     bucket 9: resolution (512, 448), count: 150                                                                                             train_util.py:968
                    INFO     bucket 10: resolution (512, 512), count: 50                                                                                             train_util.py:968
                    INFO     mean ar error (without repeats): 0.04031468409762353                                                                                    train_util.py:973
                    WARNING  clip_skip will be unexpected / SDXL学習ではclip_skipは動作しません                                                                 sdxl_train_util.py:352
                    INFO     preparing accelerator                                                                                                                train_network.py:225
accelerator device: cuda
                    INFO     loading model for process 0/1                                                                                                       sdxl_train_util.py:33
                    INFO     load Diffusers pretrained models: stabilityai/stable-diffusion-xl-base-1.0, variant=fp16                                            sdxl_train_util.py:88
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:49<00:00,  8.20s/it]
2025-01-25 12:07:17 INFO     U-Net converted to original U-Net                                                                                                  sdxl_train_util.py:124
2025-01-25 12:07:18 INFO     Enable xformers for U-Net                                                                                                              train_util.py:3040
import network module: networks.lora
2025-01-25 12:07:21 INFO     [Dataset 0]                                                                                                                            train_util.py:2323
                    INFO     caching latents.                                                                                                                       train_util.py:1095
                    INFO     checking cache validity...                                                                                                             train_util.py:1105
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 299593.14it/s]
                    INFO     caching latents...                                                                                                                     train_util.py:1144
  0%|                                                                                                                                                          | 0/24 [00:00<?, ?it/s]/home/kohya_ss/package/venv/lib/python3.10/site-packages/torch/nn/modules/conv.py:456: UserWarning: Applied workaround for CuDNN issue, install nvrtc.so (Triggered internally at ../aten/src/ATen/native/cudnn/Conv_v8.cpp:80.)
  return F.conv2d(input, weight, bias, self.stride,
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:10<00:00,  2.28it/s]
2025-01-25 12:07:42 INFO     create LoRA network. base dim (rank): 8, alpha: 1                                                                                             lora.py:928
                    INFO     neuron dropout: p=None, rank dropout: p=None, module dropout: p=None                                                                          lora.py:929
                    INFO     create LoRA for Text Encoder 1:                                                                                                              lora.py:1020
                    INFO     create LoRA for Text Encoder 2:                                                                                                              lora.py:1020
                    INFO     create LoRA for Text Encoder: 264 modules.                                                                                                   lora.py:1028
2025-01-25 12:07:43 INFO     create LoRA for U-Net: 722 modules.                                                                                                          lora.py:1036
                    INFO     enable LoRA for text encoder: 264 modules                                                                                                    lora.py:1077
                    INFO     enable LoRA for U-Net: 722 modules                                                                                                           lora.py:1082
prepare optimizer, data loader etc.
                    INFO     use 8-bit AdamW optimizer | {}                                                                                                         train_util.py:4327
running training / 学習開始
  num train images * repeats / 学習画像の数×繰り返し回数: 1200
  num reg images / 正則化画像の数: 0
  num batches per epoch / 1epochのバッチ数: 1200
  num epochs / epoch数: 2
  batch size per device / バッチサイズ: 1
  gradient accumulation steps / 勾配を合計するステップ数 = 1
  total optimization steps / 学習ステップ数: 1600
steps:   0%|                                                                                                                                                 | 0/1600 [00:00<?, ?it/s]
epoch 1/2
2025-01-25 12:07:53 INFO     epoch is incremented. current_epoch: 0, epoch: 1                                                                                        train_util.py:703
steps:  75%|████████████████████████████████████████████████████████████████████████████████████████▌                             | 1200/1600 [22:50<07:36,  1.14s/it, avr_loss=0.136]
saving checkpoint: /data/crop-test/model/last-000001.safetensors

epoch 2/2
2025-01-25 12:30:44 INFO     epoch is incremented. current_epoch: 1, epoch: 2                                                                                        train_util.py:703
steps: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1600/1600 [30:20<00:00,  1.14s/it, avr_loss=0.129]
saving checkpoint: /data/crop-test/model/last.safetensors
2025-01-25 12:38:15 INFO     model saved.                                                                                                                        train_network.py:1104
steps: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1600/1600 [30:21<00:00,  1.14s/it, avr_loss=0.129]
12:38:23-657178 INFO     Training has ended.

Generation steps

I first copied /data/crop-test/models/last.safetensors to /data. Then I create a pipeline script to create outputs with trigger word img

#!/usr/bin/env python3.10

import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float32)
pipe.load_lora_weights("last.safetensors", adapter_name="faceTest", weight_name="faceTest")
pipe.set_adapters(['faceTest'], adapter_weights=[0.75])
pipe.to('cuda')
pipe.fuse_lora(lora_scale=0.75)
pipe.enable_attention_slicing()

prompt = "cinematic photo of jennlaw, studio quality, woman standing in a room, doing a public speech, blur the background, smiling, having a positive face, wearing a v-neck black sweater, 35mm photograph, film, bokeh, professional, 4k, highly detailed"
negative_prompt = "drawing, painting, crayon, sketch, graphite, impressionist, noisy, blurry, soft, deformed, ugly nsfw, lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry"

images = pipe(prompt, 
             negative_prompt=negative_prompt, 
             guidance_scale=5,
             num_inference_steps=50,
             width=768,
             height=1024,
             num_images_per_prompt=2).images

for idx, img in enumerate(images): 
    img.save(f"outputs/{idx}.jpg")

After a while the results is generated. But it's nothing like the inputs.

These are the the inputs,

These are the outputs,

I have completely no idea what's wrong, can't find anything useful online, chatgpt didn't help either.

Does anyone know what's wrong?