Failed to generate custom object using load_lora_weights()

[Jasonli1873]

I am trying to load my clothing lora model into sd1.5 inpaint model and do virtual try-on with SAM mask. The trigger word is 's0n_j3rs3y'. There is no difference when I generate with or without lora model loaded. I am not sure what is the problem for not available to generate my trained custom object.
Here are some of the code in my project:

model_dir = "runwayml/stable-diffusion-inpainting"
lora_model_path = "/content/gdrive/MyDrive/sd/stable-diffusion-webui/models/Lora/son4.safetensors"
pipe = StableDiffusionInpaintPipeline.from_pretrained(model_dir,
                                                      scheduler=scheduler,
                                                      revision="fp16",
                                                      torch_dtype=torch.float16)
pipe.load_lora_weights("/content/gdrive/MyDrive/sd/stable-diffusion-webui/models/Lora", weight_name="son5.safetensors")
inpainting_prompts = ["A s0n_j3rs3y shirt"]
lora_scale= 0.8
for i in range(num_images_per_prompt):
    image = pipe(prompt=inpainting_prompts[i], negative_prompt=negative_prompt, guidance_scale=3, num_inference_steps=34, generator=generator,
                 image=source_image, mask_image=stable_diffusion_mask, cross_attention_kwargs={"scale": lora_scale}).images[0]
    encoded_images.append(image)

The following link is my google colab file: https://colab.research.google.com/drive/1ZQ0P9AAlYiDb7UA2BvTIwuu5bNUGgO7I?usp=sharing