Help, Dreambooth training on custom runpod serverless endpoint stops, while works perfectly on Secure cloud instance

[nayan-dhabarde]

Issue:
I tought myself in 15 days how to use diffusers repo to perform training using dreambooth for a person and day by day got better results. I was using Runpod.io for the GPU instance as I couldn't afford a GPU. My new script works perfectly fine on secure cloud from Runpod. When I tried to create an API the training stops around 30% always

What I have tried:

1. I checked with Runpod team but they were not able to help
2. Checked there Dreambooth endpoint but looks like it is not using regularization images (I have created my own set of images for realistic vision training)

How to reproduce?
Here is my launching script:

command = [f"accelerate launch --config_file \"config.yaml\" train_dreambooth.py \
        --pretrained_model_name_or_path=\"SG161222/Realistic_Vision_V4.0\"  \
        --instance_data_dir=\"smart_cropped_images\" \
        --class_data_dir=\"man_dataset\" \
        --output_dir=\"trained_model\" \
        --train_text_encoder \
        --instance_prompt=\"a photo of nayansks man\" \
        --class_prompt=\"a photo of man\" \
        --with_prior_preservation --prior_loss_weight=1.0 \
        --resolution=512 \
        --train_batch_size=1 \
        --gradient_accumulation_steps=1 \
        --checkpointing_steps=1000 \
        --learning_rate=1e-6 \
        --lr_scheduler=\"constant\" \
        --lr_warmup_steps=0 \
        --max_train_steps=1000 \
        --num_class_images=1000 "]

The worker stops and disappears, no further logs are shown, there is an inprogress response, then timed_out response
logs.txt