Training an image with a large number of masks will run out of RAM.

[unagi80]

I am using translation software, so apologies if the wording is strange.

I have a custom dataset to train segmentation.

The size of my image is 240*320 and there are about 600 objects/image in it, all annotated.

There is only one class.

Even with 5 such images in train and 2 in val, when I start learning, the RAM usage gradually increases and eventually exceeds my PC's RAM capacity (256GB).

However, this phenomenon peaks when “Plotting labels to runs~” is displayed, and is quickly resolved when the actual training loop starts.

I really want to train with more images, but this peak in RAM usage causes the program to crash.

Is there any way to avoid this phenomenon and train a large number of images?

[UltralyticsAssistant]

👋 Hello @unagi80, thank you for reaching out to Ultralytics 🚀! It sounds like you're encountering some issues with RAM usage during training. Let's try to tackle this!

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us investigate. Details about your dataset and any specific configurations would be greatly appreciated.

For large image datasets or segmentation tasks, check out our model training tips to optimize performance. This may help you handle large numbers of masks more efficiently.

In the meantime, ensure your environment is up-to-date. You can upgrade to the latest ultralytics package:

pip install -U ultralytics

Explore different environments to see if they offer improved performance for your training:

• Notebooks with free GPU:

If the issue persists, rest assured an Ultralytics engineer will assist you soon. Meanwhile, feel free to engage with the community on Discord or Discourse to gather more insights.

Looking forward to your updates! 😊

[HichTala]

When you train a model, images are loaded into RAM in batches. This means that not all images are loaded into RAM at the same time. The factor influencing the memory used is actually the batch size, not the number of images. You can reduce batch size and train a model on a larger number of images.

You can change the batch size by modifying the batch parameter.
For example:
-using python

results = model.train(rdata="coco8.yaml", batch=4)

-using CLI

yolo detect train data=coco8.yaml model=yolov8n.pt batch=4

Does that answer your question ?

[unagi80]

HichTala Thanks for the reply.

As an addendum,
The RAM I am referring to is the RAM handled by the CPU,
VRAM is different from GPU's VRAM.

I have tried batch=1 and batch=-1 but no improvement.

Once the train loop starts, processing is done well on the GPU.
VRAM management by batch processing is also good.

What I think is the problem is the phenomenon where a large amount of RAM is consumed when "Plotting labels to labels.jpg..." is displayed before the training loop starts.

After the train loop starts, the amount of RAM used decreases dramatically.

In other words, the train loop itself does not use much RAM, so I am guessing that this phenomenon is the result of some extra processing.

What we have tried is
Reducing the label data from about 600 labels per image to 10 labels per image worked fine.

Expanding the swap area (about 500 GB) worked, although it slowed down processing very much. (Once in the train loop, the swap is released and processing is very fast)

My data set has 100 images and I want to train using all of them.
It is thought that expanding the swap area will consume more RAM than it can keep up with.

[glenn-jocher]

@HichTala to address the RAM issue, try reducing the batch size in your training configuration. This should help manage memory usage more effectively. Let me know if you need further assistance!

[unagi80]

As an addendum,
The RAM I am referring to is the RAM handled by the CPU,
VRAM is different from GPU's VRAM.

I have tried batch=1 and batch=-1 but no improvement.

Once the train loop starts, processing is done well on the GPU.
VRAM management by batch processing is also good.

What I think is the problem is the phenomenon where a large amount of RAM is consumed when "Plotting labels to labels.jpg..." is displayed before the training loop starts.

After the train loop starts, the amount of RAM used decreases dramatically.

In other words, the train loop itself does not use much RAM, so I am guessing that this phenomenon is the result of some extra processing.

What we have tried is
Reducing the label data from about 600 labels per image to 10 labels per image worked fine.

Expanding the swap area (about 500 GB) worked, although it slowed down processing very much. (Once in the train loop, the swap is released and processing is very fast)

My data set has 100 images and I want to train using all of them.
It is thought that expanding the swap area will consume more RAM than it can keep up with.

[glenn-jocher]

@unagi80 to manage high RAM usage during label plotting, consider disabling label plotting by setting plots=False in your training command. This should help reduce the initial RAM spike.

[unagi80]

@glenn-jocher
Thanks for the reply.

I added plots=False to the training command and that solved the problem.

I appreciate it very much.

[glenn-jocher]

You're welcome! Glad to hear it worked for you. If you have any more questions, feel free to ask.