How to set PatchCore config to work with a bigger High-Res Dataset?

[daMichaelB]

Hello everyone,

First of all thanks for the awesome work and sharing this SOTA algorithms !

Problem Description

I am currently struggling to train PatchCore on a big dataset with High-Res images.

Just as a rough number we have 100.000 images and they are 4000 x 4000 px.

We used the default config https://github.com/openvinotoolkit/anomalib/blob/development/anomalib/models/patchcore/config.yaml and tested with different settings for

• backbone
• train_batch_size
• image_size
• precision

The training always runs Out Of Memory. The only thing we could train it was to heavily reduce the amount of images in the train/good folder and run on a small image_size.

Question

Do you have any recommendations on how to configure the PatchCore training in order to work with high resolution and many training samples ?

@samet-akcay i saw you are one of the PatchCore contributors and wanted to add you here. thanks!

[samet-akcay]

Hi @daMichaelB, thanks for reporting this. I think there are multiple issues here.

As mentioned in #110, we could utilize the GPU only for the model itself to save some memory and use CPU for other components.
For instance, for metric computation, we have AUC and adaptive F1, both of which requires significant memory to compute. We can double check if this is always handled by the GPU. Also we could add a flag to choose the metrics rather than computing and dumping all at once.

Another issue stems from an algorithm module itself, where a memory bank stores the locally-aware-patch-features for the training set. Larger dataset would store larger patch-features, which would ultimately cause OOM. We could try to utilize CPU here; however, as mentioned here in #113 , the code is not optimized for CPU yet.

[daMichaelB]

Hey @samet-akcay thank you for the clarification and the details! Do you have a recommendation which model from anomalib would be the best fit for such a use-case? Thank you!

[samet-akcay]

You could maybe give STFPM a try. Meanwhile, we'll be trying to optimize the code for larger datasets.