Iteration speed slows down exponentially

[orionflame]

This is for bugs only

Did you already ask in the discord?

Yes

You verified that this is a bug and not a feature request or question by asking in the discord?

No because no one seems to know why it happens.

Describe the bug

I started running the model, at first it's fast like 3-5s/it, then after 4 it, it's 1200s, then next iteration it's 2500s, etc. It becomes insanely slow. I don't know why.

It's using all of the 24GB VRAM of my RTX 6000 and 100% usage most of the time. Not sure what else I can do.

Now I am trying low VRAM flag.