OOM -- Where is RingAttention applied?

[orrzohar]

Hi,
I am trying to train LongVA for an alignment experiment, however, I am getting OOM. I suspect that the ring attention is not being utilized.
Where are you applying the ring attention?

Best,
Orr

[kcz358]

I think it is being used when fine tuning the text model. It is not being used when we finetuning LongVA