Question about one deepspeed-config: contiguous_gradients

[williechai]

For the code:

CogVideo/sat/configs/sft.yaml

Line 39 in 2fdc59c

contiguous_gradients: false

, contiguous_gradients is deepspeed memory optimization, which is default True. I am very curious why is it set False in CogvideoX sft procedure? And we accidentally discovered that when it is set to True, the loss will become abnormally large when training on more than 128 GPUs; so what is your motivation for disabling it? Could you please share it?

Reference:

(according to https://www.deepspeed.ai/docs/config-json/#bfloat16-training-options)

[yzy-thu]

Sorry, we didn't pay attention to this setting before. We've always used it this way. Hope someone can provide an explanation.