Whether Deepspeed-Domino is compatible with other parallel strategy?

[Andy666G]

About this work: https://arxiv.org/pdf/2409.15241

I am curious if the Domino implementation is compatible with sequence-parallel or ring-attention(context parallel),
Domino's input and weight split strategy may not be applied in sequence-parallel?

Thanks!
@GuanhuaWang

[GuanhuaWang]

Hi @Andy666G

Thanks for the interesting question. Seq-parallel is on our roadmap, and I think it is possible that domino is compatible with seq-parallel or context-parallel. Feel free to shoot me an email at [email protected], so that we can set up a meeting and discuss on this line. Thanks!