[QUESTION] Is it expected to do grad norm on dense-optimizer and moe-optimizer respectively?

[ezioliao]

If we enable expert parallelism, there will be two optimizers for dense parameters and expert parameters. When we call optimizer.step() the two optimizers perform grad-norm for their own parameters.

But if we do not enable expert parallelism, all model parameter's grad will be normed as entirely.

So my question is that the behavior of grad norm is different mathematically whether expert parallelism is turned on. Is it expetced?

[github-actions[bot]]

Marking as stale. No activity in 60 days.

[deepakn94]

I believe this behavior is fixed now.

[github-actions[bot]]

Marking as stale. No activity in 60 days.