How does cosine lr scheduler work when the prodigy optimizer is used ?

[sipie800]

The prodigy (or Dadapt) calculate lr itself. I know the lr rate argument is actually the ratio on that calculated lr. Does it also work that way if I use cosine_with_restarts? Will the ratio change as the way when a AdamW is used?

[rockerBOO]

It basically changes the LR directly, which is a component of the other adaptive calculations.

So it takes 1.0 to 0.0. You can use tensorboard or wandb to show the LR on charts. In prodigy the LR works like a multiplier.

Cosine with restarts works as you expect.