DKD: the nckd loss becomes infinity after one iteration of training.

[OuYangg]

First of all, thank the author for open source. DKD is a very good job!

I trained a binary classification model and wanted to use DKD for distillation, but after one iteration of training, the output of the student model became very strange, causing the loss to become Inf.

Some details:
iteration 1:


iteration 2:

[OuYangg]

After I adjusted the learning rate down, the loss is no longer nan, but the nckd loss is still 0.

[Zzzzz1]

NCKD loss can not be applied on binary classification tasks, since there is no "non-target class". The non-target logits will be always 1.0, may introduce some problems when calculating KLD.