NaN issue during training

[bhaswara]

Hi, I'd like to ask about the training issue I encountered. So, I just finished my training on regular nnunetv2 and didn't change any parameters from nnunetv2 where it also has gradient clipping. However, when I checked the training progress, I saw several nan in my training, but the training still can update the dice score and recover again as picture below. The final validation mean dice is also good at the end of the training. Do you know if this is normal?