The update stops and the loss does not drop

[AishuaiYao]

hello，Today I trained ade20k, but the loss stops at about 2 and I can’t go down

[Landerku]

I have the same problem, the loss seems didn't change

[hellochick]

Hi @AishuaiYao @AishuaiYao ,

If you want to re-train on ADE20k, you need to load the pre-trained weights of the backbone from PSPNet50.
For semantic segmentation tasks, it is really hard to train from scratch with any pre-trained weights.

Hope this helps.

[Landerku]

您好， 您说的非常正确，我百般尝试之下发现是由于我的学习率太高以及训练过程不够长导致的，因为缺乏相关经验，我没有想到居然需要这么久的时间。 非常感谢！ 祝您一切顺利， 刘 HsuanKung Yang <[email protected]> 于 2020年10月24日周六 下午9:43写道：

…

Hi @AishuaiYao <https://github.com/AishuaiYao> @AishuaiYao <https://github.com/AishuaiYao> , If you want to re-train on ADE20k, you need to load the pre-trained weights of the backbone from PSPNet50. For semantic segmentation tasks, it is really hard to train from scratch with any pre-trained weights. Hope this helps. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#127 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMB6VMZVS2NSE7WLUVIWYATSMLDV7ANCNFSM4NHXONUQ> .

[Landerku]

What you said is very correct. After all my attempts, I found that my learning rate was too high and the training process was not long enough. Because of my lack of relevant experience, I didn't expect that it would take so long.
And another important thing is that mu CUDA hasn't initialized because of the lack of mu GPU memory, which causes gradients can't be updated.

Hi @AishuaiYao @AishuaiYao ,

If you want to re-train on ADE20k, you need to load the pre-trained weights of the backbone from PSPNet50.
For semantic segmentation tasks, it is really hard to train from scratch with any pre-trained weights.

Hope this helps.