strange behavior in model training

[heba9004]

Hello, I need your help and experience, I would like to ask about something strange occurred when I tried a variant of Unet on your same framework, as the epochs increased the performance in terms of Jaccard index become worse (not improving as it supposed). please see below some results:

epoch 3
Jaccard for Training_phase_1_001 = 0.278841
Jaccard for Training_phase_1_002 = 0.060954
epoch 2
Jaccard for Training_phase_1_001 = 0.286575
Jaccard for Training_phase_1_002 = 0.084818
epoch 1
Jaccard for Training_phase_1_001 = 0.350196
Jaccard for Training_phase_1_002 = 0.170044

Is my learning rate low? or does I miss something somewhere?

one more thing, is it normal that training one epoch take 3.5 hours to complete using GPU?

Thank you for your help and code as well.