Diverge with augmented data

[a00achild1]

I'm trying to train a model with augmented LibriSpeech train-other-500 dataset.
The speech was augmented by adding noise and changing the tempo.
But the loss couldn't converge.
I have tried letting the learning rate small(1e-5), adding more GRU units, or adding more nodes, but none of these works.
Is anyone has a solution or suggestions?