Deepspeech forgetting previous trained data when trained on newer data

[JRMeyer]

>>> Sushantmkarande
[May 30, 2019, 5:24am]

I am constantly training new data on deepspeech. but when i train on new
data the model is not giving good accuracy for data that is previously
trained. slash
what may be reason behind this, is it because model is giving high
weights for last trained data. slash
I want to always train on new data and still want to get good prediction
on data that I first trained.

[This is an archived TTS discussion thread from discourse.mozilla.org/t/deepspeech-forgetting-previous-trained-data-when-trained-on-newer-data]

[JRMeyer]

>>> yv001
[May 30, 2019, 9:17am]

That's most likely a case of catastrophic
forgetting.
The new data influences the network too much.

Have you tried to lower the learning rate?

[Archived Post]

[JRMeyer]

>>> Sushantmkarande
[May 30, 2019, 1:49pm]

My learning rate is 0.0001 slash
is this some thing wrong with it.

[Archived Post]

[JRMeyer]

>>> lissyx
[May 30, 2019, 1:51pm]



> My learning rate is 0.0001

Try lowering it more, I was hitting something similar, going down to
1e-6 seems to behave much better.

[Archived Post]

[JRMeyer]

>>> Sushantmkarande
[May 30, 2019, 1:56pm]

thanks... slash
i will try that.

[Archived Post]

[JRMeyer]

>>> Sushantmkarande
[June 13, 2019, 5:02am]

,
slash
Hi guys... slash
well I tried 1e-6 as learning rate with around 600 sample with 8 epochs
and 1 batch size but model hardly learned anything...

[Archived Post]

[JRMeyer]

>>> yv001
[June 13, 2019, 6:59am]

You can try to go to 1e-5 with the learning rate for the same amount of
epochs.

If you see forgetting of the original data, you can take a sample of the
original data ( like 10%, depending on size of the original dataset) and
mix it with the new data.

[Archived Post]