Upgrading pre trained model to version 1

[JRMeyer]

>>> dabinat
[April 13, 2019, 5:23am]

Here's what the readme says about converting a model:

> ### Model compatibility
>
> DeepSpeech models are versioned to keep you from trying to use an
> incompatible graph with a newer client after a breaking change was
> made to the code. If you get an error saying your model file version
> is too old for the client, you should either upgrade to a newer model
> release, re-export your model from the checkpoint using a newer
> version of the code, or downgrade your client if you need to use the
> old model and can't re-export it.

But what is the exact process of re-exporting from the checkpoint? The
documentation on training seems to assume I am training the checkpoint
with additional data.

It wasn't clear exactly which arguments are needed to re-export the
model - can I do so with only slash --checkpoint_dir and slash --export_dir?

When I try to do so from the 0.4.1 model I get the following error:

> Key layer_1/bias not found in checkpoint

In another thread this issue was solved by checking out from master but
I am already using master.

[This is an archived TTS discussion thread from discourse.mozilla.org/t/upgrading-pre-trained-model-to-version-1]

[JRMeyer]

>>> dabinat
[April 14, 2019, 3:18am]

After further digging into DeepSpeech.py, it seems like this may be
something that's actually not possible to do. Filed an issue here:
mozilla/DeepSpeech#2050

[Archived Post]

[JRMeyer]

>>> reuben
[April 15, 2019, 12:14am]

The process is on a case-by-case basis, because it depends on how
backwards compatibility was broken. In this case, we had two breaking
changes:

1. Variable names were changed
2. Audio feature computation changed

The first one can be fixed by passing a custom mapping to
tf.train.Saver or loading the checkpoint manually using
tf.train.load_checkpoint. The second is more complicated, because it
means the old checkpoint is not very useful with the new code. I guess
you could try to use it to initialize weights in a new training run.

Here's some code that does the conversion if you're interested in
experimenting with the old checkpoint:
https://github.com/reuben/DeepSpeech/commits/upgrade-v0.4.1

[Archived Post]

[JRMeyer]

>>> dabinat
[April 16, 2019, 4:07am]

Thanks Reuben. I eventually managed to get it to update the model format
but I got no useful output from it.

So I gave up at that point and am just using LDC93S1 for development
purposes.

[Archived Post]

[JRMeyer]

>>> eggonlea
[October 24, 2019, 2:16pm]

Hi Reuben,

I can export the model from 0.4.1 checkpoint with alpha4, but fail with
alpha5.

Can I assume the model is still compatible until alpha4?

Or, has the model become incompatible even before alpha4 (but the code
to check model version hasn't checked in so it just doesn't complain)?

Thanks!

Add version info to exported graphs

reuben
committed 16 days ago

github.com/mozilla/DeepSpeech

#### Add version info to exported graphs

date='2019-04-02' time='13:29:57' timezone='UTC'}

[[+231.added}
-109.removed}

[Archived Post]

[JRMeyer]

>>> reuben
[April 19, 2019, 12:12am]

This is the last commit that should work with the v0.4.1 checkpoint
without any changes:
mozilla/DeepSpeech@bd7358d

[Archived Post]