Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RaspberryPi 4B source install TensorFlow version issue #193

Open
SandroMiccoli opened this issue Dec 3, 2020 · 20 comments
Open

RaspberryPi 4B source install TensorFlow version issue #193

SandroMiccoli opened this issue Dec 3, 2020 · 20 comments
Labels

Comments

@SandroMiccoli
Copy link

Describe the bug
I'm trying to train my own wake word on a RaspberryPi 4B. After installing via source, I used precise-collect to record several samples, everything worked ok, but when I try to precise-listen or to precise-train I get these WARNING messages:

With precise-listen:

$ precise-listen ../wakewords/computer/computer-en.pb
WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/__init__.py:98: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/__init__.py:98: The name tf.AttrValue is deprecated. Please use tf.compat.v1.AttrValue instead.

WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/__init__.py:98: The name tf.COMPILER_VERSION is deprecated. Please use tf.version.COMPILER_VERSION instead.

WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/__init__.py:98: The name tf.CXX11_ABI_FLAG is deprecated. Please use tf.sysconfig.CXX11_ABI_FLAG instead.

WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/__init__.py:98: The name tf.ConditionalAccumulator is deprecated. Please use tf.compat.v1.ConditionalAccumulator instead.
...

And precise-train:

$ precise-train -e 60 myword.net myword/
Using TensorFlow backend.
WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/__init__.py:98: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/__init__.py:98: The name tf.AttrValue is deprecated. Please use tf.compat.v1.AttrValue instead.

WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/__init__.py:98: The name tf.COMPILER_VERSION is deprecated. Please use tf.version.COMPILER_VERSION instead.

WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/__init__.py:98: The name tf.CXX11_ABI_FLAG is deprecated. Please use tf.sysconfig.CXX11_ABI_FLAG instead.

WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/__init__.py:98: The name tf.ConditionalAccumulator is deprecated. Please use tf.compat.v1.ConditionalAccumulator instead.

WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:3138: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/keras/optimizers.py:757: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
.....

Those two warnings appear at the beggninng but the rest of outputs seem ok. But when I try to precise-listen my new .net file, an Error appears:

 $ precise-listen myword.net
WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/__init__.py:98: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/__init__.py:98: The name tf.AttrValue is deprecated. Please use tf.compat.v1.AttrValue instead.

WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/__init__.py:98: The name tf.COMPILER_VERSION is deprecated. Please use tf.version.COMPILER_VERSION instead.

WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/__init__.py:98: The name tf.CXX11_ABI_FLAG is deprecated. Please use tf.sysconfig.CXX11_ABI_FLAG instead.

WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/__init__.py:98: The name tf.ConditionalAccumulator is deprecated. Please use tf.compat.v1.ConditionalAccumulator instead.

Using TensorFlow backend.
Traceback (most recent call last):
  File "/home/pi/mycroft-precise/.venv/bin/precise-listen", line 33, in <module>
    sys.exit(load_entry_point('mycroft-precise', 'console_scripts', 'precise-listen')())
  File "/home/pi/mycroft-precise/precise/scripts/base_script.py", line 43, in run_main
    script = cls(args)
  File "/home/pi/mycroft-precise/precise/scripts/listen.py", line 58, in __init__
    self.listener = Listener(args.model, args.chunk_size)
  File "/home/pi/mycroft-precise/precise/network_runner.py", line 107, in __init__
    self.runner = runner_cls(model_name)
  File "/home/pi/mycroft-precise/precise/network_runner.py", line 85, in __init__
    self.model = load_precise_model(model_name)
  File "/home/pi/mycroft-precise/precise/model.py", line 54, in load_precise_model
    return load_keras().models.load_model(model_name)
  File "/home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/keras/models.py", line 242, in load_model
    model_config = json.loads(model_config.decode('utf-8'))
AttributeError: 'str' object has no attribute 'decode'

I've tried download the dev branch but the same error happens.

To Reproduce
Steps to reproduce the behavior:

  1. Clone the repository
  2. Run the ./setup.sh script
  3. Activate python environment with source .venv/bin/activate
  4. Try to precise-listen any model (I used some community data and also a test of my own after collecting several recordings and generating a .net with precise-train)

Expected behavior
I expected to run my created model but got an error there.

Environment

  • Device type: Raspberry Pi 4B
  • OS: Raspbian GNU/Linux 10 (buster)
  • Linux voice 5.4.72-v7l+ #1356 SMP Thu Oct 22 13:57:51 BST 2020 armv7l GNU/Linux

Additional context
I'm using a clean buster installation. I'm using voice2json and everything works fine there.

@MatthewScholefield
Copy link
Collaborator

Thanks for the thorough report! Hmm, yeah this definitely looks like some problem related to the keras/tensorflow version.

You might be able to get it working by using code off of PR #141 since that uses Tensorflow 2 which should resolve a lot of version related issues.

Let me know if this helps.

@SandroMiccoli
Copy link
Author

Thanks for the quick reply!

Ok, got the #141 and tried running the ./setup.sh, this was the output:

$ ./setup.sh
Reading package lists... Done
Building dependency tree
Reading state information... Done
curl is already the newest version (7.64.0-4+deb10u1).
cython is already the newest version (0.29.2-2).
libatlas-base-dev is already the newest version (3.10.3-8+rpi1).
libhdf5-dev is already the newest version (1.10.4+repack-10).
libopenblas-dev is already the newest version (0.3.5+ds-3+rpi1).
libpulse-dev is already the newest version (12.2-4+deb10u1).
portaudio19-dev is already the newest version (19.6.0-1).
python3-h5py is already the newest version (2.8.0-3).
python3-scipy is already the newest version (1.1.0-7).
swig is already the newest version (3.0.12-2).
python3-pip is already the newest version (18.1-5+rpt1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Obtaining file:///home/pi/mycroft-precise/runner
Requirement already satisfied: pyaudio in ./.venv/lib/python3.7/site-packages (from precise-runner==0.3.1) (0.2.11)
Installing collected packages: precise-runner
  Attempting uninstall: precise-runner
    Found existing installation: precise-runner 0.3.1
    Uninstalling precise-runner-0.3.1:
      Successfully uninstalled precise-runner-0.3.1
  Running setup.py develop for precise-runner
Successfully installed precise-runner
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Obtaining file:///home/pi/mycroft-precise
ERROR: Could not find a version that satisfies the requirement tensorflow-gpu==2.2.0 (from mycroft-precise)
ERROR: No matching distribution found for tensorflow-gpu==2.2.0

Should I uninstall/remove something before the new install?

@MatthewScholefield
Copy link
Collaborator

MatthewScholefield commented Dec 3, 2020

Try unpinning tensorflow-gpu==2.2.0 in setup.py and rerunning. Otherwise see if you can find a built .whl pip package for the raspberry pi 4B that you can pip install file.whl within the venv.

@SandroMiccoli
Copy link
Author

SandroMiccoli commented Dec 3, 2020

Ok, so tried unpinning from setup.py but didn't work.

I found this guide on installing tensorflow-2.2.0 on RPi 4 and the installation went ok. Had to install it directly in the venv for it to work with precise though.

Tried running precise-listen and didn't give any warnings and the outputs look promissing.

I'm running precise-train on my recordings now and apparently, it's going well... Although a warning message appeared here also:

$ precise-train -e 60 myword.net myword/
Data: <TrainData wake_words=20 not_wake_words=0 test_wake_words=9 test_not_wake_words=0>
Loading wake-word...
Loading not-wake-word...
Loading wake-word...
Loading not-wake-word...
Inputs shape: (20, 29, 13)
Outputs shape: (20, 1)
Test inputs shape: (9, 29, 13)
Test outputs shape: (9, 1)
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
net (GRU)                    (None, 20)                2100
_________________________________________________________________
dense (Dense)                (None, 1)                 21
=================================================================
Total params: 2,121
Trainable params: 2,121
Non-trainable params: 0
_________________________________________________________________
Epoch 1/60
1/1 [==============================] - ETA: 0s - loss: 0.1328 - accuracy: 0.65002020-12-03 15:57:52.315038: W tensorflow/python/util/util.cc:329] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.


WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:1817: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.


1/1 [==============================] - 13s 13s/step - loss: 0.1328 - accuracy: 0.6500
Epoch 2/60
1/1 [==============================] - 13s 13s/step - loss: 0.0818 - accuracy: 0.7500
Epoch 3/60
1/1 [==============================] - 13s 13s/step - loss: 0.0745 - accuracy: 0.7500
....

After the training finishes, I'll test it out and post here how it went.

@SandroMiccoli
Copy link
Author

Well, still no success... I'm following these steps to train my own word.

Everything went well until precise-training-incremental:

$ precise-train-incremental myword.net myword/ -r data/random/
Loading wake-word...
Loading not-wake-word...
Loading wake-word...
Loading not-wake-word...
Starting file data/random/pencil sharpening.wav...
31.60919540229885%%
Saved to: myword/not-wake-word/generated/pencil sharpening-55.wav
32.758620689655174%
Saved to: myword/not-wake-word/generated/pencil sharpening-57.wav
37.35632183908046%%
Saved to: myword/not-wake-word/generated/pencil sharpening-65.wav
39.6551724137931%%
Saved to: myword/not-wake-word/generated/pencil sharpening-69.wav
48.85057471264368%%
Saved to: myword/not-wake-word/generated/pencil sharpening-85.wav
49.42528735632184%
Saved to: myword/not-wake-word/generated/pencil sharpening-86.wav
67.816091954023%3%%
Saved to: myword/not-wake-word/generated/pencil sharpening-118.wav
70.6896551724138%%
Saved to: myword/not-wake-word/generated/pencil sharpening-123.wav
72.41379310344827%
Saved to: myword/not-wake-word/generated/pencil sharpening-126.wav
72.98850574712644%
Saved to: myword/not-wake-word/generated/pencil sharpening-127.wav
Loading wake-word...
Loading not-wake-word...
Loading wake-word...
Loading not-wake-word...

Epoch 61/61
1/1 [==============================] - ETA: 0s - loss: 1.1070 - accuracy: 0.50002020-12-03 17:15:57.780733: W tensorflow/python/util/util.cc:329] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:1817: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
1/1 [==============================] - 14s 14s/step - loss: 1.1070 - accuracy: 0.5000 - val_loss: 0.1028 - val_accuracy: 0.8889
Traceback (most recent call last):
  File "/home/pi/mycroft-precise/.venv/bin/precise-train-incremental", line 33, in <module>
    sys.exit(load_entry_point('mycroft-precise', 'console_scripts', 'precise-train-incremental')())
  File "/home/pi/mycroft-precise/precise/scripts/base_script.py", line 49, in run_main
    script.run()
  File "/home/pi/mycroft-precise/precise/scripts/train_incremental.py", line 150, in run
    self.train_on_audio(fn)
  File "/home/pi/mycroft-precise/precise/scripts/train_incremental.py", line 137, in train_on_audio
    self.retrain()
  File "/home/pi/mycroft-precise/precise/scripts/train_incremental.py", line 111, in retrain
    rename(self.args.model + '.h5', self.args.model) # Rename with original
IsADirectoryError: [Errno 21] Is a directory: 'myword.net.h5' -> 'myword.net'

I followed the "method 2" in Reducing False Activations. Downloaded all the files and ran the ffmpeg script on all of them.

This is how my folder looks like after running all of these scripts:

$ ls -l
total 72
drwxr-xr-x 2 pi pi  4096 Dec  3 11:56 computer
drwxr-xr-x 3 pi pi  4096 Dec  3 16:56 data
drwxr-xr-x 2 pi pi  4096 Dec  3 11:56 heycomputer
drwxr-xr-x 5 pi pi  4096 Dec  3 12:21 myword
-rw-r--r-- 1 pi pi     2 Dec  3 17:16 myword.epoch
drwxr-xr-x 4 pi pi  4096 Dec  3 17:16 myword.net
-rw-r--r-- 1 pi pi 44072 Dec  3 17:16 myword.net.h5
-rw-r--r-- 1 pi pi   224 Dec  3 17:15 myword.net.params

@MatthewScholefield
Copy link
Collaborator

MatthewScholefield commented Dec 3, 2020

Hmm, I think I actually fixed that bug on my personal tf2 branch. Could you try checking it out with this?:

git remote add matthew https://github.com/MatthewScholefield/mycroft-precise
git fetch matthew
git checkout feature/tf2

@SandroMiccoli
Copy link
Author

SandroMiccoli commented Dec 4, 2020

After changing to your tf2 branch I left the precise-train-incremental running over night (process started at 23h and finished at 5:30). Now I tried to test the data set and this is the result:

$ precise-test myword.net myword/
Loading wake-word...
Loading not-wake-word...
Traceback (most recent call last):
  File "/home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/python/saved_model/loader_impl.py", line 98, in parse_saved_model
    saved_model.ParseFromString(file_content)
  File "/home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/google/protobuf/message.py", line 199, in ParseFromString
    return self.MergeFromString(serialized)
  File "/home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/google/protobuf/internal/python_message.py", line 1145, in MergeFromString
    if self._InternalParse(serialized, 0, length) != length:
  File "/home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/google/protobuf/internal/python_message.py", line 1195, in InternalParse
    raise message_mod.DecodeError('Field number 0 is illegal.')
google.protobuf.message.DecodeError: Field number 0 is illegal.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/mycroft-precise/.venv/bin/precise-test", line 33, in <module>    sys.exit(load_entry_point('mycroft-precise', 'console_scripts', 'precise-test')())
  File "/home/pi/mycroft-precise/precise/scripts/base_script.py", line 49, in run_main
    script.run()
  File "/home/pi/mycroft-precise/precise/scripts/test.py", line 52, in run
    predictions = Listener.find_runner(args.model)(args.model).predict(inputs)  File "/home/pi/mycroft-precise/precise/network_runner.py", line 80, in __init__
    self.model = load_precise_model(model_name)
  File "/home/pi/mycroft-precise/precise/model.py", line 55, in load_precise_model
    return load_model(model_name, custom_objects=globals())
  File "/home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/python/keras/saving/save.py", line 189, in load_model
    loader_impl.parse_saved_model(filepath)
  File "/home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/python/saved_model/loader_impl.py", line 101, in parse_saved_model
    raise IOError("Cannot parse file %s: %s." % (path_to_pb, str(e)))
OSError: Cannot parse file b'myword.net/saved_model.pb': Field number 0 is illegal..

@SandroMiccoli
Copy link
Author

I saved the output of the precise-train-incremental to a log file, here are the last lines:

...
Starting file data/random/snow_globe_turned_around.wav...
100%
Starting file data/random/STE-028.wav...
100%
Starting file data/random/snow_globe_shaken.wav...
100%
Starting file data/random/electric_roof_window_opening.wav...
100%
Starting file data/random/chirrp.wav...
100%
Starting file data/random/picooz_indoor_helicopter_flight_collage.wav...
100%
Starting file data/random/sliding_door_of_veranda_closing.wav...
100%
Starting file data/random/ritschratsch.wav...
100%
Starting file data/random/planebirdsandkitchen.wav...
100%
Starting file data/random/mnium-mnium-mnium.wav...
100%

Not sure if a specific message should end that script...

@kbromer
Copy link

kbromer commented Dec 7, 2020

Wanted to confirm I'm seeing the same issue/output on the same hardware/setup as @SandroMiccoli I have not yet tried upgrading the venv to TF2.2, were you able to get the incremental training to complete?

@SandroMiccoli
Copy link
Author

Nope, I tried 3 times but on all of them I got the same Field number 0 is illegal. message.

@kbromer
Copy link

kbromer commented Dec 9, 2020

Had success here. Trained the set with precise-train-incremental. It did error out during its training run (I believe it was due to having an existing model of the same name in the directory) but I was able to successfully retrain after clearing the existing models from previous runs with precise-train. precise-test, precise-listen succeeded as expected. precise-convert also succeeded.

Of course, the app I'm using (Rhasspy) requires .pb/.pb.param files for the integration. Anybody have suggestions on that?

Error from the precise-train-incremental:

100%
Starting file data/random/STE-032.wav...
0.0%Loading wake-word...
Loading not-wake-word...
Loading wake-word...
Loading not-wake-word...

Epoch 61/61
1/1 [==============================] - ETA: 0s - loss: 0.0608 - accuracy: 0.96312020-12-08 16:05:10.570036: W tensorflow/python/util/util.cc:329] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
WARNING:tensorflow:From /home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:1817: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
1/1 [==============================] - 14s 14s/step - loss: 0.0608 - accuracy: 0.9631 - val_loss: 1.2190 - val_accuracy: 0.2000
Traceback (most recent call last):
File "/home/pi/mycroft-precise/.venv/bin/precise-train-incremental", line 33, in
sys.exit(load_entry_point('mycroft-precise', 'console_scripts', 'precise-train-incremental')())
File "/home/pi/mycroft-precise/precise/scripts/base_script.py", line 49, in run_main
script.run()
File "/home/pi/mycroft-precise/precise/scripts/train_incremental.py", line 150, in run
self.train_on_audio(fn)
File "/home/pi/mycroft-precise/precise/scripts/train_incremental.py", line 137, in train_on_audio
self.retrain()
File "/home/pi/mycroft-precise/precise/scripts/train_incremental.py", line 111, in retrain
rename(self.args.model + '.h5', self.args.model) # Rename with original
IsADirectoryError: [Errno 21] Is a directory: 'hey-computer.net.h5' -> 'hey-computer.net'

@kbromer
Copy link

kbromer commented Dec 9, 2020

I should note: I'm using 141, not @MatthewScholefield repo/branch

@MatthewScholefield
Copy link
Collaborator

I haven't looked into field number 0 is illegal yet.

@kbromer Looks like you can set wake.engine.engine_path in your rhasspy config to your source install executable at .venv/bin/precise-engine and then you should be able to use the .tflite model file in place of the .pb.

@kbromer
Copy link

kbromer commented Dec 9, 2020

Oh wow, I'll try that, thanks!

@SandroMiccoli
Copy link
Author

@kbromer so you have the same setup? RPi 4B buster headless clean environment?

Did you have to install TensorFlow 2.2.0 "by hand" or you had success with the standard mycroft-precise installation script?

@kbromer
Copy link

kbromer commented Dec 10, 2020

Yes, only changes to the environment were booting from SSD over USB. Used the link in your comment above to install TF 2.2.

Only thing that regularly dies is the precise-train-incremental command, when it kicks off the retrain, it errors out with:

File "/home/pi/mycroft-precise/precise/scripts/train_incremental.py", line 111, in retrain rename(self.args.model + '.h5', self.args.model) # Rename with original IsADirectoryError: [Errno 21] Is a directory: 'hey-computer.net.h5' -> 'hey-computer.net'

Retraining it though with precise-train after works fine. Only other difference I can see between what we're doing is I'm using PR 141. I did have to delete my existing model and artifacts of course when shifting to 141. Happy to send you anything I've got that might be helpful in debugging. In fairness, I probably toggled some random bits in the process as well that I may be forgetting about...

@SandroMiccoli
Copy link
Author

Thanks @kbromer, still no success here...

Started with a new clean environment, installed TF2.2 via whl file, recorded some audio samples with precise-collect and now tried to run precise-train and this is what I got:

$ precise-train -e 60 myword.net myword/
Using TensorFlow backend.
Traceback (most recent call last):
  File "/home/pi/mycroft-precise/.venv/bin/precise-train", line 33, in <module>
    sys.exit(load_entry_point('mycroft-precise', 'console_scripts', 'precise-train')())
  File "/home/pi/mycroft-precise/precise/scripts/base_script.py", line 43, in run_main
    script = cls(args)
  File "/home/pi/mycroft-precise/precise/scripts/train.py", line 87, in __init__
    self.model = create_model(args.model, params)
  File "/home/pi/mycroft-precise/precise/model.py", line 76, in create_model
    model = Sequential()
  File "/home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/keras/models.py", line 405, in __init__
    name = prefix + str(K.get_uid(prefix))
  File "/home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 68, in get_uid
    graph = tf.get_default_graph()
AttributeError: module 'tensorflow' has no attribute 'get_default_graph'

I'm using the main branch for this repo though. Since I got a different error this time not sure which branch would be the most appropriate.

And TF2.2 was installed successfully (in the .venv):

$ python3
Python 3.7.3 (default, Jul 25 2020, 13:03:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.__version__
'2.2.0'
>>>

@dumblob
Copy link

dumblob commented Oct 22, 2021

@MatthewScholefield do you plan to upstrem the tf2 branch of yours?

@tystuyfzand
Copy link

tystuyfzand commented Dec 5, 2021

After changing to your tf2 branch I left the precise-train-incremental running over night (process started at 23h and finished at 5:30). Now I tried to test the data set and this is the result:

$ precise-test myword.net myword/
Loading wake-word...
Loading not-wake-word...
Traceback (most recent call last):
  File "/home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/python/saved_model/loader_impl.py", line 98, in parse_saved_model
    saved_model.ParseFromString(file_content)
  File "/home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/google/protobuf/message.py", line 199, in ParseFromString
    return self.MergeFromString(serialized)
  File "/home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/google/protobuf/internal/python_message.py", line 1145, in MergeFromString
    if self._InternalParse(serialized, 0, length) != length:
  File "/home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/google/protobuf/internal/python_message.py", line 1195, in InternalParse
    raise message_mod.DecodeError('Field number 0 is illegal.')
google.protobuf.message.DecodeError: Field number 0 is illegal.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/mycroft-precise/.venv/bin/precise-test", line 33, in <module>    sys.exit(load_entry_point('mycroft-precise', 'console_scripts', 'precise-test')())
  File "/home/pi/mycroft-precise/precise/scripts/base_script.py", line 49, in run_main
    script.run()
  File "/home/pi/mycroft-precise/precise/scripts/test.py", line 52, in run
    predictions = Listener.find_runner(args.model)(args.model).predict(inputs)  File "/home/pi/mycroft-precise/precise/network_runner.py", line 80, in __init__
    self.model = load_precise_model(model_name)
  File "/home/pi/mycroft-precise/precise/model.py", line 55, in load_precise_model
    return load_model(model_name, custom_objects=globals())
  File "/home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/python/keras/saving/save.py", line 189, in load_model
    loader_impl.parse_saved_model(filepath)
  File "/home/pi/mycroft-precise/.venv/lib/python3.7/site-packages/tensorflow/python/saved_model/loader_impl.py", line 101, in parse_saved_model
    raise IOError("Cannot parse file %s: %s." % (path_to_pb, str(e)))
OSError: Cannot parse file b'myword.net/saved_model.pb': Field number 0 is illegal..

This is super late, but I think I know the issue here. I don't have a ton of experience with tensorflow, but by the error I get as well I'm assuming it's related to the save_format change:

(.venv) tyler@Aurora:/mnt/e/Mycroft/Data$ precise-test astra.net astra/
Loading wake-word...
Loading not-wake-word...
/mnt/e/Mycroft/Precise/.venv/lib/python3.8/site-packages/tensorflow/python/saved_model/loader_impl.py:105: RuntimeWarning: Unexpected end-group tag: Not all data was converted
  saved_model.ParseFromString(file_content)
Traceback (most recent call last):
  File "/mnt/e/Mycroft/Precise/.venv/bin/precise-test", line 33, in <module>
    sys.exit(load_entry_point('mycroft-precise', 'console_scripts', 'precise-test')())
  File "/mnt/e/Mycroft/Precise/precise/scripts/base_script.py", line 49, in run_main
    script.run()
  File "/mnt/e/Mycroft/Precise/precise/scripts/test.py", line 52, in run
    predictions = Listener.find_runner(args.model)(args.model).predict(inputs)
  File "/mnt/e/Mycroft/Precise/precise/network_runner.py", line 80, in __init__
    self.model = load_precise_model(model_name)
  File "/mnt/e/Mycroft/Precise/precise/model.py", line 55, in load_precise_model
    return load_model(model_name, custom_objects=globals())
  File "/mnt/e/Mycroft/Precise/.venv/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/mnt/e/Mycroft/Precise/.venv/lib/python3.8/site-packages/keras/saving/saved_model/load.py", line 108, in load
    meta_graph_def = tf.__internal__.saved_model.parse_saved_model(
IndexError: list index (0) out of range

What it looks like is that save_format='h5' was added during the upgrade to 2.x, but it's still saving with a .pb extension which MIGHT cause it to fail attempting to load - the original train script works because it doesn't ever specify a save format that I can find (?) so it may be properly saving it. It looks like it's trying to load it with the SavedModel format instead of h5 (https://www.tensorflow.org/api_docs/python/tf/keras/Model#save) - I'm testing my theory, but I wanted to document it before it slipped my mind.

Perhaps @MatthewScholefield could confirm this? If it is the case and has a reasonable fix then I'm happy to figure it out and push it as a merge to his branch.

Edit: Looks like the fix to this would be to change the extension to .h5, and perhaps tell people to use the .h5 path (which appears to be a self-contained file?) to load the model instead of just ".net"

@rbrisita
Copy link

Anyone reading this that wants to build from source, try these steps:

  1. git clone https://github.com/MycroftAI/mycroft-precise.git
  2. cd mycroft-precise
  3. python3 -m venv .venv
  4. source .venv/bin/activate
  5. python3 -m pip install --upgrade pip
  6. pip install wheel
  7. git fetch origin pull/141/head:tf
  8. git checkout tf
  9. edit setup.py and change tensorflow-gpu to tensorflow with no pinning.
  10. ./setup.sh

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants