Error after following installation steps

[JRMeyer]

>>> fminkin
[November 27, 2017, 8:09pm]

I followed through the installation steps for cpu training deepspech, it
succeeded. slash
(interesting fact that in requirements.txt tensorflow version is not
stated, should be 1.3.0?) slash
Although the command bin/run-ldc93s1.sh failed with message:

+ '[' '!' -f data/ldc93s1/ldc93s1.csv ']'
+ echo 'Downloading and preprocessing LDC93S1 example data, saving in ./data/ldc93s1.'
Downloading and preprocessing LDC93S1 example data, saving in ./data/ldc93s1.
+ python -u bin/import_ldc93s1.py ./data/ldc93s1
Successfully downloaded LDC93S1.wav 93638 bytes.
Successfully downloaded LDC93S1.txt 62 bytes.
+ '[' -d '' ']'
++ python -c 'from xdg import BaseDirectory as xdg; print(xdg.save_data_path('deepspeech/ldc93s1'))'
+ checkpoint_dir=/home/f-minkin/.local/share/deepspeech/ldc93s1
+ python -u DeepSpeech.py --train_files data/ldc93s1/ldc93s1.csv --dev_files data/ldc93s1/ldc93s1.csv --test_files data/ldc93s1/ldc93s1.csv --train_batch_size 1 --dev_batch_size 1 --test_batch_size 1 --n_hidden 494 --epoch 50 --checkpoint_dir /home/f-minkin/.local/share/deepspeech/ldc93s1
Traceback (most recent call last):
File 'DeepSpeech.py', line 482, in
custom_op_module = tf.load_op_library(FLAGS.decoder_library_path)
File '/home/f-minkin/tmp/deepspeech-venv/local/lib/python2.7/site-packages/tensorflow/python/framework/load_library.py', line 56, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename, status)
File '/home/f-minkin/tmp/deepspeech-venv/local/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py', line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: native_client/libctc_decoder_with_kenlm.so: undefined symbol: _ZN10tensorflow10DEVICE_CPUE

Moreover if I follow the gpu instructions:

python util/taskcluster.py --target /tmp --source tensorflow --arch gpu --artifact tensorflow_gpu_warpctc-1.3.0rc0-cp27-cp27mu-linux_x86_64.whl
pip install /tmp/tensorflow_gpu_warpctc-1.3.0rc0-cp27-cp27mu-linux_x86_64.whl

I got tf exception:

+ '[' '!' -f data/ldc93s1/ldc93s1.csv ']'
+ '[' -d '' ']'
++ python -c 'from xdg import BaseDirectory as xdg; print(xdg.save_data_path('deepspeech/ldc93s1'))'
+ checkpoint_dir=/home/f-minkin/.local/share/deepspeech/ldc93s1
+ python -u DeepSpeech.py --train_files data/ldc93s1/ldc93s1.csv --dev_files data/ldc93s1/ldc93s1.csv --test_files data/ldc93s1/ldc93s1.csv --train_batch_size 1 --dev_batch_size 1 --test_batch_size 1 --n_hidden 494 --epoch 50 --checkpoint_dir /home/f-minkin/.local/share/deepspeech/ldc93s1
2017-11-27 23:07:08.360678: F tensorflow/core/platform/cpu_feature_guard.cc:35] The TensorFlow library was compiled to use AVX2 instructions, but these aren't available on your machine.
./bin/run-ldc93s1.sh: line 29: 210003 Aborted python -u DeepSpeech.py --train_files data/ldc93s1/ldc93s1.csv --dev_files data/ldc93s1/ldc93s1.csv --test_files data/ldc93s1/ldc93s1.csv --train_batch_size 1 --dev_batch_size 1 --test_batch_size 1 --n_hidden 494 --epoch 50 --checkpoint_dir '$checkpoint_dir' '$'```

[This is an archived TTS discussion thread from discourse.mozilla.org/t/error-after-following-installation-steps]

[JRMeyer]

>>> lissyx
[November 28, 2017, 7:32am]

Thanks! So your system does not have AVX2, you will have to build the
Python package from https://github.com/mozilla/tensorflow master
branch. You can follow instructions in DeepSpeech's README for that
![:slight_smile:](

[Archived Post]

[JRMeyer]

>>> lissyx
[November 28, 2017, 9:59am]

Can you also share more details about your system ? Are you using some
VM on some provider, and the output of cat /proc/cpuinfo so others can
find it ? Thanks
![:slight_smile:](

[Archived Post]

[JRMeyer]

>>> fminkin
[November 28, 2017, 12:28pm]

My machine has 32 cores, so it's a bit of a flood, I'll post link to
pastebin: slash
Link

[Archived Post]

[JRMeyer]

>>> lissyx
[November 28, 2017, 12:35pm]

Thanks, it confirms: slash
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt

There's avx but no avx2. So far, only solution is that you rebuild
the python tensorflow package, it's should only take some time but with
that kind of machine not that much.

[Archived Post]

[JRMeyer]

>>> fminkin
[November 28, 2017, 12:45pm]

Thanks a lot!

What about requirements.txt? It's stated there 'tensorflow', that has no
version, so it'll download 1.4.0? I understand, that it's not necessary
due to compilation from sources but anyway, should there be 'tensorflow
== 1.3.0'?

P.S. The information about processors setting might be very helpful to
others who wants to try it, it would be good thing to add this to
readme, in my opinion

[Archived Post]

[JRMeyer]

>>> lissyx
[November 28, 2017, 1:28pm]

It is already documented in the README that we require CPU with at least
AVX2 and FMA. Regarding the requirements.txt we should probably fix that
in a more proper way. The current issue is that since Tensorflow has no
API stability guarantee besides Python, even tensorflow==1.3.0 might
get into troubles ; people reported problems with upstream 1.3.0 and out
libctc_decoder_with_kenlm.so.

You can probably open an issue and make a PR to change that. We could
use the TaskCluster link:
https://index.taskcluster.net/v1/task/project.deepspeech.tensorflow.pip.master.gpu/artifacts/public/tensorflow_gpu_warpctc-1.3.0rc0-cp27-cp27mu-linux_x86_64.whl

This would assume that people wanting to train have a CUDA-enabled setup
(which is probably fine). Happy to review your issue and PR on that
![:slight_smile:](

[Archived Post]

[JRMeyer]

>>> fminkin
[November 28, 2017, 1:50pm]

yep, thanks! I got the CUDA-enabled setup working, tho it fails at
decoding:

Loading the LM will be faster if you build a binary file.
Reading data/lm/lm.binary
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
terminate called after throwing an instance of 'lm::FormatLoadException'
what(): native_client/kenlm/lm/read_arpa.cc:65 in void lm::ReadARPACounts(util::FilePiece&, std::vector&) threw FormatLoadException.
first non-empty line was 'version https://git-lfs.github.com/spec/v1' not slash data slash . Byte: 43
Aborted (core dumped)```

[Archived Post]

[JRMeyer]

>>> lissyx
[November 28, 2017, 2:00pm]

You have not setup git-lfs, so there is no language model, please read
the README for this.

[Archived Post]

[JRMeyer]

>>> fminkin
[November 28, 2017, 2:20pm]

Oh, it's in FAQ section and in README, I'm so sorry for not paying full
attention to it.

I'll try to make PR asap, thanks.

[Archived Post]

[JRMeyer]

>>> lissyx
[November 29, 2017, 10:40am]

FYI I'm covering similar changes for when we switch to r1.4: we will
have a task running on TaskCluster, and testing training of the model
against upstream tensorflow with our libctc_decoder_with_kenlm.so
![:slight_smile:](

[Archived Post]

[JRMeyer]

>>> fminkin
[November 29, 2017, 12:49pm]

Yeah, I saw it in Issues
![:slight_smile:](

[Archived Post]

[JRMeyer]

>>> dowwie
[November 30, 2017, 12:08pm]

I've also run into the AVX2 error and do not see avx2 in my procinfo
flags. I assume I also need to rebuild the python tensorflow package.

My setup:

-Python: python 3.6, built from source

BTW slash -- DeepSpeech discourse.mozilla.org python3 compatible, right?

[Archived Post]

[JRMeyer]

>>> lissyx
[November 30, 2017, 12:12pm]

Yes, it should all run properly with Python 3.

[Archived Post]

[JRMeyer]

>>> dowwie
[November 30, 2017, 3:27pm]

I installed tensorflow from source but it appears that I still haven't
successfully added avx2 support. It would be helpful if someone would
provide the ./configure and bazel-build commands that will surely work.

[Archived Post]

[JRMeyer]

>>> lissyx
[November 30, 2017, 3:30pm]

If you follow Tensorflow's build instruction, there's no way you endup
with a package that depends on AVX2 if you system does not. Which
configure and bazel build command did you use ?

[Archived Post]

[JRMeyer]

>>> dowwie
[November 30, 2017, 3:38pm]

Going back to the start.

I pip3 installed deepsearch. Then, when I tested it at the command line,
the error I got was

> 'The TensorFlow library was compiled to use AVX2 instructions, but
> these aren't available on your machine.'

So, I pip3 uninstalled deepsearch and tensorflow. Then, I configured and
bazel-built tensorflow. Then, I pip3 installed deepsearch, tried again,
got the same avx2 error.

My guess is that I'm not ./configuring or bazel-building correctly so
I'm asking what the recommended lines are for no-gpu configuration.

Does this make sense?

[Archived Post]

[JRMeyer]

>>> lissyx
[November 30, 2017, 3:47pm]

It makes sense, but you are doing something different than what was the
start of this topic
![:slight_smile:](
where the first poster was having issues during training. Basically, any
package you will pip install is one we did build with AVX2 support.

In your case, you need to follow the native_client/README.md about
building the Python bindings (and also building Tensorflow, it should be
all documented there). When doing so, you should have no forced
optimization and thus it should run properly.

github.com

#### mozilla/DeepSpeech/blob/master/native_client/README.md slash #build-requirements

# DeepSpeech native client, language bindings and custom decoder

This folder contains a native client for running queries on an exported DeepSpeech model, bindings for Python and Node.JS for using an exported DeepSpeech model programatically, and a CTC beam search decoder implementation that scores beams using a language model, needed for training a DeepSpeech model. We provide pre-built binaries for Linux and macOS.

## Installation

To download the pre-built binaries, use util/taskcluster.py:

<br/> python util/taskcluster.py --target /path/to/destination/folder<br/>

This will download and extract native_client.tar.xz which includes the deepspeech binary and associated libraries as well as the custom decoder OP. taskcluster.py will download binaries for the architecture of the host by default, but you can override that behavior with the --arch parameter. See the help info with python util/taskcluster.py -h for more details.

If you want the CUDA capable version of the binaries, use --arch gpu. Note that for now we don't publish CUDA-capable macOS binaries.

If you're looking to train a model, you now have a libctc_decoder_with_kenlm.so file that you can pass to the --decoder_library_path parameter of DeepSpeech.py.

## Required Dependencies

This file has been truncated. show
original

So, pip uninstall anything you installed earlier, and follow that.

Besides, out of curiosity, what is your CPU / system ?

[Archived Post]

[JRMeyer]

>>> dowwie
[November 30, 2017, 3:50pm]

Apologies for the confusion! I will try as recommended.

CPU: Intel® Core™ i7-3770 CPU 3.40GHz slash
OS: Debian 9

[Archived Post]

[JRMeyer]

>>> dowwie
[November 30, 2017, 3:57pm]

My confusion was due to the use of multiple README files in the
DeepSearch repository. You are referencing instructions in the
DeepSearch/native_client/README.me where as I was referencing the
installation instructions README in the root directory. I just noticed
the root README tells me to check out the other README if installation
fails. Oh well.

[next page
→

[Archived Post]