Installation problems (TF bindings) #59

nmiculinic · 2016-12-18T20:36:28Z

I'm trying to install tensorflow binding and have followed the steps:

cloned tf repo (without building it, tf is installed via pip)
Sucessfully installed package.
It fails on tests:

test_ctc_loss_op (unittest.loader._FailedTest) ... ERROR
test_warpctc_op (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: test_ctc_loss_op (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_ctc_loss_op
Traceback (most recent call last):
  File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/tmp/warp-ctc/tensorflow_binding/tests/test_ctc_loss_op.py", line 23, in <module>
    import warpctc_tensorflow
  File "/tmp/warp-ctc/tensorflow_binding/warpctc_tensorflow/__init__.py", line 7, in <module>
    _warpctc = tf.load_op_library(lib_file)
  File "/usr/lib/python3.5/site-packages/tensorflow/python/framework/load_library.py", line 64, in load_op_library
    None, None, error_msg, error_code)
tensorflow.python.framework.errors_impl.NotFoundError: /tmp/warp-ctc/tensorflow_binding/warpctc_tensorflow/kernels.cpython-35m-x86_64-linux-gnu.so: undefined symbol: _ZN10tensorflow7strings6StrCatB5cxx11ERKNS0_8AlphaNumE


======================================================================
ERROR: test_warpctc_op (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_warpctc_op
Traceback (most recent call last):
  File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/tmp/warp-ctc/tensorflow_binding/tests/test_warpctc_op.py", line 3, in <module>
    from warpctc_tensorflow import ctc
  File "/tmp/warp-ctc/tensorflow_binding/warpctc_tensorflow/__init__.py", line 7, in <module>
    _warpctc = tf.load_op_library(lib_file)
  File "/usr/lib/python3.5/site-packages/tensorflow/python/framework/load_library.py", line 64, in load_op_library
    None, None, error_msg, error_code)
tensorflow.python.framework.errors_impl.NotFoundError: /tmp/warp-ctc/tensorflow_binding/warpctc_tensorflow/kernels.cpython-35m-x86_64-linux-gnu.so: undefined symbol: _ZN10tensorflow7strings6StrCatB5cxx11ERKNS0_8AlphaNumE


----------------------------------------------------------------------
Ran 2 tests in 0.000s

FAILED (errors=2)
/t/w/tensorflow_binding »

Ls of directory in question:

/t/w/tensorflow_binding » ls /tmp/warp-ctc/tensorflow_binding/warpctc_tensorflow      master ✔
__pycache__  __init__.py  kernels.cpython-35m-x86_64-linux-gnu.so

The text was updated successfully, but these errors were encountered:

nmiculinic · 2016-12-18T22:33:41Z

EDIT:

Tried with python2 and python3, and tried on two computers, (local one and workstation). Problem persists.

chanil1218 · 2016-12-23T19:18:55Z

I faced same issue, and I found this link.
This error might comes from difference between gcc4 that your tensorflow binary built with and gcc5 your compilation environment for warp-ctc.

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -D_GLIBCXX_USE_CXX11_ABI=0")

Put this line into your CMakeLists.txt and just let me know it's work.

nmiculinic · 2016-12-24T00:35:29Z

I manage to fix it by using what you recommended. (in CMakeList.txt and additionally in setup.py since there is also without that flag). If I use it within tensorflow code it works fine, however unit tests still fail with error:

CUDA_HOME not found in the environment so building without GPU support. To build with GPU support please define the CUDA_HOME environment variable. This should be a path which contains include/cuda.h
running test
running egg_info
writing top-level names to warpctc_tensorflow.egg-info/top_level.txt
writing warpctc_tensorflow.egg-info/PKG-INFO
writing dependency_links to warpctc_tensorflow.egg-info/dependency_links.txt
reading manifest file 'warpctc_tensorflow.egg-info/SOURCES.txt'
writing manifest file 'warpctc_tensorflow.egg-info/SOURCES.txt'
running build_ext
copying build/lib.linux-x86_64-3.5/warpctc_tensorflow/kernels.cpython-35m-x86_64-linux-gnu.so -> warpctc_tensorflow
CUDA_HOME not found in the environment so building without GPU support. To build with GPU support please define the CUDA_HOME environment variable. This should be a path which contains include/cuda.h
running test
running egg_info
writing top-level names to warpctc_tensorflow.egg-info/top_level.txt
writing warpctc_tensorflow.egg-info/PKG-INFO
writing dependency_links to warpctc_tensorflow.egg-info/dependency_links.txt
reading manifest file 'warpctc_tensorflow.egg-info/SOURCES.txt'
writing manifest file 'warpctc_tensorflow.egg-info/SOURCES.txt'
running build_ext
copying build/lib.linux-x86_64-3.5/warpctc_tensorflow/kernels.cpython-35m-x86_64-linux-gnu.so -> warpctc_tensorflow
testBasicCPU (test_ctc_loss_op.CTCLossTest) ... ok
testBasicGPU (test_ctc_loss_op.CTCLossTest) ... Skipping GPU test, no gpus available
ok
test_session (test_ctc_loss_op.CTCLossTest)
Returns a TensorFlow Session for use in executing tests. ... ok
test_basic_cpu (test_warpctc_op.WarpCTCTest) ... /home/lpp/Downloads/git/warp-ctc/tensorflow_binding/tests/test_warpctc_op.py:16: DeprecationWarning: Please use assertEqual instead.
  self.assertEquals(activations.shape, expected_gradients.shape)
ok
test_basic_gpu (test_warpctc_op.WarpCTCTest) ... Skipping GPU test, no gpus available
ok
test_multiple_batches_cpu (test_warpctc_op.WarpCTCTest) ... ok
test_multiple_batches_gpu (test_warpctc_op.WarpCTCTest) ... Skipping GPU test, no gpus available
ok
test_session (test_warpctc_op.WarpCTCTest)
Returns a TensorFlow Session for use in executing tests. ... ok

----------------------------------------------------------------------
Ran 8 tests in 0.120s

OK
test_ctc_loss_op (unittest.loader._FailedTest) ... ERROR
test_warpctc_op (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: test_ctc_loss_op (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_ctc_loss_op
Traceback (most recent call last):
  File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/lpp/Downloads/git/warp-ctc/tensorflow_binding/tests/test_ctc_loss_op.py", line 23, in <module>
    import warpctc_tensorflow
  File "/home/lpp/Downloads/git/warp-ctc/tensorflow_binding/warpctc_tensorflow/__init__.py", line 47, in <module>
    @ops.RegisterGradient("WarpCTC")
  File "/usr/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1495, in __call__
    _gradient_registry.register(f, self._op_type)
  File "/usr/lib/python3.5/site-packages/tensorflow/python/framework/registry.py", line 62, in register
    (self._name, name, function_name, filename, line_number))
KeyError: "Registering two gradient with name 'WarpCTC' !(Previous registration was in run_commands /usr/lib/python3.5/distutils/dist.py:955)"


======================================================================
ERROR: test_warpctc_op (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_warpctc_op
Traceback (most recent call last):
  File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/lpp/Downloads/git/warp-ctc/tensorflow_binding/tests/test_warpctc_op.py", line 3, in <module>
    from warpctc_tensorflow import ctc
  File "/home/lpp/Downloads/git/warp-ctc/tensorflow_binding/warpctc_tensorflow/__init__.py", line 47, in <module>
    @ops.RegisterGradient("WarpCTC")
  File "/usr/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1495, in __call__
    _gradient_registry.register(f, self._op_type)
  File "/usr/lib/python3.5/site-packages/tensorflow/python/framework/registry.py", line 62, in register
    (self._name, name, function_name, filename, line_number))
KeyError: "Registering two gradient with name 'WarpCTC' !(Previous registration was in run_commands /usr/lib/python3.5/distutils/dist.py:955)"


----------------------------------------------------------------------
Ran 2 tests in 0.000s

FAILED (errors=2)

dylanbfox · 2017-02-08T21:25:23Z

FWIW, I ran into this issue too. I just dropped down to Ubuntu 14.04 (from 16.04), and that solved this (and other) issues.

bdaskalov · 2017-02-27T09:38:21Z

I ran into the same problem, added the flag and submitted a pull request but nobody seems to be looking at it.
#73

srilives · 2017-03-29T09:01:01Z

Hi there,

We are also getting the same error as mentioned by nmiculinic. Is there is fix? if yes, please share. Thx

megazone87 · 2017-06-01T07:22:16Z

@nmiculinic The same things happed to me. The unittest failed at the end, though the mid-process said the tests have been passed.

bidai541 · 2017-08-10T07:01:58Z

The same error too.
add this line in setup.py in line 62
extra_compile_args += [ '-D_GLIBCXX_USE_CXX11_ABI=0']
add this line in CMakeFile.txt in line 8 :
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -D_GLIBCXX_USE_CXX11_ABI=0")
(those two line means the same param may be only one could word!)
Delate the build and dist directory in warp-ctc/tensorflow_binding/
Then rebuild ; python setup.py install
This fixed!

dgdn · 2017-08-23T05:08:36Z

@bidai541 it did works!! thx

fancyerii · 2017-08-24T11:44:56Z

I face the same problem. When I add as bidai541 suggested, it seems to work. But When I run python setup.py test. it failed with:KeyError: "Registering two gradient with name 'WarpCTC' !(Previous registration was in run_commands /usr/lib/python3.5/distutils/dist.py:955)"

hyichao · 2017-09-23T02:19:57Z

@bidai541 It works and thanks for the solution. And by this I assume this error is caused by gcc5, therefore downgrading to Ubuntu14.04 works too in @dylanbfox 's case as gcc4 is installed in U14.04 meanwhile gcc5 is set as default in U16.04.
@fancyerii I assume setup.py test fails due to the script itself. You can still run your TF training code with warpctc correctly. In my case, there is a great acceleration.

hookover · 2017-11-10T10:11:50Z

@bidai541
i can't find the CMakeFile.txt file in warp-ctc dirctory and sub dircotry

hookover · 2017-11-10T10:14:43Z

@nmiculinic Have you solved this error? If yes, how to do that ?

bidai541 · 2017-11-13T01:45:34Z

@hookover Root path, pull again!

vankrelian · 2017-11-27T20:11:41Z

I follow the guidelines above to change the compiler settings in both setup.py and CMakeLists.txt. However, it didn't change anything. I still have the same error when I run the tests:

NotFoundError: /home/ekonishi/warp-ctc/tensorflow_binding/warpctc_tensorflow/kernels.so: undefined symbol: _ZTIN10tensorflow8OpKernelE

Note that I've successfully compiled and installed tensorflow 1.4 in a virtual environment.

nmiculinic changed the title ~~Installation problems~~ Installation problems (TF bindings) Dec 18, 2016

matth mentioned this issue Apr 28, 2017

Add a flag to use the old C++ ABI when compiled with newer GCC. #73

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installation problems (TF bindings) #59

Installation problems (TF bindings) #59

nmiculinic commented Dec 18, 2016 •

edited

Loading

nmiculinic commented Dec 18, 2016

chanil1218 commented Dec 23, 2016

nmiculinic commented Dec 24, 2016

dylanbfox commented Feb 8, 2017 •

edited

Loading

bdaskalov commented Feb 27, 2017

srilives commented Mar 29, 2017

megazone87 commented Jun 1, 2017

bidai541 commented Aug 10, 2017

dgdn commented Aug 23, 2017 •

edited

Loading

fancyerii commented Aug 24, 2017

hyichao commented Sep 23, 2017

hookover commented Nov 10, 2017

hookover commented Nov 10, 2017

bidai541 commented Nov 13, 2017

vankrelian commented Nov 27, 2017

Installation problems (TF bindings) #59

Installation problems (TF bindings) #59

Comments

nmiculinic commented Dec 18, 2016 • edited Loading

nmiculinic commented Dec 18, 2016

chanil1218 commented Dec 23, 2016

nmiculinic commented Dec 24, 2016

dylanbfox commented Feb 8, 2017 • edited Loading

bdaskalov commented Feb 27, 2017

srilives commented Mar 29, 2017

megazone87 commented Jun 1, 2017

bidai541 commented Aug 10, 2017

dgdn commented Aug 23, 2017 • edited Loading

fancyerii commented Aug 24, 2017

hyichao commented Sep 23, 2017

hookover commented Nov 10, 2017

hookover commented Nov 10, 2017

bidai541 commented Nov 13, 2017

vankrelian commented Nov 27, 2017

nmiculinic commented Dec 18, 2016 •

edited

Loading

dylanbfox commented Feb 8, 2017 •

edited

Loading

dgdn commented Aug 23, 2017 •

edited

Loading