Error when using structures as input

[yuanqm55]

Hi
I followed the instructions and successfully get predictions for single sequence or a fasta file of multiple sequences.
However, when I try to make predictions for structures, I get the following error:

(DeepFRI) yuanqm@gpu2:~/protein_function/DeepFRI$ python predict.py --cmap ./examples/pdb_cmaps/1S3P-A.npz -ont mf --verbose
2022-11-07 22:03:37.689676: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2022-11-07 22:03:37.689727: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-11-07 22:03:38.802448: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2022-11-07 22:03:38.839570: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:3e:00.0 name: NVIDIA GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 82 deviceMemorySize: 23.70GiB deviceMemoryBandwidth: 871.81GiB/s
2022-11-07 22:03:38.839863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties: 
pciBusID: 0000:40:00.0 name: NVIDIA GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 82 deviceMemorySize: 23.70GiB deviceMemoryBandwidth: 871.81GiB/s
2022-11-07 22:03:38.840124: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 2 with properties: 
pciBusID: 0000:b1:00.0 name: NVIDIA GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 82 deviceMemorySize: 23.70GiB deviceMemoryBandwidth: 871.81GiB/s
2022-11-07 22:03:38.840397: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 3 with properties: 
pciBusID: 0000:b5:00.0 name: NVIDIA GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 82 deviceMemorySize: 23.70GiB deviceMemoryBandwidth: 871.81GiB/s
2022-11-07 22:03:38.840530: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2022-11-07 22:03:38.840596: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory
2022-11-07 22:03:38.886881: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2022-11-07 22:03:38.887473: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2022-11-07 22:03:38.887753: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory
2022-11-07 22:03:38.887887: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory
2022-11-07 22:03:38.887957: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory
2022-11-07 22:03:38.887973: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1753] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-11-07 22:03:38.888385: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-11-07 22:03:38.903619: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3000000000 Hz
2022-11-07 22:03:38.910226: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3bd2270 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-11-07 22:03:38.910280: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2022-11-07 22:03:38.912764: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-11-07 22:03:38.912826: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      
### Computing predictions on a single protein...
Traceback (most recent call last):
  File "predict.py", line 39, in <module>
    predictor.predict(args.cmap)
  File "/home/yuanqm/protein_function/DeepFRI/deepfrier/Predictor.py", line 109, in predict
    y = self.model([A, S], training=False).numpy()[:, :, 0].reshape(-1)
  File "/home/yuanqm/anaconda3/envs/DeepFRI/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 985, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "/home/yuanqm/anaconda3/envs/DeepFRI/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py", line 386, in call
    inputs, training=training, mask=mask)
  File "/home/yuanqm/anaconda3/envs/DeepFRI/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py", line 508, in _run_internal_graph
    outputs = node.layer(*args, **kwargs)
  File "/home/yuanqm/anaconda3/envs/DeepFRI/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 985, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "/home/yuanqm/anaconda3/envs/DeepFRI/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py", line 386, in call
    inputs, training=training, mask=mask)
  File "/home/yuanqm/anaconda3/envs/DeepFRI/lib/python3.7/site-packages/tensorflow/python/keras/engine/functional.py", line 508, in _run_internal_graph
    outputs = node.layer(*args, **kwargs)
  File "/home/yuanqm/anaconda3/envs/DeepFRI/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.py", line 659, in __call__
    return super(RNN, self).__call__(inputs, **kwargs)
  File "/home/yuanqm/anaconda3/envs/DeepFRI/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 985, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "/home/yuanqm/anaconda3/envs/DeepFRI/lib/python3.7/site-packages/tensorflow/python/keras/layers/cudnn_recurrent.py", line 110, in call
    output, states = self._process_batch(inputs, initial_state)
  File "/home/yuanqm/anaconda3/envs/DeepFRI/lib/python3.7/site-packages/tensorflow/python/keras/layers/cudnn_recurrent.py", line 507, in _process_batch
    outputs, h, c, _, _ = gen_cudnn_rnn_ops.cudnn_rnnv2(**args)
  File "/home/yuanqm/anaconda3/envs/DeepFRI/lib/python3.7/site-packages/tensorflow/python/ops/gen_cudnn_rnn_ops.py", line 1740, in cudnn_rnnv2
    ctx=_ctx)
  File "/home/yuanqm/anaconda3/envs/DeepFRI/lib/python3.7/site-packages/tensorflow/python/ops/gen_cudnn_rnn_ops.py", line 1817, in cudnn_rnnv2_eager_fallback
    attrs=_attrs, ctx=ctx, name=name)
  File "/home/yuanqm/anaconda3/envs/DeepFRI/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.NotFoundError: Could not find device for node: {{node CudnnRNNV2}} = CudnnRNNV2[T=DT_FLOAT, direction="unidirectional", dropout=0, input_mode="linear_input", is_training=true, rnn_mode="lstm", seed=0, seed2=0]
All kernels registered for op CudnnRNNV2:
  device='GPU'; T in [DT_DOUBLE]
  device='GPU'; T in [DT_FLOAT]
  device='GPU'; T in [DT_HALF]
 [Op:CudnnRNNV2]

[lalalagartija]

Same error here

[Liuyungeng123]

Any solution?

[PawelSzczerbiak]

It looks like some TensorFlow incompatibility problem regarding GPU models. If you can afford longer runtime, please use CPU models from here.

[Liuyungeng123]

Thank you for your reply! I will try it!

[Liuyungeng123]

I also have an idea that we should get the branch dependabot/pip/tensorflow-gpu-2.12.0 to match the CUDA and CUDNN version