FileNotFoundError: [Errno 2] No such file or directory: '/xxx/pyramids'

[mcariou]

Hello,

I am trying to run instaGRAAL on a linux machine using docker with this command:

sudo docker run --gpus all koszullab/instagraal instagraal $OUT_HICSTUFF $ASSEMBLY $OUT_INSTA

It gave this error:

Traceback (most recent call last):
File "/usr/local/bin/instagraal", line 11, in
load_entry_point('instagraal', 'console_scripts', 'instagraal')()
File "/src/instagraal/instagraal/instagraal.py", line 1178, in main
output_folder=output_folder,
File "/src/instagraal/instagraal/instagraal.py", line 160, in init
output_folder=output_folder,
File "/src/instagraal/instagraal/simu_single.py", line 74, in init
self.select_data_set(name)
File "/src/instagraal/instagraal/simu_single.py", line 743, in select_data_set
thresh_factor=self.thresh_factor,
File "/src/instagraal/instagraal/pyramid_sparse.py", line 58, in build_and_filter
os.mkdir(all_pyramid_folder)
FileNotFoundError: [Errno 2] No such file or directory: '/xxx/pyramids'

I found this other issue #24 which seems similar, but with singularity, and attempted to reproduce the proposed solution in docker using "-v $PWD:/data" (instead of -B $PWD:/data), but obtained the same error.

The HiC matrix is produced from Hicstuff and the assembly is a fasta file (from wtdbg2).

Thanks in advance for any help and sorry if I didn't include sufficient information.
Best regards

[ABignaud]

Hi,

It seems a similar error as the one here:
#39 (comment)

Can you try changing the permissions before launching docker and providing the working directory to docker. The example of command is given in the comment of the previous issue.

[mcariou]

Oh, thank you very much, I missed this post.

It worked but I encountered a second error after some steps:

...
INFO :: import reference genome
INFO :: loading data from level = 3
INFO :: mean frag area = 611.1701049804688
INFO :: N frag duplicated = 0
INFO :: MAX ID CONTIG = 11021
INFO :: total mem used by sparse data = 11.676732
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pycuda-2021.1-py3.6-linux-x86_64.egg/pycuda/tools.py", line 470, in wrapper
return ctx_dict[cur_ctx][cache_key]
KeyError: <pycuda._driver.Context object at 0x7f044abf8710>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/instagraal", line 11, in
load_entry_point('instagraal', 'console_scripts', 'instagraal')()
File "/src/instagraal/instagraal/instagraal.py", line 1178, in main
output_folder=output_folder,
File "/src/instagraal/instagraal/instagraal.py", line 160, in init
output_folder=output_folder,
File "/src/instagraal/instagraal/simu_single.py", line 168, in init
self.pos,
File "/src/instagraal/instagraal/cuda_lib_gl_single.py", line 251, in init
self.setup_all_gpu_struct()
File "/src/instagraal/instagraal/cuda_lib_gl_single.py", line 371, in setup_all_gpu_struct
self.gpu_counter_select = ga.zeros(1, dtype=np.int32)
File "/usr/local/lib/python3.6/dist-packages/pycuda-2021.1-py3.6-linux-x86_64.egg/pycuda/gpuarray.py", line 1244, in zeros
result.fill(zero)
File "/usr/local/lib/python3.6/dist-packages/pycuda-2021.1-py3.6-linux-x86_64.egg/pycuda/gpuarray.py", line 659, in fill
func = elementwise.get_fill_kernel(self.dtype)
File "/usr/local/lib/python3.6/dist-packages/pycuda-2021.1-py3.6-linux-x86_64.egg/pycuda/tools.py", line 474, in wrapper
result = func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/pycuda-2021.1-py3.6-linux-x86_64.egg/pycuda/elementwise.py", line 566, in get_fill_kernel
"fill",
File "/usr/local/lib/python3.6/dist-packages/pycuda-2021.1-py3.6-linux-x86_64.egg/pycuda/elementwise.py", line 193, in get_elwise_kernel
arguments, operation, name, keep, options, **kwargs
File "/usr/local/lib/python3.6/dist-packages/pycuda-2021.1-py3.6-linux-x86_64.egg/pycuda/elementwise.py", line 178, in get_elwise_kernel_and_types
mod = module_builder(arguments, operation, name, keep, options, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/pycuda-2021.1-py3.6-linux-x86_64.egg/pycuda/elementwise.py", line 82, in get_elwise_module
no_extern_c=True,
File "/usr/local/lib/python3.6/dist-packages/pycuda-2021.1-py3.6-linux-x86_64.egg/pycuda/compiler.py", line 358, in init
include_dirs,
File "/usr/local/lib/python3.6/dist-packages/pycuda-2021.1-py3.6-linux-x86_64.egg/pycuda/compiler.py", line 298, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File "/usr/local/lib/python3.6/dist-packages/pycuda-2021.1-py3.6-linux-x86_64.egg/pycuda/compiler.py", line 87, in compile_plain
checksum.update(preprocess_source(source, options, nvcc).encode("utf-8"))
File "/usr/local/lib/python3.6/dist-packages/pycuda-2021.1-py3.6-linux-x86_64.egg/pycuda/compiler.py", line 59, in preprocess_source
"nvcc preprocessing of %s failed" % source_path, cmdline, stderr=stderr
pycuda.driver.CompileError: nvcc preprocessing of /tmp/tmpqlcffcwz.cu failed
[command: nvcc --preprocess -arch sm_86 -I/usr/local/lib/python3.6/dist-packages/pycuda-2021.1-py3.6-linux-x86_64.egg/pycuda/cuda /tmp/tmpqlcffcwz.cu --compiler-options -P]
[stderr:
b"nvcc fatal : Value 'sm_86' is not defined for option 'gpu-architecture'\n"]

which also looks like issue 24, but not completely.
Can this correspond to a cuda version issue? How should I check that?

Thank you very much again for your help!
Best regards

[ABignaud]

Sorry, I forgot to answer you.
Have you check, that is not due to incompatible version of pycuda. Instagraal is not working with the last versions of pycuda. You can see this issue: #42.
Best regards,

[mcariou]

Hi, I thank you again for your answer!

I struggle a little with the versions of the programs and I think that somehow I made a mess with the nvidia driver, because now gives this error:

sudo docker run --gpus all --workdir $WORKDIR --mount "type=bind,src=$WORKDIR,dst=$WORKDIR" koszullab/instagraal instagraal ~/PATH/matrice ~/PATH/assembly.fasta ~/PATH/instagraal_out/
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].
ERRO[0000] error waiting for container:

Here are informations on the nvidia driver and cuda installations:

nvidia-smi
Tue Aug 22 19:52:56 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.199.02   Driver Version: 470.199.02   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:0B:00.0 Off |                  N/A |
|  0%   37C    P8    10W / 350W |     35MiB / 24260MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1106      G   /usr/lib/xorg/Xorg                 33MiB |
+-----------------------------------------------------------------------------+

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

Do you know what I did wrong here?

Also, while the docker wasn't working, I managed to install the program locally but also got an error:

instagraal ~/PATH/matrice ~/PATH/assembly.fasta ~/PATH/instagraal_out/
INFO :: Selected_device: NVIDIA GeForce RTX 3090
Traceback (most recent call last):
  File "/usr/local/bin/instagraal", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/instagraal/instagraal.py", line 2146, in main
    p2 = window(
  File "/usr/local/lib/python3.8/dist-packages/instagraal/instagraal.py", line 208, in __init__
    self.cuda_gl_init()
  File "/usr/local/lib/python3.8/dist-packages/instagraal/instagraal.py", line 1447, in cuda_gl_init
    self.ctx_gl = cudagl.make_context(
pycuda._driver.Error: cuGLCtxCreate failed: unknown error

I noticed this error is reported elsewhere #34, #12

and from the little I understood, it might still be a pycuda version issue, but how do I chose the right version, check which one is running and change it?

I used these commands to install pycuda:

    git clone --recurse-submodules https://github.com/inducer/pycuda.git
     cd pycuda
     python3 configure.py --cuda-enable-gl --no-use-shipped-boost
     sudo python3 setup.py install

Thanks again and sorry for my lack of understanding of the answers on previous posts!
Best regards