[tsdf_node-3] process has died [pid 8271, exit code -11

[WuRobotics]

Info of my PC:
Operating system version: Ubuntu 16.04.7
ROS version: ROS Kinetic
Graphics card type: RTX 3070 (Desktop version) (Core i7 10700)(16GB ram)
Nvidia driver version: 470.63.01
CUDA version: V11.1.74
cmake version: Cmake 3.11

---

Issue description:
Hi, I'm trying to run the Yak_ros demo on my newly purchased PC with RTX 3070 GPU.
According to the ReadMe file of "Yak" and "Yak_ros", the workspace was successfully compiled with some warnings.
I source the "setup.bash" file in the "devel" folder and tried to run the demo, I got the error of : [tsdf_node-3] process has died

roslaunch yak_ros demo.launch:

Nvidia driver version:

Cuda version:

Catkin build output

---

My attempts to solve the problem
Attempt 1:
As shown in the last image, there are some warnings related to CUDA, I'm wondering maybe the CUDA 11.1 is not compatible with the Yak, or some functions in CUDA 11.1 are different from the lower version CUDA.
So I tried to downgrade the CUDA version to CUDA 9.0. After installation the CUDA 9.0

I found that the RTX 3070 is not compatible with CUDA 9.0, the Yak workspace can not be generated with Catkin build
I tried to solve the problem with: export TORCH_CUDA_ARCH_LIST="8.0"
According to https://github.com/open-mmlab/OpenPCDet/issues/591, but still no luck.

Attempt 2:
I tried to downgrade the GPU driver version and CUDA together, but still no luck.

---

Other info
The PCL version is 1.9.1, compiled from source.
The Cmake version is 3.11, compiled from source.
Other dependencies related withYak and Yak_ros ( the gl_depth_sim package, ros_industrial_cmake_boilerplate ) are working normally.

Could you please help me to check the problem and give me a hint?
Many thanks for your help!

@schornakj @gavanderhoorn @gachiemchiep

[WuRobotics]

The Catkin build output is shown below in script format:

`robotking@robotking-XPS-8940:~/yak_ros_test$ catkin build

Profile: default
Extending: [env] /opt/ros/kinetic
Workspace: /home/robotking/yak_ros_test

Build Space: [exists] /home/robotking/yak_ros_test/build
Devel Space: [exists] /home/robotking/yak_ros_test/devel
Install Space: [unused] /home/robotking/yak_ros_test/install
Log Space: [missing] /home/robotking/yak_ros_test/logs
Source Space: [exists] /home/robotking/yak_ros_test/src
DESTDIR: [unused] None

Devel Space Layout: linked
Install Space Layout: None

Additional CMake Args: None
Additional Make Args: None
Additional catkin Make Args: None
Internal Make Job Server: True
Cache Job Environments: False

Whitelisted Packages: None
Blacklisted Packages: None

Workspace configuration appears valid.

[build] Found '4' packages in 0.0 seconds.
[build] Updating package table.
Warning: generated devel space setup files have been deleted.
Starting >>> catkin_tools_prebuild
Finished <<< catkin_tools_prebuild [ 1.4 seconds ]
Starting >>> gl_depth_sim
Starting >>> ros_industrial_cmake_boilerplate
Finished <<< ros_industrial_cmake_boilerplate [ 0.6 seconds ]
Starting >>> yak
Starting >>> yak_ext
Finished <<< yak_ext [ 0.6 seconds ]

---

Warnings << yak:make /home/robotking/yak_ros_test/logs/yak/build.make.000.log
:0:20: warning: missing whitespace after the macro name
:0:20: warning: missing whitespace after the macro name
:0:20: warning: missing whitespace after the macro name
:0:20: warning: missing whitespace after the macro name
:0:20: warning: missing whitespace after the macro name
:0:20: warning: missing whitespace after the macro name
:0:20: warning: missing whitespace after the macro name
:0:20: warning: missing whitespace after the macro name
:0:20: warning: missing whitespace after the macro name
:0:20: warning: missing whitespace after the macro name
:0:20: warning: missing whitespace after the macro name
:0:20: warning: missing whitespace after the macro name
/home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp: In destructor ‘kfusion::cuda::TextureBinder::~TextureBinder()’:
/home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp:49:75: warning: ‘cudaError_t cudaUnbindTexture(const textureReference*)’ is deprecated [-Wdeprecated-declarations]
/usr/local/cuda-11.1/include/cuda_runtime_api.h:8060:46: note: declared here
extern __CUDA_DEPRECATED host cudaError_t CUDARTAPI cudaUnbindTexture(const struct textureReference texref);
^
/home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp: In instantiation of ‘kfusion::cuda::TextureBinder::TextureBinder(const kfusion::cuda::DeviceArray2D&, const texture<T, 2, readMode>&) [with T = short unsigned int; cudaTextureReadMode readMode = (cudaTextureReadMode)0u]’:
/home/robotking/yak_ros_test/src/yak/yak/src/cuda/proj_icp.cu:395:50: required from here
/home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp:18:49: warning: ‘cudaError_t cudaBindTexture2D(size_t, const texture<T, dim, readMode>&, const void*, const cudaChannelFormatDesc&, size_t, size_t, size_t) [with T = short unsigned int; int dim = 2; cudaTextureReadMode readMode = (cudaTextureReadMode)0u; cudaError_t = cudaError; size_t = long unsigned int]’ is deprecated [-Wdeprecated-declarations]
cudaSafeCall(cudaBindTexture2D(0, tex, arr.ptr(), desc, arr.cols(), arr.rows(), arr.step()));
^
/usr/local/cuda-11.1/include/cuda_runtime.h:1396:53: note: declared here
static __CUDA_DEPRECATED inline host cudaError_t cudaBindTexture2D(
^
/home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp: In instantiation of ‘kfusion::cuda::TextureBinder::TextureBinder(const kfusion::cuda::DeviceArray2D&, const texture<T, 2, readMode>&) [with T = float4; cudaTextureReadMode readMode = (cudaTextureReadMode)0u]’:
/home/robotking/yak_ros_test/src/yak/yak/src/cuda/proj_icp.cu:396:50: required from here
/home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp:18:49: warning: ‘cudaError_t cudaBindTexture2D(size_t*, const texture<T, dim, readMode>&, const void*, const cudaChannelFormatDesc&, size_t, size_t, size_t) [with T = float4; int dim = 2; cudaTextureReadMode readMode = (cudaTextureReadMode)0u; cudaError_t = cudaError; size_t = long unsigned int]’ is deprecated [-Wdeprecated-declarations]
cudaSafeCall(cudaBindTexture2D(0, tex, arr.ptr(), desc, arr.cols(), arr.rows(), arr.step()));
^
/usr/local/cuda-11.1/include/cuda_runtime.h:1396:53: note: declared here
static __CUDA_DEPRECATED inline host cudaError_t cudaBindTexture2D(
^
/usr/local/cuda-11.1/include/cuda_runtime.h: In instantiation of ‘cudaError_t cudaBindTexture2D(size_t*, const texture<T, dim, readMode>&, const void*, const cudaChannelFormatDesc&, size_t, size_t, size_t) [with T = short unsigned int; int dim = 2; cudaTextureReadMode readMode = (cudaTextureReadMode)0u; cudaError_t = cudaError; size_t = long unsigned int]’:
/home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp:18:49: required from ‘kfusion::cuda::TextureBinder::TextureBinder(const kfusion::cuda::DeviceArray2D&, const texture<T, 2, readMode>&) [with T = short unsigned int; cudaTextureReadMode readMode = (cudaTextureReadMode)0u]’
/home/robotking/yak_ros_test/src/yak/yak/src/cuda/proj_icp.cu:395:50: required from here
/usr/local/cuda-11.1/include/cuda_runtime.h:1406:27: warning: ‘cudaError_t cudaBindTexture2D(size_t*, const textureReference*, const void*, const cudaChannelFormatDesc*, size_t, size_t, size_t)’ is deprecated [-Wdeprecated-declarations]
return ::cudaBindTexture2D(offset, &tex, devPtr, &desc, width, height, pitch);
^
/usr/local/cuda-11.1/include/cuda_runtime_api.h:7956:46: note: declared here
extern __CUDA_DEPRECATED host cudaError_t CUDARTAPI cudaBindTexture2D(size_t offset, const struct textureReference texref, const void devPtr, const struct cudaChannelFormatDesc desc, size_t width, size_t height, size_t pitch);
^
/usr/local/cuda-11.1/include/cuda_runtime.h: In instantiation of ‘cudaError_t cudaBindTexture2D(size_t, const texture<T, dim, readMode>&, const void, const cudaChannelFormatDesc&, size_t, size_t, size_t) [with T = float4; int dim = 2; cudaTextureReadMode readMode = (cudaTextureReadMode)0u; cudaError_t = cudaError; size_t = long unsigned int]’:
/home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp:18:49: required from ‘kfusion::cuda::TextureBinder::TextureBinder(const kfusion::cuda::DeviceArray2D&, const texture<T, 2, readMode>&) [with T = float4; cudaTextureReadMode readMode = (cudaTextureReadMode)0u]’
/home/robotking/yak_ros_test/src/yak/yak/src/cuda/proj_icp.cu:396:50: required from here
/usr/local/cuda-11.1/include/cuda_runtime.h:1406:27: warning: ‘cudaError_t cudaBindTexture2D(size_t, const textureReference, const void*, const cudaChannelFormatDesc*, size_t, size_t, size_t)’ is deprecated [-Wdeprecated-declarations]
return ::cudaBindTexture2D(offset, &tex, devPtr, &desc, width, height, pitch);
^
/usr/local/cuda-11.1/include/cuda_runtime_api.h:7956:46: note: declared here
extern __CUDA_DEPRECATED host cudaError_t CUDARTAPI cudaBindTexture2D(size_t offset, const struct textureReference texref, const void devPtr, const struct cudaChannelFormatDesc desc, size_t width, size_t height, size_t pitch);
^
/home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp: In destructor ‘kfusion::cuda::TextureBinder::~TextureBinder()’:
/home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp:49:75: warning: ‘cudaError_t cudaUnbindTexture(const textureReference)’ is deprecated [-Wdeprecated-declarations]
/usr/local/cuda-11.1/include/cuda_runtime_api.h:8060:46: note: declared here
extern __CUDA_DEPRECATED host cudaError_t CUDARTAPI cudaUnbindTexture(const struct textureReference texref);
^
/home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp: In instantiation of ‘kfusion::cuda::TextureBinder::TextureBinder(const A&, const texture<T, 2, readMode>&, const cudaChannelFormatDesc&) [with A = kfusion::cuda::PtrStepSz; T = float; cudaTextureReadMode readMode = (cudaTextureReadMode)0u]’:
/home/robotking/yak_ros_test/src/yak/yak/src/cuda/tsdf_volume.cu:131:73: required from here
/home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp:39:49: warning: ‘cudaError_t cudaBindTexture2D(size_t, const texture<T, dim, readMode>&, const void, const cudaChannelFormatDesc&, size_t, size_t, size_t) [with T = float; int dim = 2; cudaTextureReadMode readMode = (cudaTextureReadMode)0u; cudaError_t = cudaError; size_t = long unsigned int]’ is deprecated [-Wdeprecated-declarations]
cudaSafeCall(cudaBindTexture2D(0, tex, arr.data, desc, arr.cols, arr.rows, arr.step));
^
/usr/local/cuda-11.1/include/cuda_runtime.h:1396:53: note: declared here
static __CUDA_DEPRECATED inline host cudaError_t cudaBindTexture2D(
^
/usr/local/cuda-11.1/include/cuda_runtime.h: In instantiation of ‘cudaError_t cudaBindTexture2D(size_t, const texture<T, dim, readMode>&, const void, const cudaChannelFormatDesc&, size_t, size_t, size_t) [with T = float; int dim = 2; cudaTextureReadMode readMode = (cudaTextureReadMode)0u; cudaError_t = cudaError; size_t = long unsigned int]’:
/home/robotking/yak_ros_test/src/yak/yak/include/yak/kfusion/cuda/texture_binder.hpp:39:49: required from ‘kfusion::cuda::TextureBinder::TextureBinder(const A&, const texture<T, 2, readMode>&, const cudaChannelFormatDesc&) [with A = kfusion::cuda::PtrStepSz; T = float; cudaTextureReadMode readMode = (cudaTextureReadMode)0u]’
/home/robotking/yak_ros_test/src/yak/yak/src/cuda/tsdf_volume.cu:131:73: required from here
/usr/local/cuda-11.1/include/cuda_runtime.h:1406:27: warning: ‘cudaError_t cudaBindTexture2D(size_t*, const textureReference*, const void*, const cudaChannelFormatDesc*, size_t, size_t, size_t)’ is deprecated [-Wdeprecated-declarations]
return ::cudaBindTexture2D(offset, &tex, devPtr, &desc, width, height, pitch);
^
/usr/local/cuda-11.1/include/cuda_runtime_api.h:7956:46: note: declared here
extern __CUDA_DEPRECATED host cudaError_t CUDARTAPI cudaBindTexture2D(size_t *offset, const struct textureReference *texref, const void *devPtr, const struct cudaChannelFormatDesc *desc, size_t width, size_t height, size_t pitch);
^
cd /home/robotking/yak_ros_test/build/yak; catkin build --get-env yak | catkin env -si /usr/bin/make --jobserver-fds=6,7 -j; cd -
...............................................................................
Finished <<< yak [ 17.4 seconds ]

---

Warnings << gl_depth_sim:make /home/robotking/yak_ros_test/logs/gl_depth_sim/build.make.000.log
cc1: warning: command line option ‘-std=c++14’ is valid for C++/ObjC++ but not for C
cd /home/robotking/yak_ros_test/build/gl_depth_sim; catkin build --get-env gl_depth_sim | catkin env -si /usr/bin/make --jobserver-fds=6,7 -j; cd -
...............................................................................
Finished <<< gl_depth_sim [ 19.5 seconds ]
[build] Summary: All 5 packages succeeded!
[build] Ignored: None.
[build] Warnings: 2 packages succeeded with warnings.
[build] Abandoned: None.
[build] Failed: None.
[build] Runtime: 20.9 seconds total. `

[WuRobotics]

Hello @schornakj,

I tried to re-install the Ubuntu16.04 & CUDA & Nvidia driver multiple times with different versions but got no luck...
Could you please help me to check this issue?

Thank you so much.

[WuRobotics]

Hi, @schornakj

I tried to use the Laptop with GTX 1050 GPU and accessories Nvidia Driver and CUDA, the Yak_ros demo is working normally.
But the latest RTX 3070 still got no luck...

Since I hope to scan the object with high resolution, The GTX 1050 seems not good enough, could you please check the issue?

Thank you so much.

[schornakj]

Sorry for being quiet on this issue. I don't have hardware available to test newer GPUs myself, so I probably won't be able to directly help you resolve this issue.

I do not think there are many users of this project running on CUDA 11, so there could be changes to that library since CUDA 10 that could introduce some problems. A similar issue happened during the switch from CUDA 8 to CUDA 9.

If you're able to install Ubuntu 20.04 LTS on the computer with the RTX 3070 installed, I would be interested to know if that gives better results. Ubuntu 16.04 is quite old at this point, and newer versions of CUDA have some requirements about availability of certain versions of the GCC compiler and so on that can cause problems when trying to build these projects.