FlightGoggles Renderer node fails on nvidia-docker #46
Comments
|
I can try and help, but unsure if this is a workable option. Could I have a look at the Player.log? |
Thank you! Here it is: docker@58337418fc82:~/catkin_ws$ cat /home/docker/.config/unity3d/MIT-FAST-Lab/FlightGoggles/Player.log (Filename: Line: 634) |
BhavyanshM
changed the title
|
Looks like Vulkan is not able to find a GPU. Could you confirm that you were able to install |
Yes, I think you are right, the reason is probably that the container isn't able to access the GPU: docker@58337418fc82:/$ nvidia-smi Do you see an obvious reason for such a failure in accessing the GPU from inside the container? Works fine on the host. |
|
I remember there being a condition with nvidia-docker that the versions of the nvidia driver on the local machine and inside the container should be the same otherwise it throws this error (see here) |
Wow, I was just looking at the exact same page. Alright, let me fix this. |
So, I am now able to see the docker@58337418fc82:/$ rosrun flightgoggles FlightGoggles.x86_64 /proc/self/maps: Native stacktrace: Debug info from gdb: mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb =================================================================
|
|
Could you please send the player.log? Does |
Sure. Here is the player.log: The |
|
OK, there seem to be two different errors here. Vulkan-smoketest is not picking up your display. Perhaps you need to do some X tunneling/forwarding? You also should run the single-threaded binary for FlightGoggles. |
Ahh, didn't mean to close the issue. Just hit the "Close and comment" button unintentionally. The |
|
No problem! In that case, I would advise getting |
|
|
OK, what about the output of Do you also have vulkan installed in the host OS? |
Yes, Vulkan is installed on the host OS as well. However, here is the output from inside the container for Also, here are the files in case you need to look at them. https://github.com/BhavyanshM/ROSEnv/tree/master/Melodic |
|
On simply running: Container terminal outputs the following and crashes the monitor display.
Here is the output on Player.log from this attempt to launch flightgoggles: [Vulkan init] extensions: count=16
[Vulkan init] extensions: name=VK_EXT_acquire_xlib_display, enabled=0
[Vulkan init] extensions: name=VK_EXT_debug_report, enabled=0
[Vulkan init] extensions: name=VK_EXT_direct_mode_display, enabled=0
[Vulkan init] extensions: name=VK_EXT_display_surface_counter, enabled=0
[Vulkan init] extensions: name=VK_KHR_display, enabled=1
[Vulkan init] extensions: name=VK_KHR_get_physical_device_properties2, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_surface_capabilities2, enabled=0
[Vulkan init] extensions: name=VK_KHR_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_xcb_surface, enabled=0
[Vulkan init] extensions: name=VK_KHR_xlib_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_external_fence_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_memory_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_semaphore_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_device_group_creation, enabled=0
[Vulkan init] extensions: name=VK_KHR_wayland_surface, enabled=1
[Vulkan init] extensions: name=VK_EXT_debug_utils, enabled=0
Vulkan detection: 2
Initialize engine version: 2018.3.0f2 (6e9a27477296)
GfxDevice: creating device client; threaded=1
[Vulkan init] extensions: count=16
[Vulkan init] extensions: name=VK_EXT_acquire_xlib_display, enabled=0
[Vulkan init] extensions: name=VK_EXT_debug_report, enabled=0
[Vulkan init] extensions: name=VK_EXT_direct_mode_display, enabled=0
[Vulkan init] extensions: name=VK_EXT_display_surface_counter, enabled=0
[Vulkan init] extensions: name=VK_KHR_display, enabled=1
[Vulkan init] extensions: name=VK_KHR_get_physical_device_properties2, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_surface_capabilities2, enabled=0
[Vulkan init] extensions: name=VK_KHR_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_xcb_surface, enabled=0
[Vulkan init] extensions: name=VK_KHR_xlib_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_external_fence_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_memory_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_semaphore_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_device_group_creation, enabled=0
[Vulkan init] extensions: name=VK_KHR_wayland_surface, enabled=1
[Vulkan init] extensions: name=VK_EXT_debug_utils, enabled=0
[Vulkan init] extensions: count=46
[Vulkan init] extensions: name=VK_KHR_swapchain, enabled=1
[Vulkan init] extensions: name=VK_KHR_16bit_storage, enabled=0
[Vulkan init] extensions: name=VK_KHR_bind_memory2, enabled=0
[Vulkan init] extensions: name=VK_KHR_dedicated_allocation, enabled=1
[Vulkan init] extensions: name=VK_KHR_descriptor_update_template, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_memory_requirements2, enabled=1
[Vulkan init] extensions: name=VK_KHR_image_format_list, enabled=1
[Vulkan init] extensions: name=VK_KHR_maintenance1, enabled=1
[Vulkan init] extensions: name=VK_KHR_maintenance2, enabled=0
[Vulkan init] extensions: name=VK_KHR_push_descriptor, enabled=0
[Vulkan init] extensions: name=VK_KHR_relaxed_block_layout, enabled=0
[Vulkan init] extensions: name=VK_KHR_sampler_mirror_clamp_to_edge, enabled=1
[Vulkan init] extensions: name=VK_KHR_sampler_ycbcr_conversion, enabled=0
[Vulkan init] extensions: name=VK_KHR_shader_draw_parameters, enabled=0
[Vulkan init] extensions: name=VK_KHR_storage_buffer_storage_class, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_memory, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_memory_fd, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_semaphore, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_semaphore_fd, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_fence, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_fence_fd, enabled=0
[Vulkan init] extensions: name=VK_KHR_variable_pointers, enabled=0
[Vulkan init] extensions: name=VK_KHX_device_group, enabled=0
[Vulkan init] extensions: name=VK_KHX_multiview, enabled=0
[Vulkan init] extensions: name=VK_EXT_blend_operation_advanced, enabled=0
[Vulkan init] extensions: name=VK_EXT_depth_range_unrestricted, enabled=0
[Vulkan init] extensions: name=VK_EXT_discard_rectangles, enabled=0
[Vulkan init] extensions: name=VK_EXT_display_control, enabled=0
[Vulkan init] extensions: name=VK_EXT_post_depth_coverage, enabled=0
[Vulkan init] extensions: name=VK_EXT_sample_locations, enabled=0
[Vulkan init] extensions: name=VK_EXT_sampler_filter_minmax, enabled=0
[Vulkan init] extensions: name=VK_EXT_shader_subgroup_ballot, enabled=0
[Vulkan init] extensions: name=VK_EXT_shader_subgroup_vote, enabled=0
[Vulkan init] extensions: name=VK_EXT_shader_viewport_index_layer, enabled=0
[Vulkan init] extensions: name=VK_NV_dedicated_allocation, enabled=0
[Vulkan init] extensions: name=VK_NV_fill_rectangle, enabled=0
[Vulkan init] extensions: name=VK_NV_fragment_coverage_to_color, enabled=0
[Vulkan init] extensions: name=VK_NV_framebuffer_mixed_samples, enabled=0
[Vulkan init] extensions: name=VK_NV_glsl_shader, enabled=0
[Vulkan init] extensions: name=VK_NV_clip_space_w_scaling, enabled=0
[Vulkan init] extensions: name=VK_NV_sample_mask_override_coverage, enabled=0
[Vulkan init] extensions: name=VK_NV_viewport_array2, enabled=0
[Vulkan init] extensions: name=VK_NV_viewport_swizzle, enabled=0
[Vulkan init] extensions: name=VK_NV_geometry_shader_passthrough, enabled=0
[Vulkan init] extensions: name=VK_NVX_device_generated_commands, enabled=0
[Vulkan init] extensions: name=VK_NVX_multiview_per_view_attributes, enabled=0
Vulkan API version 1.0.65
Vulkan vendor=[NVIDIA] id=[10de]
Vulkan renderer=[GeForce GTX 1070 Ti] id=[1b82]
Vulkan device type 2
Vulkan driverversion=[0x6195c000] uint=[1637203968]
Vulkan PSO: cache data not found [path=/tmp/MIT-FAST-Lab/FlightGoggles/vulkan_pso_cache.bin]
FMOD failed to initialize the output device.: "Error initializing output device. " (60)
(Filename: Line: 1398)
Forced to initialize FMOD to to the device driver's system output rate 48000, this may impact performance and/or give inconsistent experiences compared to selected sample rate 48000
(Filename: Line: 1343)
FMOD failed to initialize the output device.: "Error initializing output device. " (60)
(Filename: Line: 1398)
FMOD failed to initialize any audio devices, running on emulated software output with no sound. Please check your audio drivers and/or hardware for malfunction.
Begin MonoManager ReloadAssembly
- Completed reload, in 0.077 seconds
WARNING: Shader Unsupported: 'HDRenderPipeline/Lit' - Pass 'META' has no vertex shader
Vulkan: Internal error compiling expression: "u_xlat107 = (_EnableLightLayers != uint(0u)) ? unity_RenderingLayer.x : 3.57331108e-43;
u_xlat107!=0"
(Filename: Line: 65)
Vulkan: Internal error compiling expression: "u_xlat107 = (_EnableLightLayers != uint(0u)) ? unity_RenderingLayer.x : 3.57331108e-43;
u_xlat107!=0"
(Filename: Line: 65)
WARNING: Shader Unsupported: 'Hidden/HDRenderPipeline/TerrainLit_Basemap' - Pass 'META' has no vertex shader
WARNING: Shader Unsupported: 'HDRenderPipeline/TerrainLit' - Pass 'META' has no vertex shader
requesting resize 1024 x 768
resizing window to 1024 x 768
Desktop is 3840 x 2160 @ 60 Hz
InitializeOrResetSwapChain 1024x768 samples=1``` |
|
It looks like Unity3d is attempting to access the audio driver, one option is getting pulseaudio to work in docker. We will also try to build you a binary with sound disabled to see if that helps mitigate this issue. |
Looks like it is non-trivial to get pulseaudio to work in docker. I would really appreciate a binary with sound disabled. |
|
Sure, we will try to generate a binary with sound disabled. In the meantime would it be possible to share the Dockerfile so I can also take a quick look at it? |
Oh sure, I had actually posted it at the end of an earlier comment but here are all the files. https://github.com/BhavyanshM/ROSEnv/tree/master/Melodic Thank you so much by the way! |
|
We've just updated the binary. This build should not try to connect to the audio card. Please let us know if it works! @BhavyanshM |
Thank you @Winter-Guerra. So I tried to
Also, the |
|
Did you pull the new changes to the repo before rebuilding? 1 min download
sounds like catkin was not updated and cached your previous version.
```
cd ~/catkin_ws/src &&
wstool update &&
catkin clean &&
catkin build
```
|
So I went ahead and built the entire container all over again with just the Dockerfile and setup ROS packages for FG to remove any ambiguity on the version of FG binary being used. It took about 5 mins to download. On launching using |
|
Hi @BhavyanshM, OK, so it looks like the new Unfortunately, it looks like FlightGoggles I'm terribly sorry for the inconvenience @BhavyanshM. At the moment, it looks like it is not possible to run FlightGoggles If you'd really like to run the FlightGoggles simulation inside of Docker, you could run the FlightGoggles renderer binary outside of Docker in Linux, Windows, or MacOS and point the client IP at the ROS docker image. Currently, we only have the Linux binary published for Best, |
|
Closing due to lack of |
No worries @Winter-Guerra It is unfortunate that |
…r (issue #52). * Makes camera resolution configurable via ROS param files. Alleviates issues from #51 and #32. * Lowers takeoff thrust threshold to 9.9N (see issues #56, #45). * Adds Windows64 standalone build to releases page (requested in issues #53, #46). * Adds nominal gate location file to ROS param server. * Adds plotter for laser range finder. * Adds example stereo reconstruction pipeline launch file. Many thanks to @eatal and @varunmurali1 for their help with this patch!
|
See #131 for a solution. You'll have to script these steps into a Dockerfile. Tested with an Ubuntu-16.04 docker image using nvidia-docker2 and TITAN-V GPU connected to an Ubuntu-18.04 host.
{
"file_format_version" : "1.0.0",
"ICD": {
"library_path": "libGLX_nvidia.so.0",
"api_version" : "1.1.95"
}
}
This will allow vulcaninfo and vulkan-smoketest to run correctly. Running wget http://us.download.nvidia.com/XFree86/Linux-x86_64/418.56/NVIDIA-Linux-x86_64-418.56.run
NVIDIA-Linux-x86_64-418.56.run --extract-only
sudo cp NVIDIA-Linux-x86_64-418.56/libnvidia-glvkspirv.so.418.56 /usr/lib/x86_64-linux-gnu/ |
I understand that nvidia-docker isn't officially supported. However, I have been trying to setup flightGoggles with all its dependencies inside an nvidia-docker container. I have almost everything setup and building just fine on the container, but somehow the flightgogglesRenderer node fails when even simply running "rosrun flightgoggles FlightGoggles.x86_64":
Found path: /home/docker/catkin_ws/devel/.private/flightgoggles/lib/flightgoggles/FlightGoggles.x86_64
Mono path[0] = '/home/docker/catkin_ws/devel/.private/flightgoggles/lib/flightgoggles/FlightGoggles_Data/Managed'
Mono config path = '/home/docker/catkin_ws/devel/.private/flightgoggles/lib/flightgoggles/FlightGoggles_Data/MonoBleedingEdge/etc'
Preloaded 'ScreenSelector.so'
Display 0 'U28E590 27"': 3840x2160 (primary device).
Loading player data from /home/docker/catkin_ws/devel/.private/flightgoggles/lib/flightgoggles/FlightGoggles_Data/data.unity3d
Logging to /home/docker/.config/unity3d/MIT-FAST-Lab/FlightGoggles/Player.log
I would really appreciate any help at all, and then maybe even contribute my setup in form of a pull request. I am sure such a docker container setup will help out a lot of people that are trying hard to get Flight Goggles setup on their workstations. Thank you in advance!
The text was updated successfully, but these errors were encountered: