D435i + Qualcomm RB5 SSH Interrupt

[RuiqiZhang99]

Required Info
Camera Model	D435i
Firmware Version	05.15.01.00
Operating System & Version	Ubuntu 18.04
Kernel Version (Linux Only)	4.19.125
Platform	Quadcomm RB5
SDK Version	v2.50.0
Language	C/C++
Segment	Robot

Issue Description

Hi, there. I'm trying to develop the hardware of autonomous robot with Realsense and Quadcomm RB5. For this, I program on my laptop and establish SSH connection to RB5 with a d435i. My symptom is when I launch the realsense through /home/rb5# roslaunch ./openvins_ws/cfgs/d435i.launch, my SSH interface will get stuck here suddenly.

SUMMARY
========

PARAMETERS
 * /camera/color/image_raw/disable_pub_plugins: ['image_transport...
 * /camera/infra1/image_rect_raw/disable_pub_plugins: ['image_transport...
 * /camera/infra2/image_rect_raw/disable_pub_plugins: ['image_transport...
 * /camera/realsense2_camera/accel_fps: 250
 * /camera/realsense2_camera/accel_frame_id: camera_accel_frame
 * /camera/realsense2_camera/accel_optical_frame_id: camera_accel_opti...
 * /camera/realsense2_camera/align_depth: False
 * /camera/realsense2_camera/aligned_depth_to_color_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye1_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye2_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_infra1_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_infra2_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/allow_no_texture_points: False
 * /camera/realsense2_camera/base_frame_id: camera_link
 * /camera/realsense2_camera/calib_odom_file: 
 * /camera/realsense2_camera/clip_distance: -2.0
 * /camera/realsense2_camera/color_fps: 15
 * /camera/realsense2_camera/color_frame_id: camera_color_frame
 * /camera/realsense2_camera/color_height: 480
 * /camera/realsense2_camera/color_optical_frame_id: camera_color_opti...
 * /camera/realsense2_camera/color_width: 848
 * /camera/realsense2_camera/confidence_fps: 30
 * /camera/realsense2_camera/confidence_height: 480
 * /camera/realsense2_camera/confidence_width: 640
 * /camera/realsense2_camera/depth_fps: 15
 * /camera/realsense2_camera/depth_frame_id: camera_depth_frame
 * /camera/realsense2_camera/depth_height: 480
 * /camera/realsense2_camera/depth_optical_frame_id: camera_depth_opti...
 * /camera/realsense2_camera/depth_width: 848
 * /camera/realsense2_camera/device_type: 
 * /camera/realsense2_camera/enable_accel: True
 * /camera/realsense2_camera/enable_color: False
 * /camera/realsense2_camera/enable_confidence: True
 * /camera/realsense2_camera/enable_depth: False
 * /camera/realsense2_camera/enable_fisheye1: False
 * /camera/realsense2_camera/enable_fisheye2: False
 * /camera/realsense2_camera/enable_fisheye: False
 * /camera/realsense2_camera/enable_gyro: True
 * /camera/realsense2_camera/enable_infra1: True
 * /camera/realsense2_camera/enable_infra2: True
 * /camera/realsense2_camera/enable_infra: False
 * /camera/realsense2_camera/enable_pointcloud: False
 * /camera/realsense2_camera/enable_pose: False
 * /camera/realsense2_camera/enable_sync: False
 * /camera/realsense2_camera/filters: 
 * /camera/realsense2_camera/fisheye1_frame_id: camera_fisheye1_f...
 * /camera/realsense2_camera/fisheye1_optical_frame_id: camera_fisheye1_o...
 * /camera/realsense2_camera/fisheye2_frame_id: camera_fisheye2_f...
 * /camera/realsense2_camera/fisheye2_optical_frame_id: camera_fisheye2_o...
 * /camera/realsense2_camera/fisheye_fps: 15
 * /camera/realsense2_camera/fisheye_frame_id: camera_fisheye_frame
 * /camera/realsense2_camera/fisheye_height: 480
 * /camera/realsense2_camera/fisheye_optical_frame_id: camera_fisheye_op...
 * /camera/realsense2_camera/fisheye_width: 848
 * /camera/realsense2_camera/gyro_fps: 400
 * /camera/realsense2_camera/gyro_frame_id: camera_gyro_frame
 * /camera/realsense2_camera/gyro_optical_frame_id: camera_gyro_optic...
 * /camera/realsense2_camera/imu_optical_frame_id: camera_imu_optica...
 * /camera/realsense2_camera/infra1_frame_id: camera_infra1_frame
 * /camera/realsense2_camera/infra1_optical_frame_id: camera_infra1_opt...
 * /camera/realsense2_camera/infra2_frame_id: camera_infra2_frame
 * /camera/realsense2_camera/infra2_optical_frame_id: camera_infra2_opt...
 * /camera/realsense2_camera/infra_fps: 15
 * /camera/realsense2_camera/infra_height: 480
 * /camera/realsense2_camera/infra_rgb: False
 * /camera/realsense2_camera/infra_width: 848
 * /camera/realsense2_camera/initial_reset: True
 * /camera/realsense2_camera/json_file_path: 
 * /camera/realsense2_camera/linear_accel_cov: 0.01
 * /camera/realsense2_camera/odom_frame_id: camera_odom_frame
 * /camera/realsense2_camera/ordered_pc: False
 * /camera/realsense2_camera/pointcloud_texture_index: 0
 * /camera/realsense2_camera/pointcloud_texture_stream: RS2_STREAM_COLOR
 * /camera/realsense2_camera/pose_frame_id: camera_pose_frame
 * /camera/realsense2_camera/pose_optical_frame_id: camera_pose_optic...
 * /camera/realsense2_camera/publish_odom_tf: True
 * /camera/realsense2_camera/publish_tf: True
 * /camera/realsense2_camera/reconnect_timeout: 6.0
 * /camera/realsense2_camera/rosbag_filename: 
 * /camera/realsense2_camera/serial_no: 
 * /camera/realsense2_camera/stereo_module/exposure/1: 7500
 * /camera/realsense2_camera/stereo_module/exposure/2: 1
 * /camera/realsense2_camera/stereo_module/gain/1: 16
 * /camera/realsense2_camera/stereo_module/gain/2: 16
 * /camera/realsense2_camera/tf_publish_rate: 0.0
 * /camera/realsense2_camera/topic_odom_in: odom_in
 * /camera/realsense2_camera/unite_imu_method: linear_interpolation
 * /camera/realsense2_camera/usb_port_id: 
 * /camera/realsense2_camera/wait_for_device_timeout: -1.0
 * /camera/stereo_module/emitter_enabled: 0
 * /rosdistro: melodic
 * /rosversion: 1.14.13

NODES
  /camera/
    realsense2_camera (nodelet/nodelet)
    realsense2_camera_manager (nodelet/nodelet)

auto-starting new master
process[master]: started with pid [23766]
ROS_MASTER_URI=http://192.168.1.201:11311

setting /run_id to be1736fe-bfbc-11ef-acf4-00037f121d92
process[rosout-1]: started with pid [23777]
started core service [/rosout]
process[camera/realsense2_camera_manager-2]: started with pid [23784]
process[camera/realsense2_camera-3]: started with pid [23785]
[ INFO] [1734800285.795470986]: Initializing nodelet with 8 worker threads.
[ INFO] [1734800286.034728835]: RealSense ROS v2.3.2
[ INFO] [1734800286.034798520]: Built with LibRealSense v2.50.0
[ INFO] [1734800286.034816592]: Running with LibRealSense v2.50.0
[ INFO] [1734800286.131996499]:  

And then there is no response no matter how I operate the SSH interface. However, if I unplug the realsense directly, the SSH connection will recover immediately and these information is printed in the terminal:

[ INFO] [1734800293.007275863]:  
[ INFO] [1734800293.557777784]: Device with serial number 902512070799 was found.

[ INFO] [1734800293.558115482]: Device with physical ID 2-1.2-13 was found.
[ INFO] [1734800293.558181209]: Device with name Intel RealSense D435I was found.
[ INFO] [1734800293.559130138]: Device with port number 2-1.2 was found.
[ INFO] [1734800293.559234771]: Device USB type: 3.2
[ INFO] [1734800293.565443276]: getParameters...
[ INFO] [1734800293.635053882]: setupDevice...
[ INFO] [1734800293.635128411]: JSON file is not provided
[ INFO] [1734800293.635163775]: ROS Node Namespace: camera
[ INFO] [1734800293.635179608]: Device Name: Intel RealSense D435I
[ INFO] [1734800293.635198045]: Device Serial No: 902512070799
[ INFO] [1734800293.635211951]: Device physical port: 2-1.2-13
[ INFO] [1734800293.635225960]: Device FW version: 05.15.01.00
[ INFO] [1734800293.635244137]: Device Product ID: 0x0B3A
[ INFO] [1734800293.635268667]: Enable PointCloud: Off
[ INFO] [1734800293.635294813]: Align Depth: Off
[ INFO] [1734800293.635308302]: Sync Mode: Off
[ INFO] [1734800293.635356634]: Device Sensors: 
[ INFO] [1734800293.825806260]: Stereo Module was found.
[ INFO] [1734800293.974558636]: RGB Camera was found.
[ INFO] [1734800293.975066433]: Motion Module was found.
[ INFO] [1734800293.975296478]: (Confidence, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1734800293.975436421]: num_filters: 0
[ INFO] [1734800293.975544074]: Setting Dynamic reconfig parameters.
hwmon command 0x80( 5 0 0 0 ) failed (response -7= HW not ready)
hwmon command 0x80( 5 0 0 0 ) failed (response -7= HW not ready)
hwmon command 0x80( 5 0 0 0 ) failed (response -7= HW not ready)
hwmon command 0x80( 5 0 0 0 ) failed (response -7= HW not ready)
[ INFO] [1734800304.984651952]: Done Setting Dynamic reconfig parameters.
[ INFO] [1734800304.989569092]: infra1 stream is enabled - width: 848, height: 480, fps: 15, Format: Y8
[ INFO] [1734800304.992343225]: infra2 stream is enabled - width: 848, height: 480, fps: 15, Format: Y8
[ INFO] [1734800304.992758785]: gyro stream is enabled - fps: 400
[ INFO] [1734800304.995823586]: accel stream is enabled - fps: 250
[ INFO] [1734800304.996838451]: setupPublishers...
[ INFO] [1734800305.006692990]: Expected frequency for infra1 = 15.00000
[ INFO] [1734800305.011815540]: Expected frequency for infra2 = 15.00000
[ INFO] [1734800305.014034690]: Start publisher IMU
[ INFO] [1734800305.014696961]: setupStreams...
 21/12 16:58:25,497 WARNING [547598512512] (ds5-motion.cpp:473) IMU Calibration is not available, default intrinsic and extrinsic will be used.
[ WARN] [1734800305.553113448]: 
[ WARN] [1734800305.553393595]: frame's time domain is HARDWARE_CLOCK. Timestamps may reset periodically.
 21/12 16:58:25,648 WARNING [547163709824] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[ INFO] [1734800305.699804488]: SELECTED BASE:Depth, 0
[ INFO] [1734800305.713945665]: RealSense Node Is Up!
 21/12 16:58:25,861 WARNING [547163709824] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
 21/12 16:58:25,912 WARNING [547163709824] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
 21/12 16:58:25,963 WARNING [547163709824] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
 21/12 16:58:26,230 WARNING [547163709824] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
 21/12 16:58:26,389 WARNING [547163709824] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
 21/12 16:58:26,969 WARNING [547163709824] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
 21/12 16:59:02,333 WARNING [547163709824] (messenger-libusb.cpp:92) usb_request_queue returned error, endpoint: 135 error: No such device, number: 19

(It prints No such device since I unplug the camera, please ignore this one.)
If I run the command dmesg -w in another ssh terminal, it tells the following infomation interatively. Since the SSH connection is interrupted for some reason, this information is also printed right after the unplugging.

[15234.966955] cnss: Runtime suspend start
[15234.967276] [kworke][0x44232a4d43][16:34:54.013214] wlan: [19094:I:DP] dp_runtime_suspend: 9391: Abort suspend due to pending TX packets
[15234.967501] cnss: Runtime suspend status: -11
[15234.991962] uvcvideo: Unable to create debugfs 2-9 directory.
[15234.997467] uvcvideo 2-1.3:1.0: Entity type for entity Intel(R) RealSense(TM) Depth Ca was not initialized!
[15234.997519] uvcvideo 2-1.3:1.0: Entity type for entity Processing 2 was not initialized!
[15234.997563] uvcvideo 2-1.3:1.0: Entity type for entity Camera 1 was not initialized!
[15235.004093] input: Intel(R) RealSense(TM) Depth Ca as /devices/platform/soc/a800000.ssusb/a800000.dwc3/xhci-hcd.0.auto/usb2/2-1/2-1.3/2-1.3:1.0/input/input198
[15235.042630] uvcvideo: Unknown video format 00000050-0000-0010-8000-00aa00389b71
[15235.044253] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 435i (8086:0b3a)
[15235.151819] uvcvideo: Unable to create debugfs 2-9 directory.
[15235.160550] uvcvideo 2-1.3:1.0: Entity type for entity Intel(R) RealSense(TM) Depth Ca was not initialized!
[15235.160619] uvcvideo 2-1.3:1.0: Entity type for entity Processing 2 was not initialized!
[15235.160678] uvcvideo 2-1.3:1.0: Entity type for entity Camera 1 was not initialized!
[15235.169492] input: Intel(R) RealSense(TM) Depth Ca as /devices/platform/soc/a800000.ssusb/a800000.dwc3/xhci-hcd.0.auto/usb2/2-1/2-1.3/2-1.3:1.0/input/input199
[15235.203442] uvcvideo: Unknown video format 00000050-0000-0010-8000-00aa00389b71
[15235.203958] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 435i (8086:0b3a)
[15235.314511] uvcvideo: Unable to create debugfs 2-9 directory.
[15235.320018] uvcvideo 2-1.3:1.0: Entity type for entity Intel(R) RealSense(TM) Depth Ca was not initialized!
[15235.320087] uvcvideo 2-1.3:1.0: Entity type for entity Processing 2 was not initialized!
[15235.320146] uvcvideo 2-1.3:1.0: Entity type for entity Camera 1 was not initialized!
[15235.327685] input: Intel(R) RealSense(TM) Depth Ca as /devices/platform/soc/a800000.ssusb/a800000.dwc3/xhci-hcd.0.auto/usb2/2-1/2-1.3/2-1.3:1.0/input/input200
[15235.375869] uvcvideo: Unknown video format 00000050-0000-0010-8000-00aa00389b71
[15235.376596] uvcvideo: Found UVC 1.50 device Intel(R) RealSense(TM) Depth Camera 435i (8086:0b3a)

And normally, it will only print:

[15225.520126] [kworke][0x44185a94ab][16:34:44.566620] wlan: [19094:I:PMO] pmo_core_is_wow_applicable: 319: lpass enabled, enabling wow
[15225.520491] [kworke][0x44185ab040][16:34:44.566988] wlan: [19094:I:PMO] pmo_core_enable_wow_in_fw: 799: drv wow is enabled
[15225.520673] [kworke][0x44185ab924][16:34:44.567106] wlan: [19094:I:WMI] suspend type: WOW_IFACE_PAUSE_ENABLED
[15225.569855] cnss: Setting MHI state: SUSPEND(6)
[15225.569970] cnss: Suspending PCI link
[15225.570020] cnss: Use PCIe DRV suspend
[15225.576071] cnss: Runtime suspend status: 0
[15225.576190] cnss: Runtime resume start
[15225.576479] cnss: Resuming PCI link
[15225.579403] cnss: Set PCI link status to: 2
[15225.579493] cnss: Setting MHI state: RESUME(7)
[15225.608017] cnss: Runtime resume status: 0
[15225.608174] cnss: Runtime idle

I think the main reason for SSH problem is from dp_runtime_suspend: 9391: Abort suspend due to pending TX packets but I have no clue about it. I would appreciate it if you can provide some possible solutions.

[MartyG-RealSense]

Hi @RuiqiZhang99 There are Qualcomm RB5 installation instructions for RealSense D435i at the link below. Are the procedures recommended at that link similar to how you set up your host computer and RB5, please?

https://www.macnica.co.jp/en/business/semiconductor/articles/qualcomm/135331/