Astra Mini crashing when asked for depth images. ROS Noetic, Ubuntu 20.04, Jetson Xavier NX

[67bug]

We upgraded our ROS installation to Noetic (on the Jetson Xavier NX) and the depth stream is no longer working. The color stream works fine, but the depth does not. The binary crashes with absolutely no errors reported.

I see multiple issues talking about this:
#178
#179
#188

but there is no fix in sight. I have reached out directly to Orbbec as well and there has been no useful response. I have tried the master branch, the fix_MX400_conneted_bug branch...none of them works.

[jian-dong]

I'm sorry I didn't reproduce it. Can I have your OpenCV version? Because I believe that cv_bridge doesn't work well. Most likely, this line of code

auto image_msg =
      *(cv_bridge::CvImage(std_msgs::Header(), encoding_.at(stream_index),
                           (stream_index == DEPTH && depth_scale_ > 1) ? scaled_image : image)
            .toImageMsg());

Cause crash. Or there have any warning when you run catkin_make?

[jian-dong]

I have tested many cameras in fix_MX400_conneted_bug, but didn't reproduce this crash issue. Would you mind comment this line

auto image_msg =
      *(cv_bridge::CvImage(std_msgs::Header(), encoding_.at(stream_index),
                           (stream_index == DEPTH && depth_scale_ > 1) ? scaled_image : image)
            .toImageMsg());

in src/ob_camera_node.cpp line 610, and test whether it will crash again.

[jian-dong]

And I also need firmware_version, and you can run

 rosservice call /camera/get_version "{}"

To get the firmware version.

[67bug]

Thanks for following up. Here are some answers.

To get the firmware version

data: "{\n "firmware_version": "RD2414",\n "openni_version": "Standard 2.3.0 (Build
\ 85)-Linux-Arm (Jul 6 2023 04:44:47)",\n "ros_sdk_version": "1.2.3"\n}"

Can I have your OpenCV version?

4.5.4-8-g3e4c170df4

[67bug]

I think i have a root cause:

you are correct @jian-dong that the issue is tied to cv::resize (i think this is what you meant with line 610). Why it is failing is beyond me.

I noticed that in the past few days that you have pushed a couple of changes ac78d01 and b2b69db to ob_camera_node.cpp -- which frankly don't add up -- why is there a a dependency on depth_scale_? When is depth_scale_ used? I don't use it on our side. So with depth_scale_ essentially being 1 always, we are circumventing these lines:

if (stream_index == DEPTH && depth_scale_ > 1) {
    cv::resize(image, scaled_image, cv::Size(width * depth_scale_, height * depth_scale_), 0, 0,
               cv::INTER_NEAREST);
  }

which is good to get us going, but is bad because if depth_scale_ is ever set ot be greater than 1, the orbbec camera node will crash.

The code I had was cloned (and raised this issue against) was dated July 1, which has this bug. By adding the && depth_scale_ > 1 in the code, we have essentially hidden a latent issue.

Why cv::resize is not working is not clear to me. The code is correct. It is not a memory issue, it is not am initialization issue either. No cv::Exceptions or std::exceptions are thrown either. A run thrugh valgrind does not show an issue either. We use cv::resize in many other parts for other cameras on our product and don't have any issue with OpenCV 4.5.4.

As to the fix_MX400_conneted_bug branch, this issue is still there (link)

[jian-dong]

@67bug Thanks for your information; I had this problem on an RPI32 not long ago, so I changed this if condition.
Question about why the depth_scale_ is there, because, for Astra Mini, we have depth 320x240 aligned to color 640x480.
But the color point cloud converter only accepts the same resolution for depth and color.

So I have the last question if I change this line to

if (stream_index == DEPTH && depth_scale_ > 1) {
    cv::resize(image, scaled_image, cv::Size(width * depth_scale_, height * depth_scale_), 0, 0,
               cv::INTER_NEAREST);
  }

It's work for you?

[67bug]

for Astra Mini, we have depth 320x240 aligned to color 640x480.

I see...makes sense. We anyways use it in 320x240 mode for RGB and depth so this is not an issue for us

So I have the last question if I change this line to

if (stream_index == DEPTH && depth_scale_ > 1) {
cv::resize(image, scaled_image, cv::Size(width * depth_scale_, height * depth_scale_), 0, 0,
? cv::INTER_NEAREST);
}
It's work for you?

Yes, this works. I have to admit tough that it feels like a hack because we cannot understand why cv::resize is not working

[jian-dong]

Yes, this works. I have to admit tough that it feels like a hack because we cannot understand why cv::resize is not working

Thanks for your reply, I will dig this.

[BrendanWooAdmetal]

@67bug Thanks for your information; I had this problem on an RPI32 not long ago, so I changed this if condition. Question about why the depth_scale_ is there, because, for Astra Mini, we have depth 320x240 aligned to color 640x480. But the color point cloud converter only accepts the same resolution for depth and color.

So I have the last question if I change this line to

if (stream_index == DEPTH && depth_scale_ > 1) {
    cv::resize(image, scaled_image, cv::Size(width * depth_scale_, height * depth_scale_), 0, 0,
               cv::INTER_NEAREST);
  }

It's work for you?

I have encountered the same issue.
I see a fix is suggested here, and it worked for someone else, but to me the suggested code for the fix looks identical to the original code.
Can you clarify the fix in the code suggested here?

[BrendanWooAdmetal]

A fix has been found for my issue.
The problem was with cv_bridge versions.
ROS has cv_bridge included so it can work with images without OpenCV installed.

We also had OpenCV on our machine, and the two versions of cv_bridge were causing conflicts.
The solution was to build the vision_opencv package (which includes cv_bridge) in catkin_ws/src, so that ROS will use this version of cv_bridge consistently.

cd catkin_ws/src
git clone https://github.com/ros-perception/vision_opencv.git
cd vision_opencv
git checkout noetic
cd catkin_ws
catkin_make