Multi Astra Mini Pro support on Debian-arm64

[RaphKay-BR]

Dear Orbbec developers,
I have reached an issue about frame set checking always goes timeout with multiple Astra Mini Pro.
It always should not work when in this condition.

• 3 same firmware ( RD1007 ) flashed Astra Mini Pro on external powered USB3.0 HUB.
• SDK = Release tagged version 1.10.8 for ARM64
• Testing on RK3588 (RK3399), Debian 11 aarch64, G++ version 10.2.1
• Trying to get any frame data with multiple pipeline as explained in your example of SDK.
• Here is main() to start pthread with multiple pipeline.

vector< shared_ptr<ob::Pipeline> > o_pipes;

....

int main( int argc, char** argv )
{
    // prints SDK version 
    printf( "SDK version check : %u.%u.%u\n",
            Version::getMajor(),
            Version::getMinor(),
            Version::getPatch() );

    // Orbbec Context ..
    Context ctx;
    auto devList = ctx.queryDeviceList();
    size_t devCnt = devList->deviceCount();

    // Prevent to over counting. test device is max 3.
    if ( devCnt > 3 ) devCnt = 3;

    for( size_t cnt=0; cnt<devCnt; cnt++ )
    {
        auto dev = devList->getDevice( cnt );
        if ( dev != nullptr )
        {
            auto devInfo = dev->getDeviceInfo();
            if ( devInfo != nullptr )
            {
                snprintf( renderBoxTitle[cnt], 256,
                          "%zu\n%s\n%s",
                          cnt,
                          devInfo->name(),
                          devInfo->serialNumber() );
                // renderbox is FLTK widget, Fl_Box
                renderBox[cnt]->label( renderBoxTitle[cnt] );

                auto loc_pipe = make_shared< ob::Pipeline >(dev);
                o_pipes.push_back( loc_pipe );
            }
        }
    }

    size_t* idxs = nullptr;

    if ( devCnt > 0 )
    {
        // Start Stream ...
        #pragma omp parallel for
        for( size_t cnt=0;cnt<devCnt;cnt++ )
        {
            shared_ptr<ob::Config> cfg = make_shared<ob::Config>();
            auto depthPL = o_pipes[cnt]\
                           ->getStreamProfileList(OB_SENSOR_DEPTH);
            shared_ptr<ob::VideoStreamProfile> depthProf = nullptr;
            if ( depthPL != nullptr )
            {
                depthProf = const_pointer_cast<ob::StreamProfile> \
                            (depthPL->getProfile(OB_PROFILE_DEFAULT)) \
                            ->as<ob::VideoStreamProfile>();
                cfg->enableStream( depthProf );
                printf( "device [%zu] depth Profile enabled.\n", cnt );
            }
            else
            {
                fprintf( stderr,
                         "device [%zu] profile not found.\n", cnt );
            }

            fflush( stdout );

            auto colPL = o_pipes[cnt]->getStreamProfileList(OB_SENSOR_COLOR);
            shared_ptr<ob::VideoStreamProfile> colProf = nullptr;
            if ( colPL != nullptr )
            {
                colProf = const_pointer_cast<ob::StreamProfile> \
                          (colPL->getProfile(OB_PROFILE_DEFAULT)) \
                            ->as<ob::VideoStreamProfile>();
                cfg->enableStream( colProf );
                printf( "device [%zu] color Profile enabled.\n", cnt );
            }
            else
            {
                fprintf( stderr,
                         "device [%zu] color has issue.\n", cnt );
            }

            fflush( stdout );

            o_pipes[cnt]->start( cfg );
            printf( "pipe[ %zu ] started\n", cnt );

            fflush( stdout );
        }

        idxs = new size_t[devCnt];

        #pragma omp parallel for
        for (size_t cnt=0; cnt<devCnt; cnt++ )
        {
            idxs[cnt] = cnt;

            pthread_create( &ptt[cnt], nullptr, ptloop, &idxs[cnt] );
        }
    }

• Each device detects as well and runs each pthread as well.
• But checking frameset is always fail in pthread loop.
• Here's a difference - Single camera works well - but 2 or 3 cameras always failure.

void* ptloop( void* p )
{
    pthread_mutex_lock( &threadsmtx );
    threads++;
    pthread_mutex_unlock( &threadsmtx );

    size_t devidx = 0;

    if ( p != nullptr )
    {
        devidx = *(size_t*)p;
    }

    printf( "thread [%zu] being started.\n", devidx );
    fflush( stdout );

    // wait for a sec for window exposed.
    usleep( 1000000 );

    while( threading == true )
    {
        pthread_mutex_lock( &thrdmtx[devidx] );
        auto frameSet = o_pipes[devidx]->waitForFrames(100);
        pthread_mutex_unlock( &thrdmtx[devidx] );

        if ( frameSet != nullptr )
        {
            pthread_mutex_lock( &thrdmtx[devidx] );

            auto colFrm = frameSet->colorFrame();
            if ( colFrm != nullptr )
            {
                auto colIdx = colFrm->index();
#ifdef DEBUG_FRAME_NUMBER
                printf( "color frame [%zu] = %zu\n", devidx, colIdx );
#endif // of DEBUG_FRAME_NUMBER
                decodeFrame2RenderBox( colFrm, devidx );
            }

            auto depFrm = frameSet->depthFrame();
            if ( depFrm != nullptr )
            {
                decodeFrame2RenderBox( depFrm, devidx );
            }

            pthread_mutex_unlock( &thrdmtx[devidx] );
        }
        else
        {
            snprintf( renderBoxTitle[devidx], 256,
                      "%zu\nFailure\nWaiting frame",
                      devidx );
            renderBox[devidx]->label( renderBoxTitle[devidx] );
            renderBox[devidx]->damage( 0 );
        }

        Fl::awake();
    }

    pthread_mutex_lock( &threadsmtx );
    if ( threads > 0 ) threads--;
    pthread_mutex_unlock( &threadsmtx );

    pthread_exit( nullptr );
    return nullptr;
}

• locking pthread mutex actually not affects frameset function, just added for thread-safe.
• Always shared pointer pipeline in vector failed to get frame set as timeout. ( frameSet always nullptr )

    while( threading == true )
    {
        pthread_mutex_lock( &thrdmtx[devidx] );
        auto frameSet = o_pipes[devidx]->waitForFrames(100);
        pthread_mutex_unlock( &thrdmtx[devidx] );

May I miss something ? Or need to do something for multiple Astra Mini Pro ?
Let me know how can I solve this issue.

Regards, Raph.

[RaphKay-BR]

I have tested this on x86.64 (Ubuntu 22.04) and same.
Looks like Orbbec SDK cannot handle multiple Astra Mini Pro.
Please let me know how to handle multiple devices at once.

[RaphKay-BR]

Checked libusb debug message while hang, libusb transfer many packets to host.
I think libOrbbecSDK locked up for processing somewhere - like waiting thread mutex infinitely, can you check it ?

[RaphKay-BR]

When I connect to onboard USB 3.x port, or external powered USB 3.x HUB, it's the same, but I can trace libusb debug message like this.

and its stdout log here:

libusb bulk reading transferring completion continuously called, but Orbbec SDK seems not handle this for multiple device.

And if I connect only two devices with color frame only, it looks Orbbec SDK handle it.

Are you guyz working on it ?

[RaphKay-BR]

As I traced with libusb debug logging, libusb_transfer_completon() always successfully done, but Orbbec seems not poll() handle usbi_wait_for_events() in 60 seconds while transfer is on going.
Can you check your libusb_ features to solve this issue ?

And plus question,
Don't you're using "libusb_bulk_transfer()" in each different thread by ob::Pipeline ?
Orbbec SDK seems to cannot handle transfer callback.

[zhonghong322]

Are your three USB ports internally connected to a hub? The Astra Mini Pro is a USB 2.0 device, and if they are connected through a hub, the bandwidth might not be sufficient. Have you tried running our sample Hotplug? Is it experiencing the same issue?

[RaphKay-BR]

Are your three USB ports internally connected to a hub? The Astra Mini Pro is a USB 2.0 device, and if they are connected through a hub, the bandwidth might not be sufficient. Have you tried running our sample Hotplug? Is it experiencing the same issue?

Do you think USB2.0 devices on USB3.x HUB may issue by bandwidth ? Interesting.

[zhonghong322]

When a USB 3.0 hub connects to a USB 2.0 device, the bandwidth is limited to USB 2.0 speeds, which is a maximum theoretical throughput of 480 Mbps. This is because USB 3.0 and USB 2.0 operate on separate channels within the same cable.

USB 3.0 HUBs are backward compatible with USB 2.0 by directly adding the SS differential signal pairs.
The mechanism of a USB 3.0 HUB is essentially an SS HUB and a USB 2.0 HUB.
The SS HUB is a logical HUB, indicating that the circuit only operates at SS, forwarding the upstream port's SS data packets to the downstream ports.
This means that when a USB 3.0 HUB is connected, the host actually sees two USB devices: an SS device and an HS device.

Since 480Mbps/5Gbps can operate simultaneously, if several USB devices are plugged into a USB 3.0 HUB, where the USB 3.0 devices operate at SS and share the 5Gbps bandwidth, and the USB 2.0 devices share the 480Mbps bandwidth, both speeds can work simultaneously.

[RaphKay-BR]

You mean USB2.0 480Mbps => USB HUB (2.0) meant 480MBps/(ports-count) ? <- fixed :)

As your opinion, libsub may writes many bandwidth related warnings or any log, but not.
Did you test with libusb debug printing ? libusb bulk transfer messages seems to not speak for your description.

[RaphKay-BR]

Updated question,
Do you have any plan to adjust resolution or compressing method ( like MJPEG ), or frame-rate control for Astra Mini Pro ?
If your 480Mbps limit is right ( I actually measured to triple Astra mini pro totally limited to 310 Mbps at a HUB ), how can reduce bandwidth ?

In other case, UVC basically automatically drop frame when reached to bandwidth limit, but Astra mini pro totally drop all packets ?
If so, Astra mini pro need methods for

• pixel compression
• or, auto frame dropping
• or, reduce frame-rate
• or, reduce resolution ( like binning, maybe good for depth 1/4 dimension for roughly detect objects )

Any method availed ?

[RaphKay-BR]

Updated for Orbbec developers,
I have implemented adjust resolution and fps by checking profile list.
But still don't know why Orbbec SDK fails to wait each frames even bandwidth is 182.8125 Mbp/s and it is less than 300 Mbp/s.

Astra Mini Pro USB devices are : 17,18.23 and each device bandwidth is running on about 7800 KB/s.
Let simple calculation : 7800 x 3 KB/s = 23400 KB/s = 22.8515625 MB/s = 182.8125 Mbp/s
But waiting frame returns failure.

And tested source code is here,

multi_astras-feat-gui-thread.zip

Let me know how to use triple Astra mini pro devices on a HUB.

[zhonghong322]

Updated for Orbbec developers, I have implemented adjust resolution and fps by checking profile list. But still don't know why Orbbec SDK fails to wait each frames even bandwidth is 182.8125 Mbp/s and it is less than 300 Mbp/s.

Astra Mini Pro USB devices are : 17,18.23 and each device bandwidth is running on about 7800 KB/s. Let simple calculation : 7800 x 3 KB/s = 23400 KB/s = 22.8515625 MB/s = 182.8125 Mbp/s But waiting frame returns failure.

And tested source code is here,

multi_astras-feat-gui-thread.zip

Let me know how to use triple Astra mini pro devices on a HUB.

What are the adjusted resolution and frame rate? Bandwidth calculation should be based on resolution and frame rate. Our camera does not use the standard UVC protocol but a proprietary protocol. During protocol parsing, due to packet loss, the entire frame is discarded. Even if only one packet of a frame is lost, the entire frame will be discarded during internal verification by the SDK.

[RaphKay-BR]

@zhonghong322 ,
Resolution each are:

• Color pixel format

                        auto vPF = tmpPF->as<ob::VideoStreamProfile>();
#ifdef DEBUG
                        printf( "color Prifle [%u] : %u x %u, %u\n",
                                qx,
                                vPF->width(),
                                vPF->height(),
                                vPF->fps() );
#endif
                        if ( ( vPF->width() == 640 ) &&
                             ( vPF->height() == 480 ) &&
                             ( vPF->fps() == 10 ) &&
                             ( tmpPF->format() == OB_FORMAT_UYVY ) )

• 640x480x2(UYVY)x10Hz == 5.8 MB/s

• Depth pixels

                        auto vPF = tmpPF->as<ob::VideoStreamProfile>();
#ifdef DEBUG
                        printf( "depth Prifle [%u] : %u x %u, %u\n",
                                qx,
                                vPF->width(),
                                vPF->height(),
                                vPF->fps() );
#endif
                        if ( ( vPF->width() == 640 ) &&
                             ( vPF->height() == 480 ) &&
                             ( vPF->fps() == 10 ) )

• 640x480x2x10Hz == 5.8 MB/s

So I Think max bandwidth usage will 5.8MB/s x 2 = 11.6MB/s x 3 = 35.15MB/s == 281.25Mbp/s
And Actual bandwidth is 182.81 Mbp/s.
Which is wrong? Orrbec cameras not works even under 300Mbp/s ?
Let me know.

Regards, Raph. K.

[zhonghong322]

@zhonghong322 , Resolution each are:

• Color pixel format

                        auto vPF = tmpPF->as<ob::VideoStreamProfile>();
#ifdef DEBUG
                        printf( "color Prifle [%u] : %u x %u, %u\n",
                                qx,
                                vPF->width(),
                                vPF->height(),
                                vPF->fps() );
#endif
                        if ( ( vPF->width() == 640 ) &&
                             ( vPF->height() == 480 ) &&
                             ( vPF->fps() == 10 ) &&
                             ( tmpPF->format() == OB_FORMAT_UYVY ) )

• 640x480x2(UYVY)x10Hz == 5.8 MB/s

• Depth pixels

                        auto vPF = tmpPF->as<ob::VideoStreamProfile>();
#ifdef DEBUG
                        printf( "depth Prifle [%u] : %u x %u, %u\n",
                                qx,
                                vPF->width(),
                                vPF->height(),
                                vPF->fps() );
#endif
                        if ( ( vPF->width() == 640 ) &&
                             ( vPF->height() == 480 ) &&
                             ( vPF->fps() == 10 ) )

• 640x480x2x10Hz == 5.8 MB/s

So I Think max bandwidth usage will 5.8MB/s x 2 = 11.6MB/s x 3 = 35.15MB/s == 281.25Mbp/s And Actual bandwidth is 182.81 Mbp/s. Which is wrong? Orrbec cameras not works even under 300Mbp/s ? Let me know.

Regards, Raph. K.

1. Yes, your method for calculating bandwidth is correct, The bandwidth of our device is smaller than your calculation because the depth is compressed to 12-bit or 11-bit per pixel.
2. but with three devices connected to one hub simultaneously, the instantaneous bandwidth is too high, causing packet loss. Additionally, if only one device is connected, we generally recommend that users keep the maximum bandwidth for USB 2.0 between 240 Mbps and 270 Mbps.
3. Actually, to eliminate bandwidth issues, it's quite simple: connect each of the three devices to a separate USB port. For example, on a PC, the USB ports are usually independent. This way, run your program and see if there is still packet loss.

[RaphKay-BR]

Dear @zhonghong322 ,
I got it, Astra Mini series looks not availed for triple combination.
Anyway, my only one question is belonged to ... "Why discard packets ?"
Don't you have option to relaxed frame rate control like auto frame skipping to don't discard packets ?

[zhonghong322]

Astra mini pro does not support the congestion traffic control function. When packets are lost, the frame rate is automatically reduced, and it also does not support the Color image compression function.

[RaphKay-BR]

Dear @zhonghong322, Ok. I will forget to use multiple Astra devices on a HUB.
You can close this issue.