Throughput performace

[lhms1234]

Issue Description

Hi, community.
After some tests using srsRAN, I was wondering how performative would be this stack executed on a high-performance machine, taking into consideration the throughput achieved on the UE (got 29.2 Mbps DL and 2.45 Mbps UL). I made some tests on a performative machine too, but the result was similar (26 Mbps DL, 10.8 Mbps UL).
Even though the setup was wired and the gNB gains were calibrated, the throughput achieved on both machines was not as good as it could have been. Did you get results similar to this one? Did you do any performance optimization besides those present in the "script/srsran_performance" script?
Thanks in advance.

Setup Details

• Core (VM):
  • Stack: Open5GS;
  • Processor:
    • Intel Xeon Processor (Skylake, IBRS);
    • 8 cores;
    • CPU MHz: 2095.076.
  • RAM: 4 Gb;
  • Disk: 50 GB;
  • OS: Ubuntu 20.04 LTS;
  • Network configuration: SR-IoV used;
  • Obs: this core was able to reach 1 Gbps DL throughput, so it isn't a limitator.
• gNB:
  • Stack: srsRAN_Project;
  • Processor:
    • Cortex-A72;
    • 16 cores;
    • CPU MHz: 2000.
  • RAM: 32 Gb;
  • Disk: 108 GB;
  • OS: Ubuntu 20.04 LTS;
  • USRP: B210;
  • UHD: 4.3.0;
  • Antenna configuration: SISO;
  • RF configuration:
    • Band: 28;
    • EARFCN DL: 154000;
    • SSB EARFCN: 154090;
    • BW: 10 MHz
    • SCS: 15 KHz.
• UE: Quectel RM500Q-GL (wired connection with B210 USRP).

Expected Behavior

Get some DL throughput closer to 41 Mbps (theorical limit on this configuration).

Actual Behaviour

Got a DL throughput of 29.2 Mbps.

[andrepuschmann]

Hey @lhmscpqd - you need to share some more information about your tests. Please send logs and console trace of the gNB when you do your experiments. Also try to use UDP traffic traffic rather then TCP. There will be some improvements coming with the next release (will be out shortly) but meanwhile this might help.

On a side note - I see you're using a 16 A72 cores. What kind of system is this? Can you share more info? Is it a LX2160A?

[lhms1234]

Hello Andre, thanks for your fast reply.

Sorry about this, I'm attaching some logs of gNB (including terminal output) and iperf tests, now using UDP. Note that using UDP the throughput was much lower than using TCP (even though I use parallel streams). About the processor: yes, it's an LX2160A 16-core Arm Cortex A72. I'm using a ClearFog CX LX2. I also forgot to detail the configuration of the performative machine, here it is:

• Processor:
  • 12th Gen Intel(R) Core(TM) i9-12900K;
  • 16 cores;
  • CPU MHz: 3200.
• RAM: 32 Gb;
• Disk: 480 GB SSD;
• OS: Ubuntu 20.04 LTS;
• USRP: B210;
• UHD: 4.3.0;
• Antenna configuration: SISO;
• RF configuration:
  • Band: 28;
  • EARFCN DL: 154000;
  • SSB EARFCN: 154090;
  • BW: 10 MHz;
  • SCS: 15 KHz.

[ismagom]

Can you share your config file? When you do high performance tests, disable PCAPs. Your CPU is very powerful, you shouldn't have issues with performance.

Have you tried running the UHD benchmark tool?

On a separate note, to get the theoretical throughput you also need to configure 256QAM

[lhms1234]

Of course, here you are gnb_rf_b200_tdd_n78_20mhz.yml. I was testing with PCAPs generation disabled, but I notice that the compiled version was old (from March or April of this year) when the 256qam.yml file doesn't exist.
The result of the UHD benchmark tool was the following:

/usr/local/lib/uhd/examples/benchmark_rate --args type=b200 --rx_rate 10e6 --tx_rate 10e6

[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; UHD_4.3.0.HEAD-0-g1f8fd345
[00:00:00.000085] Creating the usrp device with: type=b200...
[INFO] [B200] Detected Device: B210
[INFO] [B200] Operating over USB 3.
[INFO] [B200] Initialize CODEC control...
[INFO] [B200] Initialize Radio control...
[INFO] [B200] Performing register loopback test... 
[INFO] [B200] Register loopback test passed
[INFO] [B200] Performing register loopback test... 
[INFO] [B200] Register loopback test passed
[INFO] [B200] Setting master clock rate selection to 'automatic'.
[INFO] [B200] Asking for clock rate 16.000000 MHz... 
[INFO] [B200] Actually got clock rate 16.000000 MHz.
Using Device: Single USRP:
  Device: B-Series Device
  Mboard 0: B210
  RX Channel: 0
    RX DSP: 0
    RX Dboard: A
    RX Subdev: FE-RX2
  RX Channel: 1
    RX DSP: 1
    RX Dboard: A
    RX Subdev: FE-RX1
  TX Channel: 0
    TX DSP: 0
    TX Dboard: A
    TX Subdev: FE-TX2
  TX Channel: 1
    TX DSP: 1
    TX Dboard: A
    TX Subdev: FE-TX1

[00:00:01.12556375] Setting device timestamp to 0...
[INFO] [B200] Asking for clock rate 40.000000 MHz... 
[INFO] [B200] Actually got clock rate 40.000000 MHz.
[00:00:01.405029246] Testing receive rate 10.000000 Msps on 1 channels
Setting TX spp to 2040
[00:00:01.427280107] Testing transmit rate 10.000000 Msps on 1 channels
[00:00:11.433972585] Benchmark complete.


Benchmark rate summary:
  Num received samples:     100222428
  Num dropped samples:      0
  Num overruns detected:    0
  Num transmitted samples:  100064040
  Num sequence errors (Tx): 0
  Num sequence errors (Rx): 0
  Num underruns detected:   0
  Num late commands:        0
  Num timeouts (Tx):        0
  Num timeouts (Rx):        0


Done!

P.S.: all this information was about the performative machine, not ClearFog.

[andrepuschmann]

But when you say "performative machine" you mean the one running the 5GC? The gNB is running on the ARM, right? It's an interesting platform because it has many cores but they aren't very powerful - it's essentially four Raspberry Pi 4s. We might have to tweak the threading model a bit to get max rate. But it should be possible. For sure from the gnb console trace we can see that the gNB isn't getting "enough" traffic from the core to even allocate all resources.

[lhms1234]

No, I mean that the performative machine is a desktop used to run srsRAN gNB as a way to compare the performance of the stack in different machines (this desktop and the ClearFog). I think the problem isn't at Core, because as I said before, this core was able to provide 1Gbps DL throughput to a Baicells gNB.

[ismagom]

Hi @lhmscpqd ,

Looking at the config, you could try the following.

1. Decrease srate to 11.52. Since you are doing 10 MHz BW, that's enough and will alleviate the UHD traffic
2. Try also removing sc12. Sometimes this is better sometimes is worse
3. Try these expert settings:

expert_phy: 
  nof_ul_threads: 4         # this will create 4 threads to process UL
  nof_pdsch_threads: 4  # this will create 4 threads to process PDSCH in parallel 

To be honest I'm not sure the bottleneck is the CPU. It seems more the transfer of samples to the USRP (RF lates/underflows). Let's try these options first, if these doesn't improve, we'll look at other options.

To enable 256qam, use this option:

  pdsch:
    mcs_table: qam256

[lhms1234]

Hi @ismagom,

After some tests permuting the parameters that you pointed out, I've noted that the USRP sample rate is strongly linked to the problem. The higher the sampling rate, the greater the instability (the other parameters didn't make that much difference).
After that, I believe you are right about the bottleneck being the transfer of samples to the USRP. I tested the stack in the same conditions with 3 UHD versions: 3.15.0, 4.0.0, and 4.3.0, but the results were almost the same. Which UHD version do you use? How to proceed to mitigate this issue?

[ismagom]

We use 4.0.2 in our lab. Can you share logs and console trace with sampling rate 11.52?

[lhms1234]

I don't found this exact version on the UHD repository, so I compiled UHD version 4.1.0.2. Here are the
logs. Note that this time the stack was more stable than before, reaching a throughput close to 40 Mbps. Unfortunately, when I tested with the same sample rate (30.72 Msps) and increased the BW (10 -> 20 Mhz), the throughput was terrible.

[pgawlowicz]

@lhms1234 any update on this issue?

[lhms1234]

Hi, @pgawlowicz .
Unfortunately, no. This activity was suspended by now. Maybe it will come back in the future.

[andrepuschmann]

Closing discussion as it's not relevant anymore to the original poster.