Amarisoft UE + srsRAN with high throughput

[Zplusless]

Hi, I'm testing multi-UE environments with Amarisoft UE + srsRAN in ZMQ mode. I have achieved multi-UE attachment for TDD(10 MHz and 20MHz) and FDD (20MHz) by modifying config files used for srsUE+srsRAN+ZMQ and the official 10-MHz configs provided in the tutorial. The configs that can successfully run (FDD + TDD/single&multi ue/ 10MHz +20MHz) is attached here for sharing(Amarisoft_UE+srsRAN.zip). However, I have encontered several problems:

1. Because there is no official config examples for ZMQ mode with bandwidth larger than 20MHz, configuring srsRAN with larger bandwidth is really hard. If I simply modify the bandwidth in fdd_20MHz_config to 50MHz, the amarisoft UE fails to work as below:
   
   ue0.log
   gnb.log

2. The throughput of TDD is much lower than FDD. For example, with 20MHz-config, FDD has around 55Mbps while TDD only has around 17Mbps.

3. If multiple amarisoft UEs run iperf3 simultaneusly, they will not achieve the expected throughput. For example, one UE can achieve 55Mbps. But with two UEs, each will have only half of the throughput. I'm not sure if it's the feature of Amarisoft UE.

4. Does ZMQ mode support MIMO?

[pgawlowicz]

We do have a tutorial on connecting Amarisoft UE in multi-UE mode (configs available):
https://docs.srsran.com/projects/project/en/latest/tutorials/source/amariUE/source/index.html#zeromq-based-setup

Did you check it?

[Zplusless]

Hi @herlesupreeth, thank you for your guidance. I've thoroughly reviewed and corrected some errors in my configuration files. Ensure consistency in the dl_nr_arfcn, ssb_nr_arfcn, and bandwidth between the gNB and UE. Now, both FDD-50MHz and TDD-100MHz systems are operational with two Amarisoft UEs attached. However, since the config files contain only basic settings, the throughput levels are not satisfactory (FDD-50MHz at 75Mbps and TDD-100MHz at 45Mbps). Should my understanding be accurate, this would be the total cell throughput. Could you please offer some advice on how to achieve higher throughput?

Specifically:

1. Would increasing the number of workers be beneficial?
2. Is ZMQ mode compatible with MIMO?
3. Are there any other parameters in the config file that could help to improve the throughput?

configs:

FDD

amariue-fdd-n7-multi-ue-50MHz.cfg.txt
gnb_zmq_fdd_50MHz.yml.txt

TDD

amariue-tdd-n78-multi-ue-100MHz.cfg.txt
gnb_zmq_tdd_100MHz.yml.txt

[herlesupreeth]

I dont see any issue in your configuration for SISO. Are 192.168.0.10 and 192.168.0.100 on the same machine or different? If there are on different machine then the link between them could be a bottleneck

With 100Mhz TDD and 1 Amarisoft UE attached I can achieve following DL speed over ZMQ (with basic configuration as in your configuration file)

   1 4601   15   1   28   262M 1323    0   0%  2.81M |  65.5   28   205k   47    0   0%      0
   1 4601   15   1   28   277M 1400    0   0%  2.79M |  65.5   28   218k   50    0   0%      0
   1 4601   15   1   28   277M 1400    0   0%  2.79M |  65.5   28   218k   50    0   0%      0

           -----------------DL-----------------------|------------------UL--------------------
 pci rnti  cqi  ri  mcs  brate   ok  nok  (%)  dl_bs | pusch  mcs  brate   ok  nok  (%)    bsr
   1 4601   15   1   28   277M 1400    0   0%  2.79M |  65.5   28   218k   50    0   0%      0
   1 4601   15   1   28   277M 1400    0   0%  2.79M |  65.5   28   218k   50    0   0%      0
   1 4601   15   1   28   277M 1400    0   0%  2.79M |  65.5   28   218k   50    0   0%      0
   1 4601   15   1   28   277M 1400    0   0%   2.8M |  65.5   28   218k   50    0   0%      0
   1 4601   15   1   28   277M 1400    0   0%   685k |  65.5   28   218k   50    0   0%      0

However, since the config files contain only basic settings, the throughput levels are not satisfactory (FDD-50MHz at 75Mbps and TDD-100MHz at 45Mbps).

These speeds are not quite low. Have you put both the machines running gNB and UE in performance mode?

Is ZMQ mode compatible with MIMO?

yes, you would need to declare as follow example (4x4 MIMO) in gNB configuration file and set number of DL and UL antennas

ru_sdr:
  device_driver: zmq                # The RF driver name.
  device_args: tx_port0=tcp://172.22.0.1:5000,tx_port1=tcp://172.22.0.1:5001,tx_port2=tcp://172.22.0.1:5002,tx_port3=tcp://172.22.0.1:5003,rx_port0=tcp://172.22.0.1:6000,rx_port1=tcp://172.22.0.1:6001,rx_port2=tcp://172.22.0.1:6002,rx_port3=tcp://172.22.0.1:6003            # Optionally pass arguments to the selected RF driver.
  srate: 122.88

cell_cfg:
  nof_antennas_dl: 4
  nof_antennas_ul: 4

and on Amarisoft UE side

  rf_driver: {
    name: "srsran",
    args: "",
    tx_port0: "tcp://172.22.0.1:6000",
    tx_port1: "tcp://172.22.0.1:6001",
    tx_port2: "tcp://172.22.0.1:6002",
    tx_port3: "tcp://172.22.0.1:6003",
    rx_port0: "tcp://172.22.0.1:5000",
    rx_port1: "tcp://172.22.0.1:5001",
    rx_port2: "tcp://172.22.0.1:5002",
    rx_port3: "tcp://172.22.0.1:5003",
    log_level: "error",
    args: "id=ue,base_srate=12288000",
  },

n_antenna_dl: 4,
n_antenna_ul: 4,

Would increasing the number of workers be beneficial?

I dont think it would be necessary

Can you provide the specs of the machine running the gNB and Amarisoft UE?

[Zplusless]

Hi @herlesupreeth, thanks for your detailed reply.

Are 192.168.0.10 and 192.168.0.100 on the same machine or different? If there are on different machine then the link between them could be a bottleneck

Amarisoft UE and gnb are deployed on different servers, which are connected with 10GE fiber. So the link may not be the bottleneck.

With 100Mhz TDD and 1 Amarisoft UE attached I can achieve following DL speed over ZMQ (with basic configuration as in your configuration file)

The 277Mbps DL throughput is acceptable in your output. The result I got (45Mbps for TDD-100MHz) is a little disappointing. 😂😂😂

These speeds are not quite low. Have you put both the machines running gNB and UE in performance mode?

Yes. I have run the srsran_performance script on gnb (ubuntu 20.04). The UE node is a little special. Its OS is Centos Stream9 (for some reason I cannot change the OS), where I got the following output when running the script:

[root@xfusion scripts]# ./srsran_performance 
Would you like to set the CPU governor to performance: [Y/n]: y
tee: '/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor': No such file or directory

I checked `/sys/devices/system/cpu/cpu0/:

[root@xfusion cpu0]# pwd
/sys/devices/system/cpu/cpu0
[root@xfusion cpu0]# ls
cache    crash_notes       driver         hotplug    node0  subsystem         topology
cpuidle  crash_notes_size  firmware_node  microcode  power  thermal_throttle  uevent

I have set CPU to performance mode in BIOS and turned off hyperthreading. Are there any other CPU configurations that I need to modify?

Besides, the CPU configurations of the two servers are as follows:

gnb server:

CPU(s):                          20
On-line CPU(s) list:             0-19
Thread(s) per core:              2
Core(s) per socket:              10
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           108
Model name:                      Intel(R) Xeon(R) D-1747NTE CPU @ 2.50GHz
Stepping:                        1
CPU MHz:                         2500.000
CPU max MHz:                     2500.0000
CPU min MHz:                     800.0000
BogoMIPS:                        5000.00

UE server:

CPU(s):                  112
  On-line CPU(s) list:   0-55
  Off-line CPU(s) list:  56-111
Vendor ID:               GenuineIntel
  BIOS Vendor ID:        Intel(R) Corporation
  Model name:            Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz
    BIOS Model name:     Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz
    CPU family:          6
    Model:               106
    Thread(s) per core:  1
    Core(s) per socket:  28
    Socket(s):           2
    Stepping:            6
    BogoMIPS:            4000.00

You would need to declare as follow example (4x4 MIMO) in gNB configuration file and set the number of DL and UL antennas.

Thanks, I'll modify the config file and have another try.

I dont think it would be necessary(increasing the number of workers)

I'm not sure. But the cores that radio and phy_worker are binding to have reached nearly 100% utilization.

In summary, I'll double-check the server CPU configuration, try to introduce MIMO and increase work num. Then I'll come back and report the results.

Thanks again!

[herlesupreeth]

I have set CPU to performance mode in BIOS and turned off hyperthreading. Are there any other CPU configurations that I need to modify?

I would disable Intel P-states in bios

On intel machine I usually follow below steps to get maximum performance

- On Intel CPU machines:

Install cpufrequtils:

sudo apt-get install cpufrequtils

Then edit the following file (if it doesn't exist, create it):

sudo nano /etc/default/cpufrequtils

And add the following line to it:

GOVERNOR="performance"

Save and exit.

Now you need to disable ondemand daemon, otherwise after you reboot the settings will be overwritten.

sudo update-rc.d ondemand disable

sudo /etc/init.d/cpufrequtils restart

Power management
Remove all power management features in the BIOS (sleep states, in particularC-states) and CPU frequency scaling (Intel SpeedStep). In some cases, you can
also do this with cpufreqtool. Also, disable hyperthreading in BIOS and make sure its turned off in Linux

1. check this using this command: watch grep \"cpu MHz\" /proc/cpuinfo
2. you must disable p-state and c-state in linux so you need to add intel_pstate=disable to the Linux boot options,
i.e GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_pstate=disable"  in /etc/default/grub, then perform update-grub. You may optionally add the following as well "processor.max_cstate=1 intel_idle.max_cstate=0 idle=poll"

Append "blacklist intel_powerclamp" to the end of /etc/modprobe.d/blacklist.conf, to blacklist the intel_powerclamp" module. If the file does not exist, create one, and add the line into it.
You also need to disable hyperthreading, CPU frequency control, C-States, P-States and any other power management from BIOS as well.

Install i7z utility to check the cpu

sudo apt-get install i7z
sudo i7z

The CPU should not change its frequency by more than 1-2 hertz and should not be any C-state other than C0

[Zplusless]

Hi @herlesupreeth, I have tested the system following your guidance:

1. Configuring both servers (UE and gnb) to get maximum performance doesn't improve the throughput significantly(no larger than 5Mbps improvement). Could you please provide the specs of the machine that can achieve 277Mbps DL throughput?

2. As you expected, increasing worker thread number does not help, either.

expert_execution:
  threads: 
    upper_phy: 
      pdsch_processor_type: auto                    # Optional TEXT (auto). Sets the PDSCH processor type. Supported: [auto, generic, concurrent, lite].
      nof_pusch_decoder_threads: 2                          # Optional UINT (1). Sets the number of threads used to encode PUSCH.
      nof_ul_threads: 2                             # Optional UINT (1). Sets the number of upprt PHY threads to proccess uplink.
      nof_dl_threads: 4                             # Optional UINT (1). Sets the number of upprt PHY threads to proccess downlink.  
    lower_phy:
      execution_profile:                            # Optional TEXT. Sets the lower physical layer executor profile. Supported: [single, dual, quad].
    ofh: 
      enable_dl_parallelization: 1                  # Optional BOOLEAN. Sets the Open Fronthaul downlink parallelization flag. Supported: [0, 1].

3. What confuses me most is MIMO even worsen the throughput(FDD-50MHz at 75Mbps, but FDD-50MHz with MIMO only has no more than 20Mbps).
   gnb_zmq_fdd_test_mimo.yml.txt

[herlesupreeth]

1. Below is the machine I used to achieve DL 277 Mbps

            .-/+oossssoo+/-.               sherle@sherle-X670E-AORUS-MASTER 
        `:+ssssssssssssssssss+:`           -------------------------------- 
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 22.04.3 LTS x86_64 
    .ossssssssssssssssssdMMMNysssso.       Host: X670E AORUS MASTER -CF 
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Kernel: 6.5.0-15-generic 
  +ssssssssshmydMMMMMMMNddddyssssssss+     Uptime: 3 hours, 22 mins 
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Packages: 2618 (dpkg), 22 (snap) 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Shell: bash 5.1.16 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Resolution: 3440x1440 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   DE: GNOME 42.9 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   WM: Mutter 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   WM Theme: Adwaita 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Theme: Yaru [GTK2/3] 
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/    Icons: Yaru [GTK2/3] 
  +sssssssssdmydMMMMMMMMddddyssssssss+     Terminal: gnome-terminal 
   /ssssssssssshdmNNNNmyNMMMMhssssss/      CPU: AMD Ryzen 7 7800X3D (16) @ 5.05 
    .ossssssssssssssssssdMMMNysssso.       GPU: AMD ATI 14:00.0 Device 164e 
      -+sssssssssssssssssyyyssss+-         GPU: NVIDIA 01:00.0 NVIDIA Corporati 
        `:+ssssssssssssssssss+:`           Memory: 13156MiB / 31219MiB 

As you expected, increasing worker thread number does not help, either.

For these setting maybe others can help you since I dont know how to properly set them. @sauka can you please take a look?

What confuses me most is MIMO even worsen the throughput(FDD-50MHz at 75Mbps, but FDD-50MHz with MIMO only has no more than 20Mbps).

I dont see any issue in the configuration file (not sure about expert_execution though

Can you try recompiling ZMQ driver for Amarisoft UE on the Amarisoft machine fetching the latest srsRAN-Project code and see if that fixes these low throughput issue? (https://docs.srsran.com/projects/project/en/latest/tutorials/source/amariUE/source/index.html#zeromq-driver-for-amarisoft-ue)

[herlesupreeth]

I see dpdk in your system info, should I try dpdk for srsRAN?

hmmm. I dont see it (maybe you got confused reading dpkg ;) ) and I havent installed it. Also, DPDK is only used when you are using ORAN-RU so I dont think its needed for ZMQ based setups.

[Zplusless]

Hi @herlesupreeth. Thanks for your replies.

I recompiled the zmq driver (the latest code in main branch r20240205-32dae89) for Amarisoft UE and change another server with an i9-12900K CPU for higher frequency. The throughput improves a little but not much. FDD-50MHz has 108Mbps and TDD-100MHz has 73Mbps. MIMO still leads to poorer throughput (less than 30Mbps for 4*4).

The detailed configuration and iperf3 logs are as follows:
server_wenhao-Amarisoft_UE+srsRAN-test.zip

[herlesupreeth]

Except for expert_execution options (which I dont think is necessary) for MIMO scenarios I dont see any issues with configuration files. Also, can you try to use UDP in iPerf3 rather than TCP (which is default)??

[pgawlowicz]

@Zplusless any update here?

[Zplusless]

Hi @pgawlowicz，A more powerful machine can really help. But it is still far from our expectations. We have turned to ru_emulator, which can achieve 1.1Gbps bitrate easily. Thanks!