Scaling UEs within an ORAN Simulation

[rcbarke]

Hello,

I am working on scaling a base simulation constructed for my research into a 15 UE scenario leveraging srsUE. Once I have successfully connected 15 UEs, my goal is to deploy a network slicing xApp I've prepared on the RIC. The configurations for network slicing have not been added in the sample files here, as I am presently focused on stabilizing the PHY layer, however, they are known.

The sample files provided within the configs folder of the srsProject repo seem to imply that SRSRAN and ZMQ are capable of scaling to hundreds of UEs in specific scenarios. Here, I'm attempting to deploy 10MHz of 1800MHz n3 at a 15kHz SCS with 52 PRBs. In a real world scenario, I would expect to be able to connect ~100 UEs to the channel.

Here is the current state of my configuration files. These are all script generated to ensure they are consistent:
15ue.zip

I am running all of my testing from a virtual machine with 50GB RAM and 25 CPU cores. I do not believe my problem is the machine spec, though I do notice significant processing load during simulation.

I am presently able to connect anywhere from 10-14 UEs during a simulation. The remaining UEs do have connectivity with the gNB and reach to changes within the rcc setting of the cu_cp (for instance, increasing the procedure timeout does cause each UE to make more RA connection attempts before camping). While the 1-5 UEs that do not connect are sending RA connection attempts into the gNB, they never receive connection responses from the gNB, so they never complete an RRC NR connection. They'll continue to make attempts to connect until the maximum number of ttis is reached, and then they camp.

I note that the PUCCH channel is extremely busy on the gNB, and the gNB occasionally even shuts down the DU without being prompted. With certain config values, I also see logs showing errors with ZMQ's buffer overflowing on both the gNB and UE.

Most of the tuning I have performed has been at the gNB level, occasionally crossing into my gnuradio script and UE conf files if I alter parameters such as the channel's sample rate.

Here are parameters I have attempted tuning to stabilize the PHY for 15 UE subscriptions:

1. cu_cp

• max_nof_ues (15-30)
• max_nof_drbs_per_ue: 29 (this needs to remain high as I would like to simulate conditions where a single UE requires a majority of the channel)
• rcc (both parameters, timeout from 300-10000 ms)
• inactivity_timer: 300-7200 (lower performs better for more UEs)

2. ru_sdr

• base_srate: See below
• srate: See below
• txgain (60-75): 65 appears optimal
• rxgain (60-75): 65 appears optimal

3. cell_cfg:

• channel_bandwidth_MHz: 5-20MHz (10MHz seems to be the most consistent and should be more than enough to support 15 UEs as noted above)
• common_scs: 15 (this appears to be locked due to the ssb_scs being locked to 15. I had the idea of trying a wider SCS at higher BW but am unable to change this setting without crashing the gNB.)

4. pdcch:

• ss0_index: 0-1
• coreset0_index: 1-6

5. prach

• prach_config_index: 1
• total_nof_ra_preambles: 64 (this is where the contention seems to occur with more UEs, I would like to set it higher however 64 appears to be the max)
• nof_ssb_per_ro: 1 (seems fixed)
• nof_cb_preambles_per_ssb: 32-64 (32 seems optimal to help with channel congestion)
• preamble_trans_max: 3-200 (this creates significant congestion and ZMQ overflow at higher values)

I have remained using n3 cells across all simulations as the frequency should be able to support the amount of UEs I am simulating. Please let me know if you disagree.

I have tried both increasing and decreasing the subcarrier spacing, channel bandwidth, and sample rate values on equivalent scaling to optimize the channel. I have already deployed a 5MHz carrier at half sample rate (25 PRBs) to aid with ZMQ buffer overflow and a 20MHz carrier at double sample rate (104 PRBs) to create more resources on the channel for UE subscription.

Is there any further tuning I can perform from the baseline I've attached, or am I reaching performance limitations for ZMQ and/or SRSRAN? Would there be another PHY config that could reliably connect 15 UEs?

Thank you for any assistance in advance!

[pgawlowicz]

You can check our example and optimized config files for supporting many UEs, for example this one for up to 256 UEs. They are for the TDD system, but some of the parameters also apply to FDD.

Additionally, you might force the UE to wait for a random period before sending another PRACH, so that UEs do not send PRACH simultaneously, thereby reducing contention in the PRACH channel. You can hardcode this behaviour here and recompile.

[rcbarke]

These configuration files are not complete configurations; they contain only the parameters that need to be adjusted for a higher number of UEs. You will need to stack multiple configuration files to set up the gNB in the desired way. For example:

sudo ./gnb -c du_rf_b200_tdd_n78_20mhz.yml -c amf.yml -c cell_cfg_max_256_ues.yml

Note that srsUE supports only SCS=15kHz, and BW=5/10/15/20 MHz.

In the case of the emulated ZMQ-based RF device, the carrier frequency does not matter. Baseband samples are simply transferred between the transmitter and receiver; there is no signal filtering based on the carrier frequency.

You do not need to change the ZMQ sample rate, but if it does not match the one used in the gNB, signal interpolation and decimation will occur. Therefore, to save CPU resources and limit IP traffic volume, it is generally advisable to adapt the ZMQ sample rate to match the gNB's sample rate.

Extremely helpful, thank you! I'll set up a 20MHz n77 carrier with 15kHz SCS at 11.12 msps to account for srsUE's spec and report back once it is configured. This alongside spacing out PRAACH signaling should go a very long way.

[pgawlowicz]

Band n77 is TDD, while 5G srsUE supports only FDD.
For a channel with BW=20MHz, you will need a sample rate of 23.04 MHz.

[rcbarke]

Band n77 is TDD, while 5G srsUE supports only FDD.

For a channel with BW=20MHz, you will need a sample rate of 23.04 MHz.

Understood, what are the specific limitations of 5G srsUE? So far we have:

• 15MHz SCS
• 5MHz, 10MHz, 15MHz, or 20MHz BW
• FDD timing configuration

Anything else?

Typically I would deploy 20MHz n77 @ 30kHz SCS in which case, yes, I would anticipate a 23.04MHz sample rate. However, if we are locked to a 15kHz SCS because of srsUE, we'd have less blocks on the channel, which is why I was thinking the sample rate would remain 11.52MHz.

If srsUE can't support TDD, I would infer that means it is limited to lower frequency, 4G/5G Nationwide NR carriers where there is overlap with LTE? Sub 3 GHz?

[pgawlowicz]

5G srsUE provides only a very basic implementation of a 5G UE. In addition to the limits you listed, there is no support for MIMO or handovers (and many more).

If your project needs a more sophisticated UE, we recommend the Amarisoft UE. You can even run it with our ZMQ-based RF-device. Please check our tutorial

[rcbarke]

5G srsUE provides only a very basic implementation of a 5G UE. In addition to the limits you listed, there is no support for MIMO or handovers (and many more).

If your project needs a more sophisticated UE, we recommend the Amarisoft UE. You can even run it with our ZMQ-based RF-device. Please check our tutorial

Thank you for your guidance, Piotr—it has been invaluable in helping me delve deeper into the SRS Project and SRS 4G architecture. After spending considerable time with the source code, I now have a stronger understanding of the current software's limitations.

With some tweaks to our gNB configuration and implementing the random wait timer within the proc_ra_nr re-attempts you suggested, we've managed to significantly reduce UE contention during network subscriptions. We still encounter occasional gridlocks with large numbers of UEs, though stability has greatly improved, and iPerf now produces more consistent bitrates over PUSCH.

Here are some of our findings:

1. Deployment Scripts: I’ve developed scripts to configure our end-to-end software stack from the base source, enabling dynamic UE management and dependency synchronization. These scripts also handle sequential launches, making network deployment more intuitive.

2. Resource Management Limitations: Our current gNB configuration supports dynamic DL PRB adjustments, but we’ve noticed SRSRAN does not yet support real-time UL PRB allocation per UE. Each UE on UL occupies the complete channel, with throughput scaling based on contention. For DL, we can adjust PRBs in real-time via the E2 interface, though there seems to be a bug where maxPRBs and initial PRBs must match to avoid network crashes.

3. Buffering Delay at gNB: During xApp development, we found that while real-time control plane adjustments are possible, traffic simulated through iPerf experiences buffering at the gNB, creating a delay that scales with congestion on both UL and DL. This affects the accuracy of deep reinforcement learning xApps, though foundational DL network slicing functions under these conditions.

4. Future Plans: To address current limitations in UL management, we’re exploring alternatives like Amarisoft UE for high-frequency network simulations. We also plan to add interference generators, pathloss simulations, and power settings for adaptive testing, though resolving the buffering delay will be essential for these simulations. In the meantime, we’ve implemented baseline resource allocation tests on the DL with nonadaptive conditions as a starting point.

Our progress with this 10MHz 1800MHz n3 carrier is promising, and we’re excited to see how far we can advance ORAN within these parameters. We also aim to explore dynamic power management at this frequency and continue investigating resource allocation at mid-band frequencies.

An enhancement to support dynamic UL PRB allocation in the SRS Project’s gNB, with E2 interface exposure, would be extremely valuable for URLLC and mmTC applications.

Best,
Ryan

[pgawlowicz]

Hi @rcbarke,
Thank you for your feedback and report.
Could you say how many srsUEs did you manage to connect in the emulated scenario with the GnuRadio signal broker?
The dynamic UL PRB allocation will soon be available in the main branch. But you can already test it using the test branch.

Best Regards,
Piotr

[rcbarke]

Hi @rcbarke,

Thank you for your feedback and report.

Could you say how many srsUEs did you manage to connect in the emulated scenario with the GnuRadio signal broker?

The dynamic UL PRB allocation will soon be available in the main branch. But you can already test it using the test branch.

Best Regards,

Piotr

Good morning! This does not appear to be software limited, with the dynamic script logic and random wait timer patch, I have been able to subscribe as many as 30 srsUEs. Note that occasionally, 1-4 UEs will still gridlock due to PRAACH contention, though this is far less frequent.

The primary limitations we've observed with large scale stress tests are CPU utilization and hard disk memory. Specifically, if the srsMetrics server and Grafana are enabled during simulation, the hard disk fills up extremely fast in larger scenarios. Unfortunately, the metrics server data is stored within private memory within a docker container, so there's no easy way to flush it to my knowledge. The host OS reports it is out of disk space, but local memory management tools aren't correctly reading the allocations within the container. Let me know if there is a method to delete records from the metrics DB to overcome this.

Regarding CPU Utilization, the host OS experiences noticeable slow down after 20 UEs or so. This was tested on a Ubuntu 20.04 Virtual Box VM with 23 CPU cores allocated and 100GB HD meomory. The host machine has a 4090 and we leveraged hardware acceleration to offload the graphics logic, which improved performance. I'm sure leveraging a larger machine would scale the UE to performance ratio.

Our research leverages a simulation with a deep reinforcement learning based implementation of network slicing, and for our initial paper, we set up a DL based scenario with 12 UEs. Stabilizing the metrics server memory with a periodic flush would be an excellent enhancement for such scenarios, as the machine learning application has to be developed and debugged while simulations are live, and we've already run into scenarios where we've exhausted memory developing. Also, the longer we can run a large simulation without the machine crashing, the more accurate the ML training becomes.

We've leverage git, increased the machine size, and reduced the simulation size during development to avoid losing data, however a method to configure the metrics server to only store data for a configurable window of time would likely be more optimal for scaling. Ex: "Last 10 minutes", "Last 30 minutes", "Last 1 hour", etc. similar to the Grafana UI.

Our source is going to remain private until after a December ICC workshop we are working towards publishing for, but after that, we can discuss sharing our implementation with the director of Clemson's IS-WiN lab.

Thank you for sharing the UL specific PRB enhancement! That's a significant improvement and I will review the test branch to check out the new code.

Best,
Ryan

[rcbarke]

Hi @rcbarke,

Thank you for your feedback and report.

Could you say how many srsUEs did you manage to connect in the emulated scenario with the GnuRadio signal broker?

The dynamic UL PRB allocation will soon be available in the main branch. But you can already test it using the test branch.

Best Regards,

Piotr

Also, another idea we plan on investigating to allievate CPU load is offloading UEs across multiple machines/VMs in larger simulations, though we've deprioritized that for now as it would not improve the hard disk utilization on the machine running srsGNB/srsMetrics server.