Latency bottleneck

[HaoxinSEU]

Hi community,

We are now investigating the latency in srsRAN. After changing the SR_period to 2 ms, we see a big improvement in the latency, and we can achieve around 14 ms RTT in ping.

However, tuning the other configurations like CPU affinities doesn't change the latency too much, and we are stuck at 14 ms RTT now. So we are wondering which part is the bottleneck for the latency? For example, which layer, which queue, or which function contributes to a large part of the latency in the gNB stack?

Since we are using an Intel NUC 12 with an i7 CPU, the computer should not be a bottleneck.

Any hints or ideas will be much appreciated! Thanks for your help!

[ismagom]

Have you tried also reducing TDD pattern periodicity, reducing k1/k2?

see eg #147

[HaoxinSEU]

Hi,

Thanks! We are doing that now. But it's hard for us to trace the packet even in the DL, as sometimes the data buffer is copied and deleted, and multi-thread and task queues make it more difficult to keep tracking one packet.

Maybe there are some better ways to do the tracking, may I ask what's your suggestion to keep track of one packet through the different layers?

[frankist]

You should breakdown the RTT latency into several parts:

1. time that it takes for a packet to travel from the GTPU until the RLC SDU queue
2. time that it takes for the RLC SDU to be popped and scheduled by the MAC scheduler via a PDSCH grant
3. time between MAC PDSCH allocation and the PDSCH actually being sent over-the-air (controllable via max_proc_delay)
4. time taken for the gNB to receive an SR
5. delay between SR being detected and PUSCH grant being allocated
6. delay between PUSCH grant being received by the PHY and the respective PDUs to travel the stack until the GTPU.

I don't think that tracking a data buffer by looking at the code is a good idea, because PDUs are subject to many transformations as they travel the stack (e.g. segmentations, concatenations). For now, I would suggest to run a ping with a large enough period so that you don't confuse consecutive pings, when looking at the logs.

[HaoxinSEU]

Thanks so much for your advice @frankist !

We will try to investigate it!

[pgawlowicz]

@HaoxinSEU Could you provide any update/feedback on this issue?

[HaoxinSEU]

Hi,

Sorry for the late reply. We followed your suggestions, and traced each operation (layer) in the debug-level log.

We found some operations in the log taking surprisingly longer than we think, and this may lead to latency. @arman-maghsoudnia will share details shortly here.