Zebra process crashes intermittently during 'config reload' on the DUT line cards

[mlok-nokia]

---

Describe the bug
On a T2 chassis line card, when we do 'sudo config reload -y', we see 'zebra' process getting crashed and generates a core. We see this issue intermittently happening. (_~ approx once in 30 attempts or so_)

We have started seeing the issue from this commit

sonic-buildimage-msft commit:
Azure/sonic-buildimage-msft@6f19e12

Following logs are seen on the bgp docker, when the crash is happening.

2023-07-09 13:59:40,064 INFO exited: zebra (terminated by SIGSEGV (core dumped); not expected)
2023-07-11 19:39:22,156 INFO exited: zebra (terminated by SIGSEGV (core dumped); not expected)

To Reproduce
Steps to reproduce the behavior:
On any T2 chassis line card, do 'sudo config reload -y' for multiple times.

Expected behavior
sudo config reload' on DUT line cards, should not cause any issue. And the line cards should come up fine with all bgp neighbors established without any crash/core files.

Actual behavior
Zebra process under bgp docker gets crashed.
Core generated
We had already raised an issue under sonic-buildimage regarding this crash, please take a look at this,
sonic-net/sonic-buildimage#15803
frr.zip
zebra.1689104360.44.0.core.gz

Screenshots

Versions

admin@ixre-egl-board1:~$ show version

SONiC Software Version: SONiC.HEAD.489499-msft-2205-ndk-d963ac161
SONiC OS Version: 11
Distribution: Debian 11.7
Kernel: 5.10.0-18-2-amd64
Build commit: d963ac161
Build date: Fri Jul  7 18:18:51 UTC 2023
Built by: gitlab-runner@sonic-bld2

Platform: x86_64-nokia_ixr7250e_36x400g-r0
HwSKU: Nokia-IXR7250E-36x100G
ASIC: broadcom
ASIC Count: 2
Serial Number: EAG2-04-210
Model Number: N/A
Hardware Revision: 56
Uptime: 15:45:52 up 1 day, 12:15,  3 users,  load average: 1.56, 1.54, 1.59
Date: Wed 12 Jul 2023 15:45:52

• FRR Version: frr_8.2.2-sonic-0_amd64.deb

Additional context

[mlok-nokia]

Attaching the symbol file and the core file
frr-dbgsym_8.2.2-sonic-0_amd64_bullseye.zip
frr-dbgsym_8.2.2-sonic-0_amd64_buster.zip
zebra.1689971652.43.1.core.gz
zebra.gz

[mlok-nokia]

After we checked the previous test history, we found this crash is seen in April OC test run.

[donaldsharp]

8.2 is now considered very old and is typically not supported from a FRR community perspective at this point in time.

First thing is that I would try this fix:
commit 0eaa652
Author: Donald Sharp [email protected]
Date: Fri May 19 09:54:05 2023 -0400

zebra: Do not allow old FPM to access freed memory after shutdown

If that doesn't work I would recommend upgrading to a newer release and then trying that to see if the problem still exists

[eqvinox]

@mlok-nokia I would recommend opening this as a bug with Sonic. Basically Sonic is trying to maintain an older stable version which is not really active in the FRRouting community anymore. We can look at this if the problem can be demonstrated in 8.5 or master, but the work to reproduce the issue on a newer version is with you / Sonic.

[abdosi]

8.2 is now considered very old and is typically not supported from a FRR community perspective at this point in time.

First thing is that I would try this fix: commit 0eaa652 Author: Donald Sharp [email protected] Date: Fri May 19 09:54:05 2023 -0400

zebra: Do not allow old FPM to access freed memory after shutdown

If that doesn't work I would recommend upgrading to a newer release and then trying that to see if the problem still exists

@mlok-nokia can you try with this patch and see if it helps.

[github-actions]

This issue is stale because it has been open 180 days with no activity. Comment or remove the autoclose label in order to avoid having this issue closed.

[frrbot]

This issue will be automatically closed in the specified period unless there is further activity.