FRR 8.0 crashes at function stream_pnt

[forsunwell]

• OS Version: ubuntu20.04
• Kernel: Linux 4.19
• FRR Version: 8.0
• When reporting a crash, provide a backtrace:

(gdb) bt
#0  0x00007f524c9db00b in raise () from /lib64/libpthread.so.0
#1  0x00007f524e1a4859 in core_handler (signo=6, siginfo=0x7f524a9b56f0, context=0x7f524a9b55c0) at lib/sigevent.c:262
#2  <signal handler called>
#3  0x00007f524c6405c9 in raise () from /lib64/libc.so.6
#4  0x00007f524c641cd8 in abort () from /lib64/libc.so.6
#5  0x00007f524e1dc7a7 in _zlog_assert_failed (xref=0x7f524e4d1a60 <_xref.16392>, extra=0x0) at lib/zlog.c:557
#6  0x00007f524e1b16b0 in stream_pnt (s=0x6a01c500) at lib/stream.c:1203
#7  0x000000000047979e in bgp_write (peer=0xa588000) at bgpd/bgp_io.c:320
#8  0x00000000004790c9 in bgp_process_writes (thread=0x7f524a9b6da0) at bgpd/bgp_io.c:139
#9  0x00007f524e1bb1b5 in thread_call (thread=0x7f524a9b6da0) at lib/thread.c:1825
#10 0x00007f524e14abb5 in fpt_run (arg=0x2ead3b0) at lib/frr_pthread.c:311
#11 0x00007f524e14a707 in frr_pthread_inner (arg=0x2ead3b0) at lib/frr_pthread.c:158
#12 0x00007f524c9d3df3 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f524c7011ad in clone () from /lib64/libc.so.6
(gdb) frame 6
#6  0x00007f524e1b16b0 in stream_pnt (s=0x6a01c500) at lib/stream.c:1203
1203  in lib/stream.c
(gdb) p /x s[0]
$74 = {next = 0x26f83800, getp = 0xffffffffffffffff, endp = 0x4b6, size = 0x4b6, data = 0x6a01c520}
(gdb)
(gdb)
(gdb)frame 7
(gdb) p peer->obuf
$18 = (struct stream_fifo *) 0x95f5680
(gdb) p peer->obuf[0]
$19 = {mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}},
    __size = '\000' <repeats 39 times>, __align = 0}, count = 49, head = 0x14f958000, tail = 0x286b1c00}
(gdb) p peer->obuf[0]->head
$20 = (struct stream *) 0x14f958000
(gdb) p peer->obuf[0]->head [0]
$21 = {next = 0x320ff3c0, getp = 0, endp = 530, size = 530, data = 0x14f958020 '\377' <repeats 16 times>, "\002\022\002"}
(gdb) p peer->obuf[0]->head [0]->next
$22 = (struct stream *) 0x320ff3c0
(gdb) p peer->obuf[0]->head [0]->next [0]
$23 = {next = 0x1794be580, getp = 0, endp = 530, size = 530, data = 0x320ff3e0 '\377' <repeats 16 times>, "\002\022\002"}
(gdb) p peer->obuf[0]->head [0]->next [0]->next [0]
$24 = {next = 0x1455c30c0, getp = 0, endp = 286, size = 286, data = 0x1794be5a0 '\377' <repeats 16 times>, "\001\036\002"}
(gdb) p peer->obuf[0]->head [0]->next [0]->next [0]->next [0]
$25 = {next = 0x426bc000, getp = 0, endp = 286, size = 286, data = 0x1455c30e0 '\377' <repeats 16 times>, "\001\036\002"}
(gdb) p peer->obuf[0]->head [0]->next [0]->next [0]->next [0]->next [0]
$26 = {next = 0xead50000, getp = 0, endp = 4006, size = 4006, data = 0x426bc020 '\377' <repeats 16 times>, "\017\246\002"}
(gdb) p peer->obuf[0]->head [0]->next [0]->next [0]->next [0]->next [0]->next [0]
$27 = {next = 0x39353700, getp = 0, endp = 4006, size = 4006, data = 0xead50020 '\377' <repeats 16 times>, "\017\246\002"}
(gdb) p peer->obuf[0]->head [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]
$28 = {next = 0x6a01c500, getp = 0, endp = 2686, size = 2686, data = 0x39353720 '\377' <repeats 16 times>, "\n~\002"}
(gdb) p peer->obuf[0]->head [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]
$29 = {next = 0x26f83800, getp = 18446744073709551615, endp = 1206, size = 1206, data = 0x6a01c520 '\377' <repeats 16 times>, "\004\266\002"}
(gdb) p peer->obuf[0]->head [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]
$32 = {next = 0x1c688ca0, getp = 18446744073709551615, endp = 1606, size = 1606, data = 0x26f83820 '\377' <repeats 16 times>, "\006F\002"}
(gdb) p peer->obuf[0]->head [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]
$33 = {next = 0x1dc693a0, getp = 18446744073709551615, endp = 185, size = 185, data = 0x1c688cc0 '\377' <repeats 16 times>}
(gdb) p peer->obuf[0]->head [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]
$34 = {next = 0x10e563260, getp = 18446744073709551615, endp = 185, size = 185, data = 0x1dc693c0 '\377' <repeats 16 times>}
(gdb) p peer->obuf[0]->head [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]
$35 = {next = 0x160b9cb10, getp = 18446744073709551615, endp = 169, size = 169, data = 0x10e563280 '\377' <repeats 16 times>}
(gdb) p peer->obuf[0]->head [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]
$36 = {next = 0x628e3900, getp = 18446744073709551615, endp = 169, size = 169, data = 0x160b9cb30 '\377' <repeats 16 times>}
(gdb) p peer->obuf[0]->head [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]->next [0]
$37 = {next = 0x250c1400, getp = 18446744073709551615, endp = 1146, size = 1146, data = 0x628e3920 '\377' <repeats 16 times>, "\004z\002"}
(gdb)

BGP crash at function stream_pnt because the value of 'getp' is incorrect, under normal circumstances, the value of 'getp' is 0, but in exceptional cases, its value is 0xffffffffffffffff.
Has anyone encountered a similar issue?

[ton31337]

8.0 is way too old. Can you test with the latest release? Even more, how can we reproduce this?

[forsunwell]

8.0 is way too old. Can you test with the latest release? Even more, how can we reproduce this?

I created 100 bgp peers, 96 peers of them received several routes, and 4 RR peers of them received 50,000 routes. And then reset the 4 RR peers randomly, the BGP crashed 3 times in one week, the backtrace is similar like the above.

[ton31337]

Can you test with 8.5.2?

[forsunwell]

It's difficult for us to update the FRR version from 8.0 to 8.5.2, is there any similar issue?

[ton31337]

Not aware of such a thing on the latest releases.

[github-actions]

This issue is stale because it has been open 180 days with no activity. Comment or remove the autoclose label in order to avoid having this issue closed.

[frrbot]

This issue will be automatically closed in the specified period unless there is further activity.