FRR 9.0.1 BGP Restart - Thread starvation #14531

SwimGeek · 2023-10-03T20:15:11Z

SwimGeek
Oct 3, 2023

Describe the bug

BGP process restarted with thread starvation messages in the log. Killed by watchfrr.

Did you check if this is a duplicate issue?
Did you test it on the latest FRRouting/frr master branch?

Versions

OS Version: Debian 11.7
Kernel: 5.10.191
FRR Version: 9.0.1

Log

Oct  3 20:56:43 cpt-ter-r1 bgpd[830]: [PA8ED-YES9R] `internet` community is deprecated
Oct  3 20:56:43 cpt-ter-r1 bgpd[830]: [PA8ED-YES9R] `internet` community is deprecated
Oct  3 20:56:43 cpt-ter-r1 bgpd[830]: [PA8ED-YES9R] `internet` community is deprecated
Oct  3 20:56:43 cpt-ter-r1 bgpd[830]: [PA8ED-YES9R] `internet` community is deprecated
Oct  3 21:26:53 cpt-ter-r1 bgpd[830]: [PA8ED-YES9R] `internet` community is deprecated
Oct  3 21:26:53 cpt-ter-r1 bgpd[830]: [PA8ED-YES9R] `internet` community is deprecated
Oct  3 21:31:50 cpt-ter-r1 bgpd[830]: [HKQ2F-8D0MY][EC 100663315] Thread Starvation: {(thread *)0x7f924c947d80 arg=0x55bf190dd0b0 timer  r=-28.699    (bgp_generate_updgrp_
packets)() (&peer->t_generate_updgrp_packets) from ../bgpd/bgp_io.c:148} was scheduled to pop greater than 4s ago
Oct  3 21:32:24 cpt-ter-r1 bgpd[830]: [HKQ2F-8D0MY][EC 100663315] Thread Starvation: {(thread *)0x55bfa3a8c890 arg=0x55bf19741e60 timer  r=-19.266    (bgp_generate_updgrp_
packets)() (&peer->t_generate_updgrp_packets) from ../bgpd/bgp_io.c:148} was scheduled to pop greater than 4s ago
Oct  3 21:33:02 cpt-ter-r1 bgpd[830]: [HKQ2F-8D0MY][EC 100663315] Thread Starvation: {(thread *)0x7f924c846de0 arg=0x55bf18eebe20 timer  r=-5.319     bgp_announce_route_ti
mer_expired() &paf->t_announce_route from ../bgpd/bgp_route.c:5142} was scheduled to pop greater than 4s ago
Oct  3 21:34:27 cpt-ter-r1 watchfrr[665]: [T58XM-TP956][EC 268435457] bgpd state -> unresponsive : no response yet to ping sent 90 seconds ago
Oct  3 21:34:27 cpt-ter-r1 watchfrr[665]: [YFT0P-5Q5YX] Forked background command [pid 2879548]: /usr/lib/frr/watchfrr.sh restart bgpd
Oct  3 21:34:28 cpt-ter-r1 bgpd[830]: [ZW1GY-R46JE] Terminating on signal
Oct  3 21:34:29 cpt-ter-r1 bgpd[830]: [QYZDQ-4PHG5][EC 100663316] Attempting to process an I/O event but for fd: 73(8) no thread to handle this!
Oct  3 21:34:47 cpt-ter-r1 watchfrr[665]: [ZE9RA-19PS5] restart bgpd child process 2879548 still running after 20 seconds, sending signal 15
Oct  3 21:34:47 cpt-ter-r1 watchfrr[665]: [SK7QP-A2GT9] restart bgpd process 2879548 terminated due to signal 15
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 196.60.9.161 6/3 (Cease/Peer De-configured) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 196.60.9.162 6/3 (Cease/Peer De-configured) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 2001:43f8:6d0::9:161 6/3 (Cease/Peer De-configured) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 2001:43f8:6d0::9:162 6/3 (Cease/Peer De-configured) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 2001:43f8:6d1::165 6/3 (Cease/Peer De-configured) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 196.60.8.42 6/3 (Cease/Peer De-configured) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 2001:43f8:6d0::42 6/3 (Cease/Peer De-configured) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 196.60.70.126 6/3 (Cease/Peer De-configured) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 2001:43f8:6d1::126 6/3 (Cease/Peer De-configured) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 196.60.70.35 6/3 (Cease/Peer De-configured) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 2001:43f8:6d1::14 6/3 (Cease/Peer De-configured) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 196.60.70.36 6/3 (Cease/Peer De-configured) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 2001:43f8:6d1::15 6/3 (Cease/Peer De-configured) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 196.60.9.161 6/2 (Cease/Administrative Shutdown) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 196.60.9.162 6/2 (Cease/Administrative Shutdown) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 2001:43f8:6d0::9:161 6/2 (Cease/Administrative Shutdown) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 2001:43f8:6d0::9:162 6/2 (Cease/Administrative Shutdown) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 2001:43f8:6d1::165 6/2 (Cease/Administrative Shutdown) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 196.60.8.42 6/2 (Cease/Administrative Shutdown) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 2001:43f8:6d0::42 6/2 (Cease/Administrative Shutdown) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 196.60.70.126 6/2 (Cease/Administrative Shutdown) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 2001:43f8:6d1::126 6/2 (Cease/Administrative Shutdown) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 196.60.70.35 6/2 (Cease/Administrative Shutdown) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 2001:43f8:6d1::14 6/2 (Cease/Administrative Shutdown) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 196.60.70.36 6/2 (Cease/Administrative Shutdown) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor 2001:43f8:6d1::15 6/2 (Cease/Administrative Shutdown) 0 bytes
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.113(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::113(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.8.212(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::212(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.105(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.110(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::105(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::110(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.105(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.110(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::9:105(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::9:110(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.161(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.162(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::9:161(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::9:162(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.8.10(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::2635(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.223.22.3(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:1f1::3(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.223.22.1(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.223.22.2(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:1f1::1(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:1f1::2(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.198(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::198(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.8.198(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::198(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.71.14(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.71.114(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::71:14(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::71:114(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.15(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.16(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::934(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::2934(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.165(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::165(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:52 cpt-ter-r1 watchfrr[665]: [YFT0P-5Q5YX] Forked background command [pid 2881005]: /usr/lib/frr/watchfrr.sh restart bgpd
Oct  3 21:36:12 cpt-ter-r1 watchfrr[665]: [ZE9RA-19PS5] restart bgpd child process 2881005 still running after 20 seconds, sending signal 15
Oct  3 21:36:12 cpt-ter-r1 watchfrr[665]: [SK7QP-A2GT9] restart bgpd process 2881005 terminated due to signal 15
Oct  3 21:37:08 cpt-ter-r1 zebra[757]: [VXKFG-8SJRV][EC 4043309121] Client 'vnc' encountered an error and is shutting down.
Oct  3 21:37:08 cpt-ter-r1 zebra[757]: [GE156-FS0MJ][EC 100663299] stream_read_try: read failed on fd 17: Connection reset by peer
Oct  3 21:37:08 cpt-ter-r1 zebra[757]: [VXKFG-8SJRV][EC 4043309121] Client 'bgp' encountered an error and is shutting down.
Oct  3 21:37:08 cpt-ter-r1 watchfrr[665]: [HD38Q-0HBRT][EC 268435457] bgpd state -> down : unexpected read error: Connection reset by peer
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.8.42(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::42(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.223.22.10(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:1f1::10(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.126(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::126(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.8.126(Unknown) in vrf default Down Neighbor deleted
:
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.162(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::9:161(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::9:162(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.8.10(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::2635(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.223.22.3(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:1f1::3(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.223.22.1(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.223.22.2(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:1f1::1(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:1f1::2(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.198(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::198(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.8.198(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::198(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.71.14(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.71.114(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::71:14(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::71:114(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.15(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.16(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::934(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::2934(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.165(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::165(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:52 cpt-ter-r1 watchfrr[665]: [YFT0P-5Q5YX] Forked background command [pid 2881005]: /usr/lib/frr/watchfrr.sh restart bgpd
Oct  3 21:36:12 cpt-ter-r1 watchfrr[665]: [ZE9RA-19PS5] restart bgpd child process 2881005 still running after 20 seconds, sending signal 15
Oct  3 21:36:12 cpt-ter-r1 watchfrr[665]: [SK7QP-A2GT9] restart bgpd process 2881005 terminated due to signal 15
Oct  3 21:37:08 cpt-ter-r1 zebra[757]: [VXKFG-8SJRV][EC 4043309121] Client 'vnc' encountered an error and is shutting down.
Oct  3 21:37:08 cpt-ter-r1 zebra[757]: [GE156-FS0MJ][EC 100663299] stream_read_try: read failed on fd 17: Connection reset by peer
Oct  3 21:37:08 cpt-ter-r1 zebra[757]: [VXKFG-8SJRV][EC 4043309121] Client 'bgp' encountered an error and is shutting down.
Oct  3 21:37:08 cpt-ter-r1 watchfrr[665]: [HD38Q-0HBRT][EC 268435457] bgpd state -> down : unexpected read error: Connection reset by peer
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.8.42(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::42(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.223.22.10(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:1f1::10(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.126(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::126(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.8.126(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::126(Unknown) in vrf default Down Neighbor deleted
:
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.162(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::9:161(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::9:162(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.8.10(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::2635(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.223.22.3(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:1f1::3(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.223.22.1(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.223.22.2(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:1f1::1(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:1f1::2(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.198(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::198(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.8.198(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::198(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.71.14(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.71.114(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::71:14(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::71:114(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.15(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.16(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::934(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::2934(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.165(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:29 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::165(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:35:52 cpt-ter-r1 watchfrr[665]: [YFT0P-5Q5YX] Forked background command [pid 2881005]: /usr/lib/frr/watchfrr.sh restart bgpd
Oct  3 21:36:12 cpt-ter-r1 watchfrr[665]: [ZE9RA-19PS5] restart bgpd child process 2881005 still running after 20 seconds, sending signal 15
Oct  3 21:36:12 cpt-ter-r1 watchfrr[665]: [SK7QP-A2GT9] restart bgpd process 2881005 terminated due to signal 15
Oct  3 21:37:08 cpt-ter-r1 zebra[757]: [VXKFG-8SJRV][EC 4043309121] Client 'vnc' encountered an error and is shutting down.
Oct  3 21:37:08 cpt-ter-r1 zebra[757]: [GE156-FS0MJ][EC 100663299] stream_read_try: read failed on fd 17: Connection reset by peer
Oct  3 21:37:08 cpt-ter-r1 zebra[757]: [VXKFG-8SJRV][EC 4043309121] Client 'bgp' encountered an error and is shutting down.
Oct  3 21:37:08 cpt-ter-r1 watchfrr[665]: [HD38Q-0HBRT][EC 268435457] bgpd state -> down : unexpected read error: Connection reset by peer
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.8.42(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::42(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.223.22.10(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:1f1::10(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.126(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::126(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.8.126(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::126(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.215(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::215(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.8.215(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::215(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.47(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.147(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::47(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::147(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.1(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::1(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.2(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.3(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::2(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::3(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.1(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::1(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.2(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.3(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::2(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::3(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.35(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::14(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.36(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::15(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.183(localhost.localdomain) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::9:183(localhost.localdomain) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.68(route-views.napafrica.routeviews.org) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::9:68(route-views.napafrica.routeviews.org) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.21(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.22(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::9:21(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::9:22(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.250.236.145(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2c0f:f748:106::51(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 197.157.79.49(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2c0f:fa90:200:2:1::29(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.20(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.70.21(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::20(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d1::21(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.23(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.24(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::9:23(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::9:24(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.72(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.60.9.73(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::9:72(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:6d0::9:73(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 196.223.22.239(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:08 cpt-ter-r1 bgpd[830]: [PXVXG-TFNNT] %ADJCHANGE: neighbor 2001:43f8:1f1::239(Unknown) in vrf default Down Neighbor deleted
Oct  3 21:37:09 cpt-ter-r1 zebra[757]: [JPSA8-5KYEA] client 31 disconnected 0 vnc routes removed from the rib
Oct  3 21:37:09 cpt-ter-r1 zebra[757]: [S929C-NZR3N] client 31 disconnected 0 vnc nhgs removed from the rib
Oct  3 21:37:10 cpt-ter-r1 zebra[757]: [JPSA8-5KYEA] client 17 disconnected 1086183 bgp routes removed from the rib
Oct  3 21:37:10 cpt-ter-r1 zebra[757]: [S929C-NZR3N] client 17 disconnected 0 bgp nhgs removed from the rib
Oct  3 21:38:13 cpt-ter-r1 watchfrr[665]: [YFT0P-5Q5YX] Forked background command [pid 2882824]: /usr/lib/frr/watchfrr.sh restart bgpd
Oct  3 21:38:13 cpt-ter-r1 zebra[757]: [V98V0-MTWPF] client 17 says hello and bids fair to announce only bgp routes vrf=0
Oct  3 21:38:13 cpt-ter-r1 zebra[757]: [V98V0-MTWPF] client 31 says hello and bids fair to announce only vnc routes vrf=0
Oct  3 21:38:14 cpt-ter-r1 bgpd[2882833]: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
Oct  3 21:38:14 cpt-ter-r1 watchfrr[665]: [QDG3Y-BY5TN] bgpd state -> up : connect succeeded
Oct  3 21:38:18 cpt-ter-r1 bgpd[2882833]: [N9HHH-F8H1M] %ADJCHANGE: neighbor 196.223.22.3(Unknown) in vrf default Up
Oct  3 21:38:18 cpt-ter-r1 bgpd[2882833]: [N9HHH-F8H1M] %ADJCHANGE: neighbor 2001:43f8:1f1::1(Unknown) in vrf default Up
Oct  3 21:38:18 cpt-ter-r1 bgpd[2882833]: [N9HHH-F8H1M] %ADJCHANGE: neighbor 196.223.22.1(Unknown) in vrf default Up
Oct  3 21:38:18 cpt-ter-r1 bgpd[2882833]: [N9HHH-F8H1M] %ADJCHANGE: neighbor 2001:43f8:1f1::3(Unknown) in vrf default Up
Oct  3 21:38:18 cpt-ter-r1 bgpd[2882833]: [N9HHH-F8H1M] %ADJCHANGE: neighbor 196.223.22.2(Unknown) in vrf default Up
Oct  3 21:38:18 cpt-ter-r1 bgpd[2882833]: [N9HHH-F8H1M] %ADJCHANGE: neighbor 2001:43f8:1f1::2(Unknown) in vrf default Up
Oct  3 21:38:18 cpt-ter-r1 bgpd[2882833]: [SKDH2-JPXCG][EC 33554502] 196.60.70.215 Unexpected afi/safi/next-hop afi: IPv4/multicast/2 in Extended Next-hop capability, ignoring
Oct  3 21:38:18 cpt-ter-r1 bgpd[2882833]: [N9HHH-F8H1M] %ADJCHANGE: neighbor 196.60.70.215(Unknown) in vrf default Up
Oct  3 21:38:18 cpt-ter-r1 bgpd[2882833]: [SKDH2-JPXCG][EC 33554502] 2001:43f8:6d1::215 Unexpected afi/safi/next-hop afi: IPv4/multicast/2 in Extended Next-hop capability, ignoring
Oct  3 21:38:18 cpt-ter-r1 bgpd[2882833]: [N9HHH-F8H1M] %ADJCHANGE: neighbor 2001:43f8:6d1::215(Unknown) in vrf default Up
Oct  3 21:38:18 cpt-ter-r1 bgpd[2882833]: [SKDH2-JPXCG][EC 33554502] 196.60.70.126 Unexpected afi/safi/next-hop afi: IPv4/multicast/2 in Extended Next-hop capability, ignoring

ahmdzaki18 · 2023-10-03T20:58:16Z

ahmdzaki18
Oct 3, 2023

Try add timeout on watchfrr_options="-T 180 -t 180"
Seems your cpu core too slow to process the prefixes and watchfrr thought bgpd hang and need to kill bgpd

2 replies

SwimGeek Oct 4, 2023
Author

Hi, could the thread starvation be caused by using RPKI?

Our monitoring shows this server has around 30% free CPU resources, but maybe an RPKI process made it unresponsive for more than 90 sec.

ton31337 Oct 4, 2023
Collaborator

it might be, but to figure this out we need more details. Like perf data and/or flamegraph.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FRR 9.0.1 BGP Restart - Thread starvation #14531

{{title}}

Replies: 1 comment 2 replies

{{title}}

{{title}}

{{title}}

Select a reply

FRR 9.0.1 BGP Restart - Thread starvation #14531

SwimGeek Oct 3, 2023

Replies: 1 comment · 2 replies

ahmdzaki18 Oct 3, 2023

SwimGeek Oct 4, 2023 Author

ton31337 Oct 4, 2023 Collaborator

SwimGeek
Oct 3, 2023

Replies: 1 comment 2 replies

ahmdzaki18
Oct 3, 2023

SwimGeek Oct 4, 2023
Author

ton31337 Oct 4, 2023
Collaborator