BGP routes getting removed while shutting down the ZAPI connection #14305

sarungprakash · 2023-08-30T15:07:47Z

sarungprakash
Aug 30, 2023

Hi all,

I have an FRR installed on a server with BGP daemon enabled, plus we have an application code that connects to Zebra over ZAPI as BGP client, so whenever the application program restarts the ZAPI connection shutdown and it ended up removing all the BGP routes from the RIB table.

Found some code ref to remove the BGP routes when the ZAPI connection shutdown:

frr/zebra/zserv.c

Line 565 in 497584a

* Deinitialize zebra client.

Is this the expected behaviour?

Output of show ip bgp is intact and providing the routes learned from BGP peer,

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.10.10.0/24    10.30.112.5              0             0 64514 i
*> 20.20.20.0/24    10.30.112.7              0             0 64515 i

but output of show ip route bgp is empty
Output of show zebra client

Client: bgp
------------------------
FD: 28
Connect Time: 01:02:19
Nexthop Registry Time: 00:01:42
Nexthop Last Update Time: 00:01:37
Client will Not be notified about it's routes status
Last Msg Rx Time: 00:00:31
Last Msg Tx Time: 00:01:26
Last Rcvd Cmd: ZEBRA_ROUTE_ADD
Last Sent Cmd: ZEBRA_INTERFACE_BFD_DEST_UPDATE

Type        Add         Update      Del
==================================================
IPv4        4           0           2
IPv6        0           0           0
Redist:v4   0           0           0
Redist:v6   0           0           0
VRF         1           0           0
Connected   4           0           0
Interface   0           0           0
Intf Addr   4           0           0
BFD peer    0           0           0
NHT v4      2           0           0
NHT v6      0           0           0
VxLAN SG    0           0           0
VNI         0           0           0
L3-VNI      0           0           0
MAC-IP      0           0           0
ES          0           0           0
ES-EVI      0           0           0
Errors: 0


Client: vnc
------------------------
FD: 31
Connect Time: 01:02:19
Not registered for Nexthop Updates
Client will Not be notified about it's routes status
Last Msg Rx Time: 01:02:19
Last Msg Tx Time: 01:02:19

Type        Add         Update      Del
==================================================
IPv4        0           0           0
IPv6        0           0           0
Redist:v4   0           0           0
Redist:v6   0           0           0
VRF         1           0           0
Connected   2           0           0
Interface   0           0           0
Intf Addr   2           0           0
BFD peer    0           0           0
NHT v4      0           0           0
NHT v6      0           0           0
VxLAN SG    0           0           0
VNI         0           0           0
L3-VNI      0           0           0
MAC-IP      0           0           0
ES          0           0           0
ES-EVI      0           0           0
Errors: 0


Client: static
------------------------
FD: 38
Connect Time: 01:02:19
Not registered for Nexthop Updates
Client will be notified about it's routes status
Last Msg Rx Time: 01:02:19
Last Msg Tx Time: 01:02:19

Type        Add         Update      Del
==================================================
IPv4        0           0           0
IPv6        0           0           0
Redist:v4   0           0           0
Redist:v6   0           0           0
VRF         1           0           0
Connected   4           0           0
Interface   0           0           0
Intf Addr   4           0           0
BFD peer    0           0           0
NHT v4      0           0           0
NHT v6      0           0           0
VxLAN SG    0           0           0
VNI         0           0           0
L3-VNI      0           0           0
MAC-IP      0           0           0
ES          0           0           0
ES-EVI      0           0           0
Errors: 0


Client: bfd
------------------------
FD: 43
Connect Time: 01:02:19
Not registered for Nexthop Updates
Client will Not be notified about it's routes status
Last Msg Rx Time: 00:01:26
Last Msg Tx Time: 00:01:26
Last Rcvd Cmd: ZEBRA_BFD_DEST_REPLAY
Last Sent Cmd: ZEBRA_BFD_DEST_REPLAY

Type        Add         Update      Del
==================================================
IPv4        0           0           0
IPv6        0           0           0
Redist:v4   0           0           0
Redist:v6   0           0           0
VRF         1           0           0
Connected   4           0           0
Interface   0           0           0
Intf Addr   4           0           0
BFD peer    0           0           0
NHT v4      0           0           0
NHT v6      0           0           0
VxLAN SG    0           0           0
VNI         0           0           0
L3-VNI      0           0           0
MAC-IP      0           0           0
ES          0           0           0
ES-EVI      0           0           0
Errors: 0


Client: bgp
------------------------
FD: 49
Connect Time: 01:01:08
Not registered for Nexthop Updates
Client will Not be notified about it's routes status
Last Msg Rx Time: 01:01:08
Last Msg Tx Time: 00:00:31
Last Rcvd Cmd: ZEBRA_REDISTRIBUTE_ADD
Last Sent Cmd: ZEBRA_REDISTRIBUTE_ROUTE_ADD

Type        Add         Update      Del
==================================================
IPv4        0           0           0
IPv6        0           0           0
Redist:v4   4           0           2
Redist:v6   0           0           0
VRF         1           0           0
Connected   0           0           0
Interface   0           0           0
Intf Addr   0           0           0
BFD peer    0           0           0
NHT v4      0           0           0
NHT v6      0           0           0
VxLAN SG    0           0           0
VNI         0           0           0
L3-VNI      0           0           0
MAC-IP      0           0           0
ES          0           0           0
ES-EVI      0           0           0
Errors: 0

Answered by mjstapp

Sep 11, 2023

so if I'm reading this correctly, you're doing a couple of things here that are ... likely to cause problems for you. are you writing your own code that uses the zapi library, and using that code of yours to connect to zebra and claim to be bgpd?

a) FRR doesn't support ZAPI as a stable api, and it may change in ways that surprise you from release to release.
b) zebra doesn't expect there to be more than one "bgpd" claiming to be bgp protocol, instance 0, and session_id 0. your closing your own process's ZAPI connection is causing zebra to clear out the real bgpd data because your code claims to be bgpd.

there are several ways to interact with FRR, with bgp and with zebra. what appears to …

View full answer

sarungprakash · 2023-09-04T07:50:20Z

sarungprakash
Sep 4, 2023
Author

Any idea reading this behaviour @ton31337 & @donaldsharp? Thanks in advance.

2 replies

ton31337 Sep 11, 2023
Collaborator

What does the log say?

sarungprakash Sep 11, 2023
Author

2023/08/30 05:35:41 ZEBRA: [VFCDB-S5FKG] connection closed socket [49]
2023/08/30 05:35:41 ZEBRA: [VXKFG-8SJRV][EC 4043309121] Client 'bgp' encountered an error and is shutting down.
2023/08/30 05:35:41 ZEBRA: [MQA3Z-XJPYF] Closing client 'bgp'
2023/08/30 05:35:41 ZEBRA: [YDZ55-W3VM6] release_daemon_table_chunks: Released 0 table chunks
2023/08/30 05:35:41 ZEBRA: [KQB7H-NPVW9] zebra/zebra_ptm.c:1365 failed to find process pid registration
2023/08/30 05:35:41 ZEBRA: [ZJPTG-6AP36] release_daemon_label_chunks: Releasing chunks for client proto bgp, instance 0, session 0
2023/08/30 05:35:41 ZEBRA: [MP8WF-MBM3F] release_daemon_label_chunks: Released 0 label chunks
2023/08/30 05:35:41 ZEBRA: [QE6V0-J8BG5] rib_delnode: (0:254):40.40.40.0/24: rn 0x558e19323750, re 0x558e19324930, removing
2023/08/30 05:35:41 ZEBRA: [QEVVE-G3FQQ] rib_meta_queue_add: (0:254):40.40.40.0/24: queued rn 0x558e19323750 into sub-queue 6
2023/08/30 05:35:41 ZEBRA: [QE6V0-J8BG5] rib_delnode: (0:254):50.50.50.0/24: rn 0x558e1933fea0, re 0x558e1934fdb0, removing
2023/08/30 05:35:41 ZEBRA: [QEVVE-G3FQQ] rib_meta_queue_add: (0:254):50.50.50.0/24: queued rn 0x558e1933fea0 into sub-queue 6
2023/08/30 05:35:41 ZEBRA: [JPSA8-5KYEA] client 49 disconnected 2 bgp routes removed from the rib
2023/08/30 05:35:41 ZEBRA: [S929C-NZR3N] client 49 disconnected 0 bgp nhgs removed from the rib
2023/08/30 05:35:41 ZEBRA: [WMJRH-3VBWX] zserv_client_free: Deleting client bgp
2023/08/30 05:35:41 ZEBRA: [NZNZ4-7P54Y] default(0:254):40.40.40.0/24: Processing rn 0x558e19323750
2023/08/30 05:35:41 ZEBRA: [ZJVZ4-XEGPF] default(0:254):40.40.40.0/24: Examine re 0x558e19324930 (bgp) status: Removed Installed flags: Recursion Selected dist 20 metric 0
2023/08/30 05:35:41 ZEBRA: [JPJF4-TGCY5] default(0:254):40.40.40.0/24: After processing: old_selected 0x558e19324930 new_selected 0x0 old_fib 0x558e19324930 new_fib 0x0
2023/08/30 05:35:41 ZEBRA: [S31W0-H281H] 0:25440.40.40.0/24: Redist del: re 0x558e19324930 (0:bgp), new re 0x0 (0:None)
2023/08/30 05:35:41 ZEBRA: [S59C1-4C5PP] default(0:254):40.40.40.0/24: Deleting route rn 0x558e19323750, re 0x558e19324930 (bgp)
2023/08/30 05:35:41 ZEBRA: [NM15X-X83N9] rib_process: (0:254):40.40.40.0/24: rn 0x558e19323750, removing re 0x558e19324930
2023/08/30 05:35:41 ZEBRA: [Y53JX-CBC5H] rib_unlink: (0:254):40.40.40.0/24: rn 0x558e19323750, re 0x558e19324930
2023/08/30 05:35:41 ZEBRA: [KT8QQ-45WQ0] rib_gc_dest: (0:?):40.40.40.0/24: removing dest from table
2023/08/30 05:35:41 ZEBRA: [QZ1V6-CRT8D] default(0:0):40.40.40.0/24 rn 0x558e19323750 dequeued from sub-queue 6
2023/08/30 05:35:41 ZEBRA: [NZNZ4-7P54Y] default(0:254):50.50.50.0/24: Processing rn 0x558e1933fea0
2023/08/30 05:35:41 ZEBRA: [ZJVZ4-XEGPF] default(0:254):50.50.50.0/24: Examine re 0x558e1934fdb0 (bgp) status: Removed Installed flags: Recursion Selected dist 20 metric 0
2023/08/30 05:35:41 ZEBRA: [JPJF4-TGCY5] default(0:254):50.50.50.0/24: After processing: old_selected 0x558e1934fdb0 new_selected 0x0 old_fib 0x558e1934fdb0 new_fib 0x0
2023/08/30 05:35:41 ZEBRA: [S31W0-H281H] 0:25450.50.50.0/24: Redist del: re 0x558e1934fdb0 (0:bgp), new re 0x0 (0:None)
2023/08/30 05:35:41 ZEBRA: [S59C1-4C5PP] default(0:254):50.50.50.0/24: Deleting route rn 0x558e1933fea0, re 0x558e1934fdb0 (bgp)
2023/08/30 05:35:41 ZEBRA: [NM15X-X83N9] rib_process: (0:254):50.50.50.0/24: rn 0x558e1933fea0, removing re 0x558e1934fdb0
2023/08/30 05:35:41 ZEBRA: [Y53JX-CBC5H] rib_unlink: (0:254):50.50.50.0/24: rn 0x558e1933fea0, re 0x558e1934fdb0
2023/08/30 05:35:41 ZEBRA: [KT8QQ-45WQ0] rib_gc_dest: (0:?):50.50.50.0/24: removing dest from table
2023/08/30 05:35:41 ZEBRA: [QZ1V6-CRT8D] default(0:0):50.50.50.0/24 rn 0x558e1933fea0 dequeued from sub-queue 6
2023/08/30 05:35:41 ZEBRA: [TQKA8-0276P] Not Notifying Owner: bgp about prefix 40.40.40.0/24(254) 3 vrf: 0
2023/08/30 05:35:41 ZEBRA: [TQKA8-0276P] Not Notifying Owner: bgp about prefix 50.50.50.0/24(254) 3 vrf: 0

mjstapp · 2023-09-11T15:54:30Z

mjstapp
Sep 11, 2023
Collaborator

so if I'm reading this correctly, you're doing a couple of things here that are ... likely to cause problems for you. are you writing your own code that uses the zapi library, and using that code of yours to connect to zebra and claim to be bgpd?

a) FRR doesn't support ZAPI as a stable api, and it may change in ways that surprise you from release to release.
b) zebra doesn't expect there to be more than one "bgpd" claiming to be bgp protocol, instance 0, and session_id 0. your closing your own process's ZAPI connection is causing zebra to clear out the real bgpd data because your code claims to be bgpd.

there are several ways to interact with FRR, with bgp and with zebra. what appears to be happening here isn't really a good approach.

1 reply

sarungprakash Sep 11, 2023
Author

makes sense, I should have thought about other approaches like FPM. Thanks @mjstapp

donaldsharp · 2023-09-11T16:03:08Z

donaldsharp
Sep 11, 2023
Maintainer

Basically as Mark is stating you are doing manipulations of the zapi protocol that were never meant to be. We are not going to be spending much time fixing ( or even addressing this type of issue ) since it is outside the scope of how we intend to use zapi

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BGP routes getting removed while shutting down the ZAPI connection #14305

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

BGP routes getting removed while shutting down the ZAPI connection #14305

sarungprakash Aug 30, 2023

Replies: 3 comments · 3 replies

sarungprakash Sep 4, 2023 Author

ton31337 Sep 11, 2023 Collaborator

sarungprakash Sep 11, 2023 Author

mjstapp Sep 11, 2023 Collaborator

sarungprakash Sep 11, 2023 Author

donaldsharp Sep 11, 2023 Maintainer

sarungprakash
Aug 30, 2023

Replies: 3 comments 3 replies

sarungprakash
Sep 4, 2023
Author

ton31337 Sep 11, 2023
Collaborator

sarungprakash Sep 11, 2023
Author

mjstapp
Sep 11, 2023
Collaborator

sarungprakash Sep 11, 2023
Author

donaldsharp
Sep 11, 2023
Maintainer