Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IPV6 multipath routes not be installed after flap another interfaces #14160

Open
alisenkov opened this issue Aug 8, 2023 · 12 comments
Open

IPV6 multipath routes not be installed after flap another interfaces #14160

alisenkov opened this issue Aug 8, 2023 · 12 comments
Labels
autoclose bgp triage Needs further investigation

Comments

@alisenkov
Copy link

alisenkov commented Aug 8, 2023 •
edited
Loading

  • frr-9.1-dev-213-g7415f1e12
  • Ubuntu 22.04.2 LTS
  • Ubuntu 5.15.0-78.85-generic 5.15.99

Hello everyone, we have problem with ECMP routes on Ubuntu 22.04, we tried FRR version 8.5, 7.1, 9.1 - problem reproduced on all versions.

config FRR:

Current configuration:
!
frr version 9.1-dev999
frr defaults traditional
hostname front3
log file /var/log/frr/frr-debug.log
log syslog warnings
no ipv6 forwarding
service integrated-vtysh-config
!
debug zebra kernel
debug zebra nexthop
!
router bgp 65119
 bgp router-id 100.64.1.1
 bgp bestpath as-path multipath-relax
 timers bgp 10 30
 neighbor pgv6-0 peer-group
 neighbor pgv6-0 remote-as 65110
 neighbor enp59s0 interface peer-group pgv6-0
 neighbor enp59s0d1 interface peer-group pgv6-0
 !
 address-family ipv4 unicast
  network 100.64.1.1/32
  neighbor pgv6-0 soft-reconfiguration inbound
  neighbor pgv6-0 route-map allow-all in
  neighbor pgv6-0 route-map Lo out
 exit-address-family
exit
!
ip prefix-list Lo0 seq 5 permit 100.64.1.1/32
!
route-map Lo permit 10
 match ip address prefix-list Lo0
exit
!
route-map Lo deny 100

route-map allow-all permit 10
exit
!

The server has two 10GE-interfaces (enp59s0, enp59s0d1) for transmit production traffic and one(eno1) for OOB(management).
We use IPV6 ND for links between Server and L3-SW, on this link establish EBGP-peering.

When normal situation - FRR and kernel have two ECMP default route to EBGP-neighbors:

front3# sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>* 0.0.0.0/0 [20/0] via fe80::c2d6:82ff:fef4:3cc4, enp59s0, weight 1, 00:23:01
                     via fe80::c2d6:82ff:fef4:428c, enp59s0d1, weight 1, 00:23:01

And this multipath default route correct install to kernel:

root@front3:/home/admin# ip ro
default nhid 22 proto bgp metric 20
	nexthop via inet6 fe80::c2d6:82ff:fef4:3cc4 dev enp59s0 weight 1
	nexthop via inet6 fe80::c2d6:82ff:fef4:428c dev enp59s0d1 weight 1

root@front3:/home/admin# ip nexthop ls
id 28 dev lo scope link proto zebra
id 29 dev br-f59be21aad6b scope link proto zebra
id 33 dev enp59s0 scope link proto zebra
id 34 group 35/36 proto zebra
id 35 via fe80::c2d6:82ff:fef4:3cc4 dev enp59s0 scope link proto zebra
id 36 via fe80::c2d6:82ff:fef4:428c dev enp59s0d1 scope link proto zebra

BUT, when we shutdown/recable interface eno1, we seems that FRR cant push same routes(default) to kernel, kernel lost default route and server becomes inaccessible for production traffic.

important point - If we do not use multipath, only one interface - after flap route correct installed to kernel, PROBLEM ACTUAL WHEN MULTIPATH USE.

also helps "service frr restart" - after restarting FRR can correct install ECMP default route to kernel:


root@front3:/home/alisenkov# ip ro
10.177.1.0/24 dev eno1 proto kernel scope link src 10.177.1.14 metric 100
10.177.1.254 dev eno1 proto dhcp scope link src 10.177.1.14 metric 100
root@front3:/home/alisenkov# service frr restart
root@front3:/home/alisenkov# ip ro
**default nhid 113 proto bgp metric 20
	nexthop via inet6 fe80::c2d6:82ff:fef4:3cc4 dev enp59s0 weight 1
	nexthop via inet6 fe80::c2d6:82ff:fef4:428c dev enp59s0d1 weight 1**
10.177.1.0/24 dev eno1 proto kernel scope link src 10.177.1.14 metric 100
10.177.1.254 dev eno1 proto dhcp scope link src 10.177.1.14 metric 100
root@front3:/home/alisenkov#

logs:

zebra logs:
2023/08/08 13:07:07 ZEBRA: [Z9NFT-4F8FQ] Intf eno1(2) has gone DOWN
2023/08/08 13:07:07 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7f0448027130, op INTF_ADDR_DEL, ifindex (2), result QUEUED
2023/08/08 13:07:07 ZEBRA: [MZPZA-W042K] zebra_if_addr_update_ctx: INTF_ADDR_DEL: ifindex eno1(2), addr fe80::4ed9:8fff:fe41:cc34/64
2023/08/08 13:07:07 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEXTHOP(105), len=60, seq=15117091, pid=1
2023/08/08 13:07:07 ZEBRA: [KBCV5-6W9G6] RTM_DELNEXTHOP ID (16)  NS 0
2023/08/08 13:07:07 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (16[17/18]) that we are still using for a route, sending it back down
2023/08/08 13:07:07 ZEBRA: [VNMVB-91G3G] _netlink_nexthop_build_group: ID (16): group 17/18
2023/08/08 13:07:07 ZEBRA: [R43C6-KYHWT] netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=16

journalctl:
Aug 08 12:34:39 front3 frrinit.sh[5715]: [5715|zebra] sending configuration
Aug 08 12:34:39 front3 frrinit.sh[5715]: [5715|zebra] done
Aug 08 12:34:39 front3 watchfrr[5682]: [QDG3Y-BY5TN] zebra state -> up : connect succeeded
Aug 08 13:00:14 front3 zebra[5697]: [HSYZM-HV7HF] Extended Error: Nexthop device is not up
Aug 08 13:00:14 front3 zebra[5697]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: Network is down, type=RTM_NEWNEXTHOP(104), seq=16, pid=2335995380
Aug 08 13:00:14 front3 zebra[5697]: [X5XE1-RS0SW][EC 4043309074] Failed to install Nexthop (21[if 2 vrfid 0]) into the kernel
Aug 08 13:00:14 front3 zebra[5697]: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (22[23/24]) that we are still using for a route, sending it back down
Aug 08 13:00:27 front3 zebra[5697]: [N5M5Y-J5BPG][EC 4043309121] Client 'static' (session id 0) encountered an error and is shutting down.
Aug 08 13:00:27 front3 zebra[5697]: [KQB7H-NPVW9] zebra/zebra_ptm.c:1333 failed to find process pid registration
Aug 08 13:00:27 front3 zebra[5697]: [KQB7H-NPVW9] zebra/zebra_ptm.c:1333 failed to find process pid registration

Aug 08 13:00:14 front3 zebra[5697]: [X5XE1-RS0SW][EC 4043309074] Failed to install Nexthop (21[if 2 vrfid 0]) into the kernel

Click to expand

2023/08/08 12:12:45 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (25[if 6 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 12:12:45 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (25[if 6 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 12:12:45 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (33[if 2 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 12:12:45 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (38[if 2 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 12:12:45 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (37[if 2 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 12:12:45 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (25[if 6 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 12:12:45 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (33[if 2 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 12:12:45 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (38[if 2 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 12:12:45 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (37[if 2 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 12:15:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (36[37/38]) that we are still using for a route, sending it back down
2023/08/08 13:00:14 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (22[23/24]) that we are still using for a route, sending it back down
2023/08/08 13:07:07 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (16[17/18]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (51[fe80::c2d6:82ff:fef4:428c if 7 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel updated a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (51[fe80::c2d6:82ff:fef4:428c if 7 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel updated a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (50[fe80::c2d6:82ff:fef4:3cc4 if 6 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel updated a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (48[if 6 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (50[fe80::c2d6:82ff:fef4:3cc4 if 6 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel updated a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (48[if 6 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (43[if 1 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (51[fe80::c2d6:82ff:fef4:428c if 7 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 13:50:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel updated a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down
2023/08/08 13:50:03 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (50[fe80::c2d6:82ff:fef4:3cc4 if 6 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 13:50:03 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel updated a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down
2023/08/08 13:50:03 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (48[if 6 vrfid 0]) that we are still using for a route, sending it back down
2023/08/08 13:50:03 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (49[50/51]) that we are still using for a route, sending it back down

Whatever

For some reason FRR is unable to set the route to the kernel, please help me to understand the reason...
@alisenkov alisenkov added the triage Needs further investigation label Aug 8, 2023
@ton31337
Copy link
Member

ton31337 commented Aug 9, 2023

BUT, when we shutdown/recable interface eno1, we seems that FRR cant push same routes(default) to kernel, kernel lost default route and server becomes inaccessible for production traffic.

You mean disabling OOB link, default routes disappear from the kernel and are not reinstalled back again until you restart FRR?

@ton31337 ton31337 added the bgp label Aug 9, 2023
@alisenkov
Copy link
Author

You mean disabling OOB link, default routes disappear from the kernel and are not reinstalled back again until you restart FRR?

yes, default ecmp route from bgp with ipv6 nexthop reinstalled back only when restart FRR.

@alisenkov
Copy link
Author

we found interest issue - https://www.spinics.net/lists/netdev/msg863121.html

> > > > > when an IPv4 route gets removed because its nexthop was deleted, the
> > > > > kernel does not send a RTM_DELROUTE netlink notifications anymore in
> > > > > 6.1. A bisect lead me to 61b91eb33a69 ("ipv4: Handle attempt to delete
> > > > > multipath route when fib_info contains an nh reference"), and
> > > > > reverting it makes it work again.

@ton31337
Copy link
Member

Can you show the full ip route output, also show ip route, show ip bgp, because I don't see any interfaces, routes related to eno1 interface.

@alisenkov
Copy link
Author

alisenkov commented Aug 15, 2023 •
edited
Loading

Can you show the full ip route output, also show ip route, show ip bgp, because I don't see any interfaces, routes related to eno1 interface.

s1# sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>* 0.0.0.0/0 [20/0] via fe80::c2d6:82ff:fef4:3cc4, ens1f1np1, weight 1, 01w5d10h
  *                  via fe80::c2d6:82ff:fef4:428c, ens1f0np0, weight 1, 01w5d10h
K>* 10.4.0.0/16 [0/100] via 10.177.1.254, eno1, src 10.177.1.31, 05w0d16h
C>* 10.177.1.0/24 is directly connected, eno1, 05w0d16h        <<<<<<<<<
C>* 192.168.100.34/32 is directly connected, lo, 05w5d18h
C>* 192.168.100.51/32 is directly connected, lo, 05w5d18h
C>* 192.168.100.54/32 is directly connected, lo, 05w5d18h
K>* 198.18.0.0/15 [0/100] via 10.177.1.254, eno1, src 10.177.1.31, 05w0d16h
s1# sh ip bgp
BGP table version is 10, local router ID is 192.168.100.34, vrf id 0
Default local pref 100, local AS 65119
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *= 0.0.0.0/0        ens1f0np0                              0 65110 ?
 *>                  ens1f1np1                              0 65110 ?
 *> 192.168.100.34/32 0.0.0.0                  0         32768 i
 *> 192.168.100.51/32 0.0.0.0                  0         32768 i
 *> 192.168.100.54/32 0.0.0.0                  0         32768 i

Displayed  4 routes and 5 total paths
s1# exit
root@s1:/home/alisenkov# ip ro
default nhid 68 via inet6 fe80::c2d6:82ff:fef4:3cc4 dev ens1f1np1 proto bgp metric 20. <<<<< its also strange because FRR has 2 ECMP routes to default, but kernel has only one
10.4.0.0/16 via 10.177.1.254 dev eno1 proto dhcp src 10.177.1.31 metric 100
10.177.1.0/24 dev eno1 proto kernel scope link src 10.177.1.31
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
198.18.0.0/15 via 10.177.1.254 dev eno1 proto dhcp src 10.177.1.31 metric 100
root@s1:/home/alisenkov#

root@s1:/home/alisenkov# ip nexthop
id 7 dev lo scope host proto zebra
id 10 dev ens1f0np0 scope link proto zebra
id 27 dev eno1 scope host proto zebra
id 28 via 10.177.1.254 dev eno1 scope link proto zebra
id 35 via fe80::c2d6:82ff:fef4:3cc4 dev ens1f1np1 scope link proto zebra
id 51 via fe80::c2d6:82ff:fef4:428c dev ens1f0np0 scope link proto zebra
id 68 group 35 proto zebra
root@s1:/home/alisenkov#

@osfrickler
Copy link

I'm seeing the same issue. To me it looks like zebra is reinstalling the nexthop groups, but the routes that were using the nhgs are not getting restored.

Some more possibly interesting notes:

  • Only IPv4 routes are affected, not IPv6
  • Only multipath routes are affected, routes with a single path will stay in the table
  • The issue only seems to happen on Ubuntu 22.04, I've not been able to reproduce it with either Debian 11 or 12

@akunszt
Copy link

akunszt commented Oct 5, 2023

It is a systemd-networkd issue at the first place. It removes the nexthop groups unconditionally. IDK which version introduced this bug but it is present in v252 and (very likely) newer versions.
Please check: systemd/systemd#29034

@huangfeilong1
Copy link

May have fixed it：#14080

@sysoleg
Copy link
Contributor

sysoleg commented Nov 30, 2023 •
edited
Loading

@alisenkov The problem seems to be related to systemd-networkd. What is the normal output of your networkctl? Please, check how your case relates to this scenario: systemd/systemd#29034 (comment)

@sysoleg
Copy link
Contributor

sysoleg commented Nov 30, 2023

@huangfeilong1 Seems completely unrelated.

@modzilla99
Copy link

modzilla99 commented Dec 1, 2023 •
edited
Loading

May have fixed it：#14080

Thanks, this fixed it for us (FRR 8.5 on Ubuntu 20.04)!

Interestingly, we don't face that problem with the system packages of systemd and frr on 22.04.

@github-actions GitHub Actions
Copy link

This issue is stale because it has been open 180 days with no activity. Comment or remove the autoclose label in order to avoid having this issue closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
autoclose bgp triage Needs further investigation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@ton31337 @sysoleg @modzilla99 @akunszt @alisenkov @osfrickler @huangfeilong1