BGP Routes not updated on kernel #10895

pantxisto · 2022-03-25T12:44:25Z

pantxisto
Mar 25, 2022

Describe the bug
BGP routes are not updated on kernel, when prompted from vtysh with "show ip bgp" the are aadvertised but when cheking zebra routes database with "show ip route" are not showed.

[ x] Did you check if this is a duplicate issue?
[ ] Did you test it on the latest FRRouting/frr master branch?

To Reproduce

Install FRR from packages. 8.2.2 Release on Ubuntu 20.04.
Configure to start automatically with zebra, staticd and bgpd.
Set bgp configuration under /etc/frr/frr.conf

 router bgp <my-asn>
  bgp router-id <router-ip>
  neighbor <master1-ip> remote-as <my-asn>
  neighbor <master2-ip> remote-as <my-asn>
  neighbor <worker1-ip> remote-as <my-asn>
  neighbor <worker2-ip> remote-as <my-asn>
  neighbor <worker3-ip> remote-as <my-asn>

systemctl daemon-reload
systemctl restart frr
On /etc/sysctl.conf add:

net.ipv4.conf.all.forwarding=1

sudo sysctl --system

Expected behavior
Routes advertised from kubernetes speakers should be updated on kernel, but are not updating. "show ip route" should show BGP advertised routes but not happening.

Screenshots
On vtysh:

show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

K>* 0.0.0.0/0 [0/0] via <gateway>, eth0 onlink, 00:39:07
C>* <router-ip>/32 is directly connected, eth0, 00:39:07

On vtysh:

show ip bgp
BGP table version is 0, local router ID is <router-ip>, vrf id 0
Default local pref 100, local AS <my-asn>
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

  Network               Next Hop            Metric LocPrf Weight Path
  i126.26.26.1/32   <master1-ip>                          0           0 ?
  i                           <master2-ip>                          0           0 ?
  i                           <worker1-ip>                          0           0 ?
  i                           <worker2-ip>                          0           0 ?
  i                           <worker3-ip>                          0           0 ?
  i126.26.26.2/32   <master1-ip>                          0           0 ?
  i                           <master2-ip>                          0           0 ?
  i                           <worker1-ip>                          0           0 ?
  i                           <worker2-ip>                          0           0 ?
  i                           <worker3-ip>                          0           0 ?

Displayed  2 routes and 10 total paths

On system:

ip route
default via <gateway> dev eth0 proto static onlink

Versions

OS Version:
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal
Kernel:
5.4.0-104-generic
FRR Version:
frr-stable 8.2.2

Additional context
It's a IBGP session between this router and a kubernetes cluster with 5 nodes using metallb.

taspelund · 2022-03-25T12:47:54Z

taspelund
Mar 25, 2022
Collaborator

Try ip nht resolve-via-default

0 replies

JustInVTime · 2022-03-25T13:24:19Z

JustInVTime
Mar 25, 2022

Same problem here on an Ubuntu 20.04 server with frr 8.2.2 as well. The routes from the kubernetes cluster are visible via show ip bgp but they don't exists in the kernel routing table.

Try ip nht resolve-via-default

I've tried this, but doesn't work

0 replies

pantxisto · 2022-03-25T13:28:43Z

pantxisto
Mar 25, 2022
Author

Wow, very fast. That solves the problem. Now another problem appear. I'm trying to curl 126.26.26.1 ip but it stays pending with no response. I've tryied to add static routes on frr.conf as next but having same behaviour, additionally "show ip routes" doens't show static routes:

ip route 126.26.26.1/32 <master1-ip>
ip route 126.26.26.1/32 <master2-ip>
ip route 126.26.26.1/32 <worker1-ip>
ip route 126.26.26.1/32 <worker2-ip>
ip route 126.26.26.1/32 <worker3-ip>

Previous stats from zebra and system now are the next:

On vtysh:

show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

K>* 0.0.0.0/0 [0/0] via <gateway>, eth0 onlink, 00:01:15
B>  126.26.26.1/32 [200/0] via <master1-ip> (recursive), weight 1, 00:01:14
  *                          via <gateway>, eth0 onlink, weight 1, 00:01:14
                           via <master2-ip> (recursive), weight 1, 00:01:14
                             via <gateway>, eth0 onlink, weight 1, 00:01:14
                           via <worker1-ip> (recursive), weight 1, 00:01:14
                             via <gateway>, eth0 onlink, weight 1, 00:01:14
                           via <worker2-ip> (recursive), weight 1, 00:01:14
                             via <gateway>, eth0 onlink, weight 1, 00:01:14
                           via <worker3-ip> (recursive), weight 1, 00:01:14
                             via <gateway>, eth0 onlink, weight 1, 00:01:14
B>  126.26.26.2/32 [200/0] via <master1-ip> (recursive), weight 1, 00:01:14
  *                          via <gateway>, eth0 onlink, weight 1, 00:01:14
                           via <master2-ip> (recursive), weight 1, 00:01:14
                             via <gateway>, eth0 onlink, weight 1, 00:01:14
                           via <worker1-ip> (recursive), weight 1, 00:01:14
                             via <gateway>, eth0 onlink, weight 1, 00:01:14
                           via <worker2-ip> (recursive), weight 1, 00:01:14
                             via <gateway>, eth0 onlink, weight 1, 00:01:14
                           via <worker3-ip> (recursive), weight 1, 00:01:14
                             via <gateway>, eth0 onlink, weight 1, 00:01:14
C>* <router-ip>/32 is directly connected, eth0, 00:01:15

On vtysh:

show ip bgp
BGP table version is 2, local router ID is <router-ip>, vrf id 0
Default local pref 100, local AS <my-asn>
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

   Network          Next Hop            Metric LocPrf Weight Path
*>i126.26.26.1/32   <master1-ip>                  0      0 ?
*=i                 <master2-ip>                   0      0 ?
*=i                 <worker1-ip>                  0      0 ?
*=i                 <worker2-ip>                     0      0 ?
*=i                 <worker3-ip>                   0      0 ?
*>i126.26.26.2/32   <master1-ip>                   0      0 ?
*=i                 <master2-ip>                   0      0 ?
*=i                 <worker1-ip>                   0      0 ?
*=i                 <worker2-ip>                     0      0 ?
*=i                 <worker3-ip>                   0      0 ?

Displayed  2 routes and 10 total paths

On system:

ip route
default via <gateway> dev eth0 proto static onlink 
126.26.26.1 nhid 87 via <gateway> dev eth0 proto bgp metric 20 onlink 
126.26.26.2 nhid 87 via <gateway> dev eth0 proto bgp metric 20 onlink

0 replies

taspelund · 2022-03-25T13:32:17Z

taspelund
Mar 25, 2022
Collaborator

@JustInVTime I don't have any data around your situation, so I have no idea if the same suggestion would be relevant for you. Can you open a new issue with details from your environment? Then we can follow up there without getting things confused between the original issue and yours

0 replies

taspelund · 2022-03-25T13:34:48Z

taspelund
Mar 25, 2022
Collaborator

@pantxisto can you ping the IP you're trying to curl? Can you run a traceroute against it? What SIP is being used for this traffic? (ip route get <IP>)

If the RIB and kernel routing table look fine, you're probably having an issue with resolving ARP for the next-hop or there's an issue with return traffic. Let's try to find where the packets are getting lost

2 replies

pantxisto Apr 3, 2022
Author

Hello @taspelund . Since yesterday I'm having the problem you described in this comment. The router is receiving advertised ips.
On vtysh with "show ip route" the output is:

Codes: K - kernel route, C - connected, S - static, R - RIP,
      O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
      T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
      f - OpenFabric,
      > - selected route, * - FIB route, q - queued, r - rejected, b - backup
      t - trapped, o - offload failure

K>* 0.0.0.0/0 [0/0] via 194.242.56.1, eth0 onlink, 2d11h36m
C>* 10.0.0.0/24 is directly connected, bgptunnel, 2d11h34m
B>* 133.33.33.33/32 [200/0] via 10.0.0.2, bgptunnel, weight 1, 00:34:20
 *                         via 10.0.0.3, bgptunnel, weight 1, 00:34:20
 *                         via 10.0.0.4, bgptunnel, weight 1, 00:34:20
 *                         via 10.0.0.5, bgptunnel, weight 1, 00:34:20
 *                         via 10.0.0.6, bgptunnel, weight 1, 00:34:20
B>* 133.33.33.34/32 [200/0] via 10.0.0.2, bgptunnel, weight 1, 00:34:20
 *                         via 10.0.0.3, bgptunnel, weight 1, 00:34:20
 *                         via 10.0.0.4, bgptunnel, weight 1, 00:34:20
 *                         via 10.0.0.5, bgptunnel, weight 1, 00:34:20
 *                         via 10.0.0.6, bgptunnel, weight 1, 00:34:20
C>* <router-node-ip>/32 is directly connected, eth0, 2d11h36m

On machine when executing the command "ip route" the output is:

ip route
default via <router-gateway-ip> dev eth0 proto static onlink 
10.0.0.0/24 dev bgptunnel proto kernel scope link src 10.0.0.1 
133.33.33.33 nhid 197 proto bgp metric 20 
	nexthop via 10.0.0.5 dev bgptunnel weight 1 
	nexthop via 10.0.0.2 dev bgptunnel weight 1 
	nexthop via 10.0.0.3 dev bgptunnel weight 1 
	nexthop via 10.0.0.4 dev bgptunnel weight 1 
	nexthop via 10.0.0.6 dev bgptunnel weight 1 
133.33.33.34 nhid 197 proto bgp metric 20 
	nexthop via 10.0.0.5 dev bgptunnel weight 1 
	nexthop via 10.0.0.2 dev bgptunnel weight 1 
	nexthop via 10.0.0.3 dev bgptunnel weight 1 
	nexthop via 10.0.0.4 dev bgptunnel weight 1 
	nexthop via 10.0.0.6 dev bgptunnel weight 1 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown

When I use the command "ping 10.0.0.2" the ping is working but when I use the command "traceroute 133.33.33.33" or "traceroute 10.0.0.2" the output is this:

traceroute 133.33.33.33
traceroute to 133.33.33.33 (133.33.33.33), 30 hops max, 60 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * *

By your words I suspect that I'm probably having an issue with resolving ARP for the next-hop or there is an issue with return traffic. This has something to do with FRRouting?

Thanks

pantxisto Apr 3, 2022
Author

Solved by using command:

sudo ufw allow from 10.0.0.1 to 133.33.33.33 (on master and worker nodes)

Thanks

pantxisto · 2022-03-25T13:53:08Z

pantxisto
Mar 25, 2022
Author

Ping is not working:

ping 126.26.26.1
PING 126.26.26.1 (126.26.26.1) 56(84) bytes of data.
^C
--- 126.26.26.1 ping statistics ---
14 packets transmitted, 0 received, 100% packet loss, time 13291ms

Traceroute gives me some nexthops but I don't know what it means 😟 :

traceroute 126.26.26.1
traceroute to 126.26.26.1 (126.26.26.1), 30 hops max, 60 byte packets
 1  ip-<unknown-ip-1>static.contabo.net (<unknown-ip-2>)  2.391 ms ip-<unknown-ip-1>.static.contabo.net (<unknown-ip-1>)  1.697 ms ip-<unknown-ip-2>static.contabo.net (<unknown-ip-2>)  1.933 ms
 2  <unknown-ip-3> (<unknown-ip-3>)  2.768 ms  2.136 ms  2.224 ms
 3  adm-bb4-link.ip.twelve99.net (<unknown-ip-4>)  148.605 ms ddf-b2-link.ip.twelve99.net (<unknown-ip-5>)  2.670 ms adm-bb4-link.ip.twelve99.net (<unknown-ip-4>)  148.597 ms
 4  * * adm-bb4-link.ip.twelve99.net (<unknown-ip-4>)  149.511 ms
 5  nyk-bb1-link.ip.twelve99.net (<unknown-ip-6>)  80.209 ms ldn-bb4-link.ip.twelve99.net (<unknown-ip-7>)  10.849 ms *
 6  nyk-bb1-link.ip.twelve99.net (<unknown-ip-6>)  80.564 ms  79.474 ms sjo-b23-link.ip.twelve99.net (<unknown-ip-8>)  148.872 ms
 7  * bbixusainc-svc075529-lag003674.ip.twelve99-cust.net (<unknown-ip-9>)  235.585 ms  235.468 ms
 8  bbixusainc-svc075529-lag003674.ip.twelve99-cust.net (<unknown-ip-9>)  236.526 ms  236.512 ms *
 9  * * *
10  * * *
11  * softbank221110197130.bbtec.net (<unknown-ip-10>)  251.702 ms  251.832 ms
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

Used SIP:

ip route get 126.26.26.1
126.26.26.1 via <gateway> dev eth0 src <router-ip> uid 1000 
    cache

If I'm having an issue resolving ARP for the next-hop or there's an issue with return traffic how can I find where the packets are getting lost?

Thank you

0 replies

taspelund · 2022-03-25T14:05:36Z

taspelund
Mar 25, 2022
Collaborator

You're seeing multiple hops so it doesn't look like an ARP issue. Multiple hops in traceroute means you have a route to get to the remote device, there are multiple intermediate routers that know how to get to the remote device (and back to yours), but some device after the last hop you see doesn't know how to get back to your device or isn't sending TTL expired ICMP errors back.

In any event, unless you haven't advertised the router-ip and there's no less specific route to get to the router-ip, the issue likely isn't on this side. I'd try adding network <router-ip>/<router-ip-mask> to your BGP config and see if that resolves the issue

0 replies

pantxisto · 2022-03-25T14:29:14Z

pantxisto
Mar 25, 2022
Author

@taspelund network <router-ip>/<router-ip-mask> does not solve it. If I traceroute kubernetes nodes it is done directly:

traceroute <master1-ip>
traceroute to <master1-ip> (<master1-ip>), 30 hops max, 60 byte packets
 1  ip-<unknown-ip-1>.static.contabo.net (<unknown-ip-1>)  0.956 ms  1.918 ms  0.886 ms
 2  ddf-b2-link.ip.twelve99.net (<master1-ip>)  0.872 ms !N * *

Does this mean that BGP speakers on each kubernetes node are the ones who are giving the problem and don't know how to get back to my device or aren't sending TTL expire ICMP errors back? Despite the speaker are on each kubernetes node is not the same to traceroute the node-ip or the advertised-ip on that node and each one have different intermediate routers?

0 replies

taspelund · 2022-03-25T14:51:24Z

taspelund
Mar 25, 2022
Collaborator

I don't know how your network is setup, so I can't really comment on the specifics.

What I see is that we have installed the routes into the kernel and they are being used by the kernel to forward the ICMP packets via the gateway you've specified -- so FRR is working correctly.

I don't know anything about your network, so I don't know if the path beyond the first hop is expected for your environment or which hop along the way may be the issue. What I do know is that every device between us and the IP you're pinging needs to have a route to our IP and their IP.

I would probably start there to make sure they all have a way to get to each of the IPs in this ping flow, or work your way backwards from the other device -- perhaps run a traceroute from the curl IP to the router-ip and see how far it gets. That should help you isolate where the issue is within your network.

0 replies

pantxisto · 2022-03-25T15:25:02Z

pantxisto
Mar 25, 2022
Author

You mean doing traceroute or a ping from router to all the ips gotten when tracerouting 126.26.26.1?
When doing a traceroute from 126.26.26.1 to router I'm only getting 1 hop and after that all the next 29 hops appear with * * *.

On metallb slack suggest that all cloud providers don't work with metallb because some of them may filter ip addresses that don’t belong to their cloud servers and recommended to use some encapsulation like GRE. Could this give problems with FRR?

0 replies

taspelund · 2022-03-25T18:01:00Z

taspelund
Mar 25, 2022
Collaborator

No, I meant logging into 126.26.26.1 and doing a traceroute to the router IP. If you're not getting anything beyond your first hop, then I'd say either that first hop doesn't have a route to the router-ip or there's some kind of filter in place.

If there isn't end-to-end reachability between 126.26.26.1 and the router-ip then you'd need some way of crossing the network via some IPs that can be forwarded, e.g. via tunneling (like GRE) or NAT.

1 reply

pantxisto Mar 27, 2022
Author

Wow, thank you very much @taspelund. Problem solved with gre tunneling and FRR working properly. I attach different discussions related with FRR and metallb for future doubts. Cloud provider filtering ips and FRR not working and setup FRR with metallb. On both links threads there is a repository referenced wich have multiple shell scripts with a vagrant file to lauch a kubernetes cluster with FRR and metallb (FRR_routing_gre_tunnel branch).

theodofi · 2022-12-18T22:48:10Z

theodofi
Dec 18, 2022

I'm installing frr version 8.4.1 on ubuntu server 22.04. Actually when I'm configuring bgp peer with frr to another bgp peer, I check it receive received route but in show ip bgp doesn't show any route. Same In the kernel ip route show.

I read the documentation, you should add line "net.ipv4.conf.all.forwarding="1 in /etc/sysctl.conf. By default, the command is not configured. It works for me

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BGP Routes not updated on kernel #10895

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 12 comments 3 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

BGP Routes not updated on kernel #10895

pantxisto Mar 25, 2022

Replies: 12 comments · 3 replies

taspelund Mar 25, 2022 Collaborator

JustInVTime Mar 25, 2022

pantxisto Mar 25, 2022 Author

taspelund Mar 25, 2022 Collaborator

taspelund Mar 25, 2022 Collaborator

pantxisto Apr 3, 2022 Author

pantxisto Apr 3, 2022 Author

pantxisto Mar 25, 2022 Author

taspelund Mar 25, 2022 Collaborator

pantxisto Mar 25, 2022 Author

taspelund Mar 25, 2022 Collaborator

pantxisto Mar 25, 2022 Author

taspelund Mar 25, 2022 Collaborator

pantxisto Mar 27, 2022 Author

theodofi Dec 18, 2022

pantxisto
Mar 25, 2022

Replies: 12 comments 3 replies

taspelund
Mar 25, 2022
Collaborator

JustInVTime
Mar 25, 2022

pantxisto
Mar 25, 2022
Author

taspelund
Mar 25, 2022
Collaborator

taspelund
Mar 25, 2022
Collaborator

pantxisto Apr 3, 2022
Author

pantxisto Apr 3, 2022
Author

pantxisto
Mar 25, 2022
Author

taspelund
Mar 25, 2022
Collaborator

pantxisto
Mar 25, 2022
Author

taspelund
Mar 25, 2022
Collaborator

pantxisto
Mar 25, 2022
Author

taspelund
Mar 25, 2022
Collaborator

pantxisto Mar 27, 2022
Author

theodofi
Dec 18, 2022