routes periodically disappear from OS RT

[doka380]

Hi colleagues,

suggest me, pls, where to look for the solution to the issue. Ubuntu 22.04.03, frr 8.5.2 (latest stable from FRR repo), kernel 5.19.0-46-generic #47~22.04.1-Ubuntu, router configuration is at the end of the message.

Upon FRR start and during some time everything is ok, but for some reasons after the time the following happens (reduced output) and can be fixed with 'clear ip ospf proc' :

While Linux RT lacks required 100.64.2.0/23 and 100.64.4.0/23 -

# ip route
default nhid 229 via 212.113.51.33 dev wan proto bgp metric 20
100.64.0.0/23 dev br-int proto kernel scope link src 100.64.0.1
100.64.97.0/24 dev wg0 proto kernel scope link src 100.64.97.1

these networks are present in FRR RT (blue and orange are renamed interfaces):

# sh ip route
[ ... ]
O   0.0.0.0/0 [110/10] via 100.64.98.3, blue, weight 1, 01:34:12
                       via 100.64.99.3, orange, weight 1, 01:34:12
B>* 0.0.0.0/0 [20/0] via 212.113.51.33, wan, weight 1, 1d00h30m
C>* 100.64.0.0/23 is directly connected, br-int, 1d00h30m
O>* 100.64.2.0/23 [110/20] via 100.64.98.2, blue, weight 1, 01:34:13
  *                        via 100.64.99.2, orange, weight 1, 01:34:13
O>* 100.64.4.0/23 [110/20] via 100.64.98.3, blue, weight 1, 01:34:13
  *                        via 100.64.99.3, orange, weight 1, 01:34:13
O   100.64.97.0/24 [110/10] is directly connected, wg0, weight 1, 01:34:24
C>* 100.64.97.0/24 is directly connected, wg0, 1d00h30m

Issuing 'clear ip ospf proc' solves the issue - these routes appear in Linux RT again, until next time.

It seems the cause of the issue is the following (note nexthops are connected), but I have no ideas what to do with this:

Aug  9 18:57:41 vishnu zebra[816303]: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (300[253/254]) that we are still using for a route, sending it back down
Aug  9 18:57:41 vishnu zebra[816303]: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (280[234/235]) that we are still using for a route, sending it back down
Aug  9 18:57:41 vishnu zebra[816303]: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (276[]) that we are still using for a route, sending it back down

The question is

any suggestions where to look for the solution to the problem? Which additional information I need to gather which can help solve it? May be I'm missing something in OS networking/kernel configuration? (I, actually, wasn't tweaking configuration except ip.forward=1). Will appreciate any recommendations on this.

Thank you

Other info

Interfaces are:

5: orange: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1496 qdisc mq state UP group default qlen 1000
    link/ether 8c:dc:d4:b1:78:ad brd ff:ff:ff:ff:ff:ff
    altname enp7s0f1
    inet 100.64.99.1/24 brd 100.64.99.255 scope global orange
       valid_lft forever preferred_lft forever
8: blue@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1496 qdisc noqueue state UP group default qlen 1000
    link/ether 14:02:ec:92:b3:91 brd ff:ff:ff:ff:ff:ff
    inet 100.64.98.1/24 brd 100.64.98.255 scope global blue
       valid_lft forever preferred_lft forever

It's always ok with OSPF neighborship - hello packets are reaching all destinations:

vishnu.utc.mygaru# sh ip ospf neigh
Neighbor ID     Pri State           Up Time         Dead Time Address         Interface                        RXmtL RqstL DBsmL
100.64.2.1        1 Full/Backup     17m52s             3.823s 100.64.98.2     blue:100.64.98.1                     0     0     0
100.64.4.1        1 Full/DR         22m59s             3.403s 100.64.98.3     blue:100.64.98.1                     0     0     0
100.64.2.1        1 Full/Backup     17m52s             3.823s 100.64.99.2     orange:100.64.99.1                   0     0     0
100.64.4.1        1 Full/DR         22m59s             3.403s 100.64.99.3     orange:100.64.99.1                   0     0     0

and following daemons are running with the following arguments -

root@vishnu:~# ps ax |grep frr
 816287 ?        S<s    0:18 /usr/lib/frr/watchfrr -d -F traditional zebra bgpd ospfd staticd
 816303 ?        S<sl   0:05 /usr/lib/frr/zebra -d -F traditional -A 127.0.0.1 -r
 816308 ?        S<sl   0:06 /usr/lib/frr/bgpd -d -F traditional -A 127.0.0.1
 816315 ?        S<s    1:19 /usr/lib/frr/ospfd -d -F traditional -A 127.0.0.1
 816318 ?        S<s    0:03 /usr/lib/frr/staticd -d -F traditional -A 127.0.0.1

Configuration is pretty simple:

frr version 8.5.2
frr defaults traditional
hostname vishnu
log syslog informational
no ipv6 forwarding
service integrated-vtysh-config
!
interface blue
 description === Blue infra ===
 ip ospf dead-interval 4
 ip ospf hello-interval 1
 no ip ospf passive
exit
!
interface orange
 description === Orange infra ===
 ip ospf dead-interval 4
 ip ospf hello-interval 1
 no ip ospf passive
exit

! this is eBGP to ISP
router bgp NNNN
[ ... ]
exit

! this is eBGP to ISP
router bgp 64512 view erspan
[ ... ]
exit

!
router ospf
 ospf router-id 100.64.0.1
 passive-interface default
 network 100.64.0.0/23 area 0
 network 100.64.97.0/24 area 0
 network 100.64.98.0/24 area 0
 network 100.64.99.0/24 area 0
 network 192.x.x.x/24 area 0
 default-information originate
exit

! few these prefix lists used for BGP
ip prefix-list [ ... ]
!
end

Thank you.

[ton31337]

Did you try enabling debug ospf zebra, debug ospf?