Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OSPF hanging on FreeBSD #14790

Closed
mwinter-osr opened this issue Nov 14, 2023 · 1 comment · Fixed by #14796
Closed

OSPF hanging on FreeBSD #14790

mwinter-osr opened this issue Nov 14, 2023 · 1 comment · Fixed by #14796
Labels
ospf regression
Milestone
9.1

Comments

@mwinter-osr
Copy link
Member

mwinter-osr commented Nov 14, 2023 •
edited
Loading

Found in the 9.1 release candidate on FreeBSD 12.3

When OSPF is configured on an interface, then the VTYSH is hanging

Steps to reproduce the issues

comp31-dut# config ter
comp31-dut(config)# interface eth2
comp31-dut(config-if)# ip address 192.168.1.101/24
comp31-dut(config-if)# no shutdown
comp31-dut(config-if)# no router ospf
comp31-dut(config)# router ospf
comp31-dut(config-router)# network 192.168.1.0/24  area 0.0.0.0

--> Vtysh hangs

Offending commit

A bisect shows that the issue was introduced with the following commit
60b7786

60b77869e554b6d2b38ea507031926f4effe390a is the first bad commit
commit 60b77869e554b6d2b38ea507031926f4effe390a
Author: Mark Stapp <[email protected]>
Date:   Thu Jun 29 13:51:18 2023 -0400

    ospfd: fix per-interface sockets
    
    Some fixes for the per-interface write sockets: better align
    opening and closing them with ospf config actions; set
    read buffer to zero since these sockets are used only for
    writing packets.
    
    Signed-off-by: Mark Stapp <[email protected]>

 ospfd/ospf_interface.c | 13 ++++++++++++-
 ospfd/ospf_network.c   | 38 ++++++++++++++++++++++++--------------
 2 files changed, 36 insertions(+), 15 deletions(-)
@mwinter-osr mwinter-osr added triage Needs further investigation ospf regression and removed triage Needs further investigation labels Nov 14, 2023
@ton31337 ton31337 added this to the 9.1 milestone Nov 14, 2023
ton31337 added a commit to opensourcerouting/frr that referenced this issue Nov 14, 2023
@ton31337
Revert "ospfd: fix per-interface sockets"
7f48678 
This reverts commit 60b7786.

A regression seen and vtysh is hanging on FreeBSD systems where OSPF is enabled
per interface.

Issue: FRRouting#14790

Signed-off-by: Donatas Abraitis <[email protected]>
@ton31337 ton31337 mentioned this issue Nov 14, 2023
Closed
@mjstapp
Copy link
Contributor

mjstapp commented Nov 14, 2023

If you've got a bsd environment where this happens, can you see if you can get an ospfd backtrace when it's hung?

donaldsharp added a commit to donaldsharp/frr that referenced this issue Nov 14, 2023
@donaldsharp
lib: Prevent infinite loop in ospf
3c5a937 
For some series of calls in FREEBSD setting the SO_RCVBUF size will
always fail under freebsd.  This is no bueno since the
setsockopt_so_recvbuf call goes into an infinite loop.

(gdb) bt
 0  setsockopt () at setsockopt.S:4
 1  0x0000000083065870 in setsockopt_so_recvbuf (sock=15, size=0) at lib/sockopt.c:26
 2  0x00000000002bd200 in ospf_ifp_sock_init (ifp=<optimized out>, ifp@entry=0x8d1dd500) at ospfd/ospf_network.c:290
 3  0x00000000002ad1e0 in ospf_if_new (ospf=0x8eefc000, ifp=0x8d1dd500, p=0x8eecf1c0) at ospfd/ospf_interface.c:276
 4  0x0000000000304ee0 in add_ospf_interface (co=0x8eecbe10, area=0x8d192100) at ospfd/ospfd.c:1115
 5  0x00000000003050fc in ospf_network_run_interface (ospf=0x8eefc000, ifp=0x8d1dd500, p=0x80ff63f8, given_area=0x8d192100)
    at ospfd/ospfd.c:1460
 6  ospf_network_run (p=0x80ff63f8, area=0x8d192100) at ospfd/ospfd.c:1474
 7  ospf_network_set (ospf=ospf@entry=0x8eefc000, p=p@entry=0x80ff63f8, area_id=..., df=<optimized out>) at ospfd/ospfd.c:1247
 8  0x00000000002e876c in ospf_network_area (self=<optimized out>, vty=0x8eef3180, argc=<optimized out>, argv=<optimized out>)
    at ospfd/ospf_vty.c:560
 9  0x0000000083006f24 in cmd_execute_command_real (vline=vline@entry=0x8eee9100, vty=vty@entry=0x8eef3180, cmd=<optimized out>,
    cmd@entry=0x0, up_level=<optimized out>) at lib/command.c:978
 10 0x0000000083006b30 in cmd_execute_command (vline=0x8eee9100, vty=vty@entry=0x8eef3180, cmd=cmd@entry=0x0, vtysh=vtysh@entry=0)
    at lib/command.c:1037
 11 0x0000000083007044 in cmd_execute (vty=vty@entry=0x8eef3180, cmd=cmd@entry=0x8eefb000 "network 192.168.64.0/24 area 0.0.0.0",
    matched=0x0, vtysh=0) at lib/command.c:1203
 12 0x000000008307e9cc in vty_command (vty=0x8eef3180, buf=0x8eefb000 "network 192.168.64.0/24 area 0.0.0.0") at lib/vty.c:594
 13 vty_execute (vty=vty@entry=0x8eef3180) at lib/vty.c:1357
 14 0x000000008307ce40 in vtysh_read (thread=<optimized out>) at lib/vty.c:2365
 15 0x0000000083073db0 in event_call (thread=thread@entry=0x80ff88a0) at lib/event.c:1965
 16 0x000000008302c604 in frr_run (master=0x8d188140) at lib/libfrr.c:1214
 17 0x000000000029c330 in main (argc=6, argv=<optimized out>) at ospfd/ospf_main.c:252
(gdb)

Force the setsockopt function to quit when the value we are passing no
longer makes any sense.

Fixes: FRRouting#14790
Signed-off-by: Donald Sharp <[email protected]>
@donaldsharp donaldsharp mentioned this issue Nov 14, 2023
Merged
donaldsharp added a commit to donaldsharp/frr that referenced this issue Nov 14, 2023
@donaldsharp
lib: Prevent infinite loop in ospf
1c0f3b6 
For some series of calls in FREEBSD setting the SO_RCVBUF size will
always fail under freebsd.  This is no bueno since the
setsockopt_so_recvbuf call goes into an infinite loop.

(gdb) bt
 0  setsockopt () at setsockopt.S:4
 1  0x0000000083065870 in setsockopt_so_recvbuf (sock=15, size=0) at lib/sockopt.c:26
 2  0x00000000002bd200 in ospf_ifp_sock_init (ifp=<optimized out>, ifp@entry=0x8d1dd500) at ospfd/ospf_network.c:290
 3  0x00000000002ad1e0 in ospf_if_new (ospf=0x8eefc000, ifp=0x8d1dd500, p=0x8eecf1c0) at ospfd/ospf_interface.c:276
 4  0x0000000000304ee0 in add_ospf_interface (co=0x8eecbe10, area=0x8d192100) at ospfd/ospfd.c:1115
 5  0x00000000003050fc in ospf_network_run_interface (ospf=0x8eefc000, ifp=0x8d1dd500, p=0x80ff63f8, given_area=0x8d192100)
    at ospfd/ospfd.c:1460
 6  ospf_network_run (p=0x80ff63f8, area=0x8d192100) at ospfd/ospfd.c:1474
 7  ospf_network_set (ospf=ospf@entry=0x8eefc000, p=p@entry=0x80ff63f8, area_id=..., df=<optimized out>) at ospfd/ospfd.c:1247
 8  0x00000000002e876c in ospf_network_area (self=<optimized out>, vty=0x8eef3180, argc=<optimized out>, argv=<optimized out>)
    at ospfd/ospf_vty.c:560
 9  0x0000000083006f24 in cmd_execute_command_real (vline=vline@entry=0x8eee9100, vty=vty@entry=0x8eef3180, cmd=<optimized out>,
    cmd@entry=0x0, up_level=<optimized out>) at lib/command.c:978
 10 0x0000000083006b30 in cmd_execute_command (vline=0x8eee9100, vty=vty@entry=0x8eef3180, cmd=cmd@entry=0x0, vtysh=vtysh@entry=0)
    at lib/command.c:1037
 11 0x0000000083007044 in cmd_execute (vty=vty@entry=0x8eef3180, cmd=cmd@entry=0x8eefb000 "network 192.168.64.0/24 area 0.0.0.0",
    matched=0x0, vtysh=0) at lib/command.c:1203
 12 0x000000008307e9cc in vty_command (vty=0x8eef3180, buf=0x8eefb000 "network 192.168.64.0/24 area 0.0.0.0") at lib/vty.c:594
 13 vty_execute (vty=vty@entry=0x8eef3180) at lib/vty.c:1357
 14 0x000000008307ce40 in vtysh_read (thread=<optimized out>) at lib/vty.c:2365
 15 0x0000000083073db0 in event_call (thread=thread@entry=0x80ff88a0) at lib/event.c:1965
 16 0x000000008302c604 in frr_run (master=0x8d188140) at lib/libfrr.c:1214
 17 0x000000000029c330 in main (argc=6, argv=<optimized out>) at ospfd/ospf_main.c:252
(gdb)

Force the setsockopt function to quit when the value we are passing no
longer makes any sense.

Fixes: FRRouting#14790
Signed-off-by: Donald Sharp <[email protected]>
mergify bot pushed a commit that referenced this issue Nov 15, 2023
@donaldsharp @mergify
lib: Prevent infinite loop in ospf
77fe290 
For some series of calls in FREEBSD setting the SO_RCVBUF size will
always fail under freebsd.  This is no bueno since the
setsockopt_so_recvbuf call goes into an infinite loop.

(gdb) bt
 0  setsockopt () at setsockopt.S:4
 1  0x0000000083065870 in setsockopt_so_recvbuf (sock=15, size=0) at lib/sockopt.c:26
 2  0x00000000002bd200 in ospf_ifp_sock_init (ifp=<optimized out>, ifp@entry=0x8d1dd500) at ospfd/ospf_network.c:290
 3  0x00000000002ad1e0 in ospf_if_new (ospf=0x8eefc000, ifp=0x8d1dd500, p=0x8eecf1c0) at ospfd/ospf_interface.c:276
 4  0x0000000000304ee0 in add_ospf_interface (co=0x8eecbe10, area=0x8d192100) at ospfd/ospfd.c:1115
 5  0x00000000003050fc in ospf_network_run_interface (ospf=0x8eefc000, ifp=0x8d1dd500, p=0x80ff63f8, given_area=0x8d192100)
    at ospfd/ospfd.c:1460
 6  ospf_network_run (p=0x80ff63f8, area=0x8d192100) at ospfd/ospfd.c:1474
 7  ospf_network_set (ospf=ospf@entry=0x8eefc000, p=p@entry=0x80ff63f8, area_id=..., df=<optimized out>) at ospfd/ospfd.c:1247
 8  0x00000000002e876c in ospf_network_area (self=<optimized out>, vty=0x8eef3180, argc=<optimized out>, argv=<optimized out>)
    at ospfd/ospf_vty.c:560
 9  0x0000000083006f24 in cmd_execute_command_real (vline=vline@entry=0x8eee9100, vty=vty@entry=0x8eef3180, cmd=<optimized out>,
    cmd@entry=0x0, up_level=<optimized out>) at lib/command.c:978
 10 0x0000000083006b30 in cmd_execute_command (vline=0x8eee9100, vty=vty@entry=0x8eef3180, cmd=cmd@entry=0x0, vtysh=vtysh@entry=0)
    at lib/command.c:1037
 11 0x0000000083007044 in cmd_execute (vty=vty@entry=0x8eef3180, cmd=cmd@entry=0x8eefb000 "network 192.168.64.0/24 area 0.0.0.0",
    matched=0x0, vtysh=0) at lib/command.c:1203
 12 0x000000008307e9cc in vty_command (vty=0x8eef3180, buf=0x8eefb000 "network 192.168.64.0/24 area 0.0.0.0") at lib/vty.c:594
 13 vty_execute (vty=vty@entry=0x8eef3180) at lib/vty.c:1357
 14 0x000000008307ce40 in vtysh_read (thread=<optimized out>) at lib/vty.c:2365
 15 0x0000000083073db0 in event_call (thread=thread@entry=0x80ff88a0) at lib/event.c:1965
 16 0x000000008302c604 in frr_run (master=0x8d188140) at lib/libfrr.c:1214
 17 0x000000000029c330 in main (argc=6, argv=<optimized out>) at ospfd/ospf_main.c:252
(gdb)

Force the setsockopt function to quit when the value we are passing no
longer makes any sense.

Fixes: #14790
Signed-off-by: Donald Sharp <[email protected]>
(cherry picked from commit 1c0f3b6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ospf regression
Projects
None yet
Milestone
9.1
Development

Successfully merging a pull request may close this issue.

lib: Prevent infinite loop in ospf donaldsharp/frr
3 participants
@mwinter-osr @ton31337 @mjstapp