Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zebra: crash when deleting a vrf with a l3vni inside #12123

Closed

Conversation

pguibert6WIND
Copy link
Member

The following crash happens when deleting a vrf with a L3VNI vxlan interface associated to that vrf:

Thread 1 (Thread 0x7fc2eb121e40 (LWP 11516)):
#0 raise (sig=) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007fc2eb774a04 in core_handler (signo=11, siginfo=0x7fff26102830, context=0x7fff26102700) at /build/tools-build-framework/output/_packages/cp-routing/src/lib/sigevent.c:262
#2
#3 0x00007fc2eb716ffb in hash_get (hash=0x55af78ece860, data=0x7fff26102be0, alloc_func=0x55af76f52f60 <zebra_evpn_alloc>) at /build/tools-build-framework/output/_packages/cp-routing/src/lib/hash.c:157
#4 0x000055af76f530f8 in zebra_evpn_add (vni=101) at /build/tools-build-framework/output/_packages/cp-routing/src/zebra/zebra_evpn.c:1023
#5 0x000055af76fb64c5 in zebra_vxlan_handle_vni_transition (zvrf=0x55af78ed0220, vni=101, add=0) at /build/tools-build-framework/output/_packages/cp-routing/src/zebra/zebra_vxlan.c:2080
#6 0x000055af76fbd94f in zebra_vxlan_vrf_delete (zvrf=0x55af78ed0220) at /build/tools-build-framework/output/_packages/cp-routing/src/zebra/zebra_vxlan.c:5364
zl3vni = 0x55af78fffb20
vni = 101
#7 0x000055af76fa13bb in zebra_vrf_delete (vrf=0x55af78ed0140) at /build/tools-build-framework/output/_packages/cp-routing/src/zebra/zebra_vrf.c:312
#8 0x00007fc2eb791330 in vrf_delete (vrf=0x55af78ed0140) at /build/tools-build-framework/output/_packages/cp-routing/src/lib/vrf.c:295
#9 0x00007fc2eb791b55 in vrf_terminate_single (vrf=0x55af78ed0140) at /build/tools-build-framework/output/_packages/cp-routing/src/lib/vrf.c:597
#10 0x00007fc2eb791c41 in vrf_terminate () at /build/tools-build-framework/output/_packages/cp-routing/src/lib/vrf.c:624
#11 0x000055af76f1c8a6 in sigint () at /build/tools-build-framework/output/_packages/cp-routing/src/zebra/main.c:191
#12 0x00007fc2eb774757 in quagga_sigevent_process () at /build/tools-build-framework/output/_packages/cp-routing/src/lib/sigevent.c:130
#13 0x00007fc2eb78cf9b in thread_fetch (m=0x55af78ccc3f0, fetch=0x7fff26102f20) at /build/tools-build-framework/output/_packages/cp-routing/src/lib/thread.c:1631
#14 0x00007fc2eb728e8c in frr_run (master=0x55af78ccc3f0) at /build/tools-build-framework/output/_packages/cp-routing/src/lib/libfrr.c:1154
#15 0x000055af76f1ce36 in main (argc=6, argv=0x7fff261031e8) at /build/tools-build-framework/output/_packages/cp-routing/src/zebra/main.c:487

Because vrf_disable is called first, zvrf->evpn_table may be freed. But because we call vrf_delete, we may still want to handle VNI transition to L2. And we dont check the evpn_table pointer.

Do the check.

Signed-off-by: Philippe Guibert [email protected]

@frrbot frrbot bot added the zebra label Oct 13, 2022
@ton31337
Copy link
Member

@Mergifyio backport dev/8.4 stable/8.3

@mergify
Copy link

mergify bot commented Oct 13, 2022

backport dev/8.4 stable/8.3

🟠 Waiting for conditions to match

  • merged [📌 backport requirement]

@NetDEF-CI
Copy link
Collaborator

NetDEF-CI commented Oct 13, 2022

Continuous Integration Result: FAILED

Continuous Integration Result: FAILED

Test incomplete. See below for issues.
CI System Testrun URL: https://ci1.netdef.org/browse/FRR-PULLREQ2-7880/

This is a comment from an automated CI system.
For questions and feedback in regards to this CI system, please feel free to email
Martin Winter - mwinter (at) opensourcerouting.org.

Get source / Pull Request: Successful

Building Stage: Successful

Basic Tests: Incomplete

Addresssanitizer topotests part 1: Incomplete (check logs for details)
Successful on other platforms/tests
  • Topotests Ubuntu 18.04 i386 part 1
  • Topotests Ubuntu 18.04 amd64 part 4
  • Topotests Ubuntu 18.04 i386 part 6
  • Addresssanitizer topotests part 2
  • Topotests Ubuntu 18.04 amd64 part 5
  • Topotests Ubuntu 18.04 amd64 part 0
  • Topotests Ubuntu 18.04 arm8 part 6
  • Topotests debian 10 amd64 part 6
  • Topotests Ubuntu 18.04 arm8 part 7
  • Topotests Ubuntu 18.04 arm8 part 1
  • Addresssanitizer topotests part 9
  • Fedora 29 rpm pkg check
  • Topotests Ubuntu 18.04 amd64 part 7
  • Topotests debian 10 amd64 part 0
  • Topotests Ubuntu 18.04 arm8 part 2
  • Addresssanitizer topotests part 3
  • Topotests debian 10 amd64 part 5
  • Topotests Ubuntu 18.04 arm8 part 8
  • Topotests Ubuntu 18.04 i386 part 7
  • CentOS 7 rpm pkg check
  • Topotests Ubuntu 18.04 i386 part 2
  • Topotests Ubuntu 18.04 amd64 part 3
  • Addresssanitizer topotests part 6
  • Topotests debian 10 amd64 part 9
  • Topotests Ubuntu 18.04 amd64 part 2
  • Topotests debian 10 amd64 part 7
  • Topotests Ubuntu 18.04 i386 part 5
  • Topotests Ubuntu 18.04 i386 part 0
  • Topotests debian 10 amd64 part 8
  • Debian 10 deb pkg check
  • Topotests Ubuntu 18.04 arm8 part 4
  • Topotests debian 10 amd64 part 3
  • Topotests Ubuntu 18.04 amd64 part 6
  • Topotests Ubuntu 18.04 arm8 part 9
  • Topotests Ubuntu 18.04 amd64 part 1
  • Ubuntu 20.04 deb pkg check
  • Topotests Ubuntu 18.04 i386 part 4
  • Ubuntu 18.04 deb pkg check
  • Topotests Ubuntu 18.04 i386 part 8
  • Debian 9 deb pkg check
  • Addresssanitizer topotests part 4
  • Topotests Ubuntu 18.04 amd64 part 9
  • Topotests Ubuntu 18.04 i386 part 3
  • Addresssanitizer topotests part 8
  • Addresssanitizer topotests part 7
  • Topotests debian 10 amd64 part 4
  • Topotests Ubuntu 18.04 arm8 part 5
  • Topotests Ubuntu 18.04 arm8 part 3
  • Topotests Ubuntu 18.04 i386 part 9
  • Addresssanitizer topotests part 5
  • Topotests debian 10 amd64 part 1
  • Topotests Ubuntu 18.04 amd64 part 8
  • Topotests Ubuntu 18.04 arm8 part 0
  • Static analyzer (clang)
  • Ubuntu 16.04 deb pkg check
  • Topotests debian 10 amd64 part 2
  • Addresssanitizer topotests part 0

@NetDEF-CI
Copy link
Collaborator

NetDEF-CI commented Oct 13, 2022

Continuous Integration Result: SUCCESSFUL

Continuous Integration Result: SUCCESSFUL

Congratulations, this patch passed basic tests

Tested-by: NetDEF / OpenSourceRouting.org CI System

CI System Testrun URL: https://ci1.netdef.org/browse/FRR-PULLREQ2-7880/

This is a comment from an automated CI system.
For questions and feedback in regards to this CI system, please feel free to email
Martin Winter - mwinter (at) opensourcerouting.org.

@NetDEF-CI
Copy link
Collaborator

NetDEF-CI commented Oct 18, 2022

Continuous Integration Result: FAILED

Continuous Integration Result: FAILED

Test incomplete. See below for issues.
CI System Testrun URL: https://ci1.netdef.org/browse/FRR-PULLREQ2-7944/

This is a comment from an automated CI system.
For questions and feedback in regards to this CI system, please feel free to email
Martin Winter - mwinter (at) opensourcerouting.org.

Get source / Pull Request: Successful

Building Stage: Successful

Basic Tests: Incomplete

Topotests Ubuntu 18.04 amd64 part 4: Incomplete (check logs for details)
Successful on other platforms/tests
  • Debian 10 deb pkg check
  • Topotests Ubuntu 18.04 arm8 part 4
  • Topotests debian 10 amd64 part 3
  • Topotests Ubuntu 18.04 amd64 part 6
  • Topotests Ubuntu 18.04 arm8 part 9
  • Topotests Ubuntu 18.04 amd64 part 1
  • Ubuntu 20.04 deb pkg check
  • Addresssanitizer topotests part 1
  • Topotests Ubuntu 18.04 i386 part 4
  • Ubuntu 18.04 deb pkg check
  • Topotests Ubuntu 18.04 i386 part 8
  • Debian 9 deb pkg check
  • Addresssanitizer topotests part 4
  • Topotests Ubuntu 18.04 amd64 part 9
  • Topotests Ubuntu 18.04 i386 part 3
  • Addresssanitizer topotests part 8
  • Addresssanitizer topotests part 7
  • Topotests debian 10 amd64 part 4
  • Topotests Ubuntu 18.04 arm8 part 5
  • Topotests Ubuntu 18.04 arm8 part 3
  • Topotests Ubuntu 18.04 i386 part 9
  • Addresssanitizer topotests part 5
  • Topotests debian 10 amd64 part 1
  • Topotests Ubuntu 18.04 amd64 part 8
  • Static analyzer (clang)
  • Topotests Ubuntu 18.04 arm8 part 0
  • Ubuntu 16.04 deb pkg check
  • Topotests debian 10 amd64 part 2
  • Addresssanitizer topotests part 0
  • Topotests Ubuntu 18.04 i386 part 1
  • Topotests Ubuntu 18.04 i386 part 6
  • Addresssanitizer topotests part 2
  • Topotests Ubuntu 18.04 amd64 part 5
  • Topotests Ubuntu 18.04 arm8 part 6
  • Topotests Ubuntu 18.04 amd64 part 0
  • Topotests debian 10 amd64 part 6
  • Topotests Ubuntu 18.04 arm8 part 7
  • Topotests Ubuntu 18.04 arm8 part 1
  • Addresssanitizer topotests part 9
  • Fedora 29 rpm pkg check
  • Topotests Ubuntu 18.04 amd64 part 7
  • Topotests debian 10 amd64 part 0
  • Topotests Ubuntu 18.04 arm8 part 2
  • Addresssanitizer topotests part 3
  • Topotests debian 10 amd64 part 5
  • Topotests Ubuntu 18.04 i386 part 7
  • Topotests Ubuntu 18.04 arm8 part 8
  • CentOS 7 rpm pkg check
  • Topotests Ubuntu 18.04 i386 part 2
  • Topotests Ubuntu 18.04 amd64 part 3
  • Addresssanitizer topotests part 6
  • Topotests debian 10 amd64 part 9
  • Topotests Ubuntu 18.04 amd64 part 2
  • Topotests debian 10 amd64 part 7
  • Topotests Ubuntu 18.04 i386 part 5
  • Topotests debian 10 amd64 part 8
  • Topotests Ubuntu 18.04 i386 part 0

@NetDEF-CI
Copy link
Collaborator

NetDEF-CI commented Oct 19, 2022

Continuous Integration Result: SUCCESSFUL

Continuous Integration Result: SUCCESSFUL

Congratulations, this patch passed basic tests

Tested-by: NetDEF / OpenSourceRouting.org CI System

CI System Testrun URL: https://ci1.netdef.org/browse/FRR-PULLREQ2-7944/

This is a comment from an automated CI system.
For questions and feedback in regards to this CI system, please feel free to email
Martin Winter - mwinter (at) opensourcerouting.org.

@@ -1019,6 +1019,8 @@ struct zebra_evpn *zebra_evpn_add(vni_t vni)
zvrf = zebra_vrf_get_evpn();
memset(&tmp_zevpn, 0, sizeof(tmp_zevpn));
tmp_zevpn.vni = vni;
if (!zvrf->evpn_table)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do this check before memset (a little bit above)?

@@ -2173,6 +2173,8 @@ static int zebra_vxlan_handle_vni_transition(struct zebra_vrf *zvrf, vni_t vni,
return 0;

zevpn = zebra_evpn_add(vni);
if (!zevpn)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can it be NULL?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it can, in the case where the hash table is not present.

The following crash happens when deleting a vrf with a L3VNI
vxlan interface associated to that vrf:

 Thread 1 (Thread 0x7fc2eb121e40 (LWP 11516)):
 #0  raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:50
 FRRouting#1  0x00007fc2eb774a04 in core_handler (signo=11, siginfo=0x7fff26102830, context=0x7fff26102700) at /build/tools-build-framework/output/_packages/cp-routing/src/lib/sigevent.c:262
 FRRouting#2  <signal handler called>
 FRRouting#3  0x00007fc2eb716ffb in hash_get (hash=0x55af78ece860, data=0x7fff26102be0, alloc_func=0x55af76f52f60 <zebra_evpn_alloc>) at /build/tools-build-framework/output/_packages/cp-routing/src/lib/hash.c:157
 FRRouting#4  0x000055af76f530f8 in zebra_evpn_add (vni=101) at /build/tools-build-framework/output/_packages/cp-routing/src/zebra/zebra_evpn.c:1023
 FRRouting#5  0x000055af76fb64c5 in zebra_vxlan_handle_vni_transition (zvrf=0x55af78ed0220, vni=101, add=0) at /build/tools-build-framework/output/_packages/cp-routing/src/zebra/zebra_vxlan.c:2080
 FRRouting#6  0x000055af76fbd94f in zebra_vxlan_vrf_delete (zvrf=0x55af78ed0220) at /build/tools-build-framework/output/_packages/cp-routing/src/zebra/zebra_vxlan.c:5364
        zl3vni = 0x55af78fffb20
        vni = 101
 FRRouting#7  0x000055af76fa13bb in zebra_vrf_delete (vrf=0x55af78ed0140) at /build/tools-build-framework/output/_packages/cp-routing/src/zebra/zebra_vrf.c:312
 FRRouting#8  0x00007fc2eb791330 in vrf_delete (vrf=0x55af78ed0140) at /build/tools-build-framework/output/_packages/cp-routing/src/lib/vrf.c:295
 FRRouting#9  0x00007fc2eb791b55 in vrf_terminate_single (vrf=0x55af78ed0140) at /build/tools-build-framework/output/_packages/cp-routing/src/lib/vrf.c:597
 FRRouting#10 0x00007fc2eb791c41 in vrf_terminate () at /build/tools-build-framework/output/_packages/cp-routing/src/lib/vrf.c:624
 FRRouting#11 0x000055af76f1c8a6 in sigint () at /build/tools-build-framework/output/_packages/cp-routing/src/zebra/main.c:191
 FRRouting#12 0x00007fc2eb774757 in quagga_sigevent_process () at /build/tools-build-framework/output/_packages/cp-routing/src/lib/sigevent.c:130
 FRRouting#13 0x00007fc2eb78cf9b in thread_fetch (m=0x55af78ccc3f0, fetch=0x7fff26102f20) at /build/tools-build-framework/output/_packages/cp-routing/src/lib/thread.c:1631
 FRRouting#14 0x00007fc2eb728e8c in frr_run (master=0x55af78ccc3f0) at /build/tools-build-framework/output/_packages/cp-routing/src/lib/libfrr.c:1154
 FRRouting#15 0x000055af76f1ce36 in main (argc=6, argv=0x7fff261031e8) at /build/tools-build-framework/output/_packages/cp-routing/src/zebra/main.c:487

Because vrf_disable is called first, zvrf->evpn_table may be freed.
But because we call vrf_delete, we may still want to handle VNI
transition to L2. And we dont check the evpn_table pointer.

Do the check.

Signed-off-by: Philippe Guibert <[email protected]>
@NetDEF-CI
Copy link
Collaborator

Continuous Integration Result: FAILED

See below for issues.
CI System Testrun URL: https://ci1.netdef.org/browse/FRR-PULLREQ2-9268/

This is a comment from an automated CI system.
For questions and feedback in regards to this CI system, please feel free to email
Martin Winter - mwinter (at) opensourcerouting.org.

Get source / Pull Request: Successful

Building Stage: Successful

Basic Tests: Failed

Topotests Ubuntu 18.04 i386 part 9: Failed (click for details) Topotests Ubuntu 18.04 i386 part 9: Unknown Log URL: https://ci1.netdef.org/browse/FRR-PULLREQ2-9268/artifact/TOPO9U18I386/TopotestLogs/ Topotests Ubuntu 18.04 i386 part 9: Unknown Log URL: https://ci1.netdef.org/browse/FRR-PULLREQ2-9268/artifact/TOPO9U18I386/TopotestDetails/ Topotests Ubuntu 18.04 i386 part 9: No useful log found
Successful on other platforms/tests
  • Addresssanitizer topotests part 3
  • Fedora 29 rpm pkg check
  • Topotests Ubuntu 18.04 amd64 part 7
  • Topotests debian 10 amd64 part 1
  • Topotests Ubuntu 18.04 i386 part 5
  • Topotests Ubuntu 18.04 i386 part 0
  • CentOS 7 rpm pkg check
  • Topotests Ubuntu 18.04 amd64 part 4
  • Addresssanitizer topotests part 2
  • Topotests Ubuntu 18.04 amd64 part 0
  • Addresssanitizer topotests part 1
  • Topotests Ubuntu 18.04 i386 part 4
  • Topotests debian 10 amd64 part 5
  • Topotests Ubuntu 18.04 amd64 part 1
  • Topotests debian 10 amd64 part 7
  • Debian 9 deb pkg check
  • Addresssanitizer topotests part 8
  • Topotests Ubuntu 18.04 amd64 part 6
  • Topotests Ubuntu 18.04 arm8 part 8
  • Ubuntu 18.04 deb pkg check
  • Topotests debian 10 amd64 part 6
  • Topotests Ubuntu 18.04 i386 part 2
  • Topotests Ubuntu 18.04 arm8 part 6
  • Topotests Ubuntu 18.04 arm8 part 1
  • Topotests Ubuntu 18.04 i386 part 7
  • Addresssanitizer topotests part 6
  • Addresssanitizer topotests part 5
  • Topotests Ubuntu 18.04 arm8 part 3
  • Topotests Ubuntu 18.04 i386 part 3
  • Topotests Ubuntu 18.04 amd64 part 2
  • Topotests Ubuntu 18.04 i386 part 8
  • Addresssanitizer topotests part 4
  • Topotests debian 10 amd64 part 3
  • Topotests debian 10 amd64 part 8
  • Addresssanitizer topotests part 0
  • Topotests Ubuntu 18.04 arm8 part 4
  • Topotests debian 10 amd64 part 4
  • Topotests debian 10 amd64 part 9
  • Topotests Ubuntu 18.04 arm8 part 9
  • Topotests Ubuntu 18.04 amd64 part 3
  • Topotests Ubuntu 18.04 arm8 part 2
  • Static analyzer (clang)
  • Topotests debian 10 amd64 part 0
  • Topotests Ubuntu 18.04 arm8 part 7
  • Addresssanitizer topotests part 9
  • Topotests debian 10 amd64 part 2
  • Topotests Ubuntu 18.04 i386 part 6
  • Topotests Ubuntu 18.04 arm8 part 0
  • Ubuntu 16.04 deb pkg check
  • Topotests Ubuntu 18.04 amd64 part 8
  • Topotests Ubuntu 18.04 arm8 part 5
  • Topotests Ubuntu 18.04 i386 part 1
  • Topotests Ubuntu 18.04 amd64 part 9
  • Addresssanitizer topotests part 7
  • Ubuntu 20.04 deb pkg check
  • Debian 10 deb pkg check
  • Topotests Ubuntu 18.04 amd64 part 5

@@ -5913,6 +5915,7 @@ void zebra_vxlan_close_tables(struct zebra_vrf *zvrf)
return;
hash_iterate(zvrf->evpn_table, zebra_evpn_vxlan_cleanup_all, zvrf);
hash_free(zvrf->evpn_table);
zvrf->evpn_table = NULL;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, why this pull request was closed? I seem to have encountered the same problem

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants