topotests: convert to exabgp 4 and python3 #14890

louis-6wind · 2023-11-28T12:10:28Z

Convert topotests to exabgp 4 and python3.

The CI migration needs to be converted to exabgp 4 for this to work

donaldsharp · 2023-11-28T14:16:32Z

@mwinter-osr -> Can you coordinate with @louis-6wind to get this done?

mwinter-osr · 2023-11-28T15:53:10Z

I had to abort the CI here. All platforms hang at the same topotest:

bgp_peer_type_multipath_relax/test_bgp_peer-type_multipath-relax.py::test_bgp_peer_type_multipath_relax_test1

Please investigate before you retry

louis-6wind · 2023-11-28T15:56:28Z

I had to abort the CI here. All platforms hang at the same topotest:
bgp_peer_type_multipath_relax/test_bgp_peer-type_multipath-relax.py::test_bgp_peer_type_multipath_relax_test1
Please investigate before you retry

This is because of the issue:

#14895

mwinter-osr · 2023-11-28T15:57:55Z

The CI already support exabgp 4. By default it's using ExaBGP 3, but if it detects the line
# NEEDS_EXABGP_4_2_11_FRR in the comment of pytest.ini then it uses exabgp 4

(This will use ExaBGP 4.2.11 with FRR related IPv6 fixes)

Please update the PR with this line in the pytest.ini

donaldsharp · 2023-11-28T16:16:11Z

exabgp used is 4.2.21

ton31337 · 2023-11-29T07:25:21Z

tests/topotests/bgp_prefix_sid2/r1/bgpd.conf

@@ -21,5 +21,7 @@ router bgp 1
 !
 address-family ipv6 vpn
  neighbor 10.0.0.101 activate
+  neighbor 10.0.0.101 route-map DENY_ALL out


Is this on purpose?

Yes it is. See commit log ebc71fe

ton31337 · 2023-11-29T07:26:41Z

tests/topotests/lib/topogen.py


    if commander is None:
        commander = Commander("exabgp", logger=logging.getLogger("exabgp"))

    def exacmd_version_ok(exacmd):
-        logger.debug("checking %s for exabgp < version 4", exacmd)
+        logger.debug("checking %s for exabgp", exacmd)


Maybe we should do a reverse check? To check if we don't use a lower version.

ton31337 · 2023-11-29T07:32:24Z

doc/developer/topotests.rst

@@ -39,8 +37,7 @@ Installing Topotest Requirements
   python3 -m pip install 'pytest-xdist>=2.3.0'
   python3 -m pip install 'scapy>=2.4.5'
   python3 -m pip install xmltodict
-   # Use python2 pip to install older ExaBGP
-   python2 -m pip install 'exabgp<4.0.0'
+   python3 -m pip install 'exabg==4.2.21'


ton31337 · 2023-11-29T07:33:20Z

docker/ubuntu-ci/Dockerfile

    python3 -m pip install wheel && \
    python3 -m pip install pytest && \
    python3 -m pip install pytest-sugar && \
    python3 -m pip install pytest-xdist && \
    python3 -m pip install "scapy>=2.4.2" && \
    python3 -m pip install xmltodict && \
    python3 -m pip install grpcio grpcio-tools && \
-    python2 -m pip install 'exabgp<4.0.0'
+    python3 -m pip install 'exabgp==4.2.21'


ton31337

Seems we still have some issues:

E AssertionError: Can't find a usable ExaBGP (must be version >= 4.2.11)

mwinter-osr · 2023-12-05T23:53:56Z

CI:rerun Fixed ExaBGP 4 invocation on non-LXC testbeds. Rerunning (still expecting more issues)

louis-6wind · 2023-12-06T10:33:19Z

ci:rerun

mwinter-osr · 2023-12-13T01:47:27Z

ci:rerun Retesting. intel systems should be fixed, arm still expected to fail

mwinter-osr

Please rebase your PR in addition to the other comments to make sure it works with latest FRR master

mwinter-osr · 2023-12-13T17:34:51Z

doc/developer/topotests.rst

@@ -39,8 +37,7 @@ Installing Topotest Requirements
   python3 -m pip install 'pytest-xdist>=2.3.0'
   python3 -m pip install 'scapy>=2.4.5'
   python3 -m pip install xmltodict
-   # Use python2 pip to install older ExaBGP
-   python2 -m pip install 'exabgp<4.0.0'
+   python3 -m pip install 'exabg>=4.2.11'


I would suggest to install exabgp 4.2.11 from our OSR repository which has the required patches:
pip3 install git+https://github.com/opensourcerouting/[email protected]

mwinter-osr · 2023-12-13T17:35:23Z

docker/ubuntu-ci/Dockerfile

    python3 -m pip install wheel && \
    python3 -m pip install pytest && \
    python3 -m pip install pytest-sugar && \
    python3 -m pip install pytest-xdist && \
    python3 -m pip install "scapy>=2.4.2" && \
    python3 -m pip install xmltodict && \
    python3 -m pip install grpcio grpcio-tools && \
-    python2 -m pip install 'exabgp<4.0.0'
+    python3 -m pip install 'exabgp>=4.2.11'


Same here on exabgp, use the OSR version with the patches

Require exabgp >= 4.2.11 to allow to newer version to run exabgp topotests. Next commits will adapt the exabgp topotests when needed. Signed-off-by: Louis Scalbert <[email protected]>

Look for python3 exabgp instead of python2. Signed-off-by: Louis Scalbert <[email protected]>

Use exabgp 4.2.11 in CI instead. Signed-off-by: Louis Scalbert <[email protected]>

Log exabgp by default in /tmp/topotests/<testname>/<peername>/exabgp.log Level is INFO. Note that in case the configuration syntax is invalid, exabgp does not log into the file and exits at startup. You can check a configuration syntax by running: > exabgp <exabgp.cfg> Signed-off-by: Louis Scalbert <[email protected]>

Create reate exabgp cli fifo even it is not used in topotests to avoid this error message: > 16:21:42 | 2290205 | cli | could not find the named pipes (exabgp.in and exabgp.out) required for the cli > 16:21:42 | 2290205 | cli | we scanned the following folders (the number is your PID): > 16:21:42 | 2290205 | cli control | - /run/exabgp/ > 16:21:42 | 2290205 | cli control | - /run/0/ > 16:21:42 | 2290205 | cli control | - /run/ > 16:21:42 | 2290205 | cli control | - /var/run/exabgp/ > 16:21:42 | 2290205 | cli control | - /var/run/0/ > 16:21:42 | 2290205 | cli control | - /var/run/ > 16:21:42 | 2290205 | cli control | - /usr/local/run/exabgp/ > 16:21:42 | 2290205 | cli control | - /usr/local/run/0/ > 16:21:42 | 2290205 | cli control | - /usr/local/run/ > 16:21:42 | 2290205 | cli control | - /usr/local/var/run/exabgp/ > 16:21:42 | 2290205 | cli control | - /usr/local/var/run/0/ > 16:21:42 | 2290205 | cli control | - /usr/local/var/run/ > 16:21:42 | 2290205 | cli control | please make them in one of the folder with the following commands: > 16:21:42 | 2290205 | cli control | > mkfifo //run/exabgp.{in,out} > 16:21:42 | 2290205 | cli control | > chmod 600 //run/exabgp.{in,out} Signed-off-by: Louis Scalbert <[email protected]>

Convert bgp_ecmp_topo1 to exabgp 4 Signed-off-by: Louis Scalbert <[email protected]>

Convert bgp_multiview_topo1 to exabgp 4 Signed-off-by: Louis Scalbert <[email protected]>

Convert bgp_vrf_md5_peering to exabgp 4 Signed-off-by: Louis Scalbert <[email protected]>

Convert bgp_vrf_netns to exabgp 4 Signed-off-by: Louis Scalbert <[email protected]>

Convert bgp_prefix_sid to exabgp 4 Signed-off-by: Louis Scalbert <[email protected]>

Convert bgp_prefix_sid2 to exabgp 4 Do not advertise prefixes to exabgp to avoid an issue where exabgp resets the bgp session with the following notification: > invalid ipv6 mpls-vpn next-hop length 48 expected 24 or 40 Signed-off-by: Louis Scalbert <[email protected]>

Convert bgp_peer_type_multipath_relax to exabgp 4 Signed-off-by: Louis Scalbert <[email protected]>

Cleanup bgp_peer_type_multipath_relax to make it more readable. Signed-off-by: Louis Scalbert <[email protected]>

Convert exabgp scripts to python3 Signed-off-by: Louis Scalbert <[email protected]>

Convert bgp_ecmp_topo1 to python3 Signed-off-by: Louis Scalbert <[email protected]>

Remove python2 support Signed-off-by: Louis Scalbert <[email protected]>

Update ubuntu template for exabgp 4 Signed-off-by: Louis Scalbert <[email protected]>

Update the documentation to require exabgp 4 Signed-off-by: Louis Scalbert <[email protected]>

Workaround an issue in bgp_peer_type_multipath_relax. Link: FRRouting#14895 Signed-off-by: Louis Scalbert <[email protected]>

frrbot bot added docker documentation labels Nov 28, 2023

github-actions bot added master size/XXL labels Nov 28, 2023

frrbot bot added the bugfix label Nov 28, 2023

louis-6wind force-pushed the exabgp4 branch from 70a1361 to 48a1b5c Compare November 28, 2023 16:06

ton31337 reviewed Nov 29, 2023

View reviewed changes

louis-6wind force-pushed the exabgp4 branch 2 times, most recently from 61d57cc to 11a7b1b Compare November 29, 2023 10:00

ton31337 reviewed Dec 1, 2023

View reviewed changes

mwinter-osr self-requested a review December 13, 2023 17:32

mwinter-osr requested changes Dec 13, 2023

View reviewed changes

louis-6wind added 7 commits December 14, 2023 10:57

topotests: require exabgp version >= 4.2.11

92c1a10

Require exabgp >= 4.2.11 to allow to newer version to run exabgp topotests. Next commits will adapt the exabgp topotests when needed. Signed-off-by: Louis Scalbert <[email protected]>

topotests: look for python3 exabgp

9589fb3

Look for python3 exabgp instead of python2. Signed-off-by: Louis Scalbert <[email protected]>

topotests: use exabgp 4.2.11 in ci

c040637

Use exabgp 4.2.11 in CI instead. Signed-off-by: Louis Scalbert <[email protected]>

topotests: convert bgp_ecmp_topo1 to exabgp 4

bc149fe

Convert bgp_ecmp_topo1 to exabgp 4 Signed-off-by: Louis Scalbert <[email protected]>

topotests: convert bgp_multiview_topo1 to exabgp 4

d228f6a

Convert bgp_multiview_topo1 to exabgp 4 Signed-off-by: Louis Scalbert <[email protected]>

louis-6wind added 9 commits December 14, 2023 10:57

topotests: convert bgp_vrf_md5_peering to exabgp 4

9598967

Convert bgp_vrf_md5_peering to exabgp 4 Signed-off-by: Louis Scalbert <[email protected]>

topotests: convert bgp_vrf_netns to exabgp 4

cfcad6f

Convert bgp_vrf_netns to exabgp 4 Signed-off-by: Louis Scalbert <[email protected]>

topotests: convert bgp_prefix_sid to exabgp 4

811ad11

Convert bgp_prefix_sid to exabgp 4 Signed-off-by: Louis Scalbert <[email protected]>

topotests: convert bgp_peer_type_multipath_relax to exabgp 4

da8333d

Convert bgp_peer_type_multipath_relax to exabgp 4 Signed-off-by: Louis Scalbert <[email protected]>

topotests: cleanup bgp_peer_type_multipath_relax

5383945

Cleanup bgp_peer_type_multipath_relax to make it more readable. Signed-off-by: Louis Scalbert <[email protected]>

topotests: convert exabgp scripts to python3

75a849f

Convert exabgp scripts to python3 Signed-off-by: Louis Scalbert <[email protected]>

topotests: convert bgp_ecmp_topo1 to python3

5d1d44d

Convert bgp_ecmp_topo1 to python3 Signed-off-by: Louis Scalbert <[email protected]>

topotests: remove python2 support

a54d18c

Remove python2 support Signed-off-by: Louis Scalbert <[email protected]>

louis-6wind force-pushed the exabgp4 branch from 11a7b1b to 09a3919 Compare December 14, 2023 09:58

louis-6wind added 3 commits December 14, 2023 11:38

docker: update ubuntu template for exabgp 4

aee4afc

Update ubuntu template for exabgp 4 Signed-off-by: Louis Scalbert <[email protected]>

doc: update for exabgp 4

3dc8b2c

Update the documentation to require exabgp 4 Signed-off-by: Louis Scalbert <[email protected]>

topotests: tmp fix delay in bgp_peer_type_multipath_relax

73ce39c

Workaround an issue in bgp_peer_type_multipath_relax. Link: FRRouting#14895 Signed-off-by: Louis Scalbert <[email protected]>

louis-6wind force-pushed the exabgp4 branch from 09a3919 to 73ce39c Compare December 14, 2023 10:44

mwinter-osr approved these changes Dec 14, 2023

View reviewed changes

mwinter-osr merged commit 0d57a9a into FRRouting:master Dec 14, 2023
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

topotests: convert to exabgp 4 and python3 #14890

topotests: convert to exabgp 4 and python3 #14890

louis-6wind commented Nov 28, 2023

donaldsharp commented Nov 28, 2023

mwinter-osr commented Nov 28, 2023

louis-6wind commented Nov 28, 2023

mwinter-osr commented Nov 28, 2023

donaldsharp commented Nov 28, 2023

ton31337 Nov 29, 2023

louis-6wind Nov 29, 2023

ton31337 Nov 29, 2023

louis-6wind Nov 29, 2023

ton31337 Nov 29, 2023

louis-6wind Nov 29, 2023

ton31337 Nov 29, 2023

louis-6wind Nov 29, 2023

ton31337 left a comment

mwinter-osr commented Dec 5, 2023

louis-6wind commented Dec 6, 2023

mwinter-osr commented Dec 13, 2023

mwinter-osr left a comment

mwinter-osr Dec 13, 2023

mwinter-osr Dec 13, 2023

topotests: convert to exabgp 4 and python3 #14890

topotests: convert to exabgp 4 and python3 #14890

Conversation

louis-6wind commented Nov 28, 2023

donaldsharp commented Nov 28, 2023

mwinter-osr commented Nov 28, 2023

louis-6wind commented Nov 28, 2023

mwinter-osr commented Nov 28, 2023

donaldsharp commented Nov 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ton31337 left a comment

Choose a reason for hiding this comment

mwinter-osr commented Dec 5, 2023

louis-6wind commented Dec 6, 2023

mwinter-osr commented Dec 13, 2023

mwinter-osr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment