test_vxlan_ecmp.py failed occasionally

[yue-fred-gao]

test_vxlan_ecmp.py failed occasionally. It appears that vpp_vxlan_tunnel_add_del of a new tunnel sometimes mistakenly returns the sw_ifindex of the vxlan tunnel deleted right before it. The suspicion is that set_reply_status in vl_api_bfd_udp_session_event_t_handler, which is unsolicited event callback, messes up the call sequence.

[linqingxuan]

I was testing vxlan-related use cases recently, and I just found it. I am checking the code. If you have code updates, please @ me and I want to learn from it. If I can fix the whole problem, I will also submit a PR. Thank you very much

[yue-fred-gao]

@linqingxuan, I have created a PR to fix this issue. With the fix, the sonic mgmt test passed 5 times in a roll. Please take a look.

[linqingxuan]

I applied this patch and it did solve the problem. I recently had a requirement to have different vxlan macs based on the inner vxlan subnet.
For example

1 2 3 are three VMs, they have different MACs, but they do not accept vxlan arp layer 2 learning, which is quite awkward. I am currently using centec and broadcom chip switches, which can accept routing with MACs. I recently read the vpp source code and it seems that it does not support this scenario. In the tofino chip, I used acl to modify the inner MAC in egress.
If this scenario should be modified in vpp? If you can give me some advice, I will be very grateful

[yue-fred-gao]

Is it inner source Mac or destination Mac that you want to change based on subnet? And again, is it subnet of source IP or destination IP?

[linqingxuan]

I have refined the content of the picture. As shown in the picture, I expect to dynamically change the inner mac of the datagram based on vni and inner ip. They are a vm under bm, and bm has closed vxlan layer 2 learning.

[yue-fred-gao]

sonic supports setting dmac in vxlan ecmp route: https://github.com/sonic-net/SONiC/blob/master/doc/vxlan/Vxlan_hld.md#22-app-db. In your examples, you need to create 3 entries in VNET_ROUTE_TUNNEL_TABLE, each matching the prefix listed above and setting dmac accordingly.

It is not supported in sonic-vpp currently but should not be too difficult. Basically, when it programs ip neighbour (https://github.com/sonic-net/sonic-platform-vpp/blob/main/docs/HLD/vxlan-vnet.md#vm-to-baremetal), use the Mac passed in from SAI attribute instead of hardcoded 00:00:00:00:00:01 or router MAC is that's configured.

Is remote vtep also sonic-vpp? you cannot treat it as l3 vxlan because that will remove inner ether header. I guess you need to map the vxlan to vlan in the remote vtep to do l2 forwarding in order to preserve the inner ether. That's is not supported in sonic-vpp.

[linqingxuan]

I looked at it carefully and you are not quite right. My problem is that the same vtep has different inner dst macs. The remote end is not sonic-vpp. It is actually an old LB service. Because there are customers on it, I can't modify its logic for the time being.

[linqingxuan]

I think I would be more suitable to find a L2 rewrite dest MAC.

[linqingxuan]

thank you very much

[yue-fred-gao]

I think you need to design it in the frame of SONiC. Otherwise, it will be too much work in control plane. With vxlan vnet tunnel route, I am not sure if you can have multiple such routes pointing to the same VTEP (same address and VNI). You can see if SONIC allows it or not. If it does, you can explore how to make it work in sonic-vpp. In vpp data plane, this should be supported. Basically, multiple ip neighbour with different nexthop address and Mac pointing to the same vxlan tunnel.
You can explore l2 rewrite. But I am not sure how to fit it into SONiC.

[linqingxuan]

I think the sonic layer is easy to modify, but the vpp layer, hahahahahaha, I just want to write a simple plug-in to modify it directly,