Using RTR with BMP triggers a RIB-DUMP upon RTR Refresh #17533
Comments
|
I'm a bit confused what's wrong yet. Could you show the full config? Do you have custom compiled version of librtr? What do you mean dumped (logs saying that)? |
Upon an RTR update the router re-sends the full rib. Causes a bit of traffic:
BMP has been activated between Week 46 and Week 47
Do you have custom compiled version of librtr?
|
|
Are you saying that once the RTR cache server refreshes the records and sends a new version (serial changes), FRR revalidates all the routes, but not the received ones? |
|
I am saying that FRR is sending the whole RIB via BMP. I have no idea why it does it, i.e., if it revalidates all routes. |
|
I tested with the master branch, and I don't see RIB being resent if the RPKI state changes. I see the same logs: But these do not trigger sending anything to BMP receivers if only RPKI state changes. I see changes in BMP only if some attributes change. Would be possible to get the logs Could you test with the master or 10.2 or 10.1 at least? |
|
Will see whether I can kick my automation to do that in the coming days; Will keep you updated. |
|
Ok, this will take some time, I guess; VyOS closed source-code access for the stable branches ( https://vyos.dev/T6781 ); Meaning I have to figure out a couple of other things first (rebuild automation and setup with other base-system+automation/port all my local-site additions to current/figure out how to patch the stale versions around enough so I can at least build a new image with a newer FRR); Will take some time. -.-' |
|
Ok, after some 'discussions' with the build system, things got deployed. I am now running 10.2* and the effect did not go away: The large in traffic is from when i upgraded the routers. With FRR now no longer choking on 'permission denied' for RAs (the extended-nexthop thing fixed in 10.2), i saw something else, though; For example, for Looking at the FRR logs, I see: Could it be that this only triggers if a specific amount of prefix changes is (at least) received via RTR? Looking back in the graphs, I have only one event since yesterday where i have three digit prefixes and did not see a dump; Specifically: Not saying I am paranoid, but that three-digit-number-not-causing-an-event-like-the-others is suspiciously below 256; The only one below 256 of the three digit ones, and the only one after which there was no event. If you think it helps I can poll the graphs and logs from all ~20 routers to see if we can inch closer to a 'number from which on it triggers'.
|
|
Are you able to see the BMP logs to verify if FRR is really sending a full-feed? Also.. you should notice |
|
Yes, we confirmed on the BMP side that FRR sends all routes. We also drew some (somewhat larger PCAPs); Nothing out of the ordinary there. Interestingly, I do not see a Looking at My colleague @TheEnbyperor also just noticed that this is set in ll#789 if the kernel is blocking, i.e., after 4k for a default datagram socket buffer. By their calculation this should hit at ca. 300 updates given around 200 bytes per update. Which maps to the observations. They probably can better chime in themselves. |
|
Tobias gave a pretty good summary, but I'll elaborate just to try and make it clearer. There exists a function if (atomic_load_explicit(&rpki_vrf->rtr_update_overflow, memory_order_seq_cst)) {
while (read(rpki_vrf->rpki_sync_socket_bgpd, &rec, sizeof(struct pfx_record)) != -1);
atomic_store_explicit(&rpki_vrf->rtr_update_overflow, 0, memory_order_seq_cst);
revalidate_all_routes(rpki_vrf);
return;
}Where is int retval = write(rpki_vrf->rpki_sync_socket_rtr, &rec, sizeof(struct pfx_record));
if (retval == -1 && (errno == EAGAIN || errno == EWOULDBLOCK))
atomic_store_explicit(&rpki_vrf->rtr_update_overflow, 1, memory_order_seq_cst);
This seems to have been introduced by 4ce8267, but I can't figure out why it'd take from 2018 until now to present itself in this way. Perhaps the send buffer on the socket should be increased? Or is something more drastic in order? |
|
What about if we scan not the whole RIB, but only the affected (by socket overflow) prefixes? I have an idea, going to validate it and see if it would work at some level better than the current approach. |
|
I am still trying to wrap my head around the issue here; my (amateur) understanding is:
To me it sounds like it would rather make sense to resolve the deadlock. Academic-code-quality-me would probably use a tmpfile instead (which--i know--comes with a ton of other caveats). But i think that resolving the deadlock situation is much more sensible than trying to work around the limits of buffer space. |
|
Would you be able to cherry-pick this commit opensourcerouting@c8f7984 and test? |
|
Building an image; Let's see. |
|
Pushed the image on the router at AMSIX; Let's see what this does. If it works the box should be showing less traffic than the rest of AS215250. |
|
so, with that patch applied no BGP session comes up; Currently drawing a pcap |
|
dies with a 'bad peer AS' notification, even though the peer AS is correctly configured; The notification even carries the correct ASN. |
|
Something unrelated at all.. Could you try installing .deb/.rpm from here https://ci1.netdef.org/browse/FRR-PULLREQ3-6403/artifact? |
|
Makes things a lot more difficult for me. But I can try; Will take some time, though, likely until after an upcoming trip, i.e., until the 17th at least. |
|
Ok, this turned out easier than anticipated. However, the .deb from the URL you provided seem to not include opensourcerouting@c8f7984 ? |
|
It should include because these artifacts are based on #17586. |
|
Hrm, it says it is, but it only lists a PIM related commit?
|
|
hrm, ok, the source does include the changes it seems; Rebooting. |
|
Used your image; Does not change the fact that no sessions come up because of the invalid-as issue I also saw when applying the patch myself. |
|
To be sure... is it fine with the master branch if you compile it? Because I don't see any relation to session not going up and my changes. |
|
I have the same behavior for 10.2; If I do not apply your patch, sessions come up; If I do apply the patch, I get an 'invalid AS'. |
|
Can you show the configuration? |
|
One of those, they are all auto generated: #17533 (comment) |
|
Hrm, shall I try what happens if I disable RTR/RPKI? |
|
sigh ffs... give me a second, i might have to burn some things. -.-' Edit: For the record; This is related to the config management system, as I suspected that to be involved at this time. |
|
ok, it appears that with your patch supplied a peer config can no longer "overwrite" the remote-as set in a peer group. Even if there is no remote-as set in a peer group... o.O Is this intentional? |
|
With the patch, for some reason, FRR no longer accepts a peer group member to set the remote AS for itself; I.e., there is no remote-as configured in the peer group, only in the peer, but i still get: Leading to a partial config load. Also tried disabling rpki/rtr to no avail. Was such a change explicitly made some when? (That would kind of break the usefulness of the peergroup feature, though...) What i also find odd is that just applying your patch (individual commit) has this effect. Does your patch somewhere initialize the remote as per peer, which may lead to the peer-group getting an empty remote-as initialized even if none is set, then leading to this issue? |
|
remote-as with peer-group was fixed here #17542 and backported to stable/10.2. I don't understand how you get the same issue ONLY applying my fix for this issue... |
|
Something you have to understand about me... at some point in the past i worked with a person that had... a... gift... to stumble into very weird bugs. Sadly, we noticed to late that it is contagious.... ;-) Let me do some specific checks to make sure which versions are running when; Will come back to you in a bit. |
|
.oO( so... in general... shall i just setup 2-3 test boxes and set them up with the associated versions and you mail me an SSH key? ) |
|
Ok, checked up;
Then again not exactly; both have been using 10.2/stable at the time of build. The working one has been built at Dec 3 14:17 UTC, the not working one Dec 4 22:33 UTC. The backported fix for the peer-group issue was merged in 0b26493 at Dec 4 12:37 UTC. Note, also, that the issue I am experiencing is mildly different from the one in #17541 The issue in that one was:
my issue is:
My expectation of how that bug could happen actually maps to how #17542 resolved the issue in #17541: By initializing the remote-as for a peer group; So, as that value is no longer empty, it can't be overwritten. I would argue that the issue observed by me hence is a regression introduced by #17542 ... Shall I file a new bug for that one? |
|
Took the liberty: #17602 |
|
Thanks, will fix that and sorry for this inconvenience... |
|
Okay... could you apply then first opensourcerouting@2797506, and then c8f7984? :) |
|
lemme bake a build before the breakfeast burrito. Will let you know in a bit. |
|
Comes up properly now. Pushing it to all other routers. |
|
So, to keep this updated: So far there were several events tat would otherwise have qualified for a route dump; But none happened. However, we see a bit less BMP traffic than expected. Will wait for input on monday where this might be coming from. |
|
Ok, further investigation shows that the patch is effective and to the best of our knowledge does not introduce any side-effects itself. From my perspective, this is good to merge and backport. |
Description
We are currently running a set (~30) of FRR routers in three ASes that send BMP to a central collector using the following configuration:
We noticed that the BMP host receives +100mbit of inbound traffic from the routers every ~30 minutes. We found that the FRR hosts dump the full RIB at that time, seemingly in sync across ASes.
All FRR hosts are configured to use the same RTR hosts:
When the event starts, the FRR daemons show consistently:
I would argue that FRR should not repush the whole RIB upon RTR refresh; Only changed routes should be sent.
Version
How to reproduce
Expected behavior
Only routes that changed their validation state are sent via BMP.
Actual behavior
The full RIB is dumped, generating notable traffic.
Additional context
No response
Checklist
The text was updated successfully, but these errors were encountered: