Fix UFFD handler never receiving remove messages when balloon device inflates
#5021
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In test_valid_handler, we had a comment to validate that the guest still works if we mess with the balloon after UFFD-based restoration. However, we didn't actually do so. Fix this by running some simple SSH command after both inflation and deflation. Signed-off-by: Patrick Roy <[email protected]>
When the balloon inflates, and the guest gives us back some pages of memory, we need to free those pages. In booted VMs, we do this with madvise(MADV_DONTNEED), and in restored VMs we do it by MAP_FIXED-ing a new VMA on top of the range-to-free. This is because if guest memory is a MAP_PRIVATE of a memory file, madvise has no effect. However, we also do this MAP_FIXED trick if the snapshot is restored with UFFD. In this case, its not needed (madvise works perfectly fine), and in fact, its wrong: If we map over the memory range, UFFD will not receive Remove events for the specified range. Fix this by only using the mmap trick for file-based restored. Fixes firecracker-microvm#4988 Signed-off-by: Patrick Roy <[email protected]>
roypat
requested review from
xmarcalx,
kalyazin,
pb8o and
Manciukic
as code owners
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5021 +/- ##
==========================================
+ Coverage 83.13% 83.14% +0.01%
==========================================
Files 245 245
Lines 26697 26704 +7
==========================================
+ Hits 22194 22203 +9
+ Misses 4503 4501 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
3 tasks
roypat
force-pushed
the
uffd-balloon-fix
branch
2 times, most recently
from
75fb5c5 to
eb07e2f
Compare
roypat
added
the
Status: Awaiting review
kalyazin
reviewed
Jan 30, 2025
roypat
force-pushed
the
uffd-balloon-fix
branch
from
eb07e2f to
b9a005f
Compare
kalyazin
reviewed
Jan 31, 2025
The crux of the issue was that UFFD gets blocked (all ioctls return -EAGAIN) when there's any `remove` events pending in the queue, which means during processing we not only need to look at the "head" of the queue, but also make sure there's no `remove` events in the "tail". Deal with these scenarios correctly by always greedily reading the entire queue, to ensure there's nothing pending, and only then processing things one-by-one. Please see the new code comments for intricacies with this approach. Fixes firecracker-microvm#4990 Signed-off-by: Patrick Roy <[email protected]>
Add entry about the UFFD fix. Signed-off-by: Patrick Roy <[email protected]>
roypat
force-pushed
the
uffd-balloon-fix
branch
from
b9a005f to
5c2dec5
Compare
kalyazin
approved these changes
Jan 31, 2025
bchalios
approved these changes
Jan 31, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When the balloon inflates, and the guest gives us back some pages of memory, we need to free those pages. In booted VMs, we do this with madvise(MADV_DONTNEED), and in restored VMs we do it by MAP_FIXED-ing a new VMA on top of the range-to-free. This is because if guest memory is a MAP_PRIVATE of a memory file, madvise has no effect.
However, we also do this MAP_FIXED trick if the snapshot is restored with UFFD. In this case, its not needed (madvise works perfectly fine), and in fact, its wrong: If we map over the memory range, UFFD will not receive Remove events for the specified range.
Fix this by only using the mmap trick for file-based restored.
After doing this, fix our test uffd handler to actually handle remove events correctly (since it never received any before, we never noticed that its was completely broken - but now that we actually exercise this in the test, it needed fixing).
License Acceptance
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.PR Checklist
tools/devtool checkstyleto verify that the PR passes theautomated style checks.
how they are solving the problem in a clear and encompassing way.
in the PR.
CHANGELOG.md.Runbook for Firecracker API changes.
integration tests.
TODO.rust-vmm.