Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change order of initialization so pinned pool is available for spill framework buffers #11889

Merged

Conversation

abellina
Copy link
Collaborator

@abellina abellina commented Dec 18, 2024

While collecting traces for #11888, I realized that the bounce buffers I am using are pageable, which is pretty bad. If I change the order of initialization, the pinned pool can be initialized before the spill framework, so we can then use pinned memory for the d2h and h2d bounced copies.

Renamed init functions slightly, as their names are not matching their function currently.

@abellina
Copy link
Collaborator Author

build

@abellina
Copy link
Collaborator Author

CI failures due to tests initializing too many times. Before my change we had all of this code under a "make sure Rmm is not initialized already" block.

Signed-off-by: Alessandro Bellina <[email protected]>
@abellina abellina force-pushed the initialize_spill_after_pinned_pool branch from 7b5c05b to 6c1178e Compare January 14, 2025 22:23
@abellina
Copy link
Collaborator Author

build

@abellina abellina requested a review from zpuller January 15, 2025 14:11
Copy link
Collaborator

@zpuller zpuller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, was able to confirm we see spill going to pinned mem

Copy link
Collaborator

@revans2 revans2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not super happy with the TestUtils.withGpuSparkSession needing to be removed. I'm not sure how to tell when reviewing something if it should be there or not. But that said it is minor.

@abellina
Copy link
Collaborator Author

abellina commented Jan 21, 2025

I'm not super happy with the TestUtils.withGpuSparkSession needing to be removed. I'm not sure how to tell when reviewing something if it should be there or not. But that said it is minor.

I don't know how to explain it other than we shouldn't start spark sessions when we don't need to submit spark tasks during a test. I think it's fairly straightforward: if your suite is all about spark tasks submitted via SQL or spark APIs, then yeah by all means set up a session.. but if your tests are really unit tests, don't use the spark session to backwards initialize singletons, just set them up directly.

@abellina abellina merged commit f6274b6 into NVIDIA:branch-25.02 Jan 21, 2025
52 checks passed
@abellina abellina deleted the initialize_spill_after_pinned_pool branch January 21, 2025 20:28
@sameerz sameerz added the performance A performance related task/issue label Jan 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance A performance related task/issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants