Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use KvikIO to enable file's fast host read and host write #17764

Draft
wants to merge 6 commits into
base: branch-25.04
Choose a base branch
from
Draft

Use KvikIO to enable file's fast host read and host write #17764

wants to merge 6 commits into from

Conversation

kingcrimsontianyu
Copy link
Contributor

@kingcrimsontianyu kingcrimsontianyu commented Jan 17, 2025 •
edited
Loading

Description

This PR makes the following improvements on I/O:

  • Remove legacy cuFile integration to simplify code maintenance. Use KvikIO to manage the GDS setting and compatibility mode.
  • Remove file utility classes and functions. Use KvikIO for all file-related operations.
  • Replace in-house implementation of host_read (for file_source) and host_write (for file_sink) with KvikIO's parallel counterpart.
  • Update the documentation on compatibility mode/GDS.

Closes #16418 and #17228

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@kingcrimsontianyu kingcrimsontianyu added libcudf Affects libcudf (C++/CUDA) code. improvement Improvement / enhancement to an existing function breaking Breaking change labels Jan 17, 2025
@kingcrimsontianyu kingcrimsontianyu self-assigned this Jan 17, 2025
@kingcrimsontianyu kingcrimsontianyu requested review from a team as code owners January 17, 2025 20:09
@github-actions github-actions bot added the CMake CMake build issue label Jan 17, 2025
@kingcrimsontianyu kingcrimsontianyu marked this pull request as draft January 17, 2025 20:09
@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Jan 17, 2025

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@kingcrimsontianyu
Copy link
Contributor Author

/ok to test

@kingcrimsontianyu kingcrimsontianyu changed the title Remove legacy cuFile integration I/O improvements Jan 17, 2025
@kingcrimsontianyu kingcrimsontianyu changed the title I/O improvements Improvements on data source and data sink Jan 18, 2025
@kingcrimsontianyu
Copy link
Contributor Author

/ok to test

@kingcrimsontianyu kingcrimsontianyu force-pushed the remove-legacy-cufile-integration branch from 1d2ed1e to 2f045b4 Compare January 18, 2025 13:35
@kingcrimsontianyu
Copy link
Contributor Author

/ok to test

@kingcrimsontianyu kingcrimsontianyu force-pushed the remove-legacy-cufile-integration branch from e6847aa to d1932df Compare January 20, 2025 19:30
@kingcrimsontianyu
Copy link
Contributor Author

/ok to test

@kingcrimsontianyu kingcrimsontianyu changed the title Improvements on data source and data sink Use KvikIO to enable file's fast host read and host write Jan 21, 2025
@kingcrimsontianyu kingcrimsontianyu force-pushed the remove-legacy-cufile-integration branch from d1932df to 183baf8 Compare January 24, 2025 04:06
@github-actions github-actions bot added Python Affects Python cuDF API. Java Affects Java cuDF API. labels Jan 24, 2025
@github-actions github-actions bot added cudf.pandas Issues specific to cudf.pandas cudf.polars Issues specific to cudf.polars pylibcudf Issues specific to the pylibcudf package labels Jan 24, 2025
@kingcrimsontianyu kingcrimsontianyu changed the base branch from branch-25.02 to branch-25.04 January 24, 2025 04:07
@kingcrimsontianyu kingcrimsontianyu removed Python Affects Python cuDF API. Java Affects Java cuDF API. cudf.pandas Issues specific to cudf.pandas cudf.polars Issues specific to cudf.polars pylibcudf Issues specific to the pylibcudf package labels Jan 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shrshi shrshi Awaiting requested review from shrshi shrshi is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

@pmattione-nvidia pmattione-nvidia Awaiting requested review from pmattione-nvidia pmattione-nvidia is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

@vuule vuule Awaiting requested review from vuule

At least 1 approving review is required to merge this pull request.

Assignees

@kingcrimsontianyu kingcrimsontianyu

Labels
breaking Breaking change CMake CMake build issue improvement Improvement / enhancement to an existing function libcudf Affects libcudf (C++/CUDA) code.
Projects
cuDF Python
Status: In Progress
libcudf
Status: No status
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Remove the cuFile (GDS) backend
1 participant
@kingcrimsontianyu