Add public APIs to Access Underlying `cudf` and `pandas` Objects from `cudf.pandas` Proxy Objects #17629

galipremsagar · 2024-12-19T09:27:43Z

Description

Fixes: #17524
Fixes: rapidsai/cuml#6232
This PR introduces methods to access the real underlying cudf and pandas objects from cudf.pandas proxy objects. These methods ensure compatibility with libraries that are cudf or pandas aware.

This PR also gives a performance boost to cudf-pandas workflows, speeds from the script posted in rapidsai/cuml#6232:

branch-25.02:

cuML Label Encoder with cuDF-Pandas took 2.00794 seconds

This PR:

cuML Label Encoder with cuDF-Pandas took 0.09284 seconds

Changes:

Added get_gpu_object() and get_cpu_object() methods.
Updated faq.md with a section explaining how to use these methods.

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

Matt711

Overall, this looks good to me. But I think we should add a bullet to the "Are there any limitations?" section of this faq.md . And it should describe the implications for users of cudf.pandas and third-party libraries that are "cudf aware." For example, they could get a cupy array (not a numpy array) when working with xgboost. What do you think?

python/cudf/cudf/pandas/fast_slow_proxy.py

galipremsagar · 2025-01-25T00:35:35Z

@vyasr @bdice @Matt711 This is ready for another round of reviews.

bdice

How we document this for users is probably the most important aspect of this PR. I gave some suggestions, let me know what you think.

docs/cudf/source/cudf_pandas/faq.md

Co-authored-by: Bradley Dice <[email protected]>

mroeschke · 2025-01-25T01:43:06Z

python/cudf/cudf/core/dataframe.py

+                and (index is None or index_extracted)
+                and (columns is None or columns_extracted)
+            ) and (dtype is None and copy is None):
+                self.__dict__.update(data.__dict__)


Could we use _mimic_inplace instead?

I feel this is much light weight that using _mimic_inplace and also copies cached attributes.

python/cudf/cudf/utils/utils.py

python/cudf/cudf/pandas/_wrappers/pandas.py

Co-authored-by: Matthew Roeschke <[email protected]>

docs/cudf/source/cudf_pandas/faq.md

python/cudf/cudf/pandas/_wrappers/numpy.py

bdice · 2025-01-27T22:46:23Z

python/cudf/cudf/pandas/_wrappers/pandas.py

+def is_cudf_pandas_dataframe(obj):
+    return is_proxy_object(obj) and isinstance(obj, DataFrame)
+
+
+def is_cudf_pandas_series(obj):
+    return is_proxy_object(obj) and isinstance(obj, Series)
+
+
+def is_cudf_pandas_index(obj):
+    return is_proxy_object(obj) and isinstance(obj, Index)


Are these functions actually useful? It seems like it would be better to just tell the user how to check this themselves with is_proxy_object + isinstance.

Yes, we could do that but my intention here was to just have single API that informs the users. It will end up being a verbose pattern in libraries that operate cudf/cudf.pandas aware.

My suggestion in a thread above was to instead implement a single method (bikeshed the name as much as you wish, I'm mostly concerned with suggesting an implementation)

def isinstance_cudf_pandas(obj, type): return is_proxy_object(obj) and isinstance(obj, type)

so that they don't need to be aware of is_proxy_object and it "feels like" they're using isinstance.

Co-authored-by: Bradley Dice <[email protected]>

docs/cudf/source/cudf_pandas/faq.md

python/cudf/cudf_pandas_tests/test_cudf_pandas.py

Co-authored-by: Bradley Dice <[email protected]>

docs/cudf/source/cudf_pandas/faq.md

python/cudf/cudf/core/dataframe.py

vyasr · 2025-01-27T23:56:09Z

python/cudf/cudf/pandas/_wrappers/pandas.py

+def is_cudf_pandas_dataframe(obj):
+    return is_proxy_object(obj) and isinstance(obj, DataFrame)
+
+
+def is_cudf_pandas_series(obj):
+    return is_proxy_object(obj) and isinstance(obj, Series)
+
+
+def is_cudf_pandas_index(obj):
+    return is_proxy_object(obj) and isinstance(obj, Index)


My suggestion in a thread above was to instead implement a single method (bikeshed the name as much as you wish, I'm mostly concerned with suggesting an implementation)

def isinstance_cudf_pandas(obj, type): return is_proxy_object(obj) and isinstance(obj, type)

so that they don't need to be aware of is_proxy_object and it "feels like" they're using isinstance.

bdice · 2025-01-28T15:45:51Z

docs/cudf/source/cudf_pandas/faq.md

+from cudf.pandas import isinstance_cudf_pandas
+
+obj = ...  # Your object here
+if isinstance_cudf_pandas(obj, "Series"):


Normally isinstance takes a class, not a string. Is that an option here, too?

responded here; #17629 (comment)

docs/cudf/source/cudf_pandas/faq.md

bdice · 2025-01-28T15:47:52Z

python/cudf/cudf/pandas/_wrappers/numpy.py

@@ -176,3 +176,7 @@ def ndarray__array_ufunc__(self, ufunc, method, *inputs, **kwargs):
    cupy._core.flags.Flags,
    _numpy_flagsobj,
 )
+
+
+def is_cudf_pandas_ndarray(obj):


Can we remove this?

bdice · 2025-01-28T15:50:20Z

python/cudf/cudf/pandas/_wrappers/pandas.py

@@ -1704,6 +1713,10 @@ def holiday_calendar_factory_wrapper(*args, **kwargs):
    )


+def isinstance_cudf_pandas(obj, type_name):
+    return is_proxy_object(obj) and obj.__class__.__name__ == type_name


Is the second half of this sufficiently precise? Can we rely on names alone? I would like to use some kind of "real" check like isinstance if that is allowable.

I think allowing real types will make this API more complex and confusing. We don't want to allow isinstance_cudf_pandas(obj, pd.DataFrame/cudf.DataFrame) as users might get confused to what they are actually checking when cudf.pandas is enabled and disabled.

xgboost has this string based checks: https://github.com/dmlc/xgboost/pull/11014/files#diff-bf11fe0b3133c5b20253ea67b82c3e576513c8079f3be355fc323e3e903d989cR852

Typically string-based checks are used as a way to avoid requiring an import of that package. We could still accept a class in the API and do a string-based check of obj.__class__.__name__ == type.__name__.

Updated the api to use this approach. 👍

Co-authored-by: Bradley Dice <[email protected]>

bdice · 2025-01-28T18:49:06Z

python/cudf/cudf/pandas/_wrappers/pandas.py

@@ -1704,6 +1713,10 @@ def holiday_calendar_factory_wrapper(*args, **kwargs):
    )


+def isinstance_cudf_pandas(obj, type):


We need API docs for public APIs.

Can I add that in a followup PR to save some CI time? I'd like to get this PR in and fix cuml ASAP.

That's fine. Please file an issue or draft PR to ensure this isn't lost.

Matt711

Thanks! I had some non-blocking questions.

python/cudf/cudf/core/column/column.py

Matt711 · 2025-01-28T19:11:32Z

python/cudf/cudf/pandas/fast_slow_proxy.py

+    def as_gpu_object(self):
+        return self._fsproxy_slow_to_fast()


Does it make sense to call isinstance_cudf_pandas inside this function and raise a friendly error message on False?

python/cudf/cudf/core/series.py

vyasr

Can we revert the changes to dataframe.py, index.py, series.py, and pandas/init.py, ? Those are now just holdovers of earlier commits. I'm preapproving so that you can merge when ready. Thanks Prem!

vyasr · 2025-01-28T19:19:15Z

docs/cudf/source/cudf_pandas/faq.md

+from cudf.pandas import isinstance_cudf_pandas
+
+obj = ...  # Your object here
+if isinstance_cudf_pandas(obj, pd.Series):


We should document if this is doing to work with cudf.Series if cudf is directly imported. It's fine if the answer is no, or if the answer is UB and we don't promise anything, we should just mention it either way.

galipremsagar · 2025-01-29T03:10:41Z

/merge

Add a public api to get fast slow objects

b5eea1f

galipremsagar added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Dec 19, 2024

galipremsagar self-assigned this Dec 19, 2024

galipremsagar requested a review from a team as a code owner December 19, 2024 09:27

galipremsagar requested review from bdice and Matt711 December 19, 2024 09:27

github-actions bot added Python Affects Python cuDF API. cudf.pandas Issues specific to cudf.pandas labels Dec 19, 2024

Matt711 requested changes Dec 19, 2024

View reviewed changes

bdice reviewed Dec 19, 2024

View reviewed changes

python/cudf/cudf/pandas/fast_slow_proxy.py Outdated Show resolved Hide resolved

galipremsagar mentioned this pull request Jan 24, 2025

[BUG] - cuML LabelEncoder is 200x slower with cuDF-Pandas vs cuDF rapidsai/cuml#6232

Open

galipremsagar added 6 commits January 24, 2025 11:19

Merge remote-tracking branch 'upstream/branch-25.02' into 17524

7bc76e5

update names and add fast paths

3cdfe94

centralize logic

34375dc

fix

31f9e99

cleanup

72ba73f

Merge branch 'branch-25.02' into 17524

3fd679f

galipremsagar added the 3 - Ready for Review Ready for review by team label Jan 25, 2025

galipremsagar requested review from Matt711, vyasr and bdice January 25, 2025 00:51

bdice reviewed Jan 25, 2025

View reviewed changes

Apply suggestions from code review

37764c2

Co-authored-by: Bradley Dice <[email protected]>

bdice approved these changes Jan 25, 2025

View reviewed changes

mroeschke reviewed Jan 25, 2025

View reviewed changes

python/cudf/cudf/utils/utils.py Outdated Show resolved Hide resolved

mroeschke reviewed Jan 25, 2025

View reviewed changes

python/cudf/cudf/pandas/_wrappers/pandas.py Outdated Show resolved Hide resolved

Apply suggestions from code review

bbe0fa4

Co-authored-by: Matthew Roeschke <[email protected]>

galipremsagar added 4 commits January 27, 2025 08:02

style

9ec0215

add is_cudf_pandas.. APIs

a3c49fd

update docs

3f06e70

Merge remote-tracking branch 'upstream/branch-25.02' into 17524

95ba799

bdice reviewed Jan 27, 2025

View reviewed changes

Apply suggestions from code review

a2e97f5

Co-authored-by: Bradley Dice <[email protected]>

galipremsagar commented Jan 27, 2025

View reviewed changes

Apply suggestions from code review

b96f8ff

Co-authored-by: Bradley Dice <[email protected]>

vyasr requested changes Jan 27, 2025

View reviewed changes

galipremsagar added 3 commits January 28, 2025 07:16

update API

0044d8f

revert cudf.pandas spilling into cudf

a77fecc

Merge branch 'branch-25.02' into 17524

aafbd31

galipremsagar requested review from vyasr and bdice January 28, 2025 15:22

bdice reviewed Jan 28, 2025

View reviewed changes

galipremsagar and others added 4 commits January 28, 2025 09:57

Update docs/cudf/source/cudf_pandas/faq.md

f15d37b

Co-authored-by: Bradley Dice <[email protected]>

cleanup

f94253d

update api

daaa5df

Merge branch 'branch-25.02' into 17524

eaa23cb

bdice reviewed Jan 28, 2025

View reviewed changes

Matt711 approved these changes Jan 28, 2025

View reviewed changes

vyasr approved these changes Jan 28, 2025

View reviewed changes

galipremsagar added 2 commits January 28, 2025 11:26

cleanup

9c58a71

Merge branch 'branch-25.02' into 17524

69837f3

galipremsagar added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 3 - Ready for Review Ready for review by team labels Jan 28, 2025

rapids-bot bot merged commit ed2f3c3 into rapidsai:branch-25.02 Jan 29, 2025
107 of 108 checks passed

galipremsagar mentioned this pull request Jan 29, 2025

Extract true gpu object from cudf.pandas proxy object rapidsai/cuml#6273

Draft

Matt711 mentioned this pull request Jan 29, 2025

Make _Series_dtype method a property #17854

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add public APIs to Access Underlying `cudf` and `pandas` Objects from `cudf.pandas` Proxy Objects #17629

Add public APIs to Access Underlying `cudf` and `pandas` Objects from `cudf.pandas` Proxy Objects #17629

galipremsagar commented Dec 19, 2024 •

edited

Loading

Matt711 left a comment

galipremsagar commented Jan 25, 2025

bdice left a comment

mroeschke Jan 25, 2025

galipremsagar Jan 27, 2025

bdice Jan 27, 2025

galipremsagar Jan 27, 2025

vyasr Jan 27, 2025

vyasr Jan 27, 2025

bdice Jan 28, 2025

galipremsagar Jan 28, 2025

bdice Jan 28, 2025

galipremsagar Jan 28, 2025

bdice Jan 28, 2025

galipremsagar Jan 28, 2025

bdice Jan 28, 2025 •

edited

Loading

galipremsagar Jan 28, 2025

bdice Jan 28, 2025

galipremsagar Jan 28, 2025 •

edited

Loading

bdice Jan 28, 2025

Matt711 left a comment

Matt711 Jan 28, 2025

vyasr left a comment

vyasr Jan 28, 2025

galipremsagar commented Jan 29, 2025

		@@ -1704,6 +1713,10 @@ def holiday_calendar_factory_wrapper(args, *kwargs):
		)


		def isinstance_cudf_pandas(obj, type):

Add public APIs to Access Underlying cudf and pandas Objects from cudf.pandas Proxy Objects #17629

Add public APIs to Access Underlying cudf and pandas Objects from cudf.pandas Proxy Objects #17629

Conversation

galipremsagar commented Dec 19, 2024 • edited Loading

Description

Checklist

Matt711 left a comment

Choose a reason for hiding this comment

galipremsagar commented Jan 25, 2025

bdice left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdice Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

galipremsagar Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Matt711 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyasr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

galipremsagar commented Jan 29, 2025

Add public APIs to Access Underlying `cudf` and `pandas` Objects from `cudf.pandas` Proxy Objects #17629

Add public APIs to Access Underlying `cudf` and `pandas` Objects from `cudf.pandas` Proxy Objects #17629

galipremsagar commented Dec 19, 2024 •

edited

Loading

bdice Jan 28, 2025 •

edited

Loading

galipremsagar Jan 28, 2025 •

edited

Loading