-
-
Notifications
You must be signed in to change notification settings - Fork 18k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix BUG: read_sql tries to convert blob/varbinary to string with pyarrow backend #60105
base: main
Are you sure you want to change the base?
Changes from all commits
2244869
bd00fc5
6a23f05
2f261c8
8f900b8
536b1ed
8965a1d
a32b4a6
f412c72
a0200d0
10ca030
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -4358,3 +4358,27 @@ def test_xsqlite_if_exists(sqlite_buildin): | |||||
(5, "E"), | ||||||
] | ||||||
drop_table(table_name, sqlite_buildin) | ||||||
|
||||||
|
||||||
@pytest.mark.parametrize("dtype_backend", ["pyarrow", "numpy_nullable", lib.no_default]) | ||||||
def test_bytes_column(sqlite_buildin, dtype_backend): | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Should test this against all databases |
||||||
pa = pytest.importorskip("pyarrow") | ||||||
""" | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is well intentioned but can you remove the docstring? We don't use them in tests. Instead, you can just add a comment pointing to the github issue number in the function body |
||||||
Regression test for (#59242) | ||||||
Bytes being returned in a column that could not be converted | ||||||
to a string would raise a UnicodeDecodeError | ||||||
when using dtype_backend='pyarrow' or dtype_backend='numpy_nullable' | ||||||
""" | ||||||
query = """ | ||||||
select cast(x'0123456789abcdef0123456789abcdef' as blob) a | ||||||
""" | ||||||
df = pd.read_sql(query, sqlite_buildin, dtype_backend=dtype_backend) | ||||||
expected = DataFrame( | ||||||
[ | ||||||
{ | ||||||
"a": b"\x01#Eg\x89\xab\xcd\xef\x01#Eg\x89\xab\xcd\xef", | ||||||
} | ||||||
], | ||||||
dtype=(pd.ArrowDtype(pa.binary()) if dtype_backend == "pyarrow" else "O"), | ||||||
) | ||||||
tm.assert_frame_equal(df, expected) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also found that this was failing for
dtype_backend=numpy_nullable
so implemented a fix and added test cases for each possibledtype_backend