Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove pyarrow dependency #582

Merged
merged 28 commits into from
Aug 8, 2024
Merged

Remove pyarrow dependency #582

merged 28 commits into from
Aug 8, 2024

Conversation

kylebarron
Copy link
Member

For #581

@kylebarron
Copy link
Member Author

kylebarron commented Jul 31, 2024

After a lot of work here and in arro3, we're pretty close to having this working! I think just three tasks!

  • table rechunking so that each Parquet buffer can have reasonably similar sizes
    for record_batch in table.to_batches(max_chunksize=max_chunksize):
  • Support for adding an array or a chunked array in table.add_column
    table = table.append_column("row_index", arange_col)
  • inferring the data type for primitive data types in from_numpy

    lonboard/lonboard/traits.py

    Lines 1008 to 1013 in ac5baac

    return fixed_size_list_array(
    Array.from_numpy(
    value.ravel("C"),
    ),
    list_size,
    )

@kylebarron
Copy link
Member Author

This is now working!
image

@kylebarron
Copy link
Member Author

All the tests now pass (locally)! arro3 isn't yet published to pypi, so that's why CI here is failing

@kylebarron kylebarron marked this pull request as ready for review August 8, 2024 21:08
@kylebarron kylebarron changed the title Start removing pyarrow dependency Remove pyarrow dependency Aug 8, 2024
@kylebarron
Copy link
Member Author

I ran through a couple examples manually and it's still working as expected!

@kylebarron kylebarron merged commit 708f8af into main Aug 8, 2024
5 checks passed
@kylebarron kylebarron deleted the kyle/remove-pyarrow branch August 8, 2024 21:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant