Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python bindings improvements #684

Closed
9 tasks done
robert3005 opened this issue Aug 23, 2024 · 3 comments
Closed
9 tasks done

Python bindings improvements #684

robert3005 opened this issue Aug 23, 2024 · 3 comments
Assignees

Comments

@robert3005
Copy link
Member

robert3005 commented Aug 23, 2024

#729

  • Generic Python Array class
  • Read Vortex file
  • Convert to Numpy/Pandas
  • Convert Vortex to Arrow
  • Convert Arrow to Vortex
  • Compress Vortex array
  • Write to disk (using LayoutWriter)

@danking
Copy link
Member

danking commented Aug 29, 2024

For Pandas, it appears we could eventually implement an ExtensionArray so that Pandas can operate directly on Vortex arrays.

@danking
Copy link
Member

danking commented Aug 30, 2024

@robert3005 Is LayoutWriter a requirement? I'm using StreamArrayWriter to write and I'm able to read that again using StreamArrayReader. All my round-trip tests have passed thus far.

@robert3005
Copy link
Member Author

The StreamArrayReader/Writer are purely for serialising arrays without much framing. LayoutWriter is our logic for writing the file format which involves encoding arrays with prunable layouts

@danking danking closed this as completed Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants