Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data formats (examples of many different ways) #55

Open
rkdarst opened this issue Jan 14, 2021 · 0 comments
Open

Data formats (examples of many different ways) #55

rkdarst opened this issue Jan 14, 2021 · 0 comments
Labels
S:code Code to evaluate, concrete and ready to go S:concept Concept to discuss and explain topic General topic idea

Comments

@rkdarst
Copy link
Member

rkdarst commented Jan 14, 2021

In our garage today, an idea came up: take one dataset, put it in as many different forms as possible:

  • apache arrow for speed of reading
  • hdf5 for similar
  • csv for archival
  • sqlite3 for incrementally updating and querying
  • column-orineted database
  • zip/tar file
  • many small files
  • etc

and then discuss the advantages and disadvantages of each of them. maybe profile them. We had a long discussion about different data formats, and having these kind of examples could make things concrete.
@rkdarst rkdarst added S:code Code to evaluate, concrete and ready to go topic General topic idea S:concept Concept to discuss and explain labels Jan 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
S:code Code to evaluate, concrete and ready to go S:concept Concept to discuss and explain topic General topic idea
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rkdarst