Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Arrow input tables #170

Closed
visr opened this issue Apr 18, 2023 · 2 comments · Fixed by #798
Closed

Support Arrow input tables #170

visr opened this issue Apr 18, 2023 · 2 comments · Fixed by #798

Comments

@visr
Copy link
Member

visr commented Apr 18, 2023

It should work already in the core, but since there are no tests let's assume it's not working. Only the output tables are written to Arrow.

The example models in conftest.py currently write all input tables to the GeoPackage file. This is what model.write(path) does. Usually Arrow input tables are mainly interesting for large (time-dependent) tables, but we need to make sure it is working in CI.

I see the class InputMixin has _write_arrow and _write_geopackage that are both not used. Instead write loops over the underlying tables and writes them one-by-one to GeoPackage. Perhaps we need something like an attribute in all tables (e.g. classes that inherit from pa.SchemaModel, that determines if it should be written to GeoPackage (the default) or Arrow.

A minimum first test is added here, which contains a utility function to convert GeoPackage tables to Arrow.
https://github.com/Deltares/Ribasim/pull/187/files#diff-dd42657be5ede608d919dd58408933bac74a4661098f89d8b48dc53adf0478d7

@visr visr added this to Ribasim Apr 18, 2023
@Hofer-Julian Hofer-Julian moved this to 🔲 Todo in Ribasim Apr 20, 2023
@Hofer-Julian Hofer-Julian changed the title support Arrow input tables Support Arrow input tables May 15, 2023
@SnippenE SnippenE added the needs-refinement Issues that are too large and need refinement label Sep 28, 2023
@SnippenE SnippenE removed the needs-refinement Issues that are too large and need refinement label Nov 7, 2023
@evetion
Copy link
Member

evetion commented Nov 7, 2023

Will add/update a testmodel that writes an arrow file in #731, which will implicitly test it in Julia as well.

@visr
Copy link
Member Author

visr commented Nov 7, 2023

Ok, nice. It'd say fixing the Julia side of things is out of scope of #731 though, it's already large enough.

@evetion evetion moved this from To do to Sprint backlog in Ribasim Nov 16, 2023
@evetion evetion moved this from Sprint backlog to 👀 In review in Ribasim Nov 17, 2023
evetion added a commit that referenced this issue Nov 17, 2023
@github-project-automation github-project-automation bot moved this from 👀 In review to ✅ Done in Ribasim Nov 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants