Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding option to return binary data #1285

Open
JPBergsma opened this issue Jul 21, 2022 · 0 comments · May be fixed by #1292
Open

Adding option to return binary data #1285

JPBergsma opened this issue Jul 21, 2022 · 0 comments · May be fixed by #1292

Comments

@JPBergsma
Copy link
Contributor

@giovannipizzi @ml-evs

For the upcoming trajectories endpoint, it would be nice to support a binary output format.

I made a small test script and managed to return the Cartesian site coordinates in the hdf5 format, which is much faster than with the JSON format. Although in the test case, there was no overhead from processing the query and such.
If we are using the JSON converter from FAST API, the JSON converter is called for each element of a list, which makes it quite slow. In addition, the numerical values have to be converted to strings, which is not necessary for the hdf5 format.

So I thought it would be good to implement the hdf5 format.
On closer inspection, I however noticed that the hdf5 format does not seem to support nested dictionaries like we use in the data.
A possible solution would be to use a dictionary instead of a list. For example, by replacing the indexes by a number or the id/name.
This would however be a deviation from the Optimade standard.

I was therefore wondering whether you think this is a problem, and if so, whether you know of other file formats that are able to efficiently store binary numbers while maintaining the Optimade data structure.

@JPBergsma JPBergsma linked a pull request Aug 4, 2022 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant