Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proxy for ironArray arrays #605

Open
FrancescAlted opened this issue Jul 18, 2022 · 1 comment
Open

Proxy for ironArray arrays #605

FrancescAlted opened this issue Jul 18, 2022 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@FrancescAlted
Copy link
Collaborator

For the Spark interface it would be nice to implement a proxy for accessing remote ironArray arrays so that we don't need a shared filesystem.

This can be achieved quite easily with a similar interface than the current proxy for Zarr objects. We just need to implement a network interface to our ironArray objects and ask for metainfo (shape, dtype...) and slices remotely.

As a proof of concept, we can use FastAPI (for the server) and requests (for the client).

@FrancescAlted FrancescAlted added the enhancement New feature or request label Jul 18, 2022
@martaiborra
Copy link
Contributor

This work is being done in inaos/iron-array-python#189. This work is very preliminary and it doesn't let compute a reduction of a remote viewed array, because it doesn't copy the array before computing the reduction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants