Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support fetching data from ORA special remote storage #71

Open
yarikoptic opened this issue Apr 28, 2022 · 0 comments
Open

Support fetching data from ORA special remote storage #71

yarikoptic opened this issue Apr 28, 2022 · 0 comments

Comments

@yarikoptic
Copy link
Member

There is git-annex-remote-ora and all supporting functionality in datalad core. Unfortunately it operates only at the full file extraction level ATM, and we would need some indexed access within a file. If I get it right , here is a relevant discussion https://stackoverflow.com/questions/4457997/indexing-random-access-to-7zip-7z-archives and also https://github.com/vasi/lzopfs (Random access to compressed files with a FUSE filesystem) might be of interest.
Not yet sure if worthwhile.

Then there would be a need in fsspec layer to provide fetching of corresponding file from ORA.

Ultimate goal of a kind: In combination with #34 (bare) and #35 (plain directory, not super dataset) could provide a quite useful setup for HPCs where there is virtually no inodes consumption -- FUSE layer would take care about providing access to heavy trees etc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant