Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Virtual datasets #8384

Open
normanrz opened this issue Feb 6, 2025 · 0 comments
Open

Virtual datasets #8384

normanrz opened this issue Feb 6, 2025 · 0 comments

Comments

@normanrz
Copy link
Member

normanrz commented Feb 6, 2025

Detailed Description

#8350 adds the storing of mag paths in the Postgres database. However, the source of truth for datasets is still the filesystem via the datastore. The next step will be to flip this source of truth. Datastores should still scan their folders and report datasets, but they should query Webknossos for accessing the datasets. This will also allow that datasets exist only in the database without a folder on disk, e.g. for fully remote datasets. Maybe this would also be a good opportunity to implement dataset IDs in the Webknossos <> datastore communication.

Motivation

  • Remote datasets: users are using more and more remote datasets; it is a bit pointlesss to store a folder with a single json file on disk for that
  • Object storage: in the future, we want to support object storage as primary storage for Webknossos datasets, which are technically also remote datasets
  • Symlinks: symlinks currently create a lot of problems for the operation of Webknossos. With the capability to compose datasets in the database, we don't have to rely on symlinks anymore
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant