Metadata extraction/storage/update/basic query

[yarikoptic]

This is a "planing" issue on adding metadata extraction and "storage" within registry to formalize what we are to develop in immediate next steps (placing details of "metadata usage" such as search aside, but keeping them in mind).

• Metadata extraction per dataset should be done using WiP https://github.com/datalad/datalad-metalad which is a replacement for "built-in" in current DataLad metadata interfaces/storage approach
  • "autodeducting" extractors should be added (datasets will likely be not configured for specific extractor) based on the data found in a dataset
  • probably will care to extract only for repositories which are datalad datasets (i.e have non-None .id)
• Since we would not be able to afford fetching all data, and in many if not most cases it would be wasteful anyways, streamed access to the files idea: datalad/git-annex aware streaming/random access access to files content datalad#4003 (comment) should be implemented
  • https://github.com/datalad/datalad-fuse was an initial attempt to implement FUSE file system using fsspec.
  • some issues which might need (or not) to be tackled on fsspec side were identified: https://github.com/intake/filesystem_spec/issues/created_by/yarikoptic
  • not yet sure 100% if seamless FUSE access would be possible or we might need to make metalad extractors become "streaming backend aware" (may be in the now https://github.com/datalad/datalad-deprecated/ `datalad.auto fashion, at least for the pure-in-Python extractors
  • administrative interfaces to clean up cached data might be needed
• Introduce to registry tracking of subdatasets
  • ATM all datasets in the registry are "independent"
  • we need to add many-to-many "subdatasets" table which would establish relationships between dataset/version's
  • figure out how to provide metadata extractors with a "super-dataset" view when needed
    • e.g. for the use cases like a BIDS dataset with each subject being a subdataset. Related issue: idea: extractor "subdataset_boundary" datalad-metalad#68
    • FUSE level might provide quite a cool way to provide such "dynamic view" across separate "flat" datasets
      • might need to become "git aware" as to be able to provide a view over specific committish tree: Make it possible to "view" a specific (not HEAD) committish tree datalad-fuse#1
• Decide on how we would store extracted JSON metadata records in the DB (or in some other "connected storage")
  • I do not think we should rely on having them purely in "instance/cache"d instances of those repositories using metalad
  • Keep in mind future needs for "simple search" or "syncing" (one way -- just export AFAIK) with an external specialized DB (e.g. graph DB)
  • Decide on "retention" of metadata across multiple versions of datasets (as registry keeps them updating to reflect current status)
    • most likely we would eventually add ability to store history (commit hexshas) for datasets registry sees
    • we might want to implement some kind of GC and policy on what metadata versions to keep
• Provide API endpoints for
  • GETing information on what metadata is available for a dataset(/version)
  • GETing specific metadata record(s)
  • very basic (grep-like) "search" end point
  • "administrative" interfaces (or end points) to
    • trigger re-extraction of metadata for a specific dataset(/version)
    • add extractor(s) to be used for a specific dataset

Attn @jwodder and @datalad/developers on what other high level components or related aspects you see I have missed or have ideas about.

• Decide on how we would store extracted JSON metadata records in the DB (or in some other "connected storage")
  • I do not think we should rely on having them purely in "instance/cache"d instances of those repositories using metalad
  • Keep in mind future needs for "simple search" or "syncing" (one way -- just export AFAIK) with an external specialized DB (e.g. graph DB)
  • Decide on "retention" of metadata across multiple versions of datasets (as registry keeps them updating to reflect current status)
    • most likely we would eventually add ability to store history (commit hexshas) for datasets registry sees
    • we might want to implement some kind of GC and policy on what metadata versions to keep
• Provide API endpoints for
  • GETing information on what metadata is available for a dataset(/version)
  • GETing specific metadata record(s)
  • very basic (grep-like) "search" end point
  • "administrative" interfaces (or end points) to
    • trigger re-extraction of metadata for a specific dataset(/version)
    • add extractor(s) to be used for a specific dataset

Attn @jwodder and @datalad/developers on what other high level components or related aspects you see I have missed or have ideas about.