Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add metadata to dtype when kind is 'O' #47

Merged
merged 4 commits into from
Apr 19, 2024
Merged

Add metadata to dtype when kind is 'O' #47

merged 4 commits into from
Apr 19, 2024

Conversation

magland
Copy link
Collaborator

@magland magland commented Apr 16, 2024

In order to get pynwb validation to pass I needed to make the tweak of this PR. It seems that h5py uses a vlen field in metadata for a numpy dtype, and pynwb inspects that. I don't understand exactly what this is about. But when I set it to bytes in this case, the validation passes.

Here's a test validation script where things are passing with this PR

import pynwb
from pynwb import CORE_NAMESPACE
from pynwb.validate import ValidatorMap
import h5py
import remfile
import lindi


def test_validate(io: pynwb.NWBHDF5IO):
    namespace = CORE_NAMESPACE
    validator = ValidatorMap(namespace=io.manager.namespace_catalog.get_namespace(name=namespace))
    builder = io.read_builder()
    validation_errors = validator.validate(builder=builder)
    if len(validation_errors) == 0:
        print('Validation passed!')
    else:
        print('Validation failed!')
        for ve in validation_errors:
            print(ve)


if __name__ == "__main__":
    # https://neurosift.app/?p=/nwb&dandisetId=000409&dandisetVersion=draft&url=https://api.dandiarchive.org/api/assets/c04f6b30-82bf-40e1-9210-34f0bcd8be24/download/

    print('')
    print('Validating with HDF5 URL...')
    h5_url = 'https://api.dandiarchive.org/api/assets/c04f6b30-82bf-40e1-9210-34f0bcd8be24/download/'
    remf = remfile.File(h5_url)
    h5f = h5py.File(remf, mode='r')
    with pynwb.NWBHDF5IO(file=h5f, mode='r', load_namespaces=True) as io:
        test_validate(io)

    print('')
    print('Validating with LINDI URL...')
    lindi_url = 'https://lindi.neurosift.org/dandi/dandisets/000409/assets/c04f6b30-82bf-40e1-9210-34f0bcd8be24/zarr.json'
    ff = lindi.LindiH5pyFile.from_reference_file_system(lindi_url)
    with pynwb.NWBHDF5IO(file=ff, mode='r', load_namespaces=True) as io:
        test_validate(io)

@magland magland requested a review from rly April 16, 2024 16:03
Base automatically changed from rfs-json to main April 16, 2024 17:15
@codecov-commenter
Copy link

codecov-commenter commented Apr 16, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.34%. Comparing base (3c4562f) to head (d269450).

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #47   +/-   ##
=======================================
  Coverage   82.33%   82.34%           
=======================================
  Files          28       28           
  Lines        1970     1971    +1     
=======================================
+ Hits         1622     1623    +1     
  Misses        348      348           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@rly rly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Let's try to validate all NWB files on DANDI and see if this will work. We should look more closely into why the logic is what it is in PyNWB...

@magland magland merged commit 97e4b6e into main Apr 19, 2024
6 checks passed
@magland magland deleted the dtype-meta-vlen branch April 19, 2024 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants