Crash when trying to retrieve attributes on an external volume

[v8oholic]

Apologies if this has been covered before, but I couldn't see any mention of it. I have just installed osxmetadata and I am experimenting with it, but I have hit a problem straight away.

I have files on an external volume which I am trying to read the attributes for. However, it crashes when creating MDItem for the file:

(tagtest) steve@MacBook-Air FFRunner % python -m osxmetadata -l /Volumes/Music/Temp/test-9026.mp4

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniconda/base/envs/facefusion/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/homebrew/Caskroom/miniconda/base/envs/facefusion/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/facefusion/lib/python3.10/site-packages/osxmetadata/__main__.py", line 1164, in <module>
    cli()  # pylint: disable=no-value-for-parameter
  File "/opt/homebrew/Caskroom/miniconda/base/envs/facefusion/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/facefusion/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/facefusion/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/facefusion/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/facefusion/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/facefusion/lib/python3.10/site-packages/osxmetadata/__main__.py", line 989, in cli
    process_files(
  File "/opt/homebrew/Caskroom/miniconda/base/envs/facefusion/lib/python3.10/site-packages/osxmetadata/__main__.py", line 1080, in process_files
    process_single_file(
  File "/opt/homebrew/Caskroom/miniconda/base/envs/facefusion/lib/python3.10/site-packages/osxmetadata/__main__.py", line 1120, in process_single_file
    md = OSXMetaData(fpath)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/facefusion/lib/python3.10/site-packages/osxmetadata/osxmetadata.py", line 96, in __init__
    raise OSError(f"Unable to create MDItem for file: {fname}")
OSError: Unable to create MDItem for file: /Volumes/Music/Temp/test-9026.mp4
(tagtest) steve@MacBook-Air FFRunner %

If I copy the same file onto the main hard disk (using finder), then it works fine:

(tagtest) steve@MacBook-Air FFRunner % python -m osxmetadata -l test-9026.mp4
/Users/steve/Applications/Python/tagtest/test-9026.mp4:
tags                      _kMDItemUserTags                  = Orange: 7
findercolor               findercolor                       = 7
attributechangedate       kMDItemAttributeChangeDate        = 2024-11-22T10:53:08.839543
audiobitrate              kMDItemAudioBitRate               = 160.0
audiochannelcount         kMDItemAudioChannelCount          = 2.0
codecs                    kMDItemCodecs                     = H.264, MPEG-4 AAC

...

stationerypad             stationerypad                     = False
(tagtest) steve@MacBook-Air FFRunner % 

Obviously I can't copy every file from the external disk, there isn't the space. Is there something I am not doing right? Is it a known problem?

It's Python 3.10.14 on Sequoia 15.1.1 and python does have access to external volumes. The filesystem is APFS (Case-sensitive, Encrypted).

I can provide any further information that I may have missed, just let me know. Thanks!

[v8oholic]

I have been able to read the attributes of the file on the external volume (com.apple.FinderInfo and com.apple.metadata:_kMDItemUserTags) using xattr. Both command line, and in python. Perhaps they don't use MDItemCreate internally, which is where osxmetatag is failing.

I haven't entirely worked out the format of the colour attributes, but I have managed a horrible hack which will filter the files according to colour tags, which is what I was trying to achieve. So for the moment I don't need to use osxmetadata. A bit weird and frustrating though.

[v8oholic]

Could this be a sandboxing issue affecting osxmetatag and not xattr? I'm not sure how that stuff works.

[RhetTbull]

I have been able to replicate this on Ventura. I do not have a machine running Sequoia. This is weird because previously on Ventura I was able to use OSXMetadata fine. (I pulled a command from my zsh history I'd previously used).

OSXMetadata uses a private Apple API for most operations unlike xattr which uses a public OS level API. It is possible Apple has changed the private API or somehow restricted the private API to a private entitlement. If so, it is weird they'd do this for external drives and not internal. (I've seen them start doing this in Sequoia with other private APIs, in particular Photos, which I also use extensively in other projects). I will look into this as time permits to see if I can figure it out as OSXMetadata is an important part of my workflow.

If you merely want to read the extended attributes then direct access to them via xattr works fine, in most cases, and earlier versions of OSXMetadata did this. However, in many cases, writing to the extended attributes does not update the actual metadata in the Spotlight database. These are managed by the private API. Hence, OSXPhotos was rewritten to use the MDItemCreate API.

The code that is failing is in the call to MDItemCreate and this is a public API. The documentation does not indicate anything has changed. The only private API access is used when writing attributes. The function may have been changed to require paths to external volumes in a different format. I will see if I can figure out what is going on.

[RhetTbull]

The code appears to work on mounted disk images and on mounted SMB shares. It only fails on directly attached external volumes which is very strange.

[RhetTbull]

So this appears to affect the native MacOS utility mdls as well:

❯ ls /Volumes/OWC_Backup/Photos/rhet/2024/10/14/IMG_0457_20241014.HEIC
 /Volumes/OWC_Backup/Photos/rhet/2024/10/14/IMG_0457_20241014.HEIC
❯ mdls /Volumes/OWC_Backup/Photos/rhet/2024/10/14/IMG_0457_20241014.HEIC
/Volumes/OWC_Backup/Photos/rhet/2024/10/14/IMG_0457_20241014.HEIC: could not find /Volumes/OWC_Backup/Photos/rhet/2024/10/14/IMG_0457_20241014.HEIC.